databuild/docs/narrative/partition-identity.md

The core question: what defines partition identity?

• Hive partitioning is great, but making every source of variation or dimension of config a partition col would be ridiculous for real applications
  • Why would it be ridiculous? Examples:
    • We use many config dims: targeting goal, modeling strategy, per-modeling strategy config (imagine flattening a struct tree), etc. Then we add another dimension to the config for an experiment. Is that a "new dataset"?
      • "all config goes through partitions" means that every config change has to be a dataset or partition pattern change.
      • Maybe this is fine, actually, as long as authors make sensible defaults a non-problem? (e.g. think of schema evolution)
    • In cases where config is internally resolved by the job,
• Whose to say the partition isn't a struct itself? s3://my/dataset/date=2025-01-01 could also be {"dataset": "s3://my/dataset", "date": "2025-01-01"}, or even {"kind": "icecream", "meta": {"toppings": ["sprinkles"]}} (though a clear requirement is that databuild does not parse partition refs)

• Should we consider the bazel config approach, where identity is the whole set of config, manifest as a hash? (not human-readable)