The core question: what defines partition identity? - Hive partitioning is great, but making every source of variation or dimension of config a partition col would be ridiculous for real applications - Why would it be ridiculous? Examples: - We use many config dims: targeting goal, modeling strategy, per-modeling strategy config (imagine flattening a struct tree), etc. Then we add another dimension to the config for an experiment. Is that a "new dataset"? - "all config goes through partitions" means that every config change has to be a dataset or partition pattern change. - Maybe this is fine, actually, as long as authors make sensible defaults a non-problem? (e.g. think of schema evolution) - In cases where config is internally resolved by the job, - Whose to say the partition isn't a struct itself? `s3://my/dataset/date=2025-01-01` could also be `{"dataset": "s3://my/dataset", "date": "2025-01-01"}`, or even `{"kind": "icecream", "meta": {"toppings": ["sprinkles"]}}` (though a clear requirement is that databuild does not parse partition refs) - Should we consider the bazel config approach, where identity is the whole set of config, manifest as a hash? (not human-readable) -