12 lines
846 B
Markdown
12 lines
846 B
Markdown
|
|
- Airflow and Luigi are OG data orchestrators that inspired databuild
|
|
- Airflow uses explicit declaration of DAG structure
|
|
- Luigi uses implicit, discovered DAG structure
|
|
- Both use DAG runs as a top-level unit of execution
|
|
- This is nice because you can see what's going to happen after the DAG run has launched
|
|
- This is not nice because you have to deal with mid-execution DAG runs during deployments - what do you do?
|
|
- Do you terminate existing dag runs and retrigger? (what if the workload is stateful? Don't do that!)
|
|
- Do you let existing dag runs finish?
|
|
- How do you deal with DAG run identity under changing DAG definition?
|
|
- These questions are all red herrings. We don't care about the DAG definition - we care about the data we want to produce.
|
|
- We should instead declare what partitions we want, and iteratively propagate
|