██████╗       ████╗   ███████████╗  ████╗
         ██╔═══██╗     ██╔██║   ╚═══██╔════╝ ██╔██║
        ██╔╝   ██║    ██╔╝██║      ██╔╝     ██╔╝██║
       ██╔╝    ██║   ██╔╝ ██║     ██╔╝     ██╔╝ ██║
      ██╔╝    ██╔╝  ██╔╝  ██║    ██╔╝     ██╔╝  ██║
     ██╔╝   ██╔═╝  █████████║   ██╔╝     █████████║
    ████████╔═╝   ██╔═════██║  ██╔╝     ██╔═════██║
    ╚═══════╝     ╚═╝     ╚═╝  ╚═╝      ╚═╝     ╚═╝

      ██████╗     ██╗   ██╗   ██╗   ██╗       █████╗
     ██╔═══██╗   ██╔╝  ██╔╝  ██╔╝  ██╔╝      ██╔══██╗
    ██╔╝   ██║  ██╔╝  ██╔╝  ██╔╝  ██╔╝      ██╔╝  ██║
   █████████╔╝ ██╔╝  ██╔╝  ██╔╝  ██╔╝      ██╔╝   ██║
  ██╔═══██╔═╝ ██╔╝  ██╔╝  ██╔╝  ██╔╝      ██╔╝   ██╔╝
 ██╔╝   ██║  ██╔╝  ██╔╝  ██╔╝  ██╔╝      ██╔╝  ██╔═╝
█████████╔╝  ██████╔═╝  ██╔╝  ████████╗ ███████╔═╝
╚════════╝   ╚═════╝    ╚═╝   ╚═══════╝ ╚══════╝

     -  - --  D E C L A R A T I V E  -- -  -
     -  - --  P A R T I T I O N E D  -- -  -
     -  - --  D A T A   B U I L D S  -- -  -

DataBuild is a trivially-deployable, partition-oriented, declarative data build system.

DataBuild is for teams at data-driven orgs who need reliable, flexible, and correct data pipelines and are tired of manually orchestrating complex dependency graphs. You define Jobs (that take input data partitions and produce output partitions), compose them into Graphs (partition dependency networks), and DataBuild handles the rest. Just ask it to build a partition, and databuild handles resolving the jobs that need to run, planning execution order, running builds concurrently, and tracking and exposing build progress. Instead of writing orchestration code that breaks when dependencies change, you focus on the data transformations while DataBuild ensures your pipelines are correct, observable, and reliable.

For important context, check out DESIGN.md, along with designs in design/. Also, check out databuild.proto for key system interfaces. Key features:

• Declarative dependencies - Ask for data, get data. Define partition dependencies and DataBuild automatically plans what jobs to run and when.

• Partition-first design - Build only what's needed. Late data arrivals and partial rebuilds work seamlessly with atomic data partitions.

• Deploy anywhere - One binary, any platform. Bazel-based builds create hermetic applications that run locally, in containers, or in the cloud.

Usage

Graph Description Methods

• Bazel targets: The foundational
• Python DSL: A more succinct method with partition patterns and decorator-based auto graph wiring. Example usage.

Examples

• Test app: color votes
  • Bazel graph description example
  • Python DSL description example
• See the podcast example BUILD file.

Ways to Use DataBuild in Production

• As a CLI build tool: You can run DataBuild builds from the command line or in a remote environment - no build event log required!
• As a standalone service: Similar to Dagster or Airflow, you can run a persistent service that you send build requests to, and which serves an API and web dashboard.
• As a cloud-native containerized build tool: Build containers from your graphs and launch scheduled builds using a container service like ECS, or even your own kubernetes cluster.

Development

Intellij

Run these to allow intellij to understand the rust source:

# Generate a Cargo.toml file so intellij can link rust src
python3 scripts/generate_cargo_toml.py
# Generate a gitignore'd rust file representing the protobuf interfaces
scripts/generate_proto_for_ide.sh

Compiling

bazel build //...

Bullet-proof compile-time correctness is essential for production reliability. Backend protobuf changes must cause predictable frontend compilation failures, preventing runtime errors. Our three-pronged approach ensures this:

1. Complete Type Chain: Proto → Rust → OpenAPI → TypeScript → Components

   • Each step uses generated types, maintaining accuracy across the entire pipeline
   • Breaking changes at any layer cause compilation failures in dependent layers

2. Consistent Data Transformation: Service boundary layer transforms API responses to dashboard types

   • Canonical frontend interfaces isolated from backend implementation details
   • Transformations handle protobuf nullability and normalize data shapes
   • Components never directly access generated API types

3. Strict TypeScript Configuration: Enforces explicit null handling and prevents implicit any types

   • strictNullChecks catches undefined property access patterns
   • noImplicitAny surfaces type safety gaps
   • Runtime type errors become compile-time failures

This system guarantees that backend interface changes are caught during TypeScript compilation, not in production.

Testing

DataBuild core testing:

bazel test //...

End to end testing:

./run_e2e_tests.sh

Test Strategy

Where possible, we make invalid state unrepresentable via rust's type system. Where that is not possible, we prefer property-testing, with a handful of bespoke tests to capture critical edge cases or important behaviors.