DataBuild

Orchestration logic changes frequently, so just don't write it.

DataBuild is a trivially-deployable, partition-oriented, declarative data build system.

For important context, check out DESIGN.md. Also, check out databuild.proto for key system interfaces.

Usage

See the podcast example BUILD file.

Development

Intellij

Run these to allow intellij to understand the rust source:

# Generate a Cargo.toml file so intellij can link rust src
python3 scripts/generate_cargo_toml.py
# Generate a gitignore'd rust file representing the protobuf interfaces
scripts/generate_proto_for_ide.sh

Compiling

bazel build //...

Bullet-proof compile-time correctness is essential for production reliability. Backend protobuf changes must cause predictable frontend compilation failures, preventing runtime errors. Our three-pronged approach ensures this:

1. Complete Type Chain: Proto → Rust → OpenAPI → TypeScript → Components

   • Each step uses generated types, maintaining accuracy across the entire pipeline
   • Breaking changes at any layer cause compilation failures in dependent layers

2. Consistent Data Transformation: Service boundary layer transforms API responses to dashboard types

   • Canonical frontend interfaces isolated from backend implementation details
   • Transformations handle protobuf nullability and normalize data shapes
   • Components never directly access generated API types

3. Strict TypeScript Configuration: Enforces explicit null handling and prevents implicit any types

   • strictNullChecks catches undefined property access patterns
   • noImplicitAny surfaces type safety gaps
   • Runtime type errors become compile-time failures

This system guarantees that backend interface changes are caught during TypeScript compilation, not in production.

Testing

DataBuild core testing:

bazel test //...

End to end testing:

./run_e2e_tests.sh