databuild/plans/10-shared-core.md

Shared Core Refactor

We want to refactor the codebase to move shared functionality into core components that are shared between interfaces (e.g. CLI and service), and which can be tested independently. The capabilities are listed in databuild/README.md, and each first level bullet represents a subcommand, with sub-bullets representing sub-sub-commands, e.g. you can run bazel-bin/mygraph.cli builds cancel c38a442d-fad3-4f74-ae3f-062e5377fe52. This should match service capabilities, e.g. get -XPOST localhost:8080/builds/c38a442d-fad3-4f74-ae3f-062e5377fe52/cancel.

These core capabilities should be factored into explicit read vs write capabilities. On the write side, it should verify the action is relevant (e.g. you can't cancel a nonexistent build, but you can request the build of an existing partition, it will just delegate), and then write the appropriate event to the BEL. Simple. On the read side, the different capabilities should be implemented by different "repositories", a'la the repository pattern. We can then handle any variation in backing database internal to the repositories (since most SQL will be valid for both SQLite, postgres, and delta).

Plan

We should take a phased approach to executing this plan. After implementing the core functionality and unit tests for each phase, we should pause and write down any potential refactoring that would benefit the system before moving onto the next phase.

Phase 1 - Implement Common Event Write Component

Goal: create a single interface for writing events to the build event log.

• Should include all existing "write" functionality, like requesting a new build, etc.
• Migrate CLI to use new write component
• Migrate service to use new write component

Phase 2 - Implement MockBuildEventLog

Goal: create a common testing tool that allows easy specification of testing conditions (e.g. BEL contents/events) to test system/graph behavior.

• Should use an in-memory sqlite database to ensure tests can be run in parallel
• Should make it very easy to specify test data (e.g. event constructors with random defaults that can be overwritten)
• Should include a trivial unit test that writes a valid event and verifies its there via real code paths.
• Design notes: shouldn't rewrite event write or repository read code; should focus on making test cases easy to describe, so then assertions can be made on repository-based queries.
• Event write and repositories should be pluggable, allowing for the MockBuildEventLog to be provided per test in a way consistent with how BEL backing databases are specified normally.

Phase 3 - Implement partitions Repository

• Create a new build event log event for partition invalidation (with reason field)
• Implement a repository in databuild/repositories/partitions/ that queries the build event log for the following capabilities
  • list
  • show
  • invalidate
• Add partitions subcommand to CLI
• Migrate or add partition capabilities to service.

Phase 4 - Implement jobs Repository

• Implement a repository in databuild/repositories/jobs/ that queries the BEL for the following capabilities
  • list
  • show
• Add jobs subcommand to CLI
• Migrate or add jobs capabilities to service.

Phase 5 - Implement tasks Repository

• Implement a "task cancel" job BEL event (with reason field)
• Implement a repository in databuild/repositories/tasks/ that queries the BEL for the following capabilities
  • list
  • show
• And add to the common write component a cancel_task method to implement this
• Add tasks subcommand to CLI
• Add service endpoint for canceling tasks
• (TODO - later we will need to implement a way to operate on the dashboard - lets do that in a later project)

Phase 6 - Implement builds Repository

• Implement a "build cancel" BEL event (with reason field)
• Implement a repository in databuild/repositories/builds/ that queries the BEL for the following capabilities
  • list
  • show
  • cancel
• Add to the common write component a cancel_build method
• Add builds subcommand to the CLI
• Migrate service endpoints to use the new shared impl
• Add service endpoint implementing build cancel
• Add a cancel button to the build status page (for in-progress builds)

Phase 7 - Testing

• Review prior work, ensure that tests have been written to cover the 90% of most important functionality for each component.
• Ensure all tests pass, and fix those that don't.
• Run e2e tests.

Phase 8 - Reflection & Next Steps

• Reflect on the work done and look for opportunities for improvement and refactoring
• Call out any "buried bodies" very explicitly (things which need to be revisited for the implementation to be complete)

Note

Do not take shortcuts. This we are building for the long term. If you have any questions, please pause and ask.