databuild/plans/20-wants-initial.md

Wants System Implementation

Overview

This plan implements the wants system described in design/wants.md, transitioning DataBuild from direct build requests to a declarative want-based model with cross-graph coordination and SLA tracking. This builds on the 3-tier BEL architecture and client-server CLI established in the previous phases.

Prerequisites

This plan assumes completion of:

• Phase 18: 3-tier BEL architecture with storage/query/client layers
• Phase 19: Client-server CLI architecture with service delegation

Implementation Phases

Phase 1: Extend BEL Storage for Wants

1. Add PartitionWantEvent to databuild.proto

   • Want event schema as defined in design/wants.md
   • Want source tracking (CLI, dashboard, scheduled, API)
   • TTL and SLA timestamp fields
   • External dependency specifications

2. Extend BELStorage Interface

   • Add append_want() method for want events
   • Extend EventFilter to support want filtering
   • Add want-specific query capabilities to storage layer

3. Implement in SQLite Storage Backend

   • Add wants table with appropriate indexes
   • Implement want filtering in list_events()
   • Schema migration logic for existing databases

Phase 2: Basic Want API in Service

1. Implement Want Management in Service

   • Service methods for creating and querying wants
   • Want lifecycle management (creation, expiration, satisfaction)
   • Integration with existing service auto-management

2. Add Want HTTP Endpoints

   • POST /api/v1/wants - Create new want
   • GET /api/v1/wants - List active wants with filtering
   • GET /api/v1/wants/{id} - Get want details
   • DELETE /api/v1/wants/{id} - Cancel want

3. CLI Want Commands

   • ./bazel-bin/my_graph.build want create <partition-ref> with SLA/TTL options
   • ./bazel-bin/my_graph.build want list with filtering options
   • ./bazel-bin/my_graph.build want status <partition-ref> for want status
   • Modify build commands to create wants via service

Phase 3: Want-Driven Build Evaluation

1. Implement Build Evaluator in Service

   • Continuous evaluation loop that checks for buildable wants
   • External dependency satisfaction checking
   • TTL expiration filtering for active wants

2. Replace Build Request Handling

   • Graph build commands create wants instead of direct build requests
   • Service background loop evaluates wants and triggers builds
   • Maintain atomic build semantics while satisfying multiple wants

3. Build Coordination Logic

   • Aggregate wants that can be satisfied by same build
   • Priority handling for urgent wants (short SLA)
   • Resource coordination across concurrent want evaluation

Phase 4: Cross-Graph Coordination

1. Implement GraphService API

   • HTTP API for cross-graph event streaming as defined in design/wants.md
   • Event filtering for efficient partition pattern subscriptions
   • Service-to-service communication for upstream dependencies

2. Upstream Dependency Configuration

   • Service configuration for upstream DataBuild instances
   • Partition pattern subscriptions to upstream graphs
   • Automatic want evaluation when upstream partitions become available

3. Cross-Graph Event Sync

   • Background sync process for upstream events
   • Triggering local build evaluation on upstream availability
   • Reliable HTTP-based coordination to avoid message loss

Phase 5: SLA Monitoring and Dashboard Integration

1. SLA Violation Tracking

   • External monitoring endpoints for SLA violations
   • Want timeline and status tracking
   • Integration with existing dashboard for want visualization

2. Want Dashboard Features

   • Want creation and monitoring UI
   • Cross-graph dependency visualization
   • SLA violation dashboard and alerting

3. Migration from Direct Builds

   • All build requests go through want system
   • Remove direct build request pathways
   • Update documentation for new build model

Benefits of Want-Based Architecture

✅ Unified Build Model

• All builds (manual, scheduled, triggered) use same want mechanism
• Complete audit trail in build event log
• Consistent SLA tracking across all build types

✅ Event-Driven Efficiency

• Builds only triggered when dependencies change
• Cross-graph coordination via efficient event streaming
• No polling for task readiness within builds

✅ Atomic Build Semantics Preserved

• Individual build requests remain all-or-nothing
• Fast failure provides immediate feedback
• Partial progress via multiple build requests over time

✅ Flexible SLA Management

• Separate business expectations (SLA) from operational limits (TTL)
• External monitoring with clear blame assignment
• Automatic cleanup of stale wants

✅ Cross-Graph Scalability

• Reliable HTTP-based coordination
• Efficient filtering via partition patterns
• Decentralized architecture with clear boundaries

Success Criteria

Phase 1: Storage Foundation

• Want events can be stored and queried in BEL storage
• EventFilter supports want-specific filtering
• SQLite backend handles want operations efficiently

Phase 2: Basic Want API

• Service can create and query wants via HTTP API
• Graph build commands work for want management
• Build commands create wants instead of direct builds

Phase 3: Want-Driven Builds

• Service background loop evaluates wants continuously
• Build evaluation triggers on want creation and external events
• TTL expiration and external dependency checking work correctly

Phase 4: Cross-Graph Coordination

• GraphService API returns filtered events for cross-graph coordination
• Upstream partition availability triggers downstream want evaluation
• Service-to-service communication is reliable and efficient

Phase 5: Complete Migration

• All builds go through want system
• Dashboard supports want creation and monitoring
• SLA violation endpoints provide monitoring integration
• Documentation reflects new want-based build model

Risk Mitigation

1. Incremental Migration: Implement wants alongside existing build system initially
2. Performance Validation: Ensure want evaluation doesn't introduce significant latency
3. Backwards Compatibility: Maintain existing build semantics during transition
4. Monitoring Integration: Provide clear observability into want lifecycle and performance