databuild/docs/design/service.md

Service

Purpose: Enable centrally hostable and human-consumable interface for databuild applications, plus efficient cross-graph coordination.

Architecture

The service provides two primary capabilities:

1. Human Interface: Web dashboard and HTTP API for build management and monitoring
2. Cross-Graph Coordination: GraphService API enabling efficient event-driven coordination between DataBuild instances

Correctness Strategy

• Rely on databuild.proto, call same shared code in core
• Fully asserted type safety from core to service to web app via Askama compile-time template checking
  • databuild.proto → prost (Rust structs) → Askama templates (compile-checked field access) → HTML
• No magic strings: enum variants and field names are checked at compile time

Cross-Graph Coordination

Services expose the GraphService API for cross-graph dependency management:

trait GraphService {
    async fn list_events(&self, since_idx: i64, filter: EventFilter, limit: i64) -> Result<EventPage>;
}

API

The purpose of the API is to enable remote, programmatic interaction with databuild applications, and to host endpoints needed by the web app.

Notes about details, context, and askama views.

Web App

The web app visualizes databuild application state via features like listing past builds, job statistics, partition liveness, build request status, etc. This section specifies the hierarchy of functions of the web app. Pages are described in visual order (generally top to bottom).

Implementation Strategy: Pure MPA with View Transitions

The web app is a traditional multi-page application (MPA) using server-side rendering. No JavaScript framework.

Stack:

• Askama: Compile-time checked HTML templates in Rust
• CSS View Transitions API: Smooth navigation animations between pages
• Plain HTML forms: For mutations (create want, cancel job, etc.)

Why this approach:

• Fast: No JS bundle to download/parse; HTML streams directly from server
• Simple: Single Rust codebase; no frontend build pipeline
• Type-safe: Askama checks template field access at compile time
• Easy to change: Templates are just HTML; Rust compiler catches breaking changes

View Transitions (supported in Chrome 126+, Safari 18.2+, Firefox 144+) provide SPA-like smooth navigation without JavaScript. The browser snapshots the old page, loads the new one, and animates matching elements:

@view-transition { navigation: auto; }
.want-card { view-transition-name: want-card; }

Trade-off: No partial page updates. Pagination and navigation reload the full page, but view transitions make this feel instant. For live updates, users refresh manually or we add <meta http-equiv="refresh"> on status pages.

General Requirements

• Nav at top of page
  • DataBuild logo in top left
  • Navigation links at the top allowing navigation to each list page:
    • Wants list page
    • Jobs list page
    • Build requests list page
    • Triggers list page
    • Build event log page
  • Graph label at top right
  • Search box for finding builds, jobs, and partitions

Home Page

Jumping off point to navigate and build.

• Aggregate statistics
  • Count of unfulfilled wants (links to want list page with filter set)
  • Count of wants past SLA (links to want list page with filter set)
  • Count of active job runs (links to job run list page with filter set)
  • Count of partitions produced in last 24h (links to partition list page with filter set)
• List recently created wants with their requested partitions (and their current status), with link to want detail page
• List of recently attempted job runs, with status, link to job run page

Build Request Page

• Show build request ID and overall status of build (colored) and "Cancel" button at top
• progress bar indicating number of: needs-build partitions, building partitions, non-live delegated partitions, and live partitions
• Summary information table
  • Requested at
  • analyze time (with datetime range)
  • build time (with datetime range)
  • number of tasks in each state (don't include sates with 0 count)
  • number of partitions in each state (don't include sates with 0 count)
• Show graph diagram of job graph (collapsable)
  • With each job and partition status color coded & linked to related run / partition
• paginated list of related build events at bottom

Wants List Page

• List recently created root want summaries, with link to want detail page

Want Summaries

• Want summaries are mini components that show:
  • Want ID
  • Comment (if any)
  • SLA status or time till breach
  • N / M live out of wanted partitions
  • Number of child wants
  • Color coded status indicator
  • Number of active job runs downstream of want
• Embedded into other pages to represent a want

Want Detail Page

• Want IDs

• Aggregate stats

• Partition summaries for wanted partitions (paginated)

• Child want summaries

• Active jobs

• Show want details

• Show children

• Show wanted partition details

• Show job runs

Job Status Page

• Job label
• "Recent Runs" select, controlling page size
• "Recent Runs Page" select - the < 1 2 3 ... N > style paginator
• Job success rate (for all selected; colored)
• Bar graph showing job execution run times for last N (selectable between 31, 100, 365)
• Recent task runs
  • With links to build request, task run, partition
  • With task result
  • With run time
  • With expandable partition metadata
• paginated list of related build events at bottom

Task Run Page

• With job label, task status, and "Cancel" button at top
• Summary information table
  • task run ID
  • output/input partitions
  • task start and end time
  • task duration
• Graph similar to build request page, all partitions and jobs not involved in this task made translucent (expandable)
• With paginated table of build events at bottom

Partition Status Page

• With PartitionRef, link to matching PartitionPattern, color-coded status, and "build" button at top
• List of tasks that produced this partition
• paginated list of related build events at bottom

PartitionPattern Page

• Paginated table of partitions that match this partition pattern, sortable by cols, including:
  • Partition ref (with link)
  • Partition pattern values
  • Partition status
  • Build request link
  • Task link (with run time next to it)

Build Event Log Page

I dunno, some people want to look at the raw thing.

• A paginated list of build event log entries

Build Event Log Pagination

This element is present on most pages, and should be reusable/pluggable for a given set of events/filters.

• Table with headers of significant fields, sorted by timestamp by default
  • With timestamp, event ID, and message field
  • With color coded event type
  • With links to build requests, jobs, and partitions where IDs are present
  • With expandable details that show the preformatted JSON event contents
  • With the < 1 2 3 ... N > style paginator
  • Page size of 100