databuild/design/service.md

Service

Purpose: Enable centrally hostable and human-consumable interface for databuild applications, plus efficient cross-graph coordination.

Architecture

The service provides two primary capabilities:

1. Human Interface: Web dashboard and HTTP API for build management and monitoring
2. Cross-Graph Coordination: GraphService API enabling efficient event-driven coordination between DataBuild instances

Correctness Strategy

• Rely on databuild.proto, call same shared code in core
• Fully asserted type safety from core to service to web app
  • Core -- databuild.proto --> service -- openapi --> web app
• No magic strings (how? protobuf doesn't have consts. enums values? code gen over yaml?)

Cross-Graph Coordination

Services expose the GraphService API for cross-graph dependency management:

trait GraphService {
    async fn list_events(&self, since_idx: i64, filter: EventFilter, limit: i64) -> Result<EventPage>;
}

API

The purpose of the API is to enable remote, programmatic interaction with databuild applications, and to host endpoints needed by the web app.

Notes about details, context, and askama views.

Web App

The web app visualizes databuild application state via features like listing past builds, job statistics, partition liveness, build request status, etc. This section specifies the hierarchy of functions of the web app. Pages are described in visual order (generally top to bottom).

General requirements:

• Nav at top of page
  • DataBuild logo in top left
  • Navigation links at the top allowing navigation to each list page:
    • Wants list page
    • Jobs list page
    • Build requests list page
    • Triggers list page
    • Build event log page
  • Graph label at top right
  • Search box for finding builds, jobs, and partitions (needs a new service API?)

The site is implemented via Aksama templating and HTMX for dynamic updates.

Home Page

Jumping off point to navigate and build.

• Aggregate statistics
  • Count of unfulfilled wants (links to want list page with filter set)
  • Count of wants past SLA (links to want list page with filter set)
  • Count of active job runs (links to job run list page with filter set)
  • Count of partitions produced in last 24h (links to partition list page with filter set)
• List recently created wants with their requested partitions (and their current status), with link to want detail page
• List of recently attempted job runs, with status, link to job run page

Build Request Page

• Show build request ID and overall status of build (colored) and "Cancel" button at top
• progress bar indicating number of: needs-build partitions, building partitions, non-live delegated partitions, and live partitions
• Summary information table
  • Requested at
  • analyze time (with datetime range)
  • build time (with datetime range)
  • number of tasks in each state (don't include sates with 0 count)
  • number of partitions in each state (don't include sates with 0 count)
• Show graph diagram of job graph (collapsable)
  • With each job and partition status color coded & linked to related run / partition
• paginated list of related build events at bottom

Wants List Page

• List recently created root want summaries, with link to want detail page

Want Summaries

• Want summaries are mini components that show:
  • Want ID
  • Comment (if any)
  • SLA status or time till breach
  • N / M live out of wanted partitions
  • Number of child wants
  • Color coded status indicator
  • Number of active job runs downstream of want
• Embedded into other pages to represent a want

Want Detail Page

• Want IDs

• Aggregate stats

• Partition summaries for wanted partitions (paginated)

• Child want summaries

• Active jobs

• Show want details

• Show children

• Show wanted partition details

• Show job runs

Job Status Page

• Job label
• "Recent Runs" select, controlling page size
• "Recent Runs Page" select - the < 1 2 3 ... N > style paginator
• Job success rate (for all selected; colored)
• Bar graph showing job execution run times for last N (selectable between 31, 100, 365)
• Recent task runs
  • With links to build request, task run, partition
  • With task result
  • With run time
  • With expandable partition metadata
• paginated list of related build events at bottom

Task Run Page

• With job label, task status, and "Cancel" button at top
• Summary information table
  • task run ID
  • output/input partitions
  • task start and end time
  • task duration
• Graph similar to build request page, all partitions and jobs not involved in this task made translucent (expandable)
• With paginated table of build events at bottom

Partition Status Page

• With PartitionRef, link to matching PartitionPattern, color-coded status, and "build" button at top
• List of tasks that produced this partition
• paginated list of related build events at bottom

PartitionPattern Page

• Paginated table of partitions that match this partition pattern, sortable by cols, including:
  • Partition ref (with link)
  • Partition pattern values
  • Partition status
  • Build request link
  • Task link (with run time next to it)

Build Event Log Page

I dunno, some people want to look at the raw thing.

• A paginated list of build event log entries

Build Event Log Pagination

This element is present on most pages, and should be reusable/pluggable for a given set of events/filters.

• Table with headers of significant fields, sorted by timestamp by default
  • With timestamp, event ID, and message field
  • With color coded event type
  • With links to build requests, jobs, and partitions where IDs are present
  • With expandable details that show the preformatted JSON event contents
  • With the < 1 2 3 ... N > style paginator
  • Page size of 100