151 lines
No EOL
6 KiB
Markdown
151 lines
No EOL
6 KiB
Markdown
|
|
# Service
|
|
Purpose: Enable centrally hostable and human-consumable interface for databuild applications, plus efficient cross-graph coordination.
|
|
|
|
## Architecture
|
|
The service provides two primary capabilities:
|
|
1. **Human Interface**: Web dashboard and HTTP API for build management and monitoring
|
|
2. **Cross-Graph Coordination**: `GraphService` API enabling efficient event-driven coordination between DataBuild instances
|
|
|
|
## Correctness Strategy
|
|
- Rely on databuild.proto, call same shared code in core
|
|
- Fully asserted type safety from core to service to web app
|
|
- Core -- databuild.proto --> service -- openapi --> web app
|
|
- No magic strings (how? protobuf doesn't have consts. enums values? code gen over yaml?)
|
|
|
|
## Cross-Graph Coordination
|
|
Services expose the `GraphService` API for cross-graph dependency management:
|
|
|
|
```rust
|
|
trait GraphService {
|
|
async fn list_events(&self, since_idx: i64, filter: EventFilter, limit: i64) -> Result<EventPage>;
|
|
}
|
|
```
|
|
|
|
## API
|
|
The purpose of the API is to enable remote, programmatic interaction with databuild applications, and to host endpoints
|
|
needed by the [web app](#web-app).
|
|
|
|
[Notes about details, context, and askama views.](https://claude.ai/share/76622c1c-7489-496e-be81-a64fef24e636)
|
|
|
|
## Web App
|
|
The web app visualizes databuild application state via features like listing past builds, job statistics,
|
|
partition liveness, build request status, etc. This section specifies the hierarchy of functions of the web app. Pages
|
|
are described in visual order (generally top to bottom).
|
|
|
|
General requirements:
|
|
- Nav at top of page
|
|
- DataBuild logo in top left
|
|
- Navigation links at the top allowing navigation to each list page:
|
|
- Wants list page
|
|
- Jobs list page
|
|
- Build requests list page
|
|
- Triggers list page
|
|
- Build event log page
|
|
- Graph label at top right
|
|
- Search box for finding builds, jobs, and partitions (needs a new service API?)
|
|
|
|
The site is implemented via Aksama templating and HTMX for dynamic updates.
|
|
|
|
### Home Page
|
|
Jumping off point to navigate and build.
|
|
- Aggregate statistics
|
|
- Count of unfulfilled wants (links to want list page with filter set)
|
|
- Count of wants past SLA (links to want list page with filter set)
|
|
- Count of active job runs (links to job run list page with filter set)
|
|
- Count of partitions produced in last 24h (links to partition list page with filter set)
|
|
- List recently created wants with their requested partitions (and their current status), with link to want detail page
|
|
- List of recently attempted job runs, with status, link to job run page
|
|
|
|
### Build Request Page
|
|
- Show build request ID and overall status of build (colored) and "Cancel" button at top
|
|
- progress bar indicating number of: needs-build partitions, building partitions, non-live delegated partitions, and
|
|
live partitions
|
|
- Summary information table
|
|
- Requested at
|
|
- analyze time (with datetime range)
|
|
- build time (with datetime range)
|
|
- number of tasks in each state (don't include sates with 0 count)
|
|
- number of partitions in each state (don't include sates with 0 count)
|
|
- Show graph diagram of job graph (collapsable)
|
|
- With each job and partition status color coded & linked to related run / partition
|
|
- [paginated](#build-event-log-pagination) list of related build events at bottom
|
|
|
|
### Wants List Page
|
|
- List recently created root want summaries, with link to want detail page
|
|
|
|
#### Want Summaries
|
|
- Want summaries are mini components that show:
|
|
- Want ID
|
|
- Comment (if any)
|
|
- SLA status or time till breach
|
|
- N / M live out of wanted partitions
|
|
- Number of child wants
|
|
- Color coded status indicator
|
|
- Number of active job runs downstream of want
|
|
- Embedded into other pages to represent a want
|
|
|
|
### Want Detail Page
|
|
- Want IDs
|
|
- Aggregate stats
|
|
-
|
|
- Partition summaries for wanted partitions (paginated)
|
|
- Child want summaries
|
|
- Active jobs
|
|
|
|
|
|
- Show want details
|
|
- Show children
|
|
- Show wanted partition details
|
|
- Show job runs
|
|
|
|
### Job Status Page
|
|
- Job label
|
|
- "Recent Runs" select, controlling page size
|
|
- "Recent Runs Page" select - the `< 1 2 3 ... N >` style paginator
|
|
- Job success rate (for all selected; colored)
|
|
- Bar graph showing job execution run times for last N (selectable between 31, 100, 365)
|
|
- Recent task runs
|
|
- With links to build request, task run, partition
|
|
- With task result
|
|
- With run time
|
|
- With expandable partition metadata
|
|
- [paginated](#build-event-log-pagination) list of related build events at bottom
|
|
|
|
### Task Run Page
|
|
- With job label, task status, and "Cancel" button at top
|
|
- Summary information table
|
|
- task run ID
|
|
- output/input partitions
|
|
- task start and end time
|
|
- task duration
|
|
- Graph similar to [build request page](#build-request-page), all partitions and jobs not involved in this task made
|
|
translucent (expandable)
|
|
- With [paginated](#build-event-log-pagination) table of build events at bottom
|
|
|
|
### Partition Status Page
|
|
- With PartitionRef, link to matching [PartitionPattern](#partitionpattern-page), color-coded status, and "build" button at top
|
|
- List of tasks that produced this partition
|
|
- [paginated](#build-event-log-pagination) list of related build events at bottom
|
|
|
|
### PartitionPattern Page
|
|
- Paginated table of partitions that match this partition pattern, sortable by cols, including:
|
|
- Partition ref (with link)
|
|
- Partition pattern values
|
|
- Partition status
|
|
- Build request link
|
|
- Task link (with run time next to it)
|
|
|
|
### Build Event Log Page
|
|
I dunno, some people want to look at the raw thing.
|
|
- A [paginated](#build-event-log-pagination) list of build event log entries
|
|
|
|
### Build Event Log Pagination
|
|
This element is present on most pages, and should be reusable/pluggable for a given set of events/filters.
|
|
- Table with headers of significant fields, sorted by timestamp by default
|
|
- With timestamp, event ID, and message field
|
|
- With color coded event type
|
|
- With links to build requests, jobs, and partitions where IDs are present
|
|
- With expandable details that show the preformatted JSON event contents
|
|
- With the `< 1 2 3 ... N >` style paginator
|
|
- Page size of 100 |