Update docs

This commit is contained in:
Stuart Axelbrooke 2025-07-09 15:05:57 -07:00
parent 5bfe885140
commit 115e8cfdb0
5 changed files with 2206 additions and 109 deletions

View file

@ -124,4 +124,8 @@ def lookup_job_for_partition(partition_ref: str) -> str:
### Common Pitfalls ### Common Pitfalls
- **Empty args**: Jobs with `"args": []` won't execute properly - **Empty args**: Jobs with `"args": []` won't execute properly
- **Wrong target refs**: Job lookup must return base targets, not `.cfg` variants - **Wrong target refs**: Job lookup must return base targets, not `.cfg` variants
- **Missing partition refs**: All outputs must be addressable via partition references - **Missing partition refs**: All outputs must be addressable via partition references
## Documentation
We use plans / designs in the [plans](./plans/) directory to anchor most large scale efforts. We create plans that are good bets, though not necessarily exhaustive, then (and this is critical) we update them after the work is completed, or after significant progress towards completion.

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

1
plans/ideas.md Normal file
View file

@ -0,0 +1 @@
- Generate a helm chart / kuztomize thing / docker compose spec?

View file

@ -5,19 +5,21 @@
## Overview ## Overview
Generate TypeScript client code from the DataBuild protobuf definitions to provide type-safe API access for the dashboard. Generate TypeScript client code from the DataBuild Build Graph Service OpenAPI specification to provide type-safe API access for the dashboard.
## Scope ## Scope
### In Scope ### In Scope
- Generate TypeScript interfaces from `databuild.proto` - Generate OpenAPI specification from Build Graph Service using aide
- Create typed client for Build Graph Service HTTP API endpoints - Create typed TypeScript client from OpenAPI spec using OpenAPI Generator
- Set up Bazel rules for protobuf-to-TypeScript generation - Set up Bazel rules for OpenAPI-to-TypeScript generation
- Implement simple custom protobuf-to-TypeScript tooling (if off-the-shelf solutions are complex) - Implement path parameter structs for proper aide OpenAPI generation
- CLI-based OpenAPI spec extraction
### Out of Scope ### Out of Scope
- Full protobuf runtime features (only need type definitions) - HTTP-based spec extraction (use CLI flag instead)
- Complex protobuf features (focus on basic message types) - External Bazel modules (use OpenAPI Generator JAR directly)
- Custom operation IDs (use auto-generated method names)
- Client-side validation (rely on server-side validation) - Client-side validation (rely on server-side validation)
## Technical Approach ## Technical Approach
@ -35,82 +37,124 @@ Key API endpoints to generate TypeScript client for:
### Generated Client Structure ### Generated Client Structure
```typescript ```typescript
// Generated types from OpenAPI spec // Generated types from OpenAPI spec (auto-generated models)
interface BuildRequest { export * from './AnalyzeRequest';
partitions: string[]; export * from './BuildRequest';
} export * from './BuildStatusResponse';
export * from './PartitionStatusResponse';
// ... all other model types
interface BuildStatusResponse { // Generated client API with auto-generated method names
build_request_id: string; export class DefaultApi {
status: string; constructor(configuration?: Configuration);
requested_partitions: string[];
created_at: number;
updated_at: number;
events: BuildEventSummary[];
}
// Generated client class
class BuildGraphClient {
constructor(baseUrl: string);
async submitBuild(request: BuildRequest): Promise<{build_request_id: string}>; // Auto-generated method names from OpenAPI paths
async getBuildStatus(id: string): Promise<BuildStatusResponse>; async apiV1BuildsPost(requestParameters: ApiV1BuildsPostRequest): Promise<BuildRequestResponse>;
async cancelBuild(id: string): Promise<{cancelled: boolean}>; async apiV1BuildsIdGet(requestParameters: ApiV1BuildsIdGetRequest): Promise<BuildStatusResponse>;
async getPartitionStatus(ref: string): Promise<PartitionStatusResponse>; async apiV1BuildsIdDelete(requestParameters: ApiV1BuildsIdDeleteRequest): Promise<any>;
async getPartitionEvents(ref: string): Promise<BuildEventSummary[]>; async apiV1PartitionsRefStatusGet(requestParameters: ApiV1PartitionsRefStatusGetRequest): Promise<PartitionStatusResponse>;
async analyzeGraph(request: AnalyzeRequest): Promise<AnalyzeResponse>; async apiV1PartitionsRefEventsGet(requestParameters: ApiV1PartitionsRefEventsGetRequest): Promise<PartitionEventsResponse>;
async apiV1AnalyzePost(requestParameters: ApiV1AnalyzePostRequest): Promise<AnalyzeResponse>;
}
// Type-safe request parameter interfaces
export interface ApiV1BuildsPostRequest {
buildRequest: BuildRequest;
}
export interface ApiV1BuildsIdGetRequest {
id: string;
} }
``` ```
### Bazel Integration ### Bazel Integration
- Create `//databuild/client:typescript` target - Create `//databuild/client:typescript` target
- Generate OpenAPI spec from Build Graph Service - Extract OpenAPI spec using CLI flag `--print-openapi-spec`
- Use OpenAPI Generator to create TypeScript client - Use OpenAPI Generator JAR to create TypeScript client
- Ensure hermetic build process - Ensure hermetic build process with JAR download
- Output client code to `databuild/client/typescript/` - Output client code to `databuild/client/typescript_generated/`
### Path Parameter Requirements
For aide to properly generate OpenAPI path parameters, all path parameters must be wrapped in structs:
```rust
#[derive(Deserialize, JsonSchema)]
pub struct BuildStatusRequest {
pub id: String,
}
pub async fn get_build_status(
Path(request): Path<BuildStatusRequest>
) -> Result<Json<BuildStatusResponse>, ...> {
// Use request.id instead of direct string
}
```
## Implementation Strategy ## Implementation Strategy
1. **Add OpenAPI Generation to Service** 1. **✅ Add OpenAPI Generation to Service**
- Add `aide` dependency for OpenAPI spec generation - ✅ Add `aide` dependency for OpenAPI spec generation
- Modify Axum service to use `aide::ApiRouter` - ✅ Modify Axum service to use `aide::ApiRouter`
- Add JsonSchema derives to request/response types - ✅ Add JsonSchema derives to request/response types
- Create OpenAPI spec endpoint - ✅ Add path parameter structs for proper aide integration
- ✅ Create CLI flag `--print-openapi-spec` for spec generation
2. **OpenAPI Spec Extraction** 2. **✅ OpenAPI Spec Extraction**
- Create Bazel rule to run service and extract OpenAPI spec - ✅ Create Bazel rule using CLI flag instead of HTTP endpoint
- Save spec as build artifact (JSON/YAML) - ✅ Extract spec using `build_graph_service --print-openapi-spec`
- ✅ Save spec as build artifact (JSON)
3. **TypeScript Client Generation** 3. **✅ TypeScript Client Generation**
- Add OpenAPI Generator to Bazel build system - ✅ Use OpenAPI Generator JAR directly (not Bazel module)
- Create Bazel rule to generate TypeScript client from spec - ✅ Create Bazel rule to generate TypeScript client from spec
- Use `typescript-fetch` generator for modern fetch-based client - ✅ Use `typescript-fetch` generator for modern fetch-based client
- ✅ Configure with camelCase naming and single request parameters
4. **Bazel Rules** 4. **Bazel Rules**
- Create `typescript_openapi_library` rule - ✅ Create `//databuild/client:typescript_client` target
- Generate client code during build - Generate client code during build with proper file copying
- Ensure proper dependency management - ✅ Ensure hermetic build with JAR download
## Deliverables ## Deliverables
- [ ] OpenAPI spec generation from Build Graph Service - [x] OpenAPI spec generation from Build Graph Service via CLI flag
- [ ] TypeScript interfaces for all API request/response types - [x] TypeScript interfaces for all API request/response types (auto-generated)
- [ ] Typed HTTP client for Build Graph Service endpoints - [x] Typed HTTP client for Build Graph Service endpoints with fetch API
- [ ] Bazel rules for automated client generation - [x] Bazel rules for automated client generation using OpenAPI Generator JAR
- [ ] Documentation for using the generated client - [x] Path parameter struct implementation for aide compatibility
## Success Criteria ## Success Criteria
- OpenAPI spec accurately reflects all service endpoints - ✅ OpenAPI spec accurately reflects all service endpoints with proper path parameters
- Generated TypeScript compiles without errors - ✅ Generated TypeScript compiles without errors using `typescript-fetch` generator
- Client provides type safety for all API endpoints - ✅ Client provides type safety for all API endpoints with auto-generated method names
- Bazel build integrates seamlessly and generates client automatically - ✅ Bazel build integrates seamlessly and generates client automatically
- Ready for use in Chunk 2 (Hello World App) - ✅ Path parameter structs enable proper aide OpenAPI generation
- ✅ Ready for use in Chunk 2 (Hello World App)
## Testing ## Testing
- Verify OpenAPI spec generation from service - ✅ Verify OpenAPI spec generation from service using CLI flag
- Verify generated TypeScript compiles - ✅ Verify generated TypeScript compiles without errors
- Test client against running Build Graph Service - ✅ Validate OpenAPI spec passes OpenAPI Generator validation
- Validate type safety with TypeScript compiler - ✅ Ensure build process is hermetic and reproducible with JAR download
- Ensure build process is hermetic and reproducible - ✅ Verify all API endpoints generate proper TypeScript method signatures
- ✅ Confirm path parameters are properly typed in generated client
## Implementation Notes
### Lessons Learned
1. **aide Path Parameter Requirements**: aide requires path parameters to be wrapped in structs with `JsonSchema` derives, not simple `Path<String>` extractors.
2. **OpenAPI Generator Bazel Integration**: The official Bazel module was not available in registry, so we used the JAR directly via genrule for better reliability.
3. **CLI vs HTTP Extraction**: Using a CLI flag for spec extraction is simpler and more reliable than starting an HTTP server.
4. **Auto-generated Method Names**: Without custom operationIds, OpenAPI Generator creates method names like `apiV1BuildsPost`. This can be improved in future iterations.
### Build Targets
- `//databuild/client:extract_openapi_spec` - Extracts OpenAPI spec JSON
- `//databuild/client:typescript_client` - Generates TypeScript client
- `//databuild/client:typescript` - Main target for consuming the client