Update docs
This commit is contained in:
parent
5bfe885140
commit
115e8cfdb0
5 changed files with 2206 additions and 109 deletions
|
|
@ -124,4 +124,8 @@ def lookup_job_for_partition(partition_ref: str) -> str:
|
|||
### Common Pitfalls
|
||||
- **Empty args**: Jobs with `"args": []` won't execute properly
|
||||
- **Wrong target refs**: Job lookup must return base targets, not `.cfg` variants
|
||||
- **Missing partition refs**: All outputs must be addressable via partition references
|
||||
- **Missing partition refs**: All outputs must be addressable via partition references
|
||||
|
||||
## Documentation
|
||||
|
||||
We use plans / designs in the [plans](./plans/) directory to anchor most large scale efforts. We create plans that are good bets, though not necessarily exhaustive, then (and this is critical) we update them after the work is completed, or after significant progress towards completion.
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
1
plans/ideas.md
Normal file
1
plans/ideas.md
Normal file
|
|
@ -0,0 +1 @@
|
|||
- Generate a helm chart / kuztomize thing / docker compose spec?
|
||||
|
|
@ -5,19 +5,21 @@
|
|||
|
||||
## Overview
|
||||
|
||||
Generate TypeScript client code from the DataBuild protobuf definitions to provide type-safe API access for the dashboard.
|
||||
Generate TypeScript client code from the DataBuild Build Graph Service OpenAPI specification to provide type-safe API access for the dashboard.
|
||||
|
||||
## Scope
|
||||
|
||||
### In Scope
|
||||
- Generate TypeScript interfaces from `databuild.proto`
|
||||
- Create typed client for Build Graph Service HTTP API endpoints
|
||||
- Set up Bazel rules for protobuf-to-TypeScript generation
|
||||
- Implement simple custom protobuf-to-TypeScript tooling (if off-the-shelf solutions are complex)
|
||||
- Generate OpenAPI specification from Build Graph Service using aide
|
||||
- Create typed TypeScript client from OpenAPI spec using OpenAPI Generator
|
||||
- Set up Bazel rules for OpenAPI-to-TypeScript generation
|
||||
- Implement path parameter structs for proper aide OpenAPI generation
|
||||
- CLI-based OpenAPI spec extraction
|
||||
|
||||
### Out of Scope
|
||||
- Full protobuf runtime features (only need type definitions)
|
||||
- Complex protobuf features (focus on basic message types)
|
||||
- HTTP-based spec extraction (use CLI flag instead)
|
||||
- External Bazel modules (use OpenAPI Generator JAR directly)
|
||||
- Custom operation IDs (use auto-generated method names)
|
||||
- Client-side validation (rely on server-side validation)
|
||||
|
||||
## Technical Approach
|
||||
|
|
@ -35,82 +37,124 @@ Key API endpoints to generate TypeScript client for:
|
|||
|
||||
### Generated Client Structure
|
||||
```typescript
|
||||
// Generated types from OpenAPI spec
|
||||
interface BuildRequest {
|
||||
partitions: string[];
|
||||
}
|
||||
// Generated types from OpenAPI spec (auto-generated models)
|
||||
export * from './AnalyzeRequest';
|
||||
export * from './BuildRequest';
|
||||
export * from './BuildStatusResponse';
|
||||
export * from './PartitionStatusResponse';
|
||||
// ... all other model types
|
||||
|
||||
interface BuildStatusResponse {
|
||||
build_request_id: string;
|
||||
status: string;
|
||||
requested_partitions: string[];
|
||||
created_at: number;
|
||||
updated_at: number;
|
||||
events: BuildEventSummary[];
|
||||
}
|
||||
|
||||
// Generated client class
|
||||
class BuildGraphClient {
|
||||
constructor(baseUrl: string);
|
||||
// Generated client API with auto-generated method names
|
||||
export class DefaultApi {
|
||||
constructor(configuration?: Configuration);
|
||||
|
||||
async submitBuild(request: BuildRequest): Promise<{build_request_id: string}>;
|
||||
async getBuildStatus(id: string): Promise<BuildStatusResponse>;
|
||||
async cancelBuild(id: string): Promise<{cancelled: boolean}>;
|
||||
async getPartitionStatus(ref: string): Promise<PartitionStatusResponse>;
|
||||
async getPartitionEvents(ref: string): Promise<BuildEventSummary[]>;
|
||||
async analyzeGraph(request: AnalyzeRequest): Promise<AnalyzeResponse>;
|
||||
// Auto-generated method names from OpenAPI paths
|
||||
async apiV1BuildsPost(requestParameters: ApiV1BuildsPostRequest): Promise<BuildRequestResponse>;
|
||||
async apiV1BuildsIdGet(requestParameters: ApiV1BuildsIdGetRequest): Promise<BuildStatusResponse>;
|
||||
async apiV1BuildsIdDelete(requestParameters: ApiV1BuildsIdDeleteRequest): Promise<any>;
|
||||
async apiV1PartitionsRefStatusGet(requestParameters: ApiV1PartitionsRefStatusGetRequest): Promise<PartitionStatusResponse>;
|
||||
async apiV1PartitionsRefEventsGet(requestParameters: ApiV1PartitionsRefEventsGetRequest): Promise<PartitionEventsResponse>;
|
||||
async apiV1AnalyzePost(requestParameters: ApiV1AnalyzePostRequest): Promise<AnalyzeResponse>;
|
||||
}
|
||||
|
||||
// Type-safe request parameter interfaces
|
||||
export interface ApiV1BuildsPostRequest {
|
||||
buildRequest: BuildRequest;
|
||||
}
|
||||
|
||||
export interface ApiV1BuildsIdGetRequest {
|
||||
id: string;
|
||||
}
|
||||
```
|
||||
|
||||
### Bazel Integration
|
||||
- Create `//databuild/client:typescript` target
|
||||
- Generate OpenAPI spec from Build Graph Service
|
||||
- Use OpenAPI Generator to create TypeScript client
|
||||
- Ensure hermetic build process
|
||||
- Output client code to `databuild/client/typescript/`
|
||||
- Extract OpenAPI spec using CLI flag `--print-openapi-spec`
|
||||
- Use OpenAPI Generator JAR to create TypeScript client
|
||||
- Ensure hermetic build process with JAR download
|
||||
- Output client code to `databuild/client/typescript_generated/`
|
||||
|
||||
### Path Parameter Requirements
|
||||
For aide to properly generate OpenAPI path parameters, all path parameters must be wrapped in structs:
|
||||
|
||||
```rust
|
||||
#[derive(Deserialize, JsonSchema)]
|
||||
pub struct BuildStatusRequest {
|
||||
pub id: String,
|
||||
}
|
||||
|
||||
pub async fn get_build_status(
|
||||
Path(request): Path<BuildStatusRequest>
|
||||
) -> Result<Json<BuildStatusResponse>, ...> {
|
||||
// Use request.id instead of direct string
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
1. **Add OpenAPI Generation to Service**
|
||||
- Add `aide` dependency for OpenAPI spec generation
|
||||
- Modify Axum service to use `aide::ApiRouter`
|
||||
- Add JsonSchema derives to request/response types
|
||||
- Create OpenAPI spec endpoint
|
||||
1. **✅ Add OpenAPI Generation to Service**
|
||||
- ✅ Add `aide` dependency for OpenAPI spec generation
|
||||
- ✅ Modify Axum service to use `aide::ApiRouter`
|
||||
- ✅ Add JsonSchema derives to request/response types
|
||||
- ✅ Add path parameter structs for proper aide integration
|
||||
- ✅ Create CLI flag `--print-openapi-spec` for spec generation
|
||||
|
||||
2. **OpenAPI Spec Extraction**
|
||||
- Create Bazel rule to run service and extract OpenAPI spec
|
||||
- Save spec as build artifact (JSON/YAML)
|
||||
2. **✅ OpenAPI Spec Extraction**
|
||||
- ✅ Create Bazel rule using CLI flag instead of HTTP endpoint
|
||||
- ✅ Extract spec using `build_graph_service --print-openapi-spec`
|
||||
- ✅ Save spec as build artifact (JSON)
|
||||
|
||||
3. **TypeScript Client Generation**
|
||||
- Add OpenAPI Generator to Bazel build system
|
||||
- Create Bazel rule to generate TypeScript client from spec
|
||||
- Use `typescript-fetch` generator for modern fetch-based client
|
||||
3. **✅ TypeScript Client Generation**
|
||||
- ✅ Use OpenAPI Generator JAR directly (not Bazel module)
|
||||
- ✅ Create Bazel rule to generate TypeScript client from spec
|
||||
- ✅ Use `typescript-fetch` generator for modern fetch-based client
|
||||
- ✅ Configure with camelCase naming and single request parameters
|
||||
|
||||
4. **Bazel Rules**
|
||||
- Create `typescript_openapi_library` rule
|
||||
- Generate client code during build
|
||||
- Ensure proper dependency management
|
||||
4. **✅ Bazel Rules**
|
||||
- ✅ Create `//databuild/client:typescript_client` target
|
||||
- ✅ Generate client code during build with proper file copying
|
||||
- ✅ Ensure hermetic build with JAR download
|
||||
|
||||
## Deliverables
|
||||
|
||||
- [ ] OpenAPI spec generation from Build Graph Service
|
||||
- [ ] TypeScript interfaces for all API request/response types
|
||||
- [ ] Typed HTTP client for Build Graph Service endpoints
|
||||
- [ ] Bazel rules for automated client generation
|
||||
- [ ] Documentation for using the generated client
|
||||
- [x] OpenAPI spec generation from Build Graph Service via CLI flag
|
||||
- [x] TypeScript interfaces for all API request/response types (auto-generated)
|
||||
- [x] Typed HTTP client for Build Graph Service endpoints with fetch API
|
||||
- [x] Bazel rules for automated client generation using OpenAPI Generator JAR
|
||||
- [x] Path parameter struct implementation for aide compatibility
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- OpenAPI spec accurately reflects all service endpoints
|
||||
- Generated TypeScript compiles without errors
|
||||
- Client provides type safety for all API endpoints
|
||||
- Bazel build integrates seamlessly and generates client automatically
|
||||
- Ready for use in Chunk 2 (Hello World App)
|
||||
- ✅ OpenAPI spec accurately reflects all service endpoints with proper path parameters
|
||||
- ✅ Generated TypeScript compiles without errors using `typescript-fetch` generator
|
||||
- ✅ Client provides type safety for all API endpoints with auto-generated method names
|
||||
- ✅ Bazel build integrates seamlessly and generates client automatically
|
||||
- ✅ Path parameter structs enable proper aide OpenAPI generation
|
||||
- ✅ Ready for use in Chunk 2 (Hello World App)
|
||||
|
||||
## Testing
|
||||
|
||||
- Verify OpenAPI spec generation from service
|
||||
- Verify generated TypeScript compiles
|
||||
- Test client against running Build Graph Service
|
||||
- Validate type safety with TypeScript compiler
|
||||
- Ensure build process is hermetic and reproducible
|
||||
- ✅ Verify OpenAPI spec generation from service using CLI flag
|
||||
- ✅ Verify generated TypeScript compiles without errors
|
||||
- ✅ Validate OpenAPI spec passes OpenAPI Generator validation
|
||||
- ✅ Ensure build process is hermetic and reproducible with JAR download
|
||||
- ✅ Verify all API endpoints generate proper TypeScript method signatures
|
||||
- ✅ Confirm path parameters are properly typed in generated client
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Lessons Learned
|
||||
|
||||
1. **aide Path Parameter Requirements**: aide requires path parameters to be wrapped in structs with `JsonSchema` derives, not simple `Path<String>` extractors.
|
||||
|
||||
2. **OpenAPI Generator Bazel Integration**: The official Bazel module was not available in registry, so we used the JAR directly via genrule for better reliability.
|
||||
|
||||
3. **CLI vs HTTP Extraction**: Using a CLI flag for spec extraction is simpler and more reliable than starting an HTTP server.
|
||||
|
||||
4. **Auto-generated Method Names**: Without custom operationIds, OpenAPI Generator creates method names like `apiV1BuildsPost`. This can be improved in future iterations.
|
||||
|
||||
### Build Targets
|
||||
|
||||
- `//databuild/client:extract_openapi_spec` - Extracts OpenAPI spec JSON
|
||||
- `//databuild/client:typescript_client` - Generates TypeScript client
|
||||
- `//databuild/client:typescript` - Main target for consuming the client
|
||||
Loading…
Reference in a new issue