# Service Interface Refactor Plan

## Motivation

The Build Graph Service HTTP API successfully implements core DataBuild functionality but has interface misalignments with the proto definitions that reduce type safety and limit functionality. Key issues:

- **Type Safety Loss**: Analysis responses use `serde_json::Value` instead of structured `JobGraph`
- **Missing Manifest Access**: Build responses only return request IDs, no way to retrieve final manifests
- **Incomplete Dashboard Support**: Missing list/metrics endpoints from design specification
- **Inconsistent Error Handling**: Different patterns between gRPC and HTTP

## Core Interface Alignment

### Fix Analysis Response Type Safety
**Problem**: `AnalyzeResponse` uses generic JSON, losing compile-time guarantees
```rust
// Current: loses type safety
pub job_graph: serde_json::Value

// Target: structured type
pub job_graph: JobGraph
```

**Critical Detail**: Ensure `JobGraph` implements `Serialize` properly and remove `serde_json::to_value()` conversion in handlers.

### Add Manifest Retrieval
**Problem**: HTTP API returns build ID immediately but no way to get final manifests (unlike proto's synchronous `GraphBuildResponse`)

**Solution**: Add `GET /api/v1/builds/:id/manifests` endpoint
- Store manifests in `BuildRequestState` during execution
- Update execute command to capture manifests from job graph results
- Enable both async (current) and manifest retrieval patterns

## Missing Core Functionality

### Dashboard Endpoints
The design specification requires but current API lacks:

1. **Recent Builds**: `GET /api/v1/builds?limit=N`
   - Enable dashboard functionality
   - Support pagination for large build histories

2. **Job Metrics**: `GET /api/v1/jobs/:label/metrics`
   - Aggregate success rates, durations, failure patterns
   - Essential for operational monitoring

3. **Query Interface**: `POST /api/v1/query`
   - Advanced querying beyond predefined endpoints
   - SQL passthrough or structured query DSL

**Implementation Strategy**: Extend event log with aggregation queries, add proper pagination support.

## Error Response Standardization

**Problem**: Current `ErrorResponse {error: String}` lacks structure and traceability

**Target**:
```rust
pub struct ErrorResponse {
    pub error: ErrorDetail,
    pub request_id: Option<String>,
    pub timestamp: i64,
}
```

**Error Code Strategy**:
- `INVALID_PARTITION_REF` - Validation failures
- `JOB_LOOKUP_FAILED` - Graph analysis issues  
- `ANALYSIS_FAILED` / `EXECUTION_FAILED` - Build process errors
- `BUILD_NOT_FOUND` - Resource not found
- `INTERNAL_ERROR` - Server errors

**Critical**: Add request ID middleware for tracing across async operations.

## Protocol Compatibility Strategy

### Dual Protocol Support
**Narrative**: Some clients need gRPC's performance/streaming, others prefer HTTP's simplicity

**Approach**: 
- Implement `DataBuildService` from proto using `tonic`
- Share core logic between HTTP and gRPC handlers
- Add sync mode option: `POST /api/v1/builds?mode=sync` returns manifests directly

**Critical Detail**: Avoid code duplication by extracting shared business logic into service layer.

## Backward Compatibility

**Philosophy**: Never break existing clients during improvements

**Strategy**:
- New endpoints alongside existing ones
- API versioning (`/api/v2`) only for incompatible changes
- Feature flags for experimental functionality
- Deprecation notices with migration paths

## Implementation Priority

### Phase 1: Core Safety (Essential)
1. Fix `AnalyzeResponse` type safety
2. Add manifest retrieval endpoint
3. Standardize error responses with request tracing

**Rationale**: These fix fundamental interface problems affecting all users.

### Phase 2: Dashboard Features (High Value)
1. Recent builds list with pagination
2. Job metrics aggregation
3. Enhanced build state tracking

**Rationale**: Enables operational monitoring and debugging workflows.

### Phase 3: Advanced Features (Complete)
1. Query execution interface
2. gRPC service implementation  
3. Sync/async mode support

**Rationale**: Achieves full design specification compliance and protocol flexibility.

## Critical Success Factors

1. **Type Safety**: All responses use structured types, not generic JSON
2. **Interface Completeness**: Every proto service method has HTTP equivalent
3. **Operational Readiness**: Dashboard endpoints support real monitoring needs
4. **Zero Breakage**: Existing clients continue working throughout transition
5. **Performance**: No regression in build execution times

## Risk Mitigation

**Breaking Changes**: Use versioning strategy and maintain parallel endpoints during transitions

**Performance Impact**: Implement caching for metrics aggregation, rate limiting for expensive queries

**Complexity Growth**: Extract shared business logic to avoid duplication between HTTP/gRPC handlers