# End-to-End Tests (Phase 1) - Design Document

## Overview

This design document outlines the implementation of comprehensive end-to-end tests for DataBuild's core capabilities. The tests will validate that CLI and Service builds produce identical results and events, ensuring consistency across different build interfaces.

## Objectives

1. **Consistency Validation**: Verify that CLI and Service builds produce identical partition events and outputs
2. **Event Verification**: Ensure expected build events are generated for both build methods
3. **Isolation**: Use separate log databases to prevent test interference
4. **Integration**: Implement as `sh_test` targets to integrate with `bazel test //...`
5. **Performance**: Design tests to minimize bazel inefficiency and execution time

## Test Scope

### Target Examples
- **Basic Graph**: Simple random number generator with sum operations
- **Podcast Reviews**: Complex multi-stage data pipeline with dependencies

### Test Scenarios

#### Basic Graph Tests
1. **Single Partition Build**
   - CLI: `bazel-bin/basic_graph.build pippin`
   - Service: `POST /api/v1/builds {"partitions": ["pippin"]}`
   - Verify: Same events, same output files

2. **Multiple Partition Build**
   - CLI: `bazel-bin/basic_graph.build pippin salem sadie`
   - Service: `POST /api/v1/builds {"partitions": ["pippin", "salem", "sadie"]}`
   - Verify: Same events, same output files

3. **Sum Partition Build**
   - CLI: `bazel-bin/basic_graph.build pippin_salem_sadie`
   - Service: `POST /api/v1/builds {"partitions": ["pippin_salem_sadie"]}`
   - Verify: Dependencies built, sum computed correctly

#### Podcast Reviews Tests
1. **Simple Pipeline**
   - CLI: `bazel-bin/podcast_reviews_graph.build "reviews/date=2020-01-01"`
   - Service: `POST /api/v1/builds {"partitions": ["reviews/date=2020-01-01"]}`
   - Verify: Raw reviews extracted correctly

2. **Complex Pipeline**
   - CLI: `bazel-bin/podcast_reviews_graph.build "daily_summaries/category=Technology/date=2020-01-01"`
   - Service: `POST /api/v1/builds {"partitions": ["daily_summaries/category=Technology/date=2020-01-01"]}`
   - Verify: Full pipeline execution with all intermediate partitions

3. **Podcasts Metadata**
   - CLI: `bazel-bin/podcast_reviews_graph.build "podcasts/all"`
   - Service: `POST /api/v1/builds {"partitions": ["podcasts/all"]}`
   - Verify: Metadata extraction and availability for downstream jobs

## Test Architecture

### Database Isolation
```
test_data/
> cli_test_db/           # CLI build event database
> service_test_db/       # Service build event database
> expected_outputs/      # Reference outputs for validation
```

### Test Structure
```
tests/
> end_to_end/
>> basic_graph_test.sh
>> podcast_reviews_test.sh
>> lib/
>>> test_utils.sh      # Common test utilities
>>> db_utils.sh        # Database comparison utilities
>>> service_utils.sh   # Service management utilities
>> BUILD                  # Bazel test targets
```

### Bazel Integration
```python
# tests/end_to_end/BUILD
sh_test(
    name = "basic_graph_e2e",
    srcs = ["basic_graph_test.sh"],
    data = [
        "//:basic_graph.build",
        "//:basic_graph.service",
        "//tests/end_to_end/lib:test_utils",
    ],
    env = {
        "TEST_DB_DIR": "$(location test_data)",
    },
    size = "medium",
)

sh_test(
    name = "podcast_reviews_e2e", 
    srcs = ["podcast_reviews_test.sh"],
    data = [
        "//:podcast_reviews_graph.build",
        "//:podcast_reviews_graph.service",
        "//tests/end_to_end/lib:test_utils",
        "//examples/podcast_reviews:data",
    ],
    env = {
        "TEST_DB_DIR": "$(location test_data)",
    },
    size = "large",
)
```

## Test Implementation Details

### Test Flow
1. **Setup**: Create isolated test databases and clean output directories
2. **CLI Build**: Execute CLI build with test database configuration
3. **Service Build**: Start service with separate test database, execute build via HTTP
4. **Comparison**: Compare build events, output files, and partition status
5. **Cleanup**: Stop services and clean test artifacts

### Event Validation
- **Event Count**: Same number of events for identical builds
- **Event Types**: Same sequence of build events (Started, Progress, Completed, etc.)
- **Event Metadata**: Same partition references, job names, and timestamps (within tolerance)
- **Event Ordering**: Proper dependency ordering maintained

### Output Validation
- **File Existence**: Same output files created
- **File Content**: Identical content (accounting for any timestamp/randomness)
- **Partition Status**: Same final partition status via API

### Service Management
```bash
# Start service with test database
start_test_service() {
    local db_path="$1"
    local port="$2"
    
    export BUILD_EVENT_LOG_DB="$db_path"
    bazel-bin/basic_graph.service --port="$port" &
    local service_pid=$!
    
    # Wait for service to be ready
    wait_for_service "http://localhost:$port/health"
    
    echo "$service_pid"
}
```

## Test Efficiency

### Basic Optimizations
- **Parallel Execution**: Tests run in parallel where possible
- **Resource Limits**: Set appropriate `size` attributes to prevent resource contention
- **Minimal Data**: Use minimal test datasets to reduce execution time

### CI/CD Integration
- **Timeout Handling**: Reasonable timeouts for service startup/shutdown
- **Retry Logic**: Retry flaky network operations
- **Artifact Collection**: Collect logs and databases on test failure

## Risk Mitigation

### Test Flakiness
- **Deterministic Randomness**: Use fixed seeds for reproducible results
- **Port Management**: Dynamic port allocation to prevent conflicts
- **Database Locking**: Proper database isolation and cleanup
- **Cleanup Guarantees**: Ensure cleanup even on test failure

## Implementation Plan

### Phase 1: Basic Framework
1. Create test directory structure
2. Implement basic test utilities
3. Create simple Basic Graph test
4. Integrate with Bazel

### Phase 2: Complete Implementation
1. Add Podcast Reviews tests
2. Implement comprehensive event validation
3. Create CI/CD integration
4. Ensure reliable test execution

## Success Criteria

1. **Consistency**: CLI and Service builds produce identical events and outputs
2. **Coverage**: All major build scenarios covered for both examples
3. **Reliability**: Tests pass consistently in CI/CD environment
4. **Integration**: Tests properly integrated with `bazel test //...`

## Future Enhancements

1. **Property-Based Testing**: Generate random partition combinations
2. **Performance Benchmarking**: Track build performance over time
3. **Chaos Testing**: Test resilience to failures and interruptions
4. **Load Testing**: Test service under concurrent build requests