195 lines
No EOL
6.6 KiB
Markdown
195 lines
No EOL
6.6 KiB
Markdown
# End-to-End Tests (Phase 1) - Design Document
|
|
|
|
## Overview
|
|
|
|
This design document outlines the implementation of comprehensive end-to-end tests for DataBuild's core capabilities. The tests will validate that CLI and Service builds produce identical results and events, ensuring consistency across different build interfaces.
|
|
|
|
## Objectives
|
|
|
|
1. **Consistency Validation**: Verify that CLI and Service builds produce identical partition events and outputs
|
|
2. **Event Verification**: Ensure expected build events are generated for both build methods
|
|
3. **Isolation**: Use separate log databases to prevent test interference
|
|
4. **Integration**: Implement as `sh_test` targets to integrate with `bazel test //...`
|
|
5. **Performance**: Design tests to minimize bazel inefficiency and execution time
|
|
|
|
## Test Scope
|
|
|
|
### Target Examples
|
|
- **Basic Graph**: Simple random number generator with sum operations
|
|
- **Podcast Reviews**: Complex multi-stage data pipeline with dependencies
|
|
|
|
### Test Scenarios
|
|
|
|
#### Basic Graph Tests
|
|
1. **Single Partition Build**
|
|
- CLI: `bazel-bin/basic_graph.build pippin`
|
|
- Service: `POST /api/v1/builds {"partitions": ["pippin"]}`
|
|
- Verify: Same events, same output files
|
|
|
|
2. **Multiple Partition Build**
|
|
- CLI: `bazel-bin/basic_graph.build pippin salem sadie`
|
|
- Service: `POST /api/v1/builds {"partitions": ["pippin", "salem", "sadie"]}`
|
|
- Verify: Same events, same output files
|
|
|
|
3. **Sum Partition Build**
|
|
- CLI: `bazel-bin/basic_graph.build pippin_salem_sadie`
|
|
- Service: `POST /api/v1/builds {"partitions": ["pippin_salem_sadie"]}`
|
|
- Verify: Dependencies built, sum computed correctly
|
|
|
|
#### Podcast Reviews Tests
|
|
1. **Simple Pipeline**
|
|
- CLI: `bazel-bin/podcast_reviews_graph.build "reviews/date=2020-01-01"`
|
|
- Service: `POST /api/v1/builds {"partitions": ["reviews/date=2020-01-01"]}`
|
|
- Verify: Raw reviews extracted correctly
|
|
|
|
2. **Complex Pipeline**
|
|
- CLI: `bazel-bin/podcast_reviews_graph.build "daily_summaries/category=Technology/date=2020-01-01"`
|
|
- Service: `POST /api/v1/builds {"partitions": ["daily_summaries/category=Technology/date=2020-01-01"]}`
|
|
- Verify: Full pipeline execution with all intermediate partitions
|
|
|
|
3. **Podcasts Metadata**
|
|
- CLI: `bazel-bin/podcast_reviews_graph.build "podcasts/all"`
|
|
- Service: `POST /api/v1/builds {"partitions": ["podcasts/all"]}`
|
|
- Verify: Metadata extraction and availability for downstream jobs
|
|
|
|
## Test Architecture
|
|
|
|
### Database Isolation
|
|
```
|
|
test_data/
|
|
> cli_test_db/ # CLI build event database
|
|
> service_test_db/ # Service build event database
|
|
> expected_outputs/ # Reference outputs for validation
|
|
```
|
|
|
|
### Test Structure
|
|
```
|
|
tests/
|
|
> end_to_end/
|
|
>> basic_graph_test.sh
|
|
>> podcast_reviews_test.sh
|
|
>> lib/
|
|
>>> test_utils.sh # Common test utilities
|
|
>>> db_utils.sh # Database comparison utilities
|
|
>>> service_utils.sh # Service management utilities
|
|
>> BUILD # Bazel test targets
|
|
```
|
|
|
|
### Bazel Integration
|
|
```python
|
|
# tests/end_to_end/BUILD
|
|
sh_test(
|
|
name = "basic_graph_e2e",
|
|
srcs = ["basic_graph_test.sh"],
|
|
data = [
|
|
"//:basic_graph.build",
|
|
"//:basic_graph.service",
|
|
"//tests/end_to_end/lib:test_utils",
|
|
],
|
|
env = {
|
|
"TEST_DB_DIR": "$(location test_data)",
|
|
},
|
|
size = "medium",
|
|
)
|
|
|
|
sh_test(
|
|
name = "podcast_reviews_e2e",
|
|
srcs = ["podcast_reviews_test.sh"],
|
|
data = [
|
|
"//:podcast_reviews_graph.build",
|
|
"//:podcast_reviews_graph.service",
|
|
"//tests/end_to_end/lib:test_utils",
|
|
"//examples/podcast_reviews:data",
|
|
],
|
|
env = {
|
|
"TEST_DB_DIR": "$(location test_data)",
|
|
},
|
|
size = "large",
|
|
)
|
|
```
|
|
|
|
## Test Implementation Details
|
|
|
|
### Test Flow
|
|
1. **Setup**: Create isolated test databases and clean output directories
|
|
2. **CLI Build**: Execute CLI build with test database configuration
|
|
3. **Service Build**: Start service with separate test database, execute build via HTTP
|
|
4. **Comparison**: Compare build events, output files, and partition status
|
|
5. **Cleanup**: Stop services and clean test artifacts
|
|
|
|
### Event Validation
|
|
- **Event Count**: Same number of events for identical builds
|
|
- **Event Types**: Same sequence of build events (Started, Progress, Completed, etc.)
|
|
- **Event Metadata**: Same partition references, job names, and timestamps (within tolerance)
|
|
- **Event Ordering**: Proper dependency ordering maintained
|
|
|
|
### Output Validation
|
|
- **File Existence**: Same output files created
|
|
- **File Content**: Identical content (accounting for any timestamp/randomness)
|
|
- **Partition Status**: Same final partition status via API
|
|
|
|
### Service Management
|
|
```bash
|
|
# Start service with test database
|
|
start_test_service() {
|
|
local db_path="$1"
|
|
local port="$2"
|
|
|
|
export BUILD_EVENT_LOG_DB="$db_path"
|
|
bazel-bin/basic_graph.service --port="$port" &
|
|
local service_pid=$!
|
|
|
|
# Wait for service to be ready
|
|
wait_for_service "http://localhost:$port/health"
|
|
|
|
echo "$service_pid"
|
|
}
|
|
```
|
|
|
|
## Test Efficiency
|
|
|
|
### Basic Optimizations
|
|
- **Parallel Execution**: Tests run in parallel where possible
|
|
- **Resource Limits**: Set appropriate `size` attributes to prevent resource contention
|
|
- **Minimal Data**: Use minimal test datasets to reduce execution time
|
|
|
|
### CI/CD Integration
|
|
- **Timeout Handling**: Reasonable timeouts for service startup/shutdown
|
|
- **Retry Logic**: Retry flaky network operations
|
|
- **Artifact Collection**: Collect logs and databases on test failure
|
|
|
|
## Risk Mitigation
|
|
|
|
### Test Flakiness
|
|
- **Deterministic Randomness**: Use fixed seeds for reproducible results
|
|
- **Port Management**: Dynamic port allocation to prevent conflicts
|
|
- **Database Locking**: Proper database isolation and cleanup
|
|
- **Cleanup Guarantees**: Ensure cleanup even on test failure
|
|
|
|
## Implementation Plan
|
|
|
|
### Phase 1: Basic Framework
|
|
1. Create test directory structure
|
|
2. Implement basic test utilities
|
|
3. Create simple Basic Graph test
|
|
4. Integrate with Bazel
|
|
|
|
### Phase 2: Complete Implementation
|
|
1. Add Podcast Reviews tests
|
|
2. Implement comprehensive event validation
|
|
3. Create CI/CD integration
|
|
4. Ensure reliable test execution
|
|
|
|
## Success Criteria
|
|
|
|
1. **Consistency**: CLI and Service builds produce identical events and outputs
|
|
2. **Coverage**: All major build scenarios covered for both examples
|
|
3. **Reliability**: Tests pass consistently in CI/CD environment
|
|
4. **Integration**: Tests properly integrated with `bazel test //...`
|
|
|
|
## Future Enhancements
|
|
|
|
1. **Property-Based Testing**: Generate random partition combinations
|
|
2. **Performance Benchmarking**: Track build performance over time
|
|
3. **Chaos Testing**: Test resilience to failures and interruptions
|
|
4. **Load Testing**: Test service under concurrent build requests |