databuild/plans/04-end-to-end-tests-1.md

End-to-End Tests (Phase 1) - Design Document

Overview

This design document outlines the implementation of comprehensive end-to-end tests for DataBuild's core capabilities. The tests will validate that CLI and Service builds produce identical results and events, ensuring consistency across different build interfaces.

Objectives

1. Consistency Validation: Verify that CLI and Service builds produce identical partition events and outputs
2. Event Verification: Ensure expected build events are generated for both build methods
3. Isolation: Use separate log databases to prevent test interference
4. Integration: Implement as sh_test targets to integrate with bazel test //...
5. Performance: Design tests to minimize bazel inefficiency and execution time

Test Scope

Target Examples

• Basic Graph: Simple random number generator with sum operations
• Podcast Reviews: Complex multi-stage data pipeline with dependencies

Test Scenarios

Basic Graph Tests

1. Single Partition Build

   • CLI: bazel-bin/basic_graph.build pippin
   • Service: POST /api/v1/builds {"partitions": ["pippin"]}
   • Verify: Same events, same output files

2. Multiple Partition Build

   • CLI: bazel-bin/basic_graph.build pippin salem sadie
   • Service: POST /api/v1/builds {"partitions": ["pippin", "salem", "sadie"]}
   • Verify: Same events, same output files

3. Sum Partition Build

   • CLI: bazel-bin/basic_graph.build pippin_salem_sadie
   • Service: POST /api/v1/builds {"partitions": ["pippin_salem_sadie"]}
   • Verify: Dependencies built, sum computed correctly

Podcast Reviews Tests

1. Simple Pipeline

   • CLI: bazel-bin/podcast_reviews_graph.build "reviews/date=2020-01-01"
   • Service: POST /api/v1/builds {"partitions": ["reviews/date=2020-01-01"]}
   • Verify: Raw reviews extracted correctly

2. Complex Pipeline

   • CLI: bazel-bin/podcast_reviews_graph.build "daily_summaries/category=Technology/date=2020-01-01"
   • Service: POST /api/v1/builds {"partitions": ["daily_summaries/category=Technology/date=2020-01-01"]}
   • Verify: Full pipeline execution with all intermediate partitions

3. Podcasts Metadata

   • CLI: bazel-bin/podcast_reviews_graph.build "podcasts/all"
   • Service: POST /api/v1/builds {"partitions": ["podcasts/all"]}
   • Verify: Metadata extraction and availability for downstream jobs

Test Architecture

Database Isolation

test_data/
> cli_test_db/           # CLI build event database
> service_test_db/       # Service build event database
> expected_outputs/      # Reference outputs for validation

Test Structure

tests/
> end_to_end/
>> basic_graph_test.sh
>> podcast_reviews_test.sh
>> lib/
>>> test_utils.sh      # Common test utilities
>>> db_utils.sh        # Database comparison utilities
>>> service_utils.sh   # Service management utilities
>> BUILD                  # Bazel test targets

Bazel Integration

# tests/end_to_end/BUILD
sh_test(
    name = "basic_graph_e2e",
    srcs = ["basic_graph_test.sh"],
    data = [
        "//:basic_graph.build",
        "//:basic_graph.service",
        "//tests/end_to_end/lib:test_utils",
    ],
    env = {
        "TEST_DB_DIR": "$(location test_data)",
    },
    size = "medium",
)

sh_test(
    name = "podcast_reviews_e2e", 
    srcs = ["podcast_reviews_test.sh"],
    data = [
        "//:podcast_reviews_graph.build",
        "//:podcast_reviews_graph.service",
        "//tests/end_to_end/lib:test_utils",
        "//examples/podcast_reviews:data",
    ],
    env = {
        "TEST_DB_DIR": "$(location test_data)",
    },
    size = "large",
)

Test Implementation Details

Test Flow

1. Setup: Create isolated test databases and clean output directories
2. CLI Build: Execute CLI build with test database configuration
3. Service Build: Start service with separate test database, execute build via HTTP
4. Comparison: Compare build events, output files, and partition status
5. Cleanup: Stop services and clean test artifacts

Event Validation

• Event Count: Same number of events for identical builds
• Event Types: Same sequence of build events (Started, Progress, Completed, etc.)
• Event Metadata: Same partition references, job names, and timestamps (within tolerance)
• Event Ordering: Proper dependency ordering maintained

Output Validation

• File Existence: Same output files created
• File Content: Identical content (accounting for any timestamp/randomness)
• Partition Status: Same final partition status via API

Service Management

# Start service with test database
start_test_service() {
    local db_path="$1"
    local port="$2"
    
    export BUILD_EVENT_LOG_DB="$db_path"
    bazel-bin/basic_graph.service --port="$port" &
    local service_pid=$!
    
    # Wait for service to be ready
    wait_for_service "http://localhost:$port/health"
    
    echo "$service_pid"
}

Test Efficiency

Basic Optimizations

• Parallel Execution: Tests run in parallel where possible
• Resource Limits: Set appropriate size attributes to prevent resource contention
• Minimal Data: Use minimal test datasets to reduce execution time

CI/CD Integration

• Timeout Handling: Reasonable timeouts for service startup/shutdown
• Retry Logic: Retry flaky network operations
• Artifact Collection: Collect logs and databases on test failure

Risk Mitigation

Test Flakiness

• Deterministic Randomness: Use fixed seeds for reproducible results
• Port Management: Dynamic port allocation to prevent conflicts
• Database Locking: Proper database isolation and cleanup
• Cleanup Guarantees: Ensure cleanup even on test failure

Implementation Plan

Phase 1: Basic Framework

1. Create test directory structure
2. Implement basic test utilities
3. Create simple Basic Graph test
4. Integrate with Bazel

Phase 2: Complete Implementation

1. Add Podcast Reviews tests
2. Implement comprehensive event validation
3. Create CI/CD integration
4. Ensure reliable test execution

Success Criteria

1. Consistency: CLI and Service builds produce identical events and outputs
2. Coverage: All major build scenarios covered for both examples
3. Reliability: Tests pass consistently in CI/CD environment
4. Integration: Tests properly integrated with bazel test //...

Future Enhancements

1. Property-Based Testing: Generate random partition combinations
2. Performance Benchmarking: Track build performance over time
3. Chaos Testing: Test resilience to failures and interruptions
4. Load Testing: Test service under concurrent build requests