Test Gap Detection Workflow (Simplified)

Efficient test coverage analysis using existing Cloud Build coverage reports

Overview

The simplified test gap detection workflow analyzes test coverage gaps by downloading and analyzing existing coverage reports from Cloud Build, rather than re-running tests. This approach is faster, more reliable, and uses the same coverage data that developers see in their development workflow.

How It Works

1. Coverage Data Sources

Frontend Coverage:

  • Primary Source: gs://aitana-public-bucket/coverage-reports/frontend-${BRANCH_NAME}/coverage-summary.json
  • Fallback Source: gs://aitana-public-bucket/coverage-badges/frontend-${BRANCH_NAME}-coverage.json
  • Generated by: Frontend Cloud Build step frontend-coverage

Backend Coverage:

  • Source: gs://aitana-public-bucket/coverage-badges/backend-api-${BRANCH_NAME}-coverage.json
  • Generated by: Backend Cloud Build step run-tests

2. Analysis Process

  1. Download Coverage Reports
    • Fetch coverage data from GCS buckets
    • Handle fallback scenarios when primary sources are unavailable
    • Convert badge format to analysis format when needed
  2. Analyze Coverage Gaps
    • Use scripts/test_coverage_analyzer.py to identify gaps
    • Categorize gaps as Critical (<50%) or Moderate (50-70%)
    • Calculate overall coverage percentage
  3. Create GitHub Issues
    • Automatically create issues when critical gaps > 2 files
    • Create issues when overall coverage < 70%
    • Update existing issues rather than creating duplicates

3. Workflow Triggers

  • Scheduled: Daily at 3 AM UTC
  • Manual: Via GitHub Actions interface or CLI
  • Branch Selection: Supports dev, test, prod branches

Manual Execution

Via GitHub CLI

# Analyze dev branch (default)
gh workflow run test-gap-detection-simple.yml --ref dev

# Analyze specific branch
gh workflow run test-gap-detection-simple.yml --ref dev -f branch_name=test

Via GitHub Web Interface

  1. Navigate to Actions → Test Gap Detection (Simplified)
  2. Click Run workflow
  3. Select branch to analyze (dev/test/prod)
  4. Click Run workflow

Local Testing

Test Coverage Analyzer Locally

# Download real coverage data
curl -s "https://storage.googleapis.com/aitana-public-bucket/coverage-badges/frontend-dev-coverage.json" -o frontend-test.json
curl -s "https://storage.googleapis.com/aitana-public-bucket/coverage-badges/backend-api-dev-coverage.json" -o backend-test.json

# Run analyzer
python3 scripts/test_coverage_analyzer.py frontend-test.json backend-test.json --output json

# Test with markdown output
python3 scripts/test_coverage_analyzer.py frontend-test.json backend-test.json > coverage-analysis.md

Test Workflow Script Integration

# Run workflow script tests
npm test src/__tests__/workflow-scripts.test.ts

# Test specific coverage analyzer functionality
npm run test:workflow-deps

Analysis Output

Coverage Metrics

Critical Gaps: Files with <50% test coverage Moderate Gaps: Files with 50-70% test coverage Overall Coverage: Combined frontend and backend coverage percentage

GitHub Issue Format

When coverage gaps are detected, the workflow creates issues with:

  • Title: 🧪 Test Coverage Gaps - {BRANCH} Branch ({COVERAGE}%)
  • Labels: test-gap, auto-test-suggestion
  • Content:
    • Coverage summary and analysis
    • Links to full coverage reports
    • Specific action items for improvement
    • Testing strategy recommendations

Example Issue Content

# 🧪 Test Coverage Analysis - dev Branch

**Generated**: 2024-01-15 03:00:00
**Branch**: dev
**Overall Coverage**: 68%
**Critical Gaps**: 3

## 📊 Coverage Summary

- Frontend Coverage: 72%
- Backend Coverage: 65%
- Critical files needing tests: 3
- Moderate gaps: 8 files

## 🔗 Coverage Reports

- [Frontend Coverage](https://storage.googleapis.com/aitana-public-bucket/coverage-reports/frontend-dev/index.html)
- [Backend Badge](https://storage.googleapis.com/aitana-public-bucket/coverage-badges/backend-api-dev-coverage.json)

## 📋 Action Items

1. **Address critical gaps** in files with <50% coverage
2. **Add integration tests** for API endpoints
3. **Improve component test coverage** for complex UI components
4. **Review backend test coverage** especially for new features

Advantages Over Complex Approach

✅ Efficiency

  • No test re-execution: Uses existing Cloud Build coverage
  • Fast execution: Downloads small JSON files vs running full test suites
  • Resource optimization: No additional compute time for tests

✅ Reliability

  • Same data source: Uses identical coverage data that developers see
  • Consistent results: No variance from test execution environment differences
  • Proven data: Coverage reports already validated by Cloud Build

✅ Simplicity

  • Single job workflow: No complex multi-step test execution
  • Clear data flow: Direct download → analyze → create issue
  • Easy debugging: Clear source of truth for coverage data

✅ Maintenance

  • No test environment setup: No Node.js, Python env complexity
  • No dependency management: Minimal workflow dependencies
  • Easy updates: Changes only affect analysis logic, not test execution

Configuration

Environment Variables

env:
  TEST_BOT_NAME: "claude"
  TEST_GAP_LABEL: "test-gap"
  AUTO_TEST_LABEL: "auto-test-suggestion"

Coverage Thresholds

  • Critical Gap Threshold: <50% coverage
  • Moderate Gap Threshold: 50-70% coverage
  • Issue Creation Threshold: >2 critical gaps OR <70% overall coverage

GCS Bucket Configuration

The workflow expects coverage reports to be available at:

  • gs://aitana-public-bucket/coverage-reports/frontend-{BRANCH}/
  • gs://aitana-public-bucket/coverage-badges/frontend-{BRANCH}-coverage.json
  • gs://aitana-public-bucket/coverage-badges/backend-api-{BRANCH}-coverage.json

Monitoring and Debugging

Workflow Execution Logs

Check GitHub Actions logs for:

  • Coverage download success/failure
  • Analysis output and results
  • Issue creation status

Coverage Data Validation

# Verify coverage data is available
curl -I "https://storage.googleapis.com/aitana-public-bucket/coverage-badges/frontend-dev-coverage.json"
curl -I "https://storage.googleapis.com/aitana-public-bucket/coverage-badges/backend-api-dev-coverage.json"

# Download and inspect coverage format
curl -s "https://storage.googleapis.com/aitana-public-bucket/coverage-badges/frontend-dev-coverage.json" | jq .

Troubleshooting Common Issues

Coverage data not found:

  • Verify Cloud Build completed successfully for the target branch
  • Check that coverage generation steps ran without errors
  • Ensure GCS permissions allow public read access

Analysis script errors:

  • Test Python scripts locally with downloaded coverage data
  • Verify coverage JSON format matches expected structure
  • Check that fallback mechanisms work correctly

Issue creation failures:

  • Verify GitHub token permissions include issues: write
  • Check that required labels exist in the repository
  • Ensure assignee user exists and has repository access

Future Enhancements

Potential Improvements

  1. Coverage Trend Analysis: Track coverage changes over time
  2. File-Specific Recommendations: Generate specific test suggestions per file
  3. Integration with PR Comments: Add coverage analysis to pull request reviews
  4. Custom Threshold Configuration: Allow branch-specific coverage thresholds
  5. Slack/Email Notifications: Alert teams when coverage drops significantly

Integration Opportunities

  • PR Status Checks: Block PRs that significantly decrease coverage
  • Dashboard Integration: Create coverage dashboard using this data
  • Automated Test Generation: Use coverage gaps to suggest test additions
  • Team Metrics: Track team-specific coverage improvement goals