Build Error Detection System

Overview

The repository includes an automated build error detection system that creates GitHub issues for Claude when real build failures occur. This system filters out false positives and ensures that only actionable build failures trigger automated issue creation.

Architecture

Components

  1. Claude Workflow (.github/workflows/claude.yml) - Main orchestration
  2. Error Detection Script (scripts/detect-build-errors.sh) - Smart error filtering
  3. GitHub Integration - Automated issue creation

Flow Diagram

Build Failure (Cloud Build) 
    ↓
GitHub Check Run Event
    ↓
Claude Workflow Triggered
    ↓
Extract Build Logs
    ↓
Run Error Detection Script
    ↓
Filter False Positives
    ↓
Real Errors Found? → Yes → Create GitHub Issue
    ↓
No → Skip Issue Creation

Error Detection Script

Location

scripts/detect-build-errors.sh

Features

Smart False Positive Filtering

The script distinguishes between real errors and false positives:

✅ Detects Real Errors:

  • AssertionError - Actual test assertion failures
  • FAILED tests/... - Failed test executions
  • ERROR: build step - Build process failures
  • step exited with non-zero status - Command failures
  • Expected function call mismatches

✅ Filters False Positives:

  • Test names containing “error” that actually passed
  • Success messages like “Build completed without errors”
  • Test descriptions with error-related keywords
  • Error handling test cases that passed

Pattern Recognition

# Real Errors (Detected)
AssertionError: Expected function to be called once. Called 0 times
FAILED tests/integration/user_validation.py::test_user_input_validation
ERROR: build step "Run tests" failed
step exited with non-zero status: 1

# False Positives (Filtered Out)
test_error_handling_validation.py::test_error_cases PASSED
test_error_recovery.py::test_error_recovery_flow PASSED
Build completed without errors.
No errors found in validation.

Usage

Command Line Usage

# Analyze a log file
./scripts/detect-build-errors.sh build.log

# Analyze from stdin
cat build.log | ./scripts/detect-build-errors.sh

# Show help
./scripts/detect-build-errors.sh --help

Exit Codes

  • 0 - No errors detected (clean build)
  • 1 - Real errors found
  • 2 - Script error (invalid arguments, etc.)

Example Output

With Real Errors:

=== BUILD ERRORS DETECTED ===
Found AssertionError (real test failure):
12:AssertionError: Expected function to be called once. Called 0 times

Found failed tests:
16:FAILED tests/integration/user_validation.py::test_user_input_validation

Total error sections found: 2
✗ Build errors found

Clean Build:

=== NO ERRORS DETECTED ===
Build logs appear clean (no real errors found)
Note: Filtered out 3 false positives (test names with 'error')
✓ No build errors detected

Claude Workflow Integration

Trigger Conditions

The workflow is triggered when:

  • Event: check_run completion
  • Branch: refs/heads/dev (dev branch only)
  • Status: failure
  • Source: Cloud Build check runs containing:
    • trigger-aitana-dev-frontend
    • trigger-aitana-dev-backend-api

Workflow Steps

  1. Checkout Repository - Access error detection scripts
  2. Make Scripts Executable - Ensure scripts can run
  3. Extract Build Logs - Get logs from GitHub check run event
  4. Run Error Detection - Execute detect-build-errors.sh on logs
  5. Analyze Results - Determine if real errors were found
  6. Create Issue - Only if real errors detected

Issue Creation Logic

// Only create issue if real errors were detected
if (errorDetectionResult.includes('Real errors detected') || 
    errorDetectionResult.includes('Script error')) {
  // Create GitHub issue with filtered logs
  await github.rest.issues.create({
    title: `Auto-Fix: ${checkRun.name} failed on dev`,
    body: issueBody,
    labels: ['auto-fix', 'claude']
  });
} else {
  // Skip issue creation for false positives
  console.log('Build failure detected but no real errors found');
}

Error Handling

The workflow includes robust error handling:

  • Script Execution Errors - Falls back to raw logs if script fails
  • Missing Logs - Handles cases where build logs aren’t available
  • Timeout Protection - Prevents hanging on large log files
  • Memory Limits - Truncates extremely large logs for GitHub issues

Issue Content

Generated Issue Format

@claude can you analyze and fix these test failures?

## Build Information
**Build URL:** [Link to Cloud Build]
**Branch:** dev
**Commit:** abc123
**Error Detection:** Real errors detected by automated analysis

## Build Logs Analysis
The following logs have been analyzed by our error detection script to filter out false positives from test names containing "error", "failed", etc.

[Filtered error logs here]


## Instructions
Please:
1. Analyze the test failures in the logs above
2. Fix the failing tests or code issues
3. Create a PR with the fixes

**Note:** This issue was created automatically when [build] failed on dev. The error detection script has filtered out likely false positives to reduce noise.

Labels Applied

  • auto-fix - Indicates automated issue creation
  • claude - Assigns to Claude for resolution

Configuration

Environment Variables

  • GH_TOKEN - GitHub token for API access (from secrets)

Script Configuration

The error detection script can be customized by modifying:

  • Error pattern regexes
  • False positive filters
  • Output formatting
  • Logging verbosity

Workflow Customization

Key configuration points in the workflow:

  • Branch filtering (currently dev only)
  • Check run name patterns
  • Log size limits
  • Issue template content

Benefits

Reduced False Positives

Before this system:

  • Issues created for test names containing “error” that passed
  • Noise from success messages mentioning errors
  • Manual triage required for every build failure

After this system:

  • Only real build failures create issues
  • Filtered, relevant error information
  • Automated noise reduction

Improved Issue Quality

  • Focused Logs - Only relevant error sections included
  • Context Preservation - Surrounding lines for debugging context
  • Actionable Content - Clear error messages and failure locations
  • Metadata - Build URLs, commit info, and detection results

Development Efficiency

  • Faster Triage - No need to manually check false positives
  • Better Focus - Claude receives only actionable issues
  • Reduced Noise - Clean issue tracker with real problems only

Troubleshooting

Common Issues

No Issues Created for Build Failures

Possible Causes:

  • Build failure was on wrong branch (not dev)
  • Check run name doesn’t match expected patterns
  • Error detection script found no real errors (all false positives)

Debug Steps:

  1. Check workflow run logs for debug output
  2. Manually run error detection script on build logs
  3. Verify check run event contains expected data

Script Fails to Execute

Possible Causes:

  • Missing execute permissions
  • Shell compatibility issues
  • Missing dependencies (grep, sed, etc.)

Debug Steps:

  1. Check script permissions in workflow logs
  2. Test script locally with sample logs
  3. Review script dependencies

False Positives Still Creating Issues

Possible Causes:

  • New error patterns not covered by filters
  • Changes in test framework output format
  • Edge cases in filtering logic

Debug Steps:

  1. Review the specific logs that triggered false positives
  2. Update filtering patterns in the script
  3. Test improved filters with sample data

Manual Testing

Test Error Detection Script

# Create test log with mixed content
cat > test-log.txt << EOF
test_error_handling.py::test_error_cases PASSED
ERROR: build step failed
FAILED tests/real_test.py::test_function
Build completed without errors
EOF

# Run detection
./scripts/detect-build-errors.sh test-log.txt

Test Workflow Locally

# Simulate workflow environment
export GITHUB_OUTPUT=/tmp/github_output
export GITHUB_SHA=test123

# Run relevant sections manually
echo "Mock build logs" | ./scripts/detect-build-errors.sh

Maintenance

Regular Tasks

Monitor Detection Accuracy

  • Review created issues for false positives
  • Check missed errors in build failures
  • Update filtering patterns as needed

Update Error Patterns

  • Add new error types as they appear
  • Refine existing patterns for better accuracy
  • Test pattern changes with historical data

Workflow Performance

  • Monitor workflow execution times
  • Optimize log processing for large builds
  • Review memory usage and limits

Evolution

The error detection system should evolve with:

  • New testing frameworks and error formats
  • Changes in build pipeline structure
  • Updates to CI/CD systems
  • New types of build failures

Consider periodic reviews of:

  • Detection accuracy metrics
  • Issue creation rates
  • Developer feedback on issue quality
  • Build failure patterns and trends