Test Auto-Fix with Claude Code

This system automatically detects backend test failures and leverages the existing Claude Code GitHub Action to fix them. Unlike complex custom solutions, this approach uses the same proven Claude Code framework that’s already integrated into the repository.

How It Works

🔄 Simple & Elegant Flow

  1. Test Failure Detection: GitHub Action monitors for backend test failures
  2. Issue Creation: Automatically creates a GitHub issue with failure details
  3. Claude Invocation: Tags @claude in the issue to trigger the existing Claude Code action
  4. Intelligent Analysis: Claude analyzes the test failures using the same AI framework you’re already using
  5. Automatic Fix: Claude creates a PR with the fixes using its proven capabilities

🎯 Zero Maintenance Approach

  • No Custom API Keys: Uses existing Claude Code integration
  • No Python Scripts: Pure GitHub Actions + Claude Code
  • No Maintenance: Leverages the existing, proven Claude Code workflow
  • Auto-Updates: Benefits from Claude Code improvements automatically

Trigger Methods

Test the system manually through GitHub Actions:

# Go to: Actions → Test Auto-Fix with Claude Code → Run workflow
# Select: Force check = true
# Optional: Add custom test failure log

2. Cloud Build Integration

Configure Cloud Build to trigger the workflow on test failures:

# Add to your Cloud Build trigger webhook
curl -X POST \
  -H "Authorization: token $GITHUB_TOKEN" \
  -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/repos/OWNER/REPO/dispatches \
  -d '{
    "event_type": "test_failure",
    "client_payload": {
      "status": "FAILURE",
      "log_url": "https://storage.cloud.google.com/bucket/log.txt",
      "build_url": "https://console.cloud.google.com/cloud-build/builds/..."
    }
  }'

3. Workflow Failure Detection

Automatically triggers when GitHub workflows fail (when backend workflows exist).

Claude Code Integration

What Makes This Powerful

The system leverages the existing anthropics/claude-code-action@beta that’s already configured in .github/workflows/claude.yml. This means:

  • Same AI Model: Uses the same Claude model that handles all your code requests
  • Same Context: Has access to the full repository context
  • Same Capabilities: All the code analysis and fixing capabilities you’re used to
  • Same Safety: Same review process through PR creation

Issue Template

When a test failure is detected, the system creates an issue like this:

@claude can you analyze and fix these test failures?

## Test Failure Details
**Build URL:** https://console.cloud.google.com/cloud-build/builds/...
**Branch:** dev
**Commit:** abc123...

## Failure Log

Step #4 - “run-tests”: tests/test_multiprocess_fix.py::test_load_with_retry_on_failure FAILED Step #4 - “run-tests”: > assert calls == 3 # Should have taken 3 attempts
Step #4 - “run-tests”: E assert 2 == 3


## Instructions
Please:
1. Analyze the test failures above
2. Identify the root causes
3. Fix the failing tests or underlying code issues  
4. Create a PR with the fixes
5. Verify the fixes work by running the tests

Claude then processes this just like any other code request you make!

Setup Instructions

1. Install the Workflow

# Move workflow to correct location
mv scripts/workflows/test-auto-fix-claude.yml .github/workflows/test-auto-fix-claude.yml

2. Test the System

  1. Go to Actions → Test Auto-Fix with Claude Code
  2. Click Run workflow
  3. Enable Force check = true
  4. Click Run workflow

This will create a test issue with the example failures from issue #779.

3. Verify Claude Response

  1. Check the created issue
  2. Verify Claude responds and analyzes the failures
  3. Check that Claude creates a PR with fixes
  4. Review and merge the fixes

4. Configure Cloud Build (Optional)

To automatically trigger on real test failures, add a webhook to your Cloud Build trigger that sends repository dispatch events on failure.

Benefits Over Custom Solutions

✅ Advantages

  • Zero API Management: No need for separate Anthropic API keys
  • Proven Reliability: Uses the same action that handles all your Claude requests
  • No Code Maintenance: No custom Python scripts to maintain
  • Automatic Updates: Benefits from Claude Code action improvements
  • Same Review Process: PRs created through the same trusted process
  • Full Repository Context: Claude has access to the entire codebase
  • Integrated Workflow: Works seamlessly with existing development process

🔧 Simplicity

  • Single Workflow File: Just one GitHub Actions file
  • Standard Integration: Uses existing Claude Code action
  • No Dependencies: No additional packages or services
  • Easy Testing: Manual trigger for validation

Example Test Failures Handled

The system can handle any test failure that Claude Code can analyze, including:

Python/pytest Failures

  • Assertion errors
  • Mock configuration issues
  • Async test problems
  • Import/dependency issues
  • Configuration mismatches

Node.js/Vitest Failures

  • Component test failures
  • TypeScript compilation errors
  • Mock setup issues
  • Integration test problems

Any Framework

Claude Code’s AI capabilities can analyze and fix failures from any testing framework by understanding the error context and codebase patterns.

Monitoring & Troubleshooting

Issue Labels

All auto-created issues are tagged with:

  • auto-fix - Indicates automatic issue creation
  • test-failure - Indicates test failure context
  • claude - Ensures Claude picks up the issue

Manual Intervention

If Claude needs clarification or assistance:

  1. Claude will ask questions in the issue comments
  2. You can provide additional context
  3. Claude will continue with the fix process

Failure Recovery

If the system doesn’t work as expected:

  1. Check the workflow run logs in Actions
  2. Verify the issue was created correctly
  3. Ensure Claude Code action has proper permissions
  4. Test with manual trigger first

Advanced Configuration

Custom Failure Detection

Modify the workflow to detect specific failure patterns:

# In test-auto-fix-claude.yml
- name: Enhanced failure detection  
  run: |
    # Add custom logic for your specific test patterns
    FAILURE_LOG=$(grep -E "FAILED|ERROR|AssertionError|TimeoutError" build_log.txt)

Custom Issue Templates

Modify the issue creation to include specific instructions:

// In GitHub Actions script
body: `@claude can you analyze and fix these test failures?

## Additional Context
- These are critical production tests
- Please prioritize safety over speed
- Include verification steps in your PR

## Test Failure Details...`

This approach gives you the power of automated test fixing while leveraging the robust, proven Claude Code framework you’re already using!