Context-Based Timeout Configuration

Overview

The Aitana backend uses a context-aware timeout system that adjusts processing time limits based on where the request originates. This ensures optimal performance for user-facing operations while allowing extended processing for background tasks.

Context Types

1. UI Context (context="ui")

• Timeout: 60 seconds
• Use Case: Interactive web interface
• Characteristics:
  • Fast response required for user experience
  • Suitable for simple queries and small documents
  • Default for all web frontend requests

2. API Context (context="api")

• Timeout: 300 seconds (5 minutes)
• Use Case: Batch processing, CLI tools, automated workflows
• Characteristics:
  • Extended processing for large documents
  • Suitable for complex extraction tasks
  • Used by Aitana CLI batch-extract command

3. Email Context (context="email")

• Timeout: 300 seconds (5 minutes)
• Use Case: Email processing, webhook handlers
• Characteristics:
  • Background processing without user waiting
  • Suitable for document attachments
  • Used by email integration system

4. Background Context (context="background")

• Timeout: 300 seconds (5 minutes)
• Use Case: Scheduled tasks, async processing
• Characteristics:
  • No user interaction required
  • Suitable for batch operations
  • Used by background workers

Implementation Details

GenAI Client Configuration

The timeout is configured in backend/genai_client.py:

def get_genai_client(context="ui"):
    """Create GenAI client with context-aware timeout"""
    
    if context in ["api", "email", "background"]:
        timeout_ms = 300000  # 5 minutes
    else:  # ui or default
        timeout_ms = 60000   # 1 minute
    
    return genai.Client(
        request_options=RequestOptions(
            timeout=timeout_ms
        )
    )

Context Flow Through System

1. Entry Point - Context is specified at request entry:

   # CLI/Batch processing
   process_assistant_request(context="api")
      
   # Web UI
   process_assistant_request(context="ui")
      
   # Email handler
   process_assistant_request(context="email")
   

2. VAC Service - Context propagates through service layer:

   async def vac_stream(**kwargs):
       request_context = kwargs.get("context", "ui")
       # Pass to model calls
   

3. Model Layer - Context reaches AI model configuration:

   async def stream_smart(..., context="ui"):
       # Configure model timeout based on context
   

4. Tool Execution - Tools inherit context from parent:

   async def file_browser_tool(..., context="ui"):
       # Tool timeout matches context
   

Usage Examples

CLI Batch Processing (5-minute timeout)

from assistant_utils import process_assistant_request

result = await process_assistant_request(
    assistant_id="my-assistant",
    user_input="Extract data from large PDF",
    context="api",  # 5-minute timeout
    documents=[{"uri": "gs://bucket/large-file.pdf"}]
)

Web UI Request (60-second timeout)

# Flask/FastAPI endpoint
@app.post("/vac/assistant/{assistant_id}")
async def assistant_endpoint(request):
    # Default context="ui" for web requests
    result = await process_assistant_request(
        assistant_id=assistant_id,
        user_input=request.question,
        context="ui"  # 60-second timeout
    )

Email Processing (5-minute timeout)

# Email webhook handler
async def process_email_attachment(attachment):
    result = await process_assistant_request(
        assistant_id="email-processor",
        user_input="Process attachment",
        context="email",  # 5-minute timeout
        documents=[attachment]
    )

Performance Implications

UI Context (60 seconds)

Pros:

• Fast feedback to users
• Prevents browser timeouts
• Better perceived performance

Cons:

• May fail on large documents
• Complex queries might timeout
• Limited processing capability

API Context (5 minutes)

Pros:

• Handles large documents
• Complex extraction possible
• Better success rate

Cons:

• Longer wait times
• Higher resource usage
• Not suitable for interactive UI

Troubleshooting

Common Issues

1. DEADLINE_EXCEEDED Errors

Symptom: 503 The model is overloaded. Please try again later.

Cause: Processing exceeded context timeout

Solutions:

• For large files, use API context instead of UI
• Reduce document size or complexity
• Simplify extraction prompts
• Use batch processing with smaller chunks

2. Wrong Context Used

Symptom: Batch processing timing out at 60 seconds

Cause: Context parameter not passed through call chain

Solution: Ensure context is passed at every level:

# ✅ Correct
await process_assistant_request(
    ...,
    context="api"
)

# ❌ Wrong (defaults to "ui")
await process_assistant_request(
    ...
    # Missing context parameter
)

3. Context Lost in Tool Chain

Symptom: Tools timeout despite API context

Cause: Tool doesn’t propagate context

Solution: Pass context to all tool calls:

async def my_tool(..., context="ui"):
    # Use context in nested calls
    await another_tool(..., context=context)

Best Practices

1. Choose Appropriate Context

• Interactive operations: Use UI context
• Batch processing: Use API context
• Background jobs: Use background context
• Email/webhooks: Use email context

2. Document Size Guidelines

| Document Size | Recommended Context | Processing Time | |————–|——————-|—————-| | < 1MB | UI | < 30 seconds | | 1-10MB | UI/API | 30-120 seconds | | 10-50MB | API | 2-5 minutes | | > 50MB | API + chunking | 5+ minutes |

3. Prompt Complexity Guidelines

| Prompt Type | Recommended Context | Example | |————|——————-|———| | Simple extraction | UI | “Extract title and date” | | Structured extraction | UI/API | “Extract all obligations with schema” | | Complex analysis | API | “Analyze legal implications and risks” | | Multi-document | API | “Compare these 5 contracts” |

4. Error Handling

Always handle timeout errors gracefully:

try:
    result = await process_assistant_request(
        ...,
        context="ui"
    )
except DeadlineExceeded:
    # Retry with longer timeout
    result = await process_assistant_request(
        ...,
        context="api"
    )

Configuration

Environment Variables

# Optional: Override default timeouts (milliseconds)
GENAI_UI_TIMEOUT_MS=60000
GENAI_API_TIMEOUT_MS=300000
GENAI_EMAIL_TIMEOUT_MS=300000
GENAI_BACKGROUND_TIMEOUT_MS=300000

Dynamic Context Selection

def get_optimal_context(file_size_mb, prompt_complexity):
    """Dynamically select context based on workload"""
    
    if file_size_mb < 1 and prompt_complexity == "simple":
        return "ui"
    elif file_size_mb < 10:
        return "ui" if prompt_complexity == "simple" else "api"
    else:
        return "api"

Migration Guide

Updating Existing Code

1. Identify timeout issues - Look for DEADLINE_EXCEEDED errors
2. Add context parameter - Update function signatures
3. Pass context through - Ensure propagation through call chain
4. Test with large files - Verify 5-minute timeout works

Example Migration

# Before (60-second timeout)
async def process_documents(files):
    for file in files:
        result = await process_assistant_request(
            user_input="Extract data",
            documents=[file]
        )

# After (5-minute timeout)
async def process_documents(files, context="api"):
    for file in files:
        result = await process_assistant_request(
            user_input="Extract data",
            documents=[file],
            context=context  # Pass context
        )

Related Documentation

• Batch Extraction Feature
• Timeout Configuration
• API Timeout Parameters
• Performance Considerations
• VAC Service Architecture