Tool Confirmation System

Overview

The tool confirmation system provides an interactive UI for users to review and modify tool configurations chosen by the assistant before execution. It uses React Context to manage state and trigger API calls with confirmed tools.

Table of Contents

• Architecture
• Usage Guide
• Implementation Flow
• Testing
• API Integration

Architecture

The tool confirmation system uses React Context to manage state and trigger API calls with confirmed tools.

Key Components

1. EmissaryContext (src/contexts/EmissaryContext.tsx)
   • Manages tool confirmation state
   • Stores pending tools for confirmation
   • Stores confirmed tool selections
2. ToolConfirmation (src/components/ToolConfirmation.tsx)
   • Displays pending tools with configurations
   • Allows users to modify tool selections
   • Triggers API call with confirmed tools
3. Emissary (src/components/Emissary.tsx)
   • Integrates confirmation flow
   • Manages message display
   • Handles API responses

State Management

The confirmation state is managed in EmissaryContext:

interface ToolConfirmationState {
  pendingToolsMessage?: string;
  pendingToolsList?: string[];
  pendingToolConfigs?: Record<string, any>;
  confirmedTools?: string[];
  confirmedToolConfigs?: Record<string, any>;
}

Usage Guide

Basic Usage

import { ToolConfirmation } from '@/components/ToolConfirmation';
import { useEmissary } from '@/contexts/EmissaryContext';

function AssistantInterface() {
  const { 
    pendingToolsMessage,
    pendingToolsList,
    pendingToolConfigs,
    resetToolConfirmation
  } = useEmissary();

  if (pendingToolsMessage && pendingToolsList) {
    return (
      <ToolConfirmation
        message={pendingToolsMessage}
        tools={pendingToolsList}
        toolConfigs={pendingToolConfigs}
        onCancel={resetToolConfirmation}
      />
    );
  }

  // Regular chat interface
  return <ChatInterface />;
}

Triggering Tool Confirmation

When the assistant response includes tool suggestions:

// In the message processing logic
if (assistantMessage.includes('TOOL_CONFIRMATION:')) {
  const toolsData = parseToolConfirmation(assistantMessage);
  
  setPendingToolsMessage(toolsData.message);
  setPendingToolsList(toolsData.tools);
  setPendingToolConfigs(toolsData.configs);
  setShowToolConfirmation(true);
}

Handling User Confirmation

When the user confirms or modifies tools:

const handleConfirm = (selectedTools: string[], configs: Record<string, any>) => {
  setConfirmedTools(selectedTools);
  setConfirmedToolConfigs(configs);
  setShowToolConfirmation(false);
  
  // Trigger API call with confirmed tools
  handleSendMessage(pendingToolsMessage, {
    tools_to_use: selectedTools,
    tool_configs: configs
  });
};

Implementation Flow

1. Initial Message Flow

sequenceDiagram
    User->>Chat: Send message
    Chat->>Backend: POST /vac/assistant/{id}
    Backend->>Assistant: Process with tools
    Assistant->>Backend: Request tool confirmation
    Backend->>Chat: Response with TOOL_CONFIRMATION
    Chat->>ToolConfirmation: Display tool options

2. Confirmation Flow

sequenceDiagram
    ToolConfirmation->>User: Show tool options
    User->>ToolConfirmation: Select/modify tools
    ToolConfirmation->>EmissaryContext: Set confirmed tools
    EmissaryContext->>Chat: Trigger API call
    Chat->>Backend: POST with confirmed tools
    Backend->>Assistant: Execute with tools
    Assistant->>Backend: Final response
    Backend->>Chat: Stream response

3. State Updates

1. Pending State: Tools awaiting confirmation

   {
     pendingToolsMessage: "I'll search for that information",
     pendingToolsList: ["google-search", "ai-search"],
     pendingToolConfigs: { "ai-search": { datastore_id: "docs" } }
   }
   

2. Confirmed State: User-approved tools

   {
     confirmedTools: ["google-search"],
     confirmedToolConfigs: {}
   }
   

3. Reset State: Clear after processing

   resetToolConfirmation();
   

Testing

Test Coverage Summary

The tool confirmation functionality has comprehensive test coverage across all layers:

Unit Tests

ToolConfirmation Component Tests

Location: src/components/__tests__/ToolConfirmation.test.tsx

describe('ToolConfirmation', () => {
  it('displays pending tools correctly');
  it('allows toggling tool selection');
  it('shows tool configurations when available');
  it('calls onConfirm with selected tools');
  it('calls onCancel when cancelled');
  it('handles empty tool list gracefully');
});

EmissaryContext Tests

Location: src/contexts/__tests__/EmissaryContext.test.tsx

describe('EmissaryContext - Tool Confirmation', () => {
  it('stores pending tools state');
  it('updates confirmed tools');
  it('resets confirmation state');
  it('triggers API call with confirmed tools');
});

Integration Tests

Full Flow Test

Location: src/components/__tests__/Emissary.integration.test.tsx

describe('Tool Confirmation Integration', () => {
  it('completes full confirmation flow', async () => {
    // 1. Send initial message
    // 2. Receive tool confirmation request
    // 3. Display confirmation UI
    // 4. User confirms tools
    // 5. API call with confirmed tools
    // 6. Display final response
  });
});

E2E Tests

Location: cypress/e2e/tool-confirmation.cy.ts

describe('Tool Confirmation E2E', () => {
  it('user can confirm suggested tools', () => {
    cy.visit('/assistant/test-id');
    cy.get('[data-testid="chat-input"]').type('Search for Python tutorials');
    cy.get('[data-testid="send-button"]').click();
    
    // Wait for tool confirmation
    cy.get('[data-testid="tool-confirmation"]').should('be.visible');
    cy.get('[data-testid="tool-google-search"]').should('be.checked');
    
    // Modify selection
    cy.get('[data-testid="tool-ai-search"]').uncheck();
    
    // Confirm
    cy.get('[data-testid="confirm-tools"]').click();
    
    // Verify API call
    cy.intercept('POST', '/vac/assistant/*').as('assistantCall');
    cy.wait('@assistantCall').its('request.body').should('deep.include', {
      tools_to_use: ['google-search']
    });
  });
});

Testing Checklist

Component Testing

• Tool display renders correctly
• Checkbox interactions work
• Configuration display toggles
• Confirm button enables/disables appropriately
• Cancel button works
• Empty states handled

State Management Testing

• Pending tools stored correctly
• Confirmed tools updated
• State resets properly
• Context values propagate

API Integration Testing

• Correct tools sent in API request
• Tool configurations included
• Response handled properly
• Error states managed

User Flow Testing

• Full confirmation flow works
• Cancel flow works
• Multiple confirmations in session
• Edge cases handled

API Integration

Request Format

When tools are confirmed, the API request includes:

{
  "question": "User's original message",
  "tools_to_use": ["google-search", "file-browser"],
  "tool_configs": {
    "file-browser": {
      "bucketUrl": "gs://my-bucket/"
    }
  },
  "assistant_id": "assistant-123",
  "user_email": "user@example.com"
}

Response Handling

The backend processes confirmed tools and returns:

{
  "answer": "Here are the search results...",
  "tools_used": ["google-search"],
  "metadata": {
    "execution_time": 2.5,
    "tool_results": {...}
  }
}

Error Handling

Common error scenarios:

1. Invalid Tools: Tool not available for assistant

   {
     "error": "Tool 'advanced-search' not available",
     "available_tools": ["google-search", "ai-search"]
   }
   

2. Configuration Error: Missing required config

   {
     "error": "Tool 'file-browser' requires 'bucketUrl' configuration"
   }
   

3. Permission Error: User lacks tool access

   {
     "error": "User does not have permission for tool 'admin-tools'"
   }
   

Best Practices

1. Clear User Communication

• Always explain why tools are needed
• Show tool descriptions
• Indicate required vs optional tools

2. Configuration Validation

• Validate configs before sending
• Provide helpful error messages
• Show default values

3. State Management

• Reset state after completion
• Handle interruptions gracefully
• Maintain consistency across components

4. Testing

• Test all user paths
• Include edge cases
• Verify API integration
• Test error scenarios

Troubleshooting

Common Issues

1. Tools not appearing: Check if pendingToolsList is set in context
2. Confirmation not triggering API: Verify handleSendMessage is called with tools
3. State not resetting: Ensure resetToolConfirmation is called after processing
4. Configuration not saved: Check that confirmedToolConfigs is passed to API

Debug Tips

// Enable debug logging
const DEBUG = process.env.NODE_ENV === 'development';

if (DEBUG) {
  console.log('Pending tools:', pendingToolsList);
  console.log('Confirmed tools:', confirmedTools);
  console.log('Tool configs:', confirmedToolConfigs);
}

Known Limitations

1. Tool configurations are not validated client-side
2. No support for conditional tool dependencies
3. Maximum of 10 tools can be displayed at once
4. Tool descriptions are limited to 200 characters

Related Documentation

• Tool Context - React context for sharing tool configurations
• Tool Selector Component - Tool selection interface
• Backend Tool System - Backend tool implementation