Structured Output Display Bug - Post-Mortem

Issue Summary

Date: September 5, 2025
Impact: StructuredOutputDisplay component was not rendering despite structured data being successfully generated and saved to Firestore
Root Cause: The convertFirestoreMessagesToLocal utility function was stripping the structuredOutput field when converting messages for display

Timeline of Investigation

1. Initial Symptoms

  • User reported that the structured extraction tool was running successfully
  • Backend logs showed structured data being generated (34 fields)
  • Firestore documents contained the structuredOutput field
  • But the StructuredOutputDisplay component was not appearing in the UI

2. Backend Investigation (Working Correctly ✅)

  • Verified structured extraction tool was generating data
  • Confirmed assistant_config.py was NOT saving structuredOutput initially
  • Fixed backend to include structuredOutput in message saves
  • Verified data was successfully saved to Firestore

3. Frontend Investigation (Found the Bug 🐛)

  • Added debug logging throughout the component chain:
    • FirebaseService.onMessages → Found messages WITH structuredOutput ✅
    • MessageStreamingContext → Messages had structuredOutput ✅
    • ChatInterface → Messages did NOT have structuredOutput ❌
    • ChatMessageItem → Never received structuredOutput ❌

4. Root Cause Discovery

The bug was in src/utils/emissaryHelpers.ts:

// BEFORE (BUG):
export function convertFirestoreMessagesToLocal(messages: Message[]): LocalChatMessage[] {
  return messages.map(msg => ({
    id: msg.id,
    sender: msg.sender,
    content: msg.content,
    timestamp: msg.timestamp,
    userName: msg.userName,
    userEmail: msg.userEmail,
    photoURL: msg.photoURL,
    read: msg.read,
    thinkingContent: msg.thinkingContent,
    traceId: msg.traceId
    // ❌ MISSING: structuredOutput field!
  } as LocalChatMessage));
}

Why This Happened

Design Flaw: Implicit Field Mapping

The convertFirestoreMessagesToLocal function explicitly maps each field instead of using spread operator. When new fields are added to the Message type, they must be manually added to this conversion function.

Type Safety Gap

TypeScript’s as LocalChatMessage cast allowed the incomplete object to pass type checking, even though structuredOutput was defined in the ChatMessage interface.

The Fix

// AFTER (FIXED):
export function convertFirestoreMessagesToLocal(messages: Message[]): LocalChatMessage[] {
  return messages.map(msg => ({
    id: msg.id,
    sender: msg.sender,
    content: msg.content,
    timestamp: msg.timestamp,
    userName: msg.userName,
    userEmail: msg.userEmail,
    photoURL: msg.photoURL,
    read: msg.read,
    thinkingContent: msg.thinkingContent,
    traceId: msg.traceId,
    structuredOutput: msg.structuredOutput // ✅ Added this line
  } as LocalChatMessage));
}

Lessons Learned

1. Use Spread Operator for Future-Proofing

Consider refactoring to:

export function convertFirestoreMessagesToLocal(messages: Message[]): LocalChatMessage[] {
  return messages.map(msg => ({
    ...msg,
    // Only override fields that need transformation
  } as LocalChatMessage));
}

2. Add Integration Tests for New Features

When adding new fields to message types, ensure tests verify the field flows through:

  • Backend → Firestore
  • Firestore → Frontend subscription
  • Frontend subscription → Component props

3. Debug Logging Strategy

The systematic addition of debug logs at each layer helped identify exactly where data was lost:

  1. Data source (Firestore subscription)
  2. State management (Context/Reducer)
  3. Component props
  4. Component rendering

Prevention Strategies

1. Automated Field Checking

Create a test that verifies all Message fields are preserved through conversion functions.

2. Type-Safe Conversions

Use stricter TypeScript configurations to catch missing fields:

// Use satisfies operator for better type checking
const converted = {
  // fields...
} satisfies LocalChatMessage;

3. Documentation

Document data flow for new fields in the message pipeline:

  • Backend saves to Firestore
  • Frontend subscribes to Firestore
  • Messages converted for display
  • Components receive and render data

Test Coverage Added

See src/utils/__tests__/emissaryHelpers.test.ts for regression tests that verify:

  1. All Message fields are preserved in conversion
  2. StructuredOutput field specifically flows through
  3. Type safety is maintained

Impact Assessment

  • Severity: Medium - Feature completely non-functional but no data loss
  • Users Affected: All users of structured extraction tool
  • Data Loss: None - data was saved correctly, just not displayed
  • Recovery: Simple code fix, no data migration needed