Building Boba: An AI Co-Pilot Application Case Study

Overview

Thoughtworks developed Boba, an experimental AI co-pilot application focused on product strategy and generative ideation. The case study provides valuable insights into building production-grade LLM applications with sophisticated user experiences beyond simple chat interfaces.

Core Application Features

• Research signals and trends analysis
• Creative matrix generation for ideation
• Scenario building and exploration
• Strategy ideation using Playing to Win framework
• Concept generation for products/features
• Visual storyboarding with integrated image generation

LLMOps Patterns and Implementation Details

Prompt Engineering & Management

• Used Langchain for prompt template management
• Implemented persona-based prompting (e.g., “visionary futurist”)
• Maintained simple, non-conditional prompt templates
• Conducted iterative prompt testing via ChatGPT before implementation

Structured Output Handling

• Enforced JSON response formats for consistent data structures
• Successfully handled complex nested JSON schemas
• Used pseudo-code schema descriptions in prompts
• Integrated with OpenAI’s Function Calling API
• Implemented response validation and parsing

Real-Time User Experience

• Implemented streaming responses using OpenAI and Langchain APIs
• Built progress monitoring capabilities
• Added ability to stop generation mid-completion
• Managed state during streaming JSON parsing
• Integrated with Vercel AI SDK for edge-ready streaming

Context Management

• Implemented selection-based context carrying
• Used tag delimiters for context specification
• Managed multi-message chat conversations
• Integrated vector stores for handling large contexts
• Built contextual conversation capabilities within specific scenarios

External Tool Integration

• Implemented Google SERP API integration
• Used Extract API for content retrieval
• Built vector store based knowledge base using HNSWLib
• Integrated OpenAI embeddings
• Created search result summarization pipeline

Technical Implementation Details

• Used RecursiveCharacterTextSplitter for text chunking
• Implemented VectorDBQAChain for question-answering
• Built in-memory vector store with HNSW graphs
• Created streaming callback handlers
• Managed JSON parsing during stream processing

User Experience Considerations

• Implemented context-aware UI elements
• Built feedback mechanisms for response iteration
• Created template-based conversation starters
• Added visibility toggles for reasoning chains
• Implemented image generation refinement capabilities

Production Challenges & Solutions

Performance Optimization

• Implemented streaming to handle long-running generations
• Used chunking for large text processing
• Optimized vector search for quick retrievals
• Managed context window limitations

Error Handling

• Built robust JSON parsing for streaming responses
• Implemented generation interruption capabilities
• Added fallback conversation channels
• Created feedback loops for response quality

Integration Architecture

• Combined multiple AI services (GPT, Stable Diffusion)
• Integrated search and content extraction services
• Built vector store infrastructure
• Implemented web UI with real-time updates

Best Practices & Recommendations

Prompt Engineering

• Test prompts in ChatGPT before implementation
• Keep templates simple and maintainable
• Use explicit schema definitions
• Implement chain-of-thought prompting

User Experience

• Show real-time progress for long operations
• Provide context selection mechanisms
• Enable iterative refinement
• Include fallback conversation options

Architecture

• Use structured response formats
• Implement streaming where appropriate
• Consider vector stores for large contexts
• Build modular prompt templates

Development Process

• Focus on UI/UX (80% of effort)
• Iterate on prompt engineering (20% of effort)
• Test with real users
• Build feedback mechanisms

Future Considerations

• Implementing long-term memory systems
• Enhancing feedback loops
• Expanding external tool integration
• Improving response quality through reinforcement learning
• Scaling vector store implementations