## Overview The Government of the City of Buenos Aires presents a compelling case study in scaling conversational AI for public service delivery through their enhancement of "Boti," an existing WhatsApp-based AI assistant. Originally launched in February 2019, Boti had already established itself as a primary communication channel, facilitating over 3 million conversations monthly. However, the government recognized an opportunity to leverage generative AI to address a specific citizen pain point: navigating the city's complex bureaucratic landscape with over 1,300 government procedures, each with unique requirements, exceptions, and nuances. The collaboration between the Buenos Aires government and AWS's Generative AI Innovation Center resulted in a sophisticated agentic AI system that demonstrates several advanced LLMOps practices, from custom guardrail implementation to novel retrieval architectures. While the case study presents impressive performance metrics, it's important to evaluate these claims within the context of the specific use case and implementation choices made. ## Technical Architecture and LLMOps Implementation The production system architecture centers around two primary components built using LangGraph and Amazon Bedrock: an input guardrail system for content safety and a government procedures agent for information retrieval and response generation. This dual-component approach reflects mature LLMOps thinking about separation of concerns and safety-first design. The input guardrail system represents a custom implementation choice over Amazon Bedrock's native guardrails. The team justified this decision based on the need for greater flexibility in optimizing for Rioplatense Spanish and monitoring specific content types. The guardrail uses an LLM classifier to categorize queries into "approved" or "blocked" categories, with subcategorization for more detailed content analysis. Approved queries include both on-topic government procedure requests and off-topic but low-risk conversational queries, while blocked queries encompass offensive language, harmful opinions, prompt injection attempts, and unethical behaviors. The reported performance shows 100% blocking of harmful queries, though the case study acknowledges occasional false positives with normal queries—a common trade-off in content moderation systems. ## Novel Reasoning Retrieval System Perhaps the most technically interesting aspect of this implementation is the reasoning retrieval system, which addresses specific challenges in government procedure disambiguation. Standard RAG approaches struggled with related procedures (such as renewing versus reprinting driver's licenses) and the typical pattern where user queries require information from a single specific procedure rather than multiple sources. The reasoning retrieval approach involves several sophisticated preprocessing and runtime steps. During preprocessing, the team created comparative summaries that capture not just basic procedural information (purpose, audience, costs, steps, requirements) but also distinguishing features relative to similar procedures. This involved clustering base summaries into small groups (average size of 5) and using LLMs to generate descriptions highlighting what makes each procedure unique within its cluster. These distinguishing descriptions are then appended to create final comparative summaries—an approach that shares conceptual similarities with Anthropic's Contextual Retrieval technique. The runtime retrieval process operates in three steps: initial semantic search retrieval of 1 to M comparative summaries, optional LLM-based reasoning for disambiguation when retrieval scores are too close (indicating potential confusion between similar procedures), and final retrieval of complete procedure documents from DynamoDB. The optional reasoning step is particularly clever from an LLMOps perspective, as it balances accuracy with latency and token usage by only applying expensive LLM reasoning when needed. ## Performance Evaluation and Results The evaluation methodology demonstrates solid LLMOps practices, using a synthetic dataset of 1,908 questions derived from known source procedures and weighting metrics based on real-world webpage visit frequency. The comparative analysis across different retrieval approaches provides valuable insights into the performance trade-offs between various embedding models and retrieval strategies. The progression from section-based chunking with Amazon Titan embeddings (baseline) to summary-based embeddings with Cohere Multilingual v3 (7.8-15.8% improvement) to reasoning retrieval with Claude 3.5 Sonnet (12.5-17.5% additional improvement reaching 98.9% top-1 accuracy) demonstrates clear performance gains, though it's worth noting these improvements come with increased computational complexity and cost. However, the evaluation approach has some limitations that should be considered. The synthetic dataset, while large, may not fully capture the diversity and complexity of real user queries. Additionally, the evaluation focuses heavily on retrieval accuracy without providing detailed analysis of end-to-end response quality, user satisfaction, or handling of edge cases. The reported linguistic accuracy (98% for voseo usage, 92% for periphrastic future) is based on subject matter expert review of a sample, but the size and selection methodology for this sample aren't specified. ## Production Deployment and Operations The system demonstrates several mature LLMOps practices in its production deployment. The use of Amazon Bedrock's Converse API provides flexibility to optimize different models for different subtasks, allowing the team to balance performance, latency, and cost across the various components. The integration with existing infrastructure (WhatsApp, DynamoDB) shows thoughtful consideration of operational complexity and user experience continuity. The prompt engineering approach incorporates several sophisticated techniques, including personality definition, response structure guidelines, and cultural localization for Rioplatense Spanish. The inclusion of sentiment analysis as metadata to route sensitive topics to different prompt templates demonstrates awareness of the contextual appropriateness required in government communications. This level of prompt engineering sophistication suggests significant investment in fine-tuning the user experience. ## Safety and Content Moderation The custom guardrail implementation raises interesting questions about the trade-offs between using platform-provided safety tools versus custom solutions. While the team achieved strong performance with their custom approach, this choice likely required significant development and maintenance overhead compared to using Amazon Bedrock's native guardrails. The decision appears justified given their specific requirements for Rioplatense Spanish optimization and detailed content monitoring, but other organizations might find different cost-benefit calculations. The parallel processing of queries through both guardrails and the main agent system reflects good safety practices, ensuring that harmful content is caught before processing. However, the case study doesn't provide detailed information about how false positives are handled in production or what feedback mechanisms exist for improving guardrail performance over time. ## Scalability and Future Considerations The system currently handles 3 million conversations monthly, which represents significant scale for a government service. However, the case study doesn't provide detailed information about infrastructure costs, latency characteristics under load, or scaling patterns. The reasoning retrieval system, while highly accurate, involves multiple LLM calls and complex preprocessing, which could present cost and latency challenges as usage grows. The outlined future improvements suggest continued evolution of the system, including exploration of speech-to-speech interactions, fine-tuning for better Rioplatense Spanish performance, and expansion to multi-agent frameworks for additional government services like account creation and appointment scheduling. These directions indicate a mature roadmap for LLMOps evolution, though they also suggest the current system is seen as a foundation rather than a complete solution. ## Critical Assessment While the case study presents impressive technical achievements and performance metrics, several aspects warrant careful consideration. The heavy reliance on AWS services (Bedrock, DynamoDB, Titan embeddings) raises questions about vendor lock-in and long-term cost management. The custom guardrail and reasoning retrieval systems, while performant, represent significant engineering overhead that might be difficult for other organizations to replicate without similar resources and expertise. The evaluation methodology, while comprehensive in scope, focuses primarily on technical metrics rather than user experience outcomes. Missing elements include user satisfaction scores, task completion rates, escalation patterns to human agents, and comparative analysis against the previous system's performance. Additionally, the case study doesn't address important operational questions like model drift monitoring, continuous evaluation processes, or incident response procedures. ## Lessons for LLMOps Practitioners This case study offers several valuable lessons for LLMOps practitioners. The reasoning retrieval approach provides a compelling solution to disambiguation challenges in domain-specific applications, though it comes with increased complexity and cost. The parallel architecture for safety and functionality demonstrates good separation of concerns, while the cultural localization efforts highlight the importance of considering linguistic and cultural context in international deployments. The collaboration model between government and cloud provider technical teams appears to have been effective, though the case study doesn't provide details about knowledge transfer, ongoing support arrangements, or long-term maintenance responsibilities. Organizations considering similar implementations should carefully evaluate their capacity for maintaining such sophisticated systems over time. Overall, this case study represents a sophisticated application of modern LLMOps practices to a real-world government service challenge, with notable technical innovations in retrieval and safety. However, the true measure of success will be long-term user adoption, operational sustainability, and the system's ability to evolve with changing citizen needs and technological capabilities.