## Overview Lexbe is a leader in legal document review software that has been operating since 2006, providing eDiscovery solutions through their cloud-based platform Lexbe Online™. The company developed Lexbe Pilot, an AI-powered Q&A assistant that represents a significant advancement in how legal professionals handle document review and analysis. This case study demonstrates a comprehensive LLMOps implementation that addresses the critical challenge of analyzing massive legal document collections ranging from 100,000 to over one million documents per case. The core problem Lexbe addressed is fundamentally an LLMOps challenge: how to deploy and scale large language models in production to handle the rigorous demands of legal document analysis where accuracy, reliability, and performance are paramount. Legal professionals face the daunting task of identifying critical evidence within vast document sets under tight deadlines, where missing key information can result in unfavorable case outcomes. Traditional keyword-based search approaches are insufficient for this task, as they often return hundreds or thousands of documents without providing the contextual understanding needed to identify truly relevant information. ## Technical Architecture and LLMOps Implementation Lexbe's solution represents a sophisticated LLMOps architecture built on Amazon Bedrock and integrated AWS services. The system employs a comprehensive RAG (Retrieval-Augmented Generation) workflow that demonstrates several key LLMOps principles in production deployment. The architecture begins with document ingestion and processing, where legal documents stored in Amazon S3 undergo text extraction using Apache Tika. This extracted text is then processed through Amazon Bedrock's embedding models, specifically Titan Text v2, to generate vector representations. The choice of embedding model and configuration was the result of extensive experimentation, with Lexbe testing multiple models including Amazon Titan and Cohere, as well as different token sizes (512 vs 1024 tokens) to optimize performance. The embedding generation and storage system demonstrates important LLMOps considerations around model selection and parameter tuning. The embeddings are stored in a vector database that enables fast semantic retrieval, while Amazon OpenSearch provides both vector and text-based indexing capabilities. This dual approach allows the system to handle both semantic similarity searches and traditional keyword-based queries, providing flexibility in how legal professionals can interact with the document corpus. The query processing pipeline showcases production-ready LLM deployment through Amazon Bedrock Knowledge Bases, which provides a fully managed RAG workflow. When users submit queries through the web interface, the system routes requests through Amazon CloudFront and an Application Load Balancer to backend services running on AWS Fargate. This serverless container approach enables horizontal scaling without infrastructure management overhead, a key consideration for handling variable workloads in legal environments. The LLM component uses Amazon Bedrock's Sonnet 3.5 model for generating coherent and accurate responses based on retrieved document context. This represents a critical LLMOps decision point where model selection directly impacts output quality and system reliability. The choice of Sonnet 3.5 reflects considerations around reasoning capabilities, context handling, and response generation quality that are essential for legal applications. ## Performance Optimization and Production Readiness One of the most compelling aspects of this case study is the detailed documentation of the iterative improvement process over eight months of development. This demonstrates real-world LLMOps practices around performance monitoring, evaluation, and continuous improvement in production environments. Lexbe established clear acceptance criteria focused on recall rates, recognizing that in legal document review, missing relevant documents can have serious consequences. The recall rate metric served as their primary production readiness benchmark, which is a sophisticated approach to LLMOps evaluation that goes beyond simple accuracy measures. The performance evolution tells a story of systematic LLMOps optimization. Starting with only a 5% recall rate in January 2024, the team achieved incremental improvements through various technical interventions. By April 2024, new features in Amazon Bedrock Knowledge Bases brought the recall rate to 36%. Parameter adjustments around token size in June 2024 increased performance to 60%, while model optimization with Titan Embed text-v2 reached 66% by August 2024. The breakthrough came in December 2024 with the introduction of reranker technology, which pushed the recall rate to 90%. This progression demonstrates important LLMOps principles around systematic evaluation, iterative improvement, and the importance of emerging technologies like rerankers in production RAG systems. ## Production Deployment and Scalability The deployment architecture demonstrates mature LLMOps practices for production systems. AWS Fargate provides containerized deployment with automatic scaling capabilities, allowing the system to handle varying workloads typical in legal environments where case loads can fluctuate significantly. The use of Amazon ECS with Linux Spot Market instances provides cost optimization, which is crucial for production LLMOps deployments where compute costs can be substantial. The system architecture includes robust security measures essential for legal applications, including encryption and role-based access controls through AWS's security framework. This addresses one of the key challenges in LLMOps for regulated industries where data privacy and security are paramount. The scalability features enable processing of document sets ranging from hundreds of thousands to over a million documents, demonstrating the system's ability to handle enterprise-scale workloads. This scalability is achieved through the combination of serverless compute (Fargate), managed services (Bedrock), and optimized storage and indexing systems. ## Advanced Capabilities and Real-World Performance The production system demonstrates sophisticated LLMOps capabilities that go beyond simple question-answering. Lexbe Pilot can generate comprehensive findings-of-fact reports spanning multiple pages with proper formatting, section headings, and hyperlinked source citations. This represents advanced document generation capabilities that require careful prompt engineering and output formatting in production LLM systems. Perhaps more impressive is the system's ability to perform deep automated inference across document collections. The example of identifying family relationships by connecting metadata from emails demonstrates sophisticated reasoning capabilities that require the LLM to synthesize information from multiple sources. This type of inferential reasoning represents advanced LLMOps implementation where the system can discover implicit connections that traditional search methods cannot identify. The system handles multilingual documents seamlessly, processing content in English, Spanish, and other languages without requiring separate processing pipelines. This multilingual capability is crucial for modern legal practices and demonstrates robust LLMOps implementation that can handle diverse data types in production. ## Collaborative Development and LLMOps Best Practices The eight-month collaboration between Lexbe and the Amazon Bedrock Knowledge Bases team represents exemplary LLMOps development practices. Weekly strategy meetings between senior teams enabled rapid iteration and continuous improvement, demonstrating the importance of close collaboration between LLMOps practitioners and platform providers. The establishment of clear acceptance criteria and systematic performance measurement throughout the development process reflects mature LLMOps practices. The focus on recall rate as the primary metric, while acknowledging the specific requirements of legal document analysis, shows sophisticated understanding of evaluation in production LLM systems. This case study represents a successful production LLMOps implementation that addresses real-world business challenges while demonstrating scalable, secure, and cost-effective deployment practices. The system's ability to transform traditional legal document review processes while maintaining the accuracy and reliability required in legal contexts showcases the potential for LLMOps to drive significant business value in regulated industries.