Halliburton partnered with AWS Generative AI Innovation Center to develop an AI-powered assistant for their Seismic Engine, a cloud-native application for seismic data processing. The traditional workflow creation process required manual configuration of approximately 100 specialized tools, which was time-consuming and required deep expertise. The solution uses Amazon Bedrock, Amazon Bedrock Knowledge Bases, Amazon Nova, and Amazon DynamoDB to transform complex workflow creation into natural language conversations. The proof-of-concept achieved workflow generation success rates of 84-97% while reducing creation time by over 95% compared to manual processes, with complete workflows delivered within 5.9-16.6 seconds.
Halliburton’s Seismic Engine case study demonstrates a production deployment of large language models to transform complex technical workflow creation in the energy sector. The company collaborated with the AWS Generative AI Innovation Center to build an AI-powered assistant that converts natural language queries into executable seismic workflows while providing question-answering capabilities for documentation. This is particularly noteworthy as an LLMOps implementation because it addresses the operational challenges of deploying multiple models with different purposes (intent routing, Q&A, and workflow generation) in a production environment that requires high accuracy and reliability for mission-critical geoscience operations.
The business problem centered on the complexity of Halliburton’s Seismic Engine, which required users to manually configure approximately 100 specialized tools to create processing workflows. This process was not only time-consuming but also created a significant barrier to entry, limiting accessibility to users with deep domain expertise. The solution needed to maintain the precision required for seismic data processing while dramatically simplifying the user experience through natural language interaction.
The production system is built around a FastAPI application deployed on AWS App Runner, which provides the scalable infrastructure for handling user queries through a streaming interface. This deployment choice reflects important LLMOps considerations around serverless deployment patterns that can scale automatically based on demand while minimizing operational overhead. The streaming interface is particularly significant from a production standpoint, as it provides immediate feedback to users rather than requiring them to wait for complete generation, improving the perceived responsiveness of the system.
The architecture employs a multi-model strategy with different models optimized for different tasks. Amazon Nova Lite handles intent routing, while Claude models (specifically Claude 3.5 Sonnet V2 and Claude 3.5 Haiku at the time of development) handle the more complex tasks of Q&A and workflow generation. This reflects a mature LLMOps approach of matching model capabilities and costs to specific use cases rather than using a single model for all tasks. The authors note that the architecture supports model upgrades without code changes, which is a critical production consideration as foundation models continue to evolve rapidly.
The intent routing mechanism represents a key LLMOps pattern for managing multi-purpose AI assistants in production. After receiving a user query, the system uses Amazon Nova Lite via the Amazon Bedrock API to classify queries into one of three categories: “Workflow_Generation” for queries related to creating or modifying workflows, “QnA” for documentation and tool-specific questions, and “General_Question” for out-of-scope queries. The selection of Amazon Nova Lite for this task demonstrates practical model selection based on the balance between accuracy and latency for a relatively straightforward classification task.
This routing layer serves several important production purposes. First, it ensures that queries are directed to the appropriate backend processing pipeline, avoiding unnecessary computational costs by not invoking complex workflow generation models for simple documentation questions. Second, it provides a mechanism for handling out-of-scope queries gracefully, which is important for production deployments where users may not always phrase queries appropriately. The implementation demonstrates how production LLM systems need orchestration layers that go beyond individual model calls.
The Q&A component demonstrates a production-ready implementation of Retrieval Augmented Generation using Amazon Bedrock Knowledge Bases, a fully managed service that handles the operational complexity of RAG workflows. The decision to use a managed service rather than building a custom RAG pipeline reflects practical LLMOps considerations around operational overhead, as the managed service handles infrastructure scaling, security, and maintenance automatically. This allows the development team to focus on solution development rather than infrastructure operations.
The knowledge base ingests two types of content with different chunking strategies based on document characteristics. Tool documentation markdown files are kept unchunked since they’re relatively short, preserving complete context for individual tools. Longer documents like Seismic Engine manuals use hierarchical chunking with default settings, which maintains parent-child relationships to balance granular retrieval with broader document context. This demonstrates thoughtful consideration of chunking strategies based on content characteristics, which is an important but often overlooked aspect of production RAG implementations.
The system uses Amazon Titan Text Embeddings V2 for embedding generation and OpenSearch Serverless as the vector database, with metadata storage including file names, URLs, and document types for downstream use. For both retrieval and response generation, the implementation uses Bedrock Knowledge Bases’ retrieve_and_generate API with Claude 3.5 Haiku as the model. The system supports multi-turn conversations by maintaining session context, and responses are formatted with inline citations for enhanced traceability. These citations are particularly important in a production setting where users need to verify information and understand the source of answers, especially for technical documentation.
The workflow generation component represents one of the more sophisticated LLMOps implementations in the case study, using LLM agents orchestrated through the LangChain framework to convert natural language into executable YAML workflows. The agent is bound with 82 tools available in Seismic Engine, and must select appropriate tools, determine execution order, and generate syntactically correct YAML based on user requirements. This is considerably more complex than simple text generation, as it requires the model to understand tool specifications, dependencies, and the domain logic of seismic data processing.
The implementation tested both Claude 3.5 Sonnet V2 and Claude 3.5 Haiku, providing different trade-offs between accuracy and speed. The models are provided with detailed tool descriptions and specifications through the LangChain tool binding mechanism, enabling them to understand each tool’s capabilities and requirements. The system considers both explicit requirements in user queries and includes necessary default parameters when specific values aren’t provided, demonstrating the practical complexity of translating natural language to precise technical specifications.
The workflow generation process supports multi-turn conversations, allowing users to iteratively refine workflows through natural language modifications. This is enabled by storing conversation history in Amazon DynamoDB, which the LLM can reference to either generate new workflows or modify existing ones. This stateful interaction pattern is important for production usability, as users rarely specify complete requirements in a single query and need the ability to refine through conversation.
The evaluation approach demonstrates mature LLMOps practices around testing and validation. The team created a comprehensive test dataset of query-workflow pairs derived from real historical workflows and validated by subject matter experts. This approach of using real-world examples rather than synthetic test cases is critical for accurately assessing production performance, particularly in specialized domains like seismic data processing where synthetic examples may not capture the full complexity and nuances of actual use cases.
The evaluation results show strong performance across both model choices. Claude Haiku 3.5 achieved 84% success rate for simple workflows and 90% for medium complexity workflows, with median generation times of 5.9 and 9.1 seconds respectively. Claude Sonnet 3.5 V2 showed superior success rates of 86% for simple and 97% for medium complexity workflows, with median generation times of 11.5 and 16.6 seconds. These results demonstrate the classic accuracy-latency trade-off in production LLM deployments, where different models can be selected based on the priority between speed and accuracy.
Critically, the evaluation included comparison against baseline human performance, showing that the solution achieves 84-97% success rates compared to 70% for new users and 85% for experienced users when creating workflows manually. The time reduction is even more dramatic, with workflows generated in 0.13-0.28 minutes compared to 2-20 minutes for human users, representing over 95% time reduction. However, it’s worth noting that these numbers come from a promotional AWS blog post, and while the methodology appears sound, independent validation would strengthen confidence in these claims.
The production implementation includes Amazon DynamoDB for managing chat history and interaction logging, which is essential for multi-turn conversation support and system observability. This represents important production considerations beyond the core model inference, as conversation state management becomes critical when users are iteratively refining complex technical workflows. The logging capability also provides important data for monitoring system performance, understanding usage patterns, and identifying areas for improvement.
The streaming response capability for both Q&A and workflow generation provides immediate feedback to users as the system processes requests. This is particularly important in production deployments where response generation may take several seconds, as streaming prevents the user interface from appearing frozen and provides progressive disclosure of the response. The implementation across both the Q&A and workflow generation paths demonstrates consistent UX considerations throughout the system architecture.
The case study provides valuable insights into production model selection and evolution. The authors explicitly note that the solution was developed using Claude 3.5 Sonnet V2 and Claude 3.5 Haiku, but these models have since been succeeded by Claude Sonnet 4.5, Claude Sonnet 4.6, and Claude Haiku 4.5. The architecture’s support for model upgrades without code changes demonstrates forward-thinking LLMOps design that anticipates the rapid evolution of foundation models.
This upgrade strategy is particularly important in production LLM systems, where model providers frequently release improved versions. By decoupling the application logic from specific model versions and using abstraction layers (in this case, Amazon Bedrock’s API), the system can benefit from model improvements without requiring application rewrites. This reduces the long-term maintenance burden and allows the system to improve over time as foundation models advance.
The deployment on AWS App Runner represents a serverless approach to LLMOps infrastructure, which automatically handles scaling based on demand without requiring manual capacity planning. This is particularly appropriate for enterprise tools where usage patterns may vary significantly over time. The FastAPI framework provides a lightweight, performant foundation for the API layer, which is well-suited for ML inference workloads.
The use of managed services throughout the stack (Bedrock for model inference, Bedrock Knowledge Bases for RAG, DynamoDB for state management, OpenSearch Serverless for vector search) reflects a build-versus-buy decision that prioritizes reduced operational overhead over maximum customization. This approach trades some flexibility for operational simplicity, which is often the right choice for enterprise deployments where the ML team may be small and focused on business value rather than infrastructure management.
The authors note that the approach generalizes well to other domains with complex, multi-step agentic workflows requiring specialized tool knowledge and configuration. This is a reasonable claim, as the architecture doesn’t contain any seismic-specific components beyond the tool specifications and documentation. The pattern of intent routing, RAG-based Q&A, and agent-based workflow generation could apply to many technical domains with complex tooling.
For future enhancements, the authors suggest exploring multi-agent architectures using frameworks like Strands Agents SDK with Amazon Bedrock AgentCore for improved accuracy through specialized sub-agents. This reflects emerging patterns in production LLM systems where complex tasks are decomposed across multiple specialized agents rather than relying on a single general-purpose agent. Such architectures can provide better accuracy by allowing each sub-agent to focus on a specific aspect of the problem, though they add coordination complexity.
While the case study demonstrates impressive results, several aspects warrant careful consideration when evaluating this as an LLMOps implementation. The evaluation was conducted on a proof-of-concept system, and the blog post doesn’t provide details about production deployment at scale, long-term reliability metrics, or how the system handles edge cases and failures in real-world usage. The impressive performance metrics (84-97% success rates, 95% time reduction) should be viewed in the context of this being an AWS-published case study promoting their services, though the evaluation methodology appears sound.
The system’s reliance on managed AWS services (Bedrock, Knowledge Bases, App Runner, etc.) provides operational benefits but also creates vendor lock-in and may limit customization options. Organizations considering similar implementations should weigh these trade-offs based on their specific requirements around portability, cost optimization, and the need for fine-grained control over the infrastructure.
The workflow generation accuracy of 84-97% is strong, but the 3-16% failure rate means that users still need sufficient expertise to verify generated workflows before execution, particularly for critical seismic processing operations. This positions the system as an accelerator and accessibility tool rather than a complete replacement for human expertise, which is an appropriate and realistic positioning for current LLM capabilities in specialized technical domains.
The case study provides limited discussion of ongoing monitoring, evaluation in production, or how model performance is tracked over time. These are critical aspects of LLMOps that extend beyond initial deployment, and production implementations would need robust monitoring to detect model drift, track success rates in real usage, and identify opportunities for improvement.
Overall, this case study represents a well-architected production LLM deployment that thoughtfully addresses key LLMOps challenges including model selection, evaluation, multi-model orchestration, state management, and deployment infrastructure. While some claims should be viewed with appropriate skepticism given the promotional nature of the content, the technical approach demonstrates mature practices that would be valuable for organizations building similar agent-based systems for complex technical workflows.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Toyota Motor North America (TMNA) and Toyota Connected built a generative AI platform to help dealership sales staff and customers access accurate vehicle information in real-time. The problem was that customers often arrived at dealerships highly informed from internet research, while sales staff lacked quick access to detailed vehicle specifications, trim options, and pricing. The solution evolved from a custom RAG-based system (v1) using Amazon Bedrock, SageMaker, and OpenSearch to retrieve information from official Toyota data sources, to a planned agentic platform (v2) using Amazon Bedrock AgentCore with Strands agents and MCP servers. The v1 system achieved over 7,000 interactions per month across Toyota's dealer network, with citation-backed responses and legal compliance built in, while v2 aims to enable more dynamic actions like checking local vehicle availability.
Moody's Analytics, a century-old financial institution serving over 1,500 customers across 165 countries, transformed their approach to serving high-stakes financial decision-making by evolving from a basic RAG chatbot to a sophisticated multi-agent AI system on AWS. Facing challenges with unstructured financial data (PDFs with complex tables, charts, and regulatory documents), context window limitations, and the need for 100% accuracy in billion-dollar decisions, they architected a serverless multi-agent orchestration system using Amazon Bedrock, specialized task agents, custom workflows supporting up to 400 steps, and intelligent document processing pipelines. The solution processes over 1 million tokens daily in production, achieving 60% faster insights and 30% reduction in task completion times while maintaining the precision required for credit ratings, risk intelligence, and regulatory compliance across credit, climate, economics, and compliance domains.
