Fetcherr, a company deploying decision-making systems across enterprises, presents an architecture that combines large language models with specialized "market models" for high-stakes quantitative business decisions, particularly in pricing and demand forecasting. The company argues that while LLMs excel at orchestration and reasoning, they are insufficient for reliable quantitative decisions because they are primarily trained on text rather than market dynamics. Their solution uses LLMs to orchestrate a multi-agent system where specialized sub-agents have access to proprietary deep learning models trained on market data (demand, pricing, competition), forecast tools, and constrained optimization capabilities. In a demonstrated experiment comparing their approach to a vanilla Claude agent on airline pricing decisions, the Fetcherr system with market models recommended profitable price decreases based on elasticity analysis, while the unaided LLM agent incorrectly recommended price increases by confusing correlation for causation, resulting in an estimated 6% revenue uplift versus an 8% revenue loss respectively.
Fetcherr is a company founded by Uri Shalmy, a PhD in computational neuroscience with experience building algorithmic trading systems. The company deploys quantitative decision-making systems across various enterprises, with particular focus on pricing, demand forecasting, and revenue optimization. The case study presents two workshop sessions that detail Fetcherr’s architectural approach to production AI systems: one providing conceptual overview and motivation, the other diving into technical implementation details.
The core thesis presented challenges the prevailing assumption that large language models alone are sufficient for high-stakes business decisions. Drawing an analogy to neuroscience, the speakers argue that just as the human brain has specialized regions for processing different types of sensory input (visual cortex for images, auditory cortex for sound), business decision-making systems should have specialized components trained on business-specific data rather than relying solely on language-centric architectures.
Central to Fetcherr’s approach is what they call the “market model” - a proprietary deep learning model that sits alongside but distinct from large language models. While LLMs are trained primarily on text and visual models on images, market models are trained directly on market dynamics: transaction data, pricing histories, competitive actions, demand patterns, and other business-relevant signals.
The market model is conceptually compared to world models in robotics and simulation, but instead of modeling physical environments, it models business environments. The fundamental data structure is multi-dimensional, with dimensions representing time of transaction, delivery time, product attributes, customer class attributes, and other relevant features. Rather than tokens or pixels, the model works with what they call “voxels” - probabilistic representations of the likelihood of specific events like transactions, cancellations, or competitor repricing actions.
The technical implementation uses multivariate time series forecasting with deep neural networks. The architecture consumes historical data (known past information like previous demand and pricing) along with exogenous variables or covariates that include both past features (historical product attributes, past prices) and future-known features (holidays, weather forecasts, day-of-week information, static product identifiers). The model predicts not just point forecasts but probabilistic distributions, allowing for uncertainty quantification through either parameterized distributions like multivariate normal distributions or through quantile forecasting.
A significant focus is placed on interpretability and human-understandable outputs from the market model. The speakers demonstrate how they visualize the multi-dimensional output space by fixing certain dimensions and varying others. For example, they show demand surfaces where the x-axis represents own pricing, the y-axis represents competitor pricing, and color intensity represents expected demand. These visualizations reveal intuitive patterns: blue areas (low demand) where own price is significantly higher than competitors, red areas (high demand) where own price is competitive, and equilibrium lines where customers are price-indifferent.
The model incorporates attention mechanisms similar to modern transformer architectures, allowing engineers to inspect what the model pays attention to. One example shows how the model automatically redistributed attention when a new competitor entered the market mid-year, intelligently adjusting based on competitor attributes without explicit programming of this behavior.
Temporal dynamics are also visualized, such as showing how demand surfaces change in correlation with fuel prices in the airline industry, demonstrating the model’s ability to capture complex market relationships.
Beyond forecasting, the architecture includes constrained optimization capabilities. The mathematical formulation involves maximizing expected reward (typically revenue, calculated as demand times price) over feasible action sets, subject to both hard and soft constraints. Examples of constraints include price bounds (never exceeding certain thresholds), variance constraints (avoiding large consecutive pricing changes), and ordering constraints (ensuring economy seats are never priced higher than business class).
The implementation uses standard optimization libraries like SciPy’s minimize functions, but the key is that these optimizers consume the market model’s probabilistic forecasts to evaluate counterfactual scenarios. The system can simulate different pricing policies, evaluate expected outcomes under each, and select optimal actions while respecting business constraints.
The speakers demonstrate revenue landscapes showing how optimal pricing changes as constraints are introduced. A global maximum might exist at one price point, but hard constraints could force the optimizer to settle for a local maximum that satisfies business rules.
The production system architecture combines LLM capabilities with the specialized market models through a multi-agent framework. This represents a sophisticated approach to LLMOps where the LLM serves as orchestrator rather than the sole intelligence.
The architecture uses Claude’s agent SDK (described as similar to Claude Code) and implements the Model Context Protocol (MCP) for tool integration. The system defines multiple MCP servers: one for data capabilities (fetching historical data from cloud storage like GCP buckets), another for market model capabilities (prediction, optimization, evaluation tools), and others for specific analytical functions.
Four distinct agents are configured with different roles and tool access:
Revenue Manager Agent: The orchestrating agent with a system prompt appropriate for a revenue management role. It has access to all sub-agents and coordinates the overall decision-making process.
Market Dynamics Analyst: Has access to market model prediction tools and can invoke forecasting capabilities to understand how demand and other market factors will evolve.
Pricing Policy Analyst: Focuses on optimization tools, taking market forecasts and determining optimal pricing decisions under constraints.
QA Analyst: Equipped with evaluation tools to assess prediction uncertainty, validate recommendations, and flag potential issues in the analysis.
This division of labor is intentional - each sub-agent has access only to tools relevant to its specialty, avoiding confusion and ensuring appropriate expertise is applied to each aspect of the decision.
Each capability is wrapped as a tool with structured inputs and outputs. For example, the predict_market_dynamics tool uses Pydantic for input validation, ensuring the agent provides properly formatted configuration before invoking the actual prediction capability. The tool returns structured output including quantile forecasts, elasticity estimates, revenue projections, and uncertainty measures.
A critical aspect of the production system is “grounding” - tracking and logging when and how the agent uses specific tools. The implementation uses hook matchers that intercept post-tool-use events, allowing custom logging behavior. This grounding serves multiple purposes: validating that agents use the intended capabilities, debugging reasoning paths, and building audit trails of decision-making processes.
The hooks capture which tools were called, by which agent, with what parameters, and what results were returned. This metadata is essential for production monitoring and for understanding agent behavior in operational settings.
The most compelling demonstration is a head-to-head comparison on a real client problem in the airline industry. Both agents receive identical data: a year-over-year analysis showing demand decreased 22% for a specific product while prices increased over 60%. The question posed is simply: “Is the price right?”
Vanilla Claude Agent Approach: The unaided agent used its general capabilities - writing Python scripts, performing statistical analysis with Pandas, calculating correlations, and examining historical patterns. However, it made a critical error in confusing correlation for causation. It analyzed correlation across products that should not have been correlated together and concluded that “demand elasticity is negligible” with “correlation of nearly zero.” Based on this flawed analysis, it recommended increasing prices by $200 (from $1,100 to $1,300), reasoning that price increases carry “near zero demand destruction risk.” The simulation estimated this recommendation would result in an 8% revenue decline.
Market Model Agent Approach: The Fetcherr agent invoked its market model prediction tools and analyzed demand elasticity at the current $1,100 price point. It determined that elasticity was greater than 1 in absolute value, meaning demand was elastic - every 1% price reduction would yield approximately 1.4% more bookings, more than offsetting the fare reduction in revenue terms. The unconstrained optimization found the maximum revenue at $828, but the constrained optimizer (respecting business rules about minimum prices and maximum price changes) recommended $863. The revenue manager rounded this to $850 for simplicity. The simulation estimated this recommendation would yield over 6% revenue uplift.
Process Differences: The grounding logs revealed significant differences in tool usage. The vanilla agent wrote code, produced CSVs, and performed standard data analysis. The market model agent used prediction tools extensively, invoked optimization under constraints, and crucially, the QA analyst flagged uncertainty in predictions for one of six products in the market. This triggered the revenue manager to re-dispatch the pricing analyst to re-examine that subset, showing inter-agent communication and iterative refinement. For the high-uncertainty product, the system flagged it for human review rather than making an automated recommendation - the vanilla agent had no such uncertainty assessment mechanism and shipped recommendations for all products regardless of confidence.
The code walkthrough reveals production-level practices. The market model training uses Ray for hyperparameter tuning with configurations for input size, hidden layers, learning rates, and other neural network parameters. Feature engineering carefully separates static features (cabin class, flight direction), known-past features (historical demand, past prices), and known-future features (holidays, weather, temporal information).
The training process uses distribution-based loss functions appropriate for probabilistic forecasting rather than point prediction. While the speakers mention compatibility with auto-research frameworks (likely AutoML approaches), they explicitly choose not to use fully automated hyperparameter optimization, preferring “consistency for the pre-trained model” and “guarantees” over maximum automation. This reflects production pragmatism - the value of predictable, validated model behavior over potentially marginal performance gains from black-box optimization.
The time series framework follows patterns from Darts (an open-source library from Unit8), though the speakers clarify they are not affiliated, simply recommending it for best practices in time series forecasting.
Data engineering includes encoding numerical and categorical features, handling multiple aggregations of forecasts (mean across horizon, revenue calculations), and computing derived metrics like elasticity and maximum revenue points from the base demand predictions.
This case study reveals several important LLMOps principles for production systems:
Specialized Models for Specialized Tasks: The core argument is architectural - LLMs should not be expected to handle all intelligence requirements. For quantitative decision-making requiring accurate forecasting and optimization, domain-specific models trained on relevant data outperform general-purpose language models. The LLM’s role is orchestration, reasoning about the problem structure, and coordinating specialized capabilities.
Trust and Reliability Through Delegation: The system design explicitly separates “things the LLM is good at” (natural language understanding, problem decomposition, coordinating workflows) from “things that require guarantees” (mathematical optimization, probabilistic forecasting, uncertainty quantification). By delegating the latter to deterministic tools and validated models, the system achieves reliability that pure LLM approaches struggle to match.
Multi-Agent Patterns in Production: The architecture demonstrates practical multi-agent orchestration where agents have different roles, different tool access, and communicate with each other. The QA agent’s ability to flag issues that trigger re-analysis by other agents shows emergent quality control behavior.
Grounding and Observability: Production LLM systems require extensive instrumentation. The grounding hooks that track tool usage provide essential visibility into agent reasoning and decision-making, enabling debugging, validation, and continuous improvement.
Evaluation Through Counterfactual Simulation: Rather than relying solely on the agent’s stated reasoning, the system includes simulation capabilities to evaluate recommendations objectively. The revenue landscape simulations provide quantitative estimates of recommendation quality independent of the agent’s confidence.
Structured I/O and Validation: Use of Pydantic for tool input validation and structured output formats ensures agents interact with capabilities in well-defined ways, reducing errors and improving reliability.
The presentation draws interesting parallels between biological systems and AI architectures. Just as evolution specialized brain regions for different sensory modalities and cognitive functions, the speakers argue that business decision-making systems should specialize components for different business concepts. Language models treating price, demand, and supply as mere tokens (just as they treat any other words) fails to give these fundamental business concepts the “first-class citizen” status they deserve.
This biological inspiration extends to the concept of “market sensation” - the system’s ability to sense and forecast market dynamics analogously to how sensory systems perceive the physical world. The market model serves as a specialized perceptual system for the business environment.
The speakers explicitly frame this work as part of a transition “from practices that are based on general harness engineering to practices that are more enterprise grade truly reliable agent operations.” This positions the case study at the frontier of LLMOps maturity - moving beyond simple prompt engineering or even basic tool use toward sophisticated multi-agent systems with specialized capabilities, proper observability, and production reliability guarantees.
While the demonstration is compelling, several aspects warrant balanced consideration. The comparison uses a single agent (Claude) as the baseline, and it is possible that other approaches - such as few-shot prompting with worked examples of elasticity analysis, or fine-tuning on business decision-making tasks - might improve LLM performance. The speakers acknowledge this is “one example” rather than comprehensive evaluation across diverse scenarios.
The market model itself requires substantial data for training and ongoing maintenance. Organizations adopting this architecture would need sufficient historical transaction data, the ML engineering capability to train and update these models, and the domain expertise to design appropriate constraints and objectives. This represents significant upfront and ongoing investment beyond simply deploying LLM agents.
The architectural complexity introduces operational overhead. Managing multiple models (LLMs plus market models), multiple agents with different toolsets, MCP servers, grounding infrastructure, and simulation capabilities creates a sophisticated system requiring specialized expertise to operate and debug.
Nevertheless, for enterprises making high-stakes quantitative decisions where errors have significant financial impact, the demonstrated performance gap - a 14 percentage point swing in estimated revenue impact between the approaches - suggests the investment may be justified. The case study makes a strong argument that for certain production use cases, hybrid architectures combining LLM orchestration with specialized domain models represent the current state of the art in LLMOps.
This panel discussion features three AI-native companies—Delphi (personal AI profiles), Seam AI (sales/marketing automation agents), and APIsec (API security testing)—discussing their journeys building production LLM systems over three years. The companies address infrastructure evolution from single-shot prompting to fully agentic systems, the shift toward serverless and scalable architectures, managing costs at scale (including burning through a trillion OpenAI tokens), balancing deterministic workflows with model autonomy, and measuring ROI through outcome-based metrics rather than traditional productivity gains. Key technical themes include moving away from opinionated architectures to let models reason autonomously, implementing state machines for high-confidence decisions, using tools like Pydantic AI and Logfire for instrumentation, and leveraging Pinecone for vector search at scale.
OpenAI's Forward Deployed Engineering (FDE) team embeds with enterprise customers to solve high-value problems using LLMs, aiming for production deployments that generate tens of millions to billions in value. The team works on complex use cases across industries—from wealth management at Morgan Stanley to semiconductor verification and automotive supply chain optimization—building custom solutions while extracting generalizable patterns that inform OpenAI's product development. Through an "eval-driven development" approach combining LLM capabilities with deterministic guardrails, the FDE team has grown from 2 to 52 engineers in 2025, successfully bridging the gap between AI capabilities and enterprise production requirements while maintaining focus on zero-to-one problem solving rather than long-term consulting engagements.
Benchling, a 14-year-old platform for life sciences R&D data management, launched Benchling AI six months ago to bring intelligent agents to scientific workflows. The problem scientists face is the time-consuming nature of drug discovery, from initial experiments to FDA submissions, involving manual data entry, analysis, and report writing. Benchling AI addresses this through a chat-based agent interface that leverages their extensive historical data repository to help scientists find relevant experiments, design new tests, analyze results, and generate regulatory reports. The system uses multiple model families in parallel for critical tasks like data entry, employs custom-built harnesses tailored to scientific workflows rather than coding-focused architectures, and integrates agent skills that function like standard operating procedures. Early results suggest the potential to reduce drug discovery timelines by 2x through eliminating workflow bottlenecks and enabling more efficient experimental design.
