BMW Research is developing a multi-agent in-car voice assistant system that faces the critical challenge of reliable agent orchestration and delegation decisions at runtime. The problem centers on determining which specialized agent should handle specific tasks in constantly changing contexts, where multiple agents may be capable of solving the same problem but with different tradeoffs in cost, latency, and user experience. The proposed solution emphasizes treating agent delegation as fundamentally different from simple tool selection, recognizing that agents can initiate autonomous loops and that multiple valid execution paths may exist for the same user request. Rather than forcing canonical ground truth labels, the approach advocates for context-aware delegation decisions and evaluation frameworks that assess not just task success but also the quality of the chosen path, including metrics around latency, cost, and human-machine collaboration.
BMW Research is exploring the operational challenges of deploying multi-agent LLM systems in production automotive environments, specifically focusing on in-car voice assistants. Daniel Homola from BMW Research presented work centered on a deceptively simple but operationally critical question: in a multi-agent system, who should handle each piece of work? The presentation addresses the fundamental LLMOps challenge of making reliable orchestration and delegation decisions repeatedly at runtime in production environments where context is constantly changing.
The use case grounds the theoretical considerations in a practical multi-agent in-car voice assistant system that coordinates several specialized agents including a navigation agent for route finding, car control agents for manipulating vehicle functions like windows, seats, and climate controls, and GUI agents (computer use agents) that can operate applications through the vehicle’s screen interface similar to how human users would. An orchestrator sits above these specialized agents, coordinating execution and sometimes combining results from multiple agents.
A critical insight from this work is that agent delegation in production multi-agent systems is fundamentally different from simple tool selection, despite superficial similarities. This distinction has important implications for how these systems are engineered, evaluated, and operated in production.
Traditional tool selection problems are typically evaluated against a single ground truth label where one correct tool is identified for a given input. However, agent delegation in production environments exhibits several characteristics that make it more complex. First, overlap between agent capabilities is normal and expected rather than an error condition. Multiple agents may be capable of solving the same task, and the choice between them involves tradeoffs in cost, latency, user experience, and other production considerations beyond mere capability matching.
The example provided illustrates this well: when a user says “Play some jazz” in the vehicle, the system faces a genuine choice. If the media screen is empty, using a dedicated media API might be the optimal path. However, if there is already a song or playlist visible on the screen, the GUI agent could simply tap the appropriate interface element. Both approaches would successfully complete the user’s request, but they represent different tradeoffs in terms of latency, user experience, and system resource utilization. In production, forcing such scenarios into a single canonical ground truth would eliminate the system’s ability to make context-appropriate optimization decisions.
Another fundamental distinction highlighted is that while agent delegation may be implemented using tool call mechanisms at the API level, agents are not tools in the traditional sense. Tool calls are bounded operations with predictable execution patterns and guaranteed termination. Agent delegation, by contrast, is unbounded and can initiate autonomous loops with unpredictable execution characteristics.
When control is delegated to an agent in production, that agent may reason about the problem, ask for clarification from the user, encounter obstacles and get stuck, resume execution later after receiving additional information, or potentially never return cleanly to the orchestrator. This means orchestration is not simply about selecting the right capability but also about deciding when it is safe and useful to let another autonomous loop take over what is effectively unbounded control of part of the system.
This has direct implications for production reliability and system monitoring. Traditional tool calling patterns allow for straightforward timeout handling and error recovery. Agent delegation requires more sophisticated oversight mechanisms that can detect when an agent has entered an unproductive state, determine whether intervention is needed, and potentially transfer control to an alternative agent or execution path.
The work emphasizes that reliable production systems must make delegation decisions based on full runtime context rather than static routing rules. The same user utterance may warrant different delegation decisions depending on current system state, user history, visible interface elements, vehicle status, and other contextual factors.
For example, the navigation agent would clearly be the appropriate choice for “navigate me to Munich,” but requests like “play some jazz” represent genuine ambiguity where multiple valid execution trajectories exist. The optimal choice depends on runtime context that may not be available during system design or training. Production systems must be able to adapt their delegation decisions dynamically based on this runtime context rather than forcing inputs into predetermined categories.
This context-awareness extends beyond simple state checking to include optimization for different operational objectives. Sometimes the system might optimize for minimum latency to provide the fastest user response. In other scenarios, optimizing for human-machine collaboration and user experience might be more appropriate, even if it requires slightly longer execution time. The production system needs to make these optimization choices dynamically based on current priorities and constraints.
The distinction between agent delegation and tool selection has significant implications for how multi-agent systems are evaluated and benchmarked in production environments. Traditional benchmarks that simply assess whether the final task succeeded are insufficient for evaluating the quality of agent orchestration and delegation decisions.
BMW’s approach advocates for evaluation frameworks that consider the entire execution path, not just the final outcome. This includes assessing whether the chosen path was reasonable given the context, measuring the cost and latency characteristics of the execution, and evaluating the user experience provided by the selected delegation strategy. A successful task completion that took twice as long as necessary or consumed excessive computational resources represents a different quality outcome than an efficient execution, and evaluation frameworks need to capture these distinctions.
Creating appropriate benchmarks requires datasets that include full runtime context rather than just input-output pairs. The evaluation data must capture the state of the vehicle systems, visible interface elements, user interaction history, and other contextual factors that influence optimal delegation decisions. Without this context, benchmarks cannot meaningfully assess whether the system made good choices given the actual operational constraints.
The presentation proposes an approach to building and evaluating multi-agent systems at enterprise scale that reflects organizational structure and ownership patterns. In a large organization like BMW, different teams own different agents and have responsibility for their development, maintenance, and performance. The proposed approach has each team that owns an agent build their own benchmark for that agent’s capabilities and performance characteristics.
These individual agent benchmarks can then be composed into shared benchmarks that evaluate core collaboration and coordination patterns between agents. This compositional approach to benchmarking mirrors the compositional architecture of the multi-agent system itself and allows teams to maintain ownership of their components while ensuring that orchestration and delegation patterns work correctly across the integrated system.
The shared benchmarks focus specifically on making orchestration paths and delegation decisions measurable, context-aware, and reliable. This reflects the operational reality that in production deployment, the quality of orchestration often matters more than the capabilities of individual agents. A system with excellent specialized agents that makes poor delegation decisions will perform worse in production than a system with adequate agents that orchestrates them effectively.
The work makes specific reference to GUI agents or computer use agents as one type of specialized agent in the multi-agent system. These agents operate applications through screen interfaces similar to how human users would, rather than through dedicated APIs. The presentation references a previous talk at the AI Engineer conference that explored the motivation behind the GUI agent paradigm for enterprise applications.
In the context of the in-car voice assistant, GUI agents represent an interesting delegation option because they can handle tasks through the same interface elements that would be visible to human users. This creates the overlap scenarios where a task could be accomplished either through a specialized API-based agent or through GUI interaction. The choice between these approaches involves tradeoffs in reliability, latency, and user experience that must be resolved at runtime based on current context.
Throughout the presentation, there is consistent emphasis on production reliability as the driving concern. The patterns for connecting agents like handoffs, agents-as-tools, and routing through classifiers are described as means to an end rather than ends in themselves. Different enterprises may use different naming conventions and pattern taxonomies, but fundamentally they all reduce to the same core operational challenge: making good delegation decisions repeatedly at runtime under constantly changing conditions.
The work acknowledges that many coordination patterns exist in practice, including handoff patterns where control transfers to another agent, agents-as-tools patterns where the orchestrator delegates work and waits for results, and router or classifier patterns where each user turn is dispatched to a specialist. However, the emphasis is that all of these patterns ultimately depend on the quality of the underlying delegation decision. An elegant pattern implemented with poor delegation logic will produce unreliable production behavior.
The presentation concludes with three critical questions that should guide the development and operation of multi-agent systems in production environments. First, who handles this work? This is the fundamental delegation decision that must be made reliably at runtime. Second, what path should the selected agents take together? This addresses orchestration quality and the coordination of multi-step processes across agents. Third, how do we know it was good? This emphasizes the need for measurable evaluation criteria that go beyond simple task success to assess the quality of execution paths.
These questions reflect a pragmatic LLMOps perspective focused on operational reliability rather than theoretical elegance. They push system designers and operators to think concretely about decision quality, execution efficiency, and measurement frameworks that can validate production behavior.
While this presentation comes from BMW Research and naturally presents their approach in a favorable light, the core insights about agent delegation versus tool selection appear sound and reflect real operational challenges in production multi-agent systems. The emphasis on context-aware runtime decisions and evaluation frameworks that consider execution quality rather than just task success aligns with broader industry experience deploying complex LLM systems.
The proposed approach of compositional benchmarking where individual teams own agent-level benchmarks that compose into system-level orchestration benchmarks is pragmatic and reflects how large organizations actually develop complex systems. However, the presentation does not provide concrete results demonstrating that this approach works better than alternatives, nor does it share specific metrics from production deployments that would validate the claims about reliability and performance.
The distinction drawn between bounded tool calls and unbounded agent delegation is conceptually useful, though the practical implications for production systems may vary depending on implementation details. Many production systems impose timeouts and other constraints on agent behavior that partially bound what is theoretically unbounded control, and the presentation does not deeply explore these practical mitigation strategies.
Overall, this represents thoughtful work on real operational challenges in deploying multi-agent LLM systems in production automotive environments, though it is presented at a conceptual level without detailed implementation specifics or quantitative validation of the proposed approaches.
ListenLabs, a platform for analyzing user research at scale, built a sophisticated multi-agent system that processes hundreds to thousands of user interviews, surveys, and focus group feedback. The company evolved from basic retrieval-augmented generation to a complex architecture featuring three primary agents: a study creation agent (Composer) that collaboratively builds discussion guides with users through an artifact-based interface, an interview agent that conducts voice-based multimodal conversations with participants, and a research agent that analyzes large volumes of qualitative data to generate insights, charts, video clips, and PowerPoint presentations. Their system demonstrates advanced LLMOps practices including parallelized sub-agent execution for processing hundreds of interviews simultaneously, custom evaluation agents for quality control, contextual prompt engineering, code execution in sandboxes, and sophisticated trace analysis for continuous improvement. The platform handles the complete lifecycle from study design through data collection to automated analysis and reporting.
Earmark built a productivity suite for product teams that transforms meeting conversations into finished work in real-time, addressing the problem of endless context-switching and manual follow-up work that plagues modern product development. Founded by Mark Barb and Sandon, who both came from the product management SaaS space, Earmark uses live transcription and multiple parallel AI agents to generate product specs, tickets, summaries, and other artifacts during meetings rather than after them. The company pivoted from an Apple Vision Pro communication training tool to a web-based real-time meeting assistant after discovering through 60 customer interviews that few people actually prepare for presentations. With 78% of survey respondents saying they'd be "super bummed" if the product disappeared, Earmark has achieved strong product-market fit by focusing specifically on product managers, engineering leaders, and adjacent roles who spend most of their time in back-to-back meetings with different audiences and deliverables.
Sierra, an AI agent platform company, discusses their comprehensive approach to deploying LLMs in production for customer service automation across voice and chat channels. The company addresses fundamental challenges in productionizing AI agents including non-deterministic behavior, latency requirements, and quality assurance through novel solutions like simulation-based testing that runs thousands of parallel test scenarios, speculative execution for voice latency optimization, and constellation-based multi-model orchestration where 10-20 different models handle various aspects of each conversation. Their outcome-based pricing model aligns incentives with customer success, while their hybrid no-code/code platform enables both business and technical teams to collaboratively build, test, and deploy agents. The platform serves large enterprise customers across multiple industries, with agents handling millions of customer interactions in production environments.
