## Overview and Use Case Context Google Research's Wayfinding AI represents a production-oriented research prototype that explores how LLMs can be deployed to help people navigate complex health information. The case study centers on a fundamental design challenge in LLMOps: how to move beyond simple question-answering paradigms to create AI systems that actively understand user context and guide conversations toward more helpful outcomes. This work is particularly notable because it demonstrates an iterative, user-centered approach to deploying LLMs in a sensitive domain (healthcare information) where accuracy, relevance, and user trust are paramount. The use case addresses a well-documented problem: people searching for health information online often struggle to articulate their concerns effectively and are overwhelmed by generic, non-personalized content. The research team's hypothesis was that by designing an AI agent to proactively seek context through clarifying questions—mimicking how healthcare professionals interact with patients—they could deliver more tailored and helpful information experiences. This represents a departure from the typical LLM deployment pattern of providing comprehensive answers immediately. ## Technical Architecture and LLMOps Implementation The Wayfinding AI is built on top of Gemini, specifically using Gemini 2.5 Flash as the baseline model in their comparative studies. The system implements a sophisticated prompt engineering approach that orchestrates three core behaviors at each conversational turn. First, it generates up to three targeted clarifying questions designed to systematically reduce ambiguity about the user's health concern. Second, it produces a "best-effort" answer based on the information available so far, acknowledging that this answer can be improved with additional context. Third, it provides transparent reasoning that explains how the user's latest responses have refined the AI's understanding and answer. From an LLMOps perspective, this architecture requires careful prompt design to balance multiple objectives simultaneously: asking relevant questions, providing provisional answers, maintaining conversational coherence, and explaining reasoning—all while ensuring medical appropriateness and safety. The system must dynamically adapt its questioning strategy based on the conversation history and the specific health topic, which suggests sophisticated context management and potentially multi-step prompting or agent-based architectures. The interface design itself is a critical component of the production system. The team created a two-column layout that separates the interactive conversational stream (left column, containing user messages and clarifying questions) from the informational content (right column, containing best-effort answers and detailed explanations). This design decision emerged from user research showing that clarifying questions could be easily missed when embedded in long paragraphs of text. This highlights an important LLMOps lesson: the interface through which LLM outputs are presented can be as critical as the model outputs themselves for user experience and system effectiveness. ## User Research and Iterative Development Process The development process exemplifies best practices in LLMOps, particularly around evaluation and iteration. The team conducted four separate mixed-method user experience studies with a total of 163 participants. The formative research phase involved 33 participants and revealed critical insights that shaped the system design. Participants reported struggling to articulate health concerns, describing their search process as throwing "all the words in there" to see what comes back. This insight directly informed the proactive questioning strategy. During prototype testing, the team discovered that users strongly preferred a "deferred-answer" approach where the AI asks questions before providing comprehensive answers, with one participant noting it "feels more like the way it would work if you talk to a doctor." However, the research also revealed important failure modes: engagement dropped when questions were poorly formulated, irrelevant, or difficult to find in the interface. This kind of qualitative feedback is invaluable for production LLM systems, as it identifies not just what works but under what conditions systems fail. The evaluation methodology itself demonstrates sophisticated LLMOps practices. The final validation study used a randomized within-subjects design with 130 participants, where each person interacted with both the Wayfinding AI and a baseline Gemini model. This controlled comparison allowed for direct measurement of the intervention's impact while controlling for individual variation in health topics and information needs. Participants were required to spend at least 3 minutes with each system and could optionally share their conversation histories, balancing research needs with privacy considerations. ## Evaluation Framework and Metrics The team measured six specific dimensions of user experience: helpfulness, relevance of questions asked, tailoring to situation, goal understanding, ease of use, and efficiency of getting useful information. This multi-dimensional evaluation framework goes beyond simple satisfaction scores to capture specific aspects of LLM performance that matter for production deployment. The results showed that Wayfinding AI was significantly preferred for helpfulness, relevance, goal understanding, and tailoring, though not necessarily for ease of use or efficiency—an honest acknowledgment that the more sophisticated interaction pattern may involve some tradeoffs. Beyond preference metrics, the team analyzed conversation patterns themselves, finding that Wayfinding AI conversations were notably longer (4.96 turns on average vs. 3.29 for baseline when understanding symptom causes) and had different structural characteristics. They visualized conversation flows using Sankey diagrams showing that Wayfinding AI conversations contained substantially more user responses to clarifying questions. This behavioral evidence complements the self-reported preferences and demonstrates that the system actually changes how users interact with the AI, not just how they perceive it. From an LLMOps perspective, this evaluation approach is noteworthy because it combines quantitative metrics (turn counts, preference ratings), qualitative feedback (open-ended responses about what users learned), and behavioral analysis (conversation flow patterns). This triangulation provides robust evidence for production readiness that goes beyond typical accuracy metrics. It also demonstrates the importance of human evaluation for conversational AI systems where traditional offline metrics may not capture user experience quality. ## Production Considerations and Challenges While presented as a research prototype, the case study reveals several important production considerations. The team was careful to exclude certain types of health inquiries (details provided in the paper), implement informed consent processes, and instruct participants not to share identifying information. These safeguards reflect the regulatory and ethical considerations necessary when deploying LLMs in healthcare contexts, even for informational purposes. The team also addresses potential limitations transparently. They note that effectiveness depends heavily on execution quality—poorly formulated questions or irrelevant clarifications damage user engagement. This suggests that production deployment would require robust quality assurance mechanisms, potentially including human review of generated questions, continuous monitoring of conversation quality, and mechanisms to detect and recover from poor interactions. The paper's emphasis on "transparent reasoning" also reflects awareness that healthcare applications require explainability, which may necessitate additional prompt engineering or architectural components beyond basic LLM generation. The two-column interface represents another production consideration: it departs from familiar chat interfaces, which could affect adoption. While the study showed users still preferred the system overall, the team acknowledges this may involve ease-of-use tradeoffs. This raises questions about how to introduce novel interaction patterns in production while managing user expectations and learning curves. ## Context Management and Conversational State A critical but somewhat implicit aspect of the LLMOps implementation is how the system maintains conversational context across multiple turns while managing increasingly complex user histories. Each turn must incorporate the user's original question, all subsequent clarifying questions, all user responses, and previous best-effort answers to generate appropriate follow-up questions and updated answers. This context management challenge is central to any multi-turn conversational system and likely requires careful prompt design to keep context within model token limits while maintaining coherence. The system's ability to explain "how the user's latest answers have helped refine the previous answer" suggests it maintains explicit models of information gain across turns. This could be implemented through meta-prompting (asking the LLM to reflect on how new information changes its understanding) or through separate reasoning steps that track which aspects of ambiguity have been resolved. Either approach represents sophisticated prompt engineering beyond simple question-answer patterns. ## Broader LLMOps Lessons and Implications This case study offers several valuable lessons for LLMOps practitioners. First, it demonstrates that user research and iterative design are as important as model selection for production LLM systems. The Wayfinding AI's success stems not primarily from using a different model but from careful design of interaction patterns based on user needs. Second, it shows that familiar interaction patterns (immediate comprehensive answers) are not always optimal, and that departing from defaults can yield significant user experience improvements when grounded in research. Third, the work highlights that evaluation for production systems must go beyond accuracy metrics to capture user experience dimensions like relevance, tailoring, and goal understanding. Fourth, it demonstrates the value of comparative studies with baseline systems to isolate the impact of specific design choices. Finally, it reinforces that interface design and information architecture are integral to LLMOps, not separate concerns—the two-column layout was essential to making the proactive questioning approach work. The team's approach to transparency and limitations is also worth noting. They acknowledge that this is a research prototype, identify specific conditions under which the approach fails, and describe exclusion criteria and safeguards. This balanced presentation, while potentially reflecting the academic context, provides a model for how to communicate about LLM systems responsibly in production contexts. The team's work suggests that successful deployment of LLMs for complex information needs requires not just powerful models but careful attention to conversational design, user research, multi-dimensional evaluation, and appropriate interface design—a holistic LLMOps approach that integrates technical and human factors considerations throughout the development lifecycle.