## Overview Portola developed Tolan, an AI companion application designed to serve as an "alien best friend" for users seeking authentic, non-romantic AI relationships. This case study, sourced from Braintrust (a platform vendor promoting their evaluation tools), describes how Portola structured their LLMOps workflow to enable non-technical subject matter experts to drive quality improvements in production. While the source material is promotional in nature, it offers valuable insights into operationalizing LLMs for subjective, emotionally complex domains where traditional automated evaluation approaches fall short. The fundamental LLMOps challenge Portola faced was maintaining conversation quality in a system where success metrics are inherently subjective and context-dependent. Unlike typical chatbots or productivity assistants with measurable task completion rates, Tolan's success depends on creating genuine emotional connections—a quality that resists straightforward quantification. This required Portola to develop an operational workflow that leveraged human domain expertise at the center of their quality assurance and iteration processes. ## Technical Architecture and Complexity The case study reveals that Portola's prompting pipeline integrates multiple complex components that make traditional evaluation approaches challenging. Their system combines memory retrieval systems, dynamically generated user context, real-time voice processing, and multimodal inputs (including photos users share) into a cohesive conversation flow. This architectural complexity means that isolated unit testing or simple input-output evaluation misses the emergent qualities that make conversations feel authentic. The memory system specifically requires handling nuanced retrieval patterns that mirror how human friends actually remember things—not perfect recall, but contextually appropriate surfacing of remembered details at natural moments in conversation. This represents a sophisticated implementation challenge that goes beyond simple RAG (retrieval-augmented generation) patterns, requiring careful orchestration of what gets stored, retrieved, and integrated into prompts at runtime. ## The Quality Challenge: Beyond Automated Metrics Portola identified three critical factors for building user trust that highlight the limitations of automated evaluation in their domain. First, authentic memory requires not just technical accuracy in retrieval, but subjective appropriateness in what gets remembered and when it surfaces. Second, authentic mirroring of user emotions and communication styles involves vocabulary choices, pacing, and emotional resonance that can't be reduced to simple scoring functions. Third, avoiding the "AI uncanny valley" requires constant monitoring for patterns that signal artificial behavior—like excessive binary choice questions or overuse of trendy slang. The case study acknowledges that "a lot of what we're working on is really squishy stuff," in the words of their behavioral researcher. This represents an important and balanced recognition that not all LLM quality can or should be measured through automated evals. While this could be seen as a limitation in their LLMOps maturity, it's more accurately characterized as domain-appropriate methodology for emotionally complex applications. The tension between quantifiable metrics and subjective quality is a genuine challenge in conversational AI, particularly for applications targeting emotional connection rather than task completion. ## Workflow Design: Pattern Identification and Dataset Curation Portola's operational workflow begins with systematic log review by their behavioral researcher, Lily Doyle, who spends approximately an hour daily examining chat logs using Braintrust's interface. This represents a deliberate investment in human-in-the-loop observability, treating production logs as the primary data source for quality issues rather than relying solely on user-reported problems or automated anomaly detection. When recurring patterns emerge—whether through log review, user feedback, or focus group sessions—the team creates problem-specific datasets tagged with the identified issue. Examples from the case study include datasets for "somatic therapy" (unwanted therapeutic questioning patterns), "or-questions" (excessive binary choices), and "gen-z-lingo" (inappropriate slang usage). These datasets range from 10 to 200 examples and are deliberately not comprehensive test suites but focused collections of real conversation examples demonstrating specific problems. This approach represents a pragmatic departure from traditional ML evaluation methodology, which typically emphasizes comprehensive, stable test sets. Portola's rationale—"It feels useless to come up with a golden dataset. We're on a different model, we've changed the prompt eight times. Things change so fast"—reflects the reality of rapid LLM development cycles where models and prompts evolve faster than traditional test infrastructure can accommodate. The tradeoff here is important to note: this approach sacrifices regression testing coverage and reproducibility in favor of agility and relevance to current production behavior. Whether this tradeoff is appropriate depends heavily on the application domain, release velocity, and risk tolerance. The technical implementation leverages Braintrust's dataset management capabilities, which provide several operational advantages according to the case study. Focused iteration on specific behavioral patterns makes improvement measurement more tractable than holistic quality scoring. Fresh data reflecting current product state avoids the staleness problem common in long-lived test suites. Rapid response to new issues becomes possible without updating comprehensive evaluation frameworks. Context preservation through trace storage maintains full conversation history, which is critical for evaluating conversational AI where context windows and conversation flow matter enormously. ## Playground-Based Iteration and Manual Review Once datasets are curated, Portola's workflow moves to side-by-side prompt comparison in playground environments. The behavioral researcher manually reviews outputs from current versus candidate prompts, assessing conversation quality through domain expertise rather than automated scoring. This manual evaluation is positioned not as a temporary workaround pending better automated evals, but as the appropriate primary methodology for their domain. The playground serves as the primary workspace where domain experts load curated datasets, run comparison tests between prompt versions, review outputs holistically considering tone and emotional intelligence, document specific failures, and iterate on prompt refinements. This represents a fairly mature prompt engineering workflow that treats prompts as first-class artifacts requiring careful testing before deployment, even if that testing is primarily manual. The case study's emphasis on manual review is both a strength and potential limitation. On one hand, it acknowledges the genuine difficulty of automating evaluation for subjective, context-dependent quality in emotionally complex domains. Human judgment from experts in behavioral research, storytelling, and conversation design is likely to catch nuances that automated metrics miss. On the other hand, this approach has scaling limitations—manual review of prompt changes requires expert time that grows with product complexity and doesn't provide the regression testing coverage that automated evaluations offer. The case study doesn't address how this workflow scales as the product grows more complex or the team expands, nor does it discuss version control, reproducibility, or how manual judgments are documented for future reference. ## Deployment and Production Management The final component of Portola's workflow is their prompts-as-code infrastructure, which enables subject matter experts to deploy prompt changes directly to production once satisfied with playground results. According to the case study, their "science fiction writer can sit down, see something he doesn't like, test against it very quickly, and deploy his change to production." This represents significant operational autonomy for non-technical domain experts, eliminating engineering handoffs from the quality improvement cycle. From an LLMOps perspective, this workflow demonstrates several mature practices. Prompts are treated as code artifacts that can be versioned, tested (albeit manually), and deployed through presumably structured processes. Domain experts have direct access to deployment tooling, suggesting either sophisticated access controls and safety rails or high trust in expert judgment (or both). The elimination of engineering bottlenecks in the prompt iteration cycle represents genuine operational efficiency. However, the case study leaves important questions unanswered about production safety and risk management. What safeguards exist to prevent problematic deployments? How are prompt versions tracked and rollbacks handled when issues emerge in production? What monitoring exists to detect quality regressions after deployment? Is there any staged rollout or A/B testing of prompt changes, or do changes go directly to all users? These operational details matter significantly for assessing the maturity and robustness of the LLMOps approach, but the promotional nature of the source material likely led to their omission. ## Results and Performance Improvements The case study claims a 4x improvement in prompt iteration velocity, measured as weekly prompt iterations compared to the previous workflow requiring coordination between subject matter experts and engineers. This is presented as the primary quantitative outcome, along with qualitative improvements in conversation quality across multiple dimensions: memory system behavior and recall patterns, natural conversation flow and question patterns, brand voice consistency, and appropriate handling of sensitive topics. These results should be interpreted with appropriate skepticism given the promotional source. The 4x iteration velocity improvement is plausible given the removal of engineering handoffs, but the metric itself (number of iterations) doesn't directly measure quality improvements or user satisfaction. More iterations could indicate more effective problem-solving, but could also reflect thrashing or lack of confidence in changes. The case study provides no quantitative evidence of actual conversation quality improvements, user retention metrics, or satisfaction scores that would validate whether the increased iteration velocity translated to better user outcomes. The qualitative improvements described—better memory behavior, more natural conversation flow, improved brand voice—are difficult to evaluate without baseline comparisons or user data. The workflow's value in quickly addressing model transitions and identifying regressions when switching models is noteworthy and represents a genuine operational capability, but again lacks quantitative validation. ## Key Operational Insights and Tradeoffs The Portola case study offers several valuable insights for LLMOps practitioners working in similar domains, though each comes with important tradeoffs worth considering. The emphasis on not forcing automated evals for qualitative work acknowledges that conversation quality, emotional intelligence, and brand voice in certain domains genuinely resist automated measurement. This is an important counterpoint to the industry push toward comprehensive automated evaluation frameworks. However, the tradeoff is reduced regression testing coverage, difficulty scaling human review, and challenges in comparing approaches objectively. A balanced approach might combine automated evals for tractable aspects (like avoiding specific problematic patterns) with manual review for holistic quality assessment. The strategy of creating problem-specific datasets rather than comprehensive test suites maintains agility and keeps evaluation data fresh and relevant to current production behavior. This is particularly valuable in fast-moving LLM development where models and prompts change frequently. The tradeoff is less systematic regression testing and potential blind spots where issues aren't actively being monitored. Organizations should consider whether their domain and risk profile can tolerate this tradeoff or whether comprehensive test coverage is essential. The embrace of manual review for high-stakes domains recognizes that when building AI for emotionally complex contexts like companion relationships or mental health support, expert human judgment is essential for quality assurance. This is a defensible position for these specific domains, but requires significant resource investment in expert time and has inherent scaling limitations. The case study recommends budgeting time for subject matter experts to spend hours reviewing real usage, which represents a substantial operational commitment. The empowerment of non-technical domain experts to own the full cycle from problem identification to production deployment represents a mature DevOps philosophy applied to LLMOps. This removes bottlenecks and positions quality decisions with those who best understand the domain. However, it requires significant infrastructure investment to provide accessible tools, appropriate safety rails, and clear deployment processes that non-technical experts can navigate confidently. ## Critical Assessment and Balanced Perspective While the case study provides valuable insights, its promotional nature requires careful interpretation. The source material is published by Braintrust, the platform vendor that Portola uses, and serves to showcase Braintrust's capabilities in supporting manual review workflows. This context means certain aspects are likely emphasized while potential challenges, failures, or limitations are underreported. The workflow described represents a reasonable approach for early-stage products in subjective, emotionally complex domains where rapid iteration matters more than comprehensive testing. However, several aspects would benefit from additional operational rigor as the product matures. The heavy reliance on manual review creates scaling challenges as user base and product complexity grow. The lack of detailed discussion around production safety, monitoring, and rollback procedures suggests either an immature deployment process or editorial decisions to omit these details. The absence of quantitative user outcome metrics makes it difficult to assess whether the operational improvements actually translated to better user experiences. The case study's emphasis on "empowering non-technical experts" is valuable but potentially overstated. The workflow still requires significant technical infrastructure—trace storage, dataset management, playground environments, prompts-as-code deployment—which engineers built and maintain. The non-technical experts operate within carefully constructed rails, which is appropriate but different from true end-to-end autonomy. The partnership between engineers (building infrastructure) and domain experts (operating within it) is the actual success story, rather than eliminating engineering involvement entirely. From a maturity perspective, Portola demonstrates strong observability practices, systematic problem identification from production logs, structured prompt testing workflows, and version-controlled prompt deployment. These are positive indicators of LLMOps maturity. However, the apparent lack of automated regression testing, unclear production safety mechanisms, and heavy resource requirements for manual review represent areas where the approach may face challenges scaling or may be inappropriate for higher-risk applications. ## Applicability and Lessons for Other Organizations The Portola workflow is most applicable to organizations building LLM applications in subjective, emotionally complex domains where conversation quality and emotional intelligence matter more than task completion metrics. Examples might include mental health support, coaching applications, creative collaboration tools, or other companion-style interfaces. For these applications, the investment in domain expert review time and acceptance of manual evaluation may be appropriate. The approach is less suitable for applications with clear success metrics, high-volume transaction requirements, strict regulatory compliance needs, or where comprehensive regression testing is essential. Organizations in finance, healthcare (clinical applications), legal, or other regulated domains would need significantly more automated evaluation, audit trails, and safety mechanisms than described in this case study. The key lesson is that LLMOps workflows should match the domain characteristics and organizational context. There's no one-size-fits-all approach. Organizations should carefully consider their quality requirements, risk tolerance, resource availability, and scaling needs when deciding how much to invest in automated versus manual evaluation, comprehensive versus problem-specific testing, and engineering-mediated versus expert-direct deployment workflows. The case study's value lies less in providing a universal blueprint and more in demonstrating that legitimate alternatives exist to standard ML evaluation methodology for certain domains. The emphasis on empowering domain experts, accepting manual review for appropriate use cases, and prioritizing rapid iteration over comprehensive testing represents a valid operational philosophy for specific contexts. However, organizations should carefully assess whether their context genuinely matches these characteristics or whether the approach's limitations would create unacceptable risks.