This presentation addresses the challenge of managing disorganized personal notes and research materials by building an LLM-powered knowledge management system. The solution involves using voice transcription tools for rapid note capture, LLM agents to enrich and interconnect notes through automated tagging and backlinking, automated wiki generation to organize concepts and entities, and visualization tools to create graph views of knowledge connections. The system runs on scheduled automation in cloud environments, transforming raw markdown notes into an interconnected knowledge base that surfaces forgotten insights and makes personal research navigable through Wikipedia-style browsing of one's own thoughts.
This case study presents a comprehensive approach to building production LLM systems for personal knowledge management, demonstrated by Ben Holmes from Warp. The core problem being addressed is the common challenge of maintaining disorganized notes across various platforms like Apple Notes, where valuable insights and research become scattered and difficult to retrieve. The solution architecture involves multiple LLM-powered automation pipelines that transform raw markdown notes into an interconnected knowledge base with wikis, backlinks, and visual graph representations.
What makes this particularly relevant to LLMOps is the emphasis on production-ready automation rather than one-off LLM interactions. The system runs scheduled jobs in cloud environments, processes file-based data structures, and maintains consistency across multiple processing passes. The presenter demonstrates a custom-built application called Hubble for note navigation and shows how the entire workflow operates with minimal manual intervention once configured.
The foundation of the system begins with efficient data capture. The presenter advocates strongly for voice transcription as the fastest method to get thoughts into the system, citing approximately 200 words per minute versus typing speeds. Two specific tools are highlighted for local voice-to-text processing: Handy, an open-source tool, and Voice Inc, a paid tool with a one-time $20 lifetime fee. Both run local models that stay on-device, eliminating subscription costs and privacy concerns associated with cloud-based services.
The emphasis on local models for transcription is noteworthy from an LLMOps perspective, as it demonstrates a hybrid architecture where some processing happens locally for speed and privacy, while more complex reasoning tasks are delegated to cloud-based LLMs. The voice transcription produces raw markdown files that serve as the input to subsequent LLM processing pipelines. These raw materials can come from various sources: meeting transcripts, podcast listening notes, research reading, or stream-of-consciousness thoughts captured via mobile apps.
The presenter demonstrates this with personal examples, including notes from the Acquired podcast about Walt Disney’s founding and Ferrari’s history. The raw transcripts are intentionally unstructured, rambling, and without perfect formatting because the goal is rapid capture rather than immediate organization. This creates a workflow where humans focus on what they do best (generating ideas and context) while LLMs handle what they excel at (pattern recognition, categorization, and connection-making).
The first major LLM processing stage is note enrichment, implemented as an agent skill called “enrich note.” This skill transforms raw markdown files into structured, interconnected documents. The enrichment process involves several distinct operations that demonstrate careful prompt engineering and system design.
First, the system adds timestamps to track when notes were enriched, which serves a dual purpose: preventing redundant processing and enabling the system to identify which notes need attention in batch operations. This is a practical LLMOps consideration that prevents wasted API calls and compute resources on already-processed content.
Second, the system generates tags based on a controlled vocabulary stored in a reference folder. This is a particularly interesting design decision that addresses a common challenge with LLM systems: hallucination and inconsistency in categorization. Rather than allowing the LLM to invent new tags arbitrarily, the system provides a concrete list and instructs the agent to be “reluctant” to add new tags, though it can if it identifies genuine new patterns. The presenter specifically notes that Claude “loves to get creative,” so explicit instructions are needed to constrain this behavior. This demonstrates sophisticated prompt engineering that balances flexibility with consistency.
Third, the enrichment process includes web research to identify and verify sources. All major agent platforms now include web search capabilities, though the presenter specifically mentions Exa as an alternative tool for this functionality. The system attempts to find URLs for podcast episodes, articles, or other source materials referenced in the notes, adding bibliographic metadata automatically.
Fourth, and perhaps most valuable, the system generates backlinks by searching through the entire note corpus for related content using key term search. This creates a Wikipedia-style navigation experience where users can follow connections between related ideas. The system identifies thematically similar notes and creates bidirectional links, enabling serendipitous rediscovery of forgotten connections.
The presenter demonstrates running this enrichment skill using Warp’s interface with the GLM 5.2 model, showing that the system works with various LLM providers including open-weight models. The ability to swap models demonstrates good abstraction in the system architecture, allowing for experimentation with different capabilities and cost profiles.
Beyond individual note enrichment, the system can generate comprehensive wikis that aggregate and organize related notes by topic. This capability is inspired by a gist from Andrej Karpathy about LLM knowledge bases, which has gained significant traction in the community. The wiki generation takes a raw directory of markdown files and produces structured pages organizing people, places, concepts, organizations, and sources.
The presenter demonstrates two complete wikis: one focused on AI news and research, another on religious texts from studying the Bible. The AI wiki includes expected entries like major AI organizations and researchers, but also unexpected connections like Adam Neely, a jazz musician who has discussed AI’s impact on the music industry. This demonstrates the system’s ability to find cross-domain connections that might not be obvious through manual organization.
Each wiki entry includes references back to the raw notes that informed it, maintaining traceability from synthesized knowledge back to original sources. The wiki structure includes dedicated sections for different entity types, making it easy to browse by category. For the religious text wiki, this means having a people section with entries for various biblical figures, each linked to relevant notes and connected to related characters or events.
From an LLMOps perspective, the wiki generation represents a more complex task than simple note enrichment. The system must analyze multiple documents, identify entities across them, determine relationships, and generate coherent summaries. The presenter describes this as simply providing the agent with a description of the desired wiki structure along with Karpathy’s original instructions, then letting it generate the output. This demonstrates the power of modern LLMs for complex document synthesis tasks, though the presentation doesn’t dive deeply into error handling, quality validation, or edge cases that would be important in production systems.
A critical aspect of making this system production-ready is automated scheduling that runs enrichment and wiki generation without manual intervention. The presenter discusses two approaches to automation: local automation using tools like Codex app, and cloud-based automation using Warp’s oz.dev platform. The distinction is important: local automation requires the laptop to be running at scheduled times, while cloud automation can run independently.
The cloud automation architecture involves syncing markdown files to a cloud sandbox, typically powered by Docker, running the agent workflows, then syncing results back. The presenter uses the Obsidian headless CLI for synchronization, which provides automatic bidirectional sync without requiring manual git operations. This is positioned as more convenient than git-based approaches, though git would also work for users who prefer that workflow.
The automation is configured to run on daily or weekly schedules, with two primary workflows: note enrichment and wiki updates. The enrichment automation specifically looks for notes that lack enrichment timestamps, processes only those, then syncs the results back. This incremental processing approach is more efficient than reprocessing the entire corpus each time.
The presenter shows actual run logs from these automations, demonstrating that the system successfully processes notes overnight and presents updated wikis and enriched notes in the morning. This “daily paper” metaphor captures the vision: waking up to a fresh synthesis of yesterday’s thoughts and research, automatically organized and connected.
From an LLMOps standpoint, several production concerns are evident in this design. The use of timestamps for idempotency ensures consistent results even if automations run multiple times. The cloud sandbox approach provides isolation and reproducibility. The scheduled execution means the system can leverage off-peak hours when compute resources might be cheaper or more available. However, the presentation doesn’t address monitoring, error recovery, cost management, or quality assurance, which would all be important considerations for production deployment at scale.
The final layer of the system involves generating visualizations that provide different views into the knowledge base. The presenter demonstrates a graph view that shows notes as nodes connected by their relationships, clustered by topic area. This visualization was created by simply instructing an agent to “build with HTML and Tailwind some sort of graph view,” demonstrating the power of code-generating LLMs for UI development.
The graph view clusters notes into recognizable topic areas: scattered thoughts in the center, books, startup founding, AI and engineering, and faith and scripture. The visualization is interactive, allowing users to click on nodes to see the actual note content and how it connects to other ideas. The presenter even demonstrates modifying the visualization on the fly, changing the layout to look like a space constellation as a visual theme change.
Beyond graph views, the system can generate other visualizations like burndown charts for habit tracking based on note-taking frequency. The key insight is that because all the underlying data is in structured markdown files, LLMs can generate arbitrary HTML visualizations tailored to specific analytical needs.
The presentation showcases a custom application called Hubble that serves as the primary interface for this system. Hubble is described as free and open source, positioned as an “Apple Notes that’s agent accessible.” The application can display both markdown notes with their enrichments and backlinks, as well as HTML visualizations. This unified interface makes the system practical for daily use rather than requiring context-switching between multiple tools.
While the presentation is light on deep architectural details, several technical decisions emerge as important to the LLMOps implementation. The system uses markdown as its core data format, which provides several advantages: human readability, git-friendliness, tool compatibility with existing editors like Obsidian, and ease of parsing for LLMs. The file-based architecture makes the system simple to backup, version, and migrate.
The presenter demonstrates using Claude for some tasks, specifically mentioning its tendency to be overly creative with tag generation, which required explicit prompt instructions to constrain. The system also works with GLM 5.2, demonstrating model-agnostic design. This flexibility allows for optimization based on task requirements: perhaps using cheaper, faster models for simple enrichment and more capable models for complex wiki generation.
Web search capabilities are integrated for source research, with Exa mentioned as a specific tool option. The Obsidian headless CLI provides synchronization infrastructure. Docker provides containerization for cloud execution. These tool choices reflect a pragmatic approach to building with existing, well-supported components rather than building everything from scratch.
The agent framework appears to be built around the concept of “skills” that can be invoked with specific instructions and file paths. The enrich note skill demonstrates a clear separation between the skill definition and its execution environment, allowing the same skill to run locally for manual testing or in the cloud for scheduled automation.
Throughout the presentation, several prompt engineering techniques emerge as important to system reliability. The controlled vocabulary for tags, with explicit instructions to be reluctant about adding new ones, prevents tag explosion. The timestamp-based tracking of enrichment status provides a simple mechanism for incremental processing. The reference to Karpathy’s wiki instructions suggests using proven prompt patterns rather than inventing everything from scratch.
The presenter mentions instructing agents to use web tools for source research, to search for related notes using key term search, and to organize wiki content by entity types. These instructions are presumably encoded in the skill definitions, though the full prompts aren’t shown in detail. The ability to show actual execution logs demonstrates that these instructions are sufficiently clear for the LLMs to follow consistently.
One interesting tension in the system is between automation and control. The presenter wants the system to run automatically overnight but also wants to ensure quality and consistency in the outputs. The solution appears to be careful prompt engineering that sets clear expectations and constraints, combined with periodic human review of the results. This reflects a mature understanding of LLM capabilities: they’re powerful and useful but require thoughtful guidance to produce reliable results.
While the presentation focuses on demonstrating successful workflows, several production considerations are implied but not deeply explored. Error handling, for instance, isn’t discussed: what happens if the synchronization fails, if an LLM API is unavailable, if a note has formatting that confuses the parser, or if web search can’t find a source? Robust production systems need graceful degradation and retry logic for these scenarios.
Cost management is another consideration. Running daily automations across potentially hundreds of notes with multiple LLM API calls could become expensive. The system would benefit from cost tracking, budget alerts, and optimization strategies like caching or batching. The flexibility to use different models for different tasks (cheaper models for simple enrichment, more capable models for wiki generation) provides one avenue for cost optimization.
Quality assurance is mentioned only implicitly through the timestamp tracking and controlled vocabularies. More sophisticated systems might include validation steps, human review queues for ambiguous cases, or A/B testing of different prompt strategies. The ability to view execution logs provides some visibility, but structured monitoring and alerting would be important for production reliability.
Data privacy and security aren’t addressed in the presentation, though the mention of local voice transcription models suggests some awareness of these concerns. For personal notes, privacy is important, and the choice of where to run processing (locally, cloud sandbox, third-party APIs) has implications for data exposure.
While the presentation focuses on personal knowledge management, the presenter explicitly mentions workplace applications. Meeting transcripts could be automatically enriched with participant information and links to related meetings. Customer success teams could maintain wikis of client interactions. Research teams could organize literature reviews. The underlying architecture is flexible enough to adapt to various knowledge-intensive workflows.
The system represents a broader pattern in LLMOps: using LLMs not for customer-facing chatbots but for internal knowledge work and automation. These use cases often have different requirements than consumer applications: they can tolerate longer processing times in exchange for batch efficiency, they benefit from human oversight rather than fully autonomous operation, and they need integration with existing tools and workflows rather than net-new interfaces.
The emphasis on open source components and model flexibility also reflects important trends in LLMOps. Organizations increasingly want to avoid vendor lock-in and maintain control over their knowledge infrastructure. The ability to use local models for some tasks and cloud models for others, to swap between Claude and open-weight alternatives, and to run on self-hosted infrastructure provides strategic flexibility.
The presentation positions this work within a broader community of practice around LLM knowledge bases. The reference to Karpathy’s gist, the mention of Obsidian users, and the open-sourcing of the Hubble application all suggest an ecosystem of tools and techniques being shared and refined. The presenter encourages contributions to Hubble and implies that many of the components shown could be adapted or extended by others.
This community-driven approach accelerates innovation but also creates challenges for production deployment. Open source tools may have documentation gaps, breaking changes, or security vulnerabilities. The rapid pace of evolution in the LLM space means that best practices are still emerging. Organizations deploying these systems need to balance the benefits of leveraging community innovation with the stability requirements of production operations.
The presentation concludes with a call to action for attendees to try the tools and contribute to the ecosystem, reinforcing the collaborative nature of this space. The Warp booth and side event mentions suggest commercial interest in these capabilities, potentially as part of Warp’s platform offerings. This reflects a common pattern where open source experimentation informs commercial product development, with benefits flowing in both directions.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Notion, a knowledge work platform serving enterprise customers, spent multiple years (2022-2026) iterating through four to five complete rebuilds of their agent infrastructure before shipping Custom Agents to production. The core problem was enabling users to automate complex workflows across their workspaces while maintaining enterprise-grade reliability, security, and cost efficiency. Their solution involved building a sophisticated agent harness with progressive tool disclosure, SQL-like database abstractions, markdown-based interfaces optimized for LLM consumption, and a comprehensive evaluation framework. The result was a production system handling over 100 tools, serving majority-agent traffic for search, and enabling workflows like automated bug triaging, email processing, and meeting notes capture that fundamentally changed how their company and customers operate.
Wix developed two interconnected AI systems to address the challenge of searching and understanding code across thousands of repositories and services in a large organization. The first system, OctoCode, is an MCP-based tool with 90,000 downloads and 5,000 weekly active users that helps developers search repositories, understand dependencies, and navigate complex codebases. The second system, Bilbo, is an enterprise service that orchestrates multiple AI agents to investigate bugs and perform deep research across the organization's technical stack, integrating with GitLab, databases, logs, documentation, and other internal systems. Both systems employ sophisticated prompt engineering, context management, sub-agent architectures, and custom tooling protocols to handle the complexity of enterprise-scale code search and investigation while managing token limits and maintaining response quality.
