## Overview Ramp Research represents a production deployment of an agentic AI system designed to democratize data access within a fintech company. The case study describes how Ramp built an internal AI analyst agent to address what they call the "data bottleneck" – a situation where data questions were funneled through a single on-call analyst, resulting in hours of delay and ultimately discouraging employees from asking data questions altogether. The system was launched in early August 2025 and by mid-September had processed over 1,800 data questions across more than 1,200 conversations with 300 distinct users, representing a 10-20x increase in question volume compared to their traditional help channel. ## Problem Context and Business Case The problem Ramp faced is common in data-driven organizations: as scale increases, data questions create a bottleneck. Each question becomes a request in a help channel (`#help-data`), requiring an analyst to navigate multiple tools (Looker, Snowflake, dbt documentation) to formulate an answer. This process typically took hours, which meant decision windows narrowed, and more critically, many questions were never asked at all because employees hesitated to add to the queue. Ramp rejected the notion that this bottleneck was an inevitable consequence of organizational scale and instead saw it as an engineering problem suitable for an AI solution. ## Architectural Approach: Agentic System Design The core technical insight behind Ramp Research is the use of an agentic architecture rather than a simpler retrieval-augmented generation (RAG) approach. The team recognized that their analytics warehouse contains thousands of tables and views, and many questions require row-level inspection to answer correctly. Rather than relying exclusively on generic compression methods like keyword or vector search, they designed the agent with tools that allow it to inspect column values, branch based on findings, and backtrack when necessary – essentially reasoning through data the way a human analyst would. This represents a more sophisticated approach to the problem than simple semantic search over documentation would provide. The agent doesn't just retrieve relevant context; it actively explores the data space using programmatic tools to arrive at answers. This tool-using capability is central to the system's effectiveness and distinguishes it from simpler RAG implementations. ## Context Layer and Knowledge Management A critical component of Ramp Research's architecture is what they call the "context layer." The team recognized that large-scale data without context is nearly unusable. At Ramp, critical data context lives across three systems: dbt (data build tool), Looker (business intelligence), and Snowflake (data warehouse). The team aggregated and indexed metadata from these sources, allowing the agent to fetch the right models and construct precise queries. However, metadata alone proved insufficient. The agent struggled to connect its understanding of the data schema to the various business domains within Ramp (e.g., pricing, transactions, customer accounts). Much of this domain knowledge was tacit, residing with domain experts on the analytics team. To address this, Ramp had domain owners write technical documentation covering their respective areas. These documents were organized into a file system that Ramp Research can access on-demand, presumably through some form of retrieval mechanism, though the exact implementation details aren't specified. This combination of structured metadata from analytics tools and unstructured domain documentation represents a hybrid knowledge management approach. It's worth noting that maintaining this context layer requires ongoing effort – documentation must be kept current, and metadata must be refreshed – which represents an operational cost that the case study mentions they're working to automate further. ## Interface Design: Slack as Platform Ramp chose Slack as the primary interface for their AI agent, which is a pragmatic choice given that Slack already serves as their internal communication hub. The system operates through a dedicated channel (`#ramp-research-beta`) which had grown to over 500 members by the time of publication. This channel approach allowed the team to gather feedback during development and iterate on the product with a community of engaged users. Two specific interface features stand out as important for the production system: **Data Previews**: Initially, users had to open external tools (Redash or Snowflake) to inspect data and verify the agent's answers. The team added in-thread CSV previews, allowing users to validate results without leaving Slack. This is particularly important for less technical users who may not be comfortable with SQL or business intelligence tools. From an LLMOps perspective, this reduces friction in the user experience and likely increases trust in the system's outputs by making verification easier. **Multi-turn Conversations**: The team made each Slack thread stateful, enabling users to have back-and-forth conversations with the agent. This allows for clarification of intent, collaborative problem-solving in threads, and reasoning through complex problems iteratively. Importantly, the team notes that this conversational capability also improved the agent's end-to-end performance, suggesting that the ability to ask follow-up questions helps the agent better understand user intent and deliver more accurate results. Beyond the dedicated beta channel, the system is deployed as a Slack app that can be added to other channels, enabling teams to integrate it into existing workflows. Examples include using it in alert channels to diagnose failed transactions and in project channels to help scope new features. ## Evaluation Strategy Evolution The case study provides valuable insight into how Ramp iterated on their evaluation approach, which is particularly instructive for LLMOps practitioners. They went through three distinct phases: **Phase 1: Human-in-the-Loop Per Question**: Their initial approach was to ping domain owners for every question in their domain. This didn't scale because effort still increased linearly with request volume, essentially recreating the original bottleneck they were trying to solve. **Phase 2: End-to-End Concept Testing**: They shifted from evaluating individual questions to evaluating the context layer itself. Working with domain experts, they identified high-priority concepts in each domain and wrote end-to-end tests. These tests could identify when Ramp Research passed or failed on important scenarios but provided little diagnostic information about why failures occurred, making it difficult to iterate and improve. **Phase 3: Intermediate Step Validation**: The team built a custom Python mini-framework within their dbt project that asserts not just on final answers but on intermediate steps. This includes validating expected tool calls, table references, and query structure. This granular testing approach enabled them to close the feedback loop effectively: update context, run tests, and confirm that changes actually improved performance. This evolution demonstrates a mature approach to LLMOps evaluation. Rather than relying solely on end-to-end accuracy metrics, they built infrastructure to understand the agent's reasoning process, which is essential for debugging and iterative improvement of agentic systems. The integration with dbt is also notable – by embedding tests in their data transformation layer, they can ensure that changes to the data warehouse don't break the agent's capabilities. ## Production Metrics and Impact The case study provides concrete metrics that offer insight into the system's production performance and adoption: - Over 1,800 data questions answered since early August launch (approximately 6 weeks) - More than 1,200 distinct conversations - 300 unique users - 500+ members in the beta Slack channel - In the last 4 weeks before publication, Ramp Research answered 1,476 questions in the beta channel compared to 66 in the traditional help channel These numbers indicate strong adoption and suggest that the system has successfully shifted behavior. The 10-20x increase in question volume that Ramp cites is particularly significant. They argue that most of this growth represents questions that previously "died in drafts or never left someone's head" – latent demand that was suppressed by the friction of the old process. From a business impact perspective, Ramp frames the value in terms of decision quality rather than direct cost savings. They use a "counting cards" analogy: a small improvement in decision quality across thousands of decisions (pricing adjustments, go-to-market filters, feature rollouts) compounds into material business value. This framing is more sophisticated than simply claiming the agent "replaces" analysts, and it's more aligned with how AI systems typically create value – augmenting human decision-making rather than fully automating tasks. The case study also mentions downstream benefits for customers: faster answers from account managers, better bug isolation, and sharper roadmap decisions. When validation of hypotheses becomes trivial, teams can move with more conviction and less rework. ## Privacy and Security Considerations The case study includes a brief but important note: Ramp Research does not have access to any personally identifiable information (PII). This is a critical design constraint for a system operating in the finance industry, which must handle sensitive customer data. The fact that the agent can still answer a wide variety of data questions without PII access suggests that their data architecture segregates PII from aggregated analytics data, which is a common pattern in financial services. However, the case study doesn't provide details on other security measures, such as access control (can all users ask questions about all data domains?), audit logging, or how they prevent the agent from leaking sensitive business information. These are important considerations for any production LLM system, particularly in regulated industries. ## Limitations and Critical Assessment While the case study presents impressive results, several aspects warrant careful consideration: **Verification and Accuracy**: The case study doesn't provide explicit accuracy metrics. While they mention that users can validate results through CSV previews and that they've built comprehensive test suites, we don't know what percentage of answers are actually correct, how often users verify results, or what happens when the agent produces incorrect answers. The 10-20x increase in question volume is meaningful, but without accuracy data, it's difficult to fully assess whether these are high-quality answers or whether users are accepting potentially flawed outputs because they're fast. **Maintenance Burden**: The system depends heavily on the context layer – documentation, metadata, and domain knowledge. The case study acknowledges this is "an incredibly valuable technical asset" and mentions automating its maintenance as a future goal, which suggests it currently requires manual effort. The operational cost of keeping domain documentation current and ensuring metadata stays synchronized could be substantial. **Scope Limitations**: The agent operates only on the analytics warehouse, explicitly excluding PII and presumably excluding operational databases. This means there are entire categories of questions it cannot answer. Additionally, the system is designed for internal use only, so we can't assess how it would perform with external users who lack organizational context. **Selection Bias**: The 300 users and 500+ beta channel members likely represent relatively data-savvy employees who are comfortable asking questions in a public channel. The system may not work as well for less technical users or those with more basic data literacy, though the CSV preview feature suggests they're trying to address this. **Generalization Claims**: The case study positions Ramp Research as "the first step towards building the future of work at Ramp, where agents and people work collaboratively." This is promotional language typical of company blog posts. While the specific solution appears solid, the broader claims about the future of work should be taken as aspirational rather than demonstrated outcomes. ## Future Directions The case study mentions two key areas for future development: **Headless API**: Teams are beginning to use Ramp Research for automated workflows like generating customer case studies and detecting fraud patterns. Providing a headless API would allow teams to integrate the agent into custom applications and automated pipelines, moving beyond the conversational interface. **Automated Context Maintenance**: The team wants to automate the maintenance and improvement of the context layer, which would reduce operational overhead and potentially allow them to expand coverage beyond the analytics database. These directions suggest the team is thinking about Ramp Research as a platform rather than just a chat interface, which is a mature perspective on LLMOps. ## LLMOps Takeaways This case study illustrates several important LLMOps principles: The value of agentic architectures over simple RAG for complex, exploratory tasks where the search space is large and answers require multi-step reasoning. The decision to give the agent tools for inspection, branching, and backtracking rather than relying solely on retrieval was central to the system's capabilities. The importance of a hybrid knowledge layer that combines structured metadata with unstructured domain documentation. Neither alone would have been sufficient. The evolution from human-in-the-loop evaluation to intermediate step validation demonstrates how evaluation strategies must mature as systems scale. Testing intermediate reasoning steps rather than just final outputs is crucial for debugging and improving agentic systems. The choice of interface matters significantly for adoption. By meeting users where they already work (Slack) and adding conveniences like in-thread data previews, Ramp reduced friction and increased trust. Multi-turn conversational capability serves dual purposes: it improves user experience and appears to improve agent performance by enabling intent clarification. The business value framing around decision quality at scale rather than headcount replacement is both more realistic and more compelling for agentic systems that augment rather than replace human judgment. Overall, while the case study has a promotional tone typical of company blog posts and lacks some technical depth (specific models used, latency characteristics, detailed accuracy metrics), it provides a credible account of deploying an agentic AI system in production within a finance company. The iterative approach to evaluation, the thoughtful interface design, and the clear metrics on adoption and usage patterns make this a valuable reference for organizations considering similar internal AI tools.