Financial services organizations are struggling to move AI from proof-of-concept to production at scale, with many pilots failing to reach deployment. CardWorks, a subprime credit card company, successfully implemented a multi-agent system using Anthropic's Claude and AWS/Snowflake infrastructure to analyze macroeconomic impacts on delinquency rates, reducing analysis time from one week to five minutes. The solution combined natural language processing with specialized agents for energy, financial, and internal banking data. Meanwhile, TCS is working with major banks to reimagine entire value chains using 20+ autonomous agents, such as transforming commercial customer onboarding processes to reduce dropouts, targeting 70-80% efficiency gains rather than incremental 20-30% improvements through process transformation rather than mere automation.
This case study presents multiple perspectives on deploying LLM-based systems in financial services, featuring insights from TCS (a consulting and systems integration firm), Anthropic (the LLM provider), and CardWorks (a subprime credit card company). The discussion centers on the persistent challenge of moving generative AI from pilot projects to production-scale deployments, with particular focus on multi-agent systems and enterprise transformation. According to McKinsey 2025 data referenced in the panel, 88% of organizations have implemented AI in at least one line of business, but the transition from simple use cases to complex AI systems remains challenging.
The key insight emerging from this case study is the distinction between simple, localized AI use cases that can reach production in 3-4 months versus complex “AI systems” involving multiple coordinated agents that require substantially more effort and organizational readiness. The panel emphasizes that while early AI implementations focused on natural language workflows and customer-facing applications with limited integration requirements, the industry is now pivoting toward interconnected agent-based systems that demand fundamentally different approaches to data, integration, governance, and organizational structure.
CardWorks provides the most concrete production deployment example in this case study. As a credit card company serving the subprime market with customers below 700 credit scores, CardWorks relies heavily on data analytics for risk analysis and marketing. The company identified that modelers were spending 25-30% of their time simply collating data from multiple sources before they could begin actual analysis work.
The specific use case implemented involved analyzing the impact of oil prices on delinquency rates. Previously, this analysis required approximately one week of effort from skilled modelers. CardWorks built a multi-agent solution that went live approximately two weeks before this panel discussion. The architecture leveraged their existing AWS and Snowflake infrastructure, with subscription access to curated macroeconomic data through Snowflake’s data marketplace. The implementation used Anthropic’s Claude as the underlying language model.
The multi-agent architecture consisted of a main agent handling natural language processing to evaluate queries, with three specialized sub-agents: an energy sector agent, a financial sector agent, and a Medic Bank agent for internal CardWorks data. The challenge these agents addressed was the sheer scale and complexity of the macroeconomic data, which spans from the 1970s to present with annual, quarterly, and daily granularity. A business user could pose a natural language query such as “Give me the impact of Medic Bank portfolio with the card to the price increases of oil and how it impacts delinquency rate,” and the system would coordinate across the three agents to retrieve, join, and analyze the relevant data.
The production results were significant: what previously took one week was completed in five minutes, with the system generating appropriate SQL queries and returning an actionable answer that oil prices do impact delinquency rates with a three-month lag. This represents not just a speed improvement but a fundamental shift in how data scientists can interact with complex, multi-source datasets. The chief data officer emphasized that while agent building existed before, having the output operationalized and reliably available represents the true production value.
The panel identified several critical barriers preventing AI pilots from reaching production scale. Integration complexity emerged as the primary challenge, with TCS reporting that in typical AI projects, only 40% of effort goes to building the AI solution itself while 60% is consumed by integration work. In some cases, even successful pilots were abandoned because the business case for the required integration work did not justify the investment. This challenge is particularly acute for the “AI systems” category of use cases that require coordination across multiple platforms and data sources.
Data readiness represents the second major challenge. While proof-of-concept environments work with curated, clean data, production systems must handle the full complexity of real-world enterprise data. Many organizations lack both AI-ready data and the capability to manage the complete data lifecycle required for production LLM systems. CardWorks’ chief data officer emphasized that poor data foundation remains one of the most significant but least exciting aspects of successful deployment. The company is actively working with TCS to use generative AI to migrate legacy SAS code to SQL and Python to modernize their data infrastructure and improve integration capabilities with cloud-based AI tools.
Metadata quality emerged as a critical but often overlooked requirement for production LLM systems, particularly for natural language querying of business data. The CardWorks example illustrated this challenge: historical modeling work might use abbreviations like “DQ 30” that models won’t recognize without training or proper metadata indicating this means “delinquency 30 days.” The solution requires not just technical definitions but business-user-friendly descriptions that account for the variety of ways humans might phrase questions. CardWorks leveraged third-party tools to enhance metadata and noted that successful external data sources like their macroeconomic data subscription had invested heavily in metadata tables that make it easy for LLMs to recognize entities and avoid hallucinations.
Cross-functional team composition presents another barrier. AI projects cannot be staffed solely with AI specialists. They require subject matter experts from platform teams, business experts who can re-architect workflows, and coordination across organizational boundaries. Many organizations structure AI initiatives with centralized teams that can slow progress, whereas a federated model with business unit control over use cases combined with centralized governance appears more effective based on TCS’s client experience.
AI governance and model risk management constitute the fourth major challenge. Organizations need centralized control planes for governance rather than duplicating controls within individual use cases. This includes model lifecycle management, risk management frameworks, and determining where controls should be placed in the architecture. The panel emphasized the importance of having security and compliance teams as partners throughout development rather than notifying them after systems are built, which inevitably surfaces blockers late in the process.
A central theme in the case study is the distinction between using AI to incrementally improve existing processes versus fundamentally reimagining business workflows. TCS advocates for a “right to left” approach that starts with the desired business value and works backward to reimagine the process, rather than analyzing existing workflows and identifying where AI can be inserted. This “AI first” approach requires returning to first principles about what a process should accomplish.
The most ambitious example discussed involves TCS working with one of the largest banks to reimagine the entire commercial customer onboarding process. Rather than focusing on cost reduction, this initiative addresses the high dropout rate in commercial banking onboarding caused by customer frustration with lengthy processes. The solution involves 20+ autonomous agents working together to create a seamless end-to-end experience. The approach “flips” the typical optimization mindset: instead of targeting 10-30% productivity improvements by tweaking existing processes, the team envisions an almost humanless, fully autonomous process and then identifies where human intervention is actually necessary. This shift in perspective enables 70-80% efficiency gains rather than incremental improvements.
Another example involves credit memo generation in commercial lending, which currently takes 1-4 weeks. Early pilots suggest 40-60% time reduction is achievable. Document processing, which permeates virtually every banking and financial services workflow, is being transformed through agentic document processing solutions that can handle the variety and complexity of financial documentation.
The customer onboarding transformation is particularly illustrative of the LLMOps challenges involved. The system requires sophisticated integration across multiple backend systems, careful orchestration of 20+ agents, governance mechanisms to ensure regulatory compliance, and graduated autonomy approaches where the system starts with heavy human oversight and progressively gains independence as trust is established through monitored performance.
The panel provided specific recommendations for organizational structure and strategy. Anthropic’s representative emphasized the importance of single-threaded leadership, with one executive owning AI overall and sub-leaders owning specific use cases within business units. These leaders are responsible for reporting progress, blockers, and value creation. This structure contrasts with group-based or committee approaches that tend to slow decision-making and dilute accountability.
For prioritizing among many potential use cases, the recommendation is to create a simple two-axis chart of complexity versus business impact. Organizations should start with low-complexity, high-business-impact cases to achieve early wins, build momentum, and develop organizational capabilities before tackling the most complex and interesting use cases. Starting with the most ambitious use cases often results in nine-month cycles of frustration, blocked by security, compliance, and integration challenges while teams simultaneously learn new technology. Building “calluses and skills” with simpler implementations prepares teams for complexity.
The most successful organizations in AI adoption, according to Anthropic’s observations, share a common thread: top-down executive mandate for broad staff adoption of AI tools in daily work. Even before the ecosystem was as mature as it currently is, these organizations mandated usage levels that forced teams to learn the edges, understand the tools, and climb the learning curve. As teams became proficient, they identified creative automation opportunities and began queuing work to LLMs, focusing human effort on highest-value activities. This creates a virtuous cycle where successful teams present their approaches to others, spreading adoption organically but supported by executive mandate.
Another characteristic of successful organizations is setting realistic acceptance criteria rather than demanding perfection. If a system can automate 80% of a task, enabling humans to augment the remaining 20%, this is vastly preferable to humans doing 100% manually. The analogy offered: if you need to travel 10 miles, driving nine and walking one is far better than walking all 10. This pragmatic approach to autonomy and augmentation accelerates deployment and value capture.
A sophisticated approach to managing the autonomy-versus-control tradeoff emerged from the discussion. TCS advocates for a graduated autonomy model where systems begin with complete human oversight and progressively gain autonomy as they demonstrate reliable performance. This requires continuous monitoring of agent behavior and performance metrics that inform decisions about expanding autonomy boundaries.
The comparison to coding assistants illustrates this principle: over the past year, code generated by AI has improved dramatically in quality. Users who have worked with these tools over time develop trust and require less time reviewing output, though review remains necessary. The same trust-building dynamic applies to enterprise agent systems. Starting with heavily supervised agents performing repetitive tasks, organizations can expand autonomy as patterns prove reliable.
Critical to this approach is establishing proper guardrails, policies, and constitutional AI principles that fence agent behavior. A centralized control plane that manages these constraints across all agents prevents inconsistent governance and enables systematic monitoring. Escalation mechanisms are essential: when an agent encounters patterns it hasn’t observed before or situations that exceed predetermined thresholds, the system should escalate to human judgment rather than proceeding with uncertain actions.
While the case study doesn’t provide exhaustive architectural details, several technology components are identified. CardWorks’ implementation runs on AWS infrastructure with Snowflake as the data platform, utilizing Snowflake’s data marketplace for macroeconomic data. Anthropic’s Claude serves as the foundation model powering the multi-agent system. The architecture separates concerns between a main natural language processing agent and specialized domain agents that understand specific data domains.
The panel references the broader ecosystem maturation enabling production deployments, including SDKs, agent frameworks, and MCP. Anthropic is working with hyperscalers like AWS and systems integrators like TCS to provide building blocks beyond the core model. The model serves as the “brain” but requires orchestration tools, data access mechanisms, and context management to function in production environments.
TCS mentioned their approach of creating rapid build processes that translate ideas into defined outcomes, followed by scaling with appropriate platforms and integration infrastructure. This suggests a phased methodology moving from discovery and visioning through rapid prototyping to production scaling, with platform engineering as a distinct phase.
CardWorks’ ongoing data modernization effort illustrates a pragmatic LLMOps use case: using generative AI to accelerate the migration from legacy systems to modern cloud infrastructure. The company is conducting a proof-of-concept with TCS to convert legacy SAS code to SQL, Python, and other open-source languages. This addresses both the integration challenges and the reality that on-premise SAS infrastructure doesn’t integrate well with cloud-based generative AI tools.
This represents a meta-application of LLMOps: using LLMs to prepare the data and system infrastructure necessary for more sophisticated LLM applications. Legacy systems in 35-year-old financial institutions create complex dependencies and technical debt that block AI adoption. Automating the translation and modernization of this legacy code base can accelerate the broader AI transformation agenda.
While the case study provides valuable insights into real-world LLM deployments in financial services, several caveats warrant consideration. The CardWorks implementation, while impressive in speed improvement, addresses a relatively contained use case involving data retrieval and analysis rather than decision-making or customer-facing interactions. The five-minute response time, while far better than one week, still represents batch-style analysis rather than real-time interaction. The claim that results were “phenomenal” should be understood in context: the system successfully identified a correlation with a three-month lag, but the case study doesn’t discuss validation of this finding, monitoring for drift in the relationship, or how this insight integrates into production risk models.
The discussion of 20+ agents reimagining commercial customer onboarding is aspirational rather than a completed deployment. The panel doesn’t provide details on current progress, specific challenges encountered, or metrics demonstrating the 70-80% improvement target. This represents TCS positioning their capabilities and vision rather than documented production results. The timeline, complexity, and actual outcomes remain to be seen.
The emphasis on integration consuming 60% of project effort highlights that LLM deployment success depends heavily on traditional enterprise architecture and integration capabilities rather than AI-specific concerns. Organizations without strong integration competencies and modern data infrastructure will struggle regardless of LLM capabilities. This suggests that LLMOps success may be more about general engineering and architecture maturity than AI-specific factors.
The recommendations around organizational structure, while sensible, are presented without rigorous evidence or comparison of outcomes across different organizational models. The observation that successful firms have executive mandates for adoption could reflect correlation rather than causation—perhaps organizations with strong executive leadership and change management capabilities succeed across multiple dimensions, with AI being just one example.
The graduated autonomy approach is conceptually sound but operationally complex. The case study doesn’t address how organizations define appropriate thresholds for expanding autonomy, how they measure agent reliability in ways that inform these decisions, or how they handle the organizational change management required as humans cede control progressively to automated systems. The comparison to coding assistants may oversimplify, as coding assistance impacts individual developer workflows while autonomous enterprise agents affect cross-functional processes with compliance and regulatory implications.
Despite these caveats, the case study reveals genuine progress toward production LLM deployments. CardWorks’ implementation demonstrates several maturity indicators: integration with enterprise data platforms, handling of real business questions in production, multi-agent coordination for complex queries, and deployment timelines measured in weeks rather than months for a well-scoped use case. The emphasis on metadata quality and data foundation work reflects hard-won operational experience rather than theoretical concerns.
The shift in discussion from simple chatbot and summarization use cases to multi-agent systems performing actual work indicates ecosystem maturation. The integration of LLMs with enterprise data platforms like Snowflake, the availability of curated third-party data with LLM-friendly metadata, and the development of agent orchestration frameworks all contribute to making production deployments more feasible than they were even a year prior.
The acknowledgment of failures and abandoned projects due to integration costs demonstrates realistic assessment rather than purely optimistic promotion. The candid discussion of challenges, the emphasis on unsexy but critical concerns like metadata and legacy modernization, and the recognition that most project effort goes to integration rather than AI itself all suggest practitioners grappling with real operational issues rather than simply marketing AI capabilities.
This case study illuminates several important LLMOps considerations for financial services and likely other highly regulated industries. First, the 60% integration effort statistic suggests that LLMOps tooling and platforms should prioritize integration capabilities, pre-built connectors, and orchestration frameworks rather than focusing exclusively on model serving, monitoring, and retraining. The bottleneck isn’t model operations in isolation but the coordination between models and existing enterprise systems.
Second, metadata management emerges as a critical but underserved aspect of LLMOps infrastructure. Organizations need systematic approaches to creating business-user-friendly, LLM-compatible metadata across their data assets. This likely requires dedicated tooling, processes, and organizational roles that bridge data governance, semantic modeling, and LLM engineering.
Third, the graduated autonomy model implies requirements for sophisticated monitoring, escalation, and progressive deployment capabilities in LLMOps platforms. Systems need telemetry that captures not just model performance but agent decision quality, the ability to define and modify autonomy boundaries based on demonstrated reliability, and escalation workflows that route edge cases to human judgment while capturing those examples for future model improvement.
Fourth, the emphasis on cross-functional teams and federated organizational models suggests that LLMOps platforms must support collaboration across diverse roles—business users, subject matter experts, data engineers, ML engineers, compliance specialists—with appropriate abstractions and interfaces for each. Purely code-centric MLOps approaches may not translate well to LLMOps in enterprise settings where business users need to participate in defining agent behavior, reviewing outputs, and providing feedback.
Finally, the focus on process transformation rather than automation implies that LLMOps extends beyond technical operations to include business process redesign methodologies, change management capabilities, and value measurement frameworks that capture business outcomes rather than just technical metrics. Organizations attempting LLM deployments may need to combine traditional MLOps disciplines with business process management and transformation consulting capabilities to achieve the 70-80% improvements discussed rather than incremental gains.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Digits, a company providing automated accounting services for startups and small businesses, implemented production-scale LLM agents to handle complex workflows including vendor hydration, client onboarding, and natural language queries about financial books. The company evolved from a simple 200-line agent implementation to a sophisticated production system incorporating LLM proxies, memory services, guardrails, observability tooling (Phoenix from Arize), and API-based tool integration using Kotlin and Golang backends. Their agents achieve a 96% acceptance rate on classification tasks with only 3% requiring human review, handling approximately 90% of requests asynchronously and 10% synchronously through a chat interface.
Stripe, processing approximately 1.3% of global GDP, has evolved from traditional ML-based fraud detection to deploying transformer-based foundation models for payments that process every transaction in under 100ms. The company built a domain-specific foundation model treating charges as tokens and behavior sequences as context windows, ingesting tens of billions of transactions to power fraud detection, improving card-testing detection from 59% to 97% accuracy for large merchants. Stripe also launched the Agentic Commerce Protocol (ACP) jointly with OpenAI to standardize how agents discover and purchase from merchant catalogs, complemented by internal AI adoption reaching 8,500 employees daily using LLM tools, with 65-70% of engineers using AI coding assistants and achieving significant productivity gains like reducing payment method integrations from 2 months to 2 weeks.
