## Case Study Overview This case study presents Salesforce's solution to critical performance and reliability challenges in their AI inference infrastructure, specifically focusing on their AI Metadata Service (AIMS). The case demonstrates how metadata management bottlenecks can significantly impact large-scale AI deployment and how strategic caching architectures can resolve these issues while maintaining system resilience. Salesforce operates a multi-cloud, multi-tenant architecture where AI applications like Agentforce require tenant-specific configuration for every inference request. Each tenant may use different AI providers (OpenAI's ChatGPT or Salesforce's internal models) with unique tuning parameters, making metadata retrieval essential for proper request routing and context application. The AIMS serves as the central repository for this configuration data, making it a critical dependency in the AI stack where latency and availability directly impact end-user experience. ## Technical Problem Analysis The initial architecture suffered from several interconnected issues that are common in production AI systems. The primary challenge was that every AI inference request required real-time metadata retrieval from AIMS, which depended on a shared backend database and multiple downstream services including the CDP Admin Service. This created a synchronous dependency chain that introduced significant performance penalties. The latency impact was substantial, with metadata retrieval alone contributing approximately 400 milliseconds to P90 latency per request. In the context of AI inference workflows, this represented a significant bottleneck, contributing to overall end-to-end latency that reached 15,000 milliseconds at the P90 percentile. This level of latency is particularly problematic for AI applications like Agentforce where users expect responsive interactions. The reliability risks were equally concerning. The system exhibited classic single-point-of-failure characteristics where database degradation, restarts, or vertical scaling operations would impact all inference requests. A notable production incident occurred when the shared database experienced resource exhaustion across multiple dimensions - CPU, RAM, IOPS, disk space, and connection limits - resulting in approximately 30 minutes of disrupted metadata fetches that halted inference workflows across the platform. The "noisy neighbor" problem was another significant challenge. Because AIMS shared its database with other CDP services, resource contention from high usage by other services frequently degraded AIMS performance. This architectural decision, while potentially cost-effective, created unpredictable performance characteristics that were difficult to manage operationally. ## Solution Architecture and Implementation The engineering team's solution centered on implementing a sophisticated multi-layered caching strategy that addressed both latency and reliability concerns. The approach recognized that AI metadata changes infrequently, making caching a highly effective strategy once properly implemented. The first layer consists of local caching within the AIMS Client, which is integrated into the AI Gateway service. This L1 cache provides immediate access to metadata for the most commonly accessed configurations. The design prioritizes cache hits for frequently accessed tenant configurations, ensuring that the majority of requests can be served without any network calls to the backend services. The second layer implements service-level caching within the AIMS itself, designed to avoid unnecessary database and downstream service calls. This L2 cache serves as a resilience buffer that can maintain service continuity even during complete backend outages. The L2 cache stores longer-lived metadata and configuration data, and since most configurations change infrequently, it can safely serve potentially stale but valid responses during backend failures. The team made TTLs configurable per use case, recognizing that different types of metadata have different freshness requirements. Model preferences might remain static for weeks, while trust attributes may require more frequent updates. This flexibility allows clients to set appropriate expiration windows for both L1 and L2 caches based on their specific tolerance for stale data, preserving operational flexibility while protecting core system behaviors. ## Technical Implementation Details The implementation leveraged Salesforce's existing Scone framework to ensure consistency across services. The team introduced SmartCacheable annotations (both reactive and non-reactive variants) that enabled other teams to adopt caching capabilities without implementing custom caching logic. This standardization approach helped enforce shared guardrails for consistency and cache expiry while streamlining adoption across different service teams. Background refresh logic was implemented to proactively update cache entries before expiration, reducing the likelihood of cache misses during normal operations. Observability hooks were integrated to monitor cache age, track usage patterns, and enable proactive cache invalidation when necessary. These monitoring capabilities proved essential for operational visibility and troubleshooting. The system includes sophisticated alerting mechanisms that trigger when service behavior shifts to increased L2 cache usage, signaling potential issues in the backend infrastructure before they become critical user-facing problems. This represents a shift from reactive incident response to preventative operations management. ## Performance Results and Impact The performance improvements achieved through this caching strategy were substantial and measurable. Client-side caching reduced configuration fetch latency by over 98%, dropping from approximately 400 milliseconds to sub-millisecond response times for L1 cache hits. When L1 cache expires, the server-side L2 cache responds in approximately 15 milliseconds, still representing a significant improvement over the original architecture. The end-to-end request latency improvement was notable, reducing from 15,000 milliseconds to 11,000 milliseconds at the P90 percentile, representing a 27% reduction in overall response times. For AI applications like Agentforce where multiple AI calls may be chained together, these faster metadata lookups contributed to significantly quicker agent responses and enhanced overall system responsiveness. From a reliability perspective, the L2 cache architecture maintained 65% system availability even during complete backend outages that would have previously resulted in total service disruption. This resilience improvement transformed what could have been complete service outages into periods of continued operation with graceful degradation. ## Operational and Business Impact The implementation significantly improved operational characteristics of Salesforce's AI infrastructure. During a recent database connection timeout incident, both AIMS and customer-facing Agentforce services remained stable thanks to the L2 cache, while database infrastructure was scaled up and issues resolved without customer impact. This type of operational resilience has become a core part of the system architecture. The solution also addressed cost efficiency concerns by reducing the load on shared database infrastructure and minimizing the impact of resource contention from other services. By serving the majority of requests from cache, the system reduced database query load and associated infrastructure costs. ## Technical Lessons and Considerations This case study illustrates several important principles for LLMOps implementations at scale. The critical importance of treating metadata services as first-class infrastructure components that require the same level of attention to performance and reliability as the AI models themselves becomes clear. Metadata retrieval may seem like a minor component, but when it becomes a bottleneck, it can significantly impact overall system performance. The multi-layered caching approach demonstrates how different cache layers can serve complementary purposes - L1 for performance optimization and L2 for resilience and availability. The configurability of TTLs based on data characteristics and use case requirements provides operational flexibility while maintaining system safety. The integration of comprehensive observability and alerting capabilities proved essential for successful operation. The ability to monitor cache hit ratios, detect shifts in usage patterns, and proactively alert on potential backend issues enables teams to maintain high service quality while operating complex distributed systems. However, the case study also highlights some limitations and considerations that should be evaluated when assessing this approach. The improvement in availability to 65% during outages, while significant, still represents partial service degradation rather than full availability. The reliance on potentially stale data during outages introduces data consistency considerations that may not be acceptable for all types of AI applications. The success of this caching strategy depends heavily on the relatively static nature of AI configuration metadata. Applications with more dynamic configuration requirements might not achieve similar benefits and could require different architectural approaches. The operational complexity of managing multi-layered caches with different TTLs and refresh patterns also introduces additional system complexity that teams need to manage effectively.