## Overview Qovery, a DevOps automation platform serving over 200 organizations, embarked on building an agentic DevOps copilot in February 2025 to eliminate the manual grunt work associated with infrastructure management. The company's vision was to create an AI assistant capable of understanding developer intent and autonomously executing complex infrastructure tasks. This case study provides insights into their four-phase technical evolution from a basic rule-based system to a sophisticated agentic AI system with memory and recovery capabilities. The DevOps copilot represents a significant advancement in LLMOps implementation within the infrastructure automation domain. Rather than building a simple chatbot, Qovery focused on creating what they describe as "DevOps automation with a brain" - a system that can handle complex, multi-step workflows while maintaining reliability and context across conversations. ## Technical Evolution and LLMOps Implementation ### Phase 1: Basic Intent-to-Tool Mapping The initial implementation followed a straightforward approach where user intents were mapped directly to specific tools. For example, a request like "Stop all dev environments after 6pm" would trigger the stop-env tool through hardcoded mapping logic. This phase represents a classical rule-based approach to AI automation, where each intent required explicit programming and tool invocation was deterministic. While this approach provided predictability and ease of implementation, it quickly revealed scalability limitations. Every new use case required manual coding, and complex workflows couldn't be handled without explicit chaining logic. The system lacked the flexibility to handle unexpected user requests or adapt to novel scenarios, which is crucial for production LLM deployments. ### Phase 2: Dynamic Agentic Architecture The second phase marked a fundamental shift toward true agentic behavior. Instead of relying on predefined mappings, the system was redesigned to analyze user input and dynamically plan sequences of tool invocations. This represents a core LLMOps principle - moving from static rule-based systems to adaptive AI-driven workflows. The agentic architecture introduced several key components that are essential for production LLM systems. Each tool was designed with clear input/output interfaces, ensuring stateless operation and independent testability. This modular approach is crucial for LLMOps as it allows for individual tool optimization and versioning without affecting the overall system. The system became capable of handling unexpected user needs by reasoning about available tools and their combinations. For instance, questions like "How can I optimize my Dockerfile?" or "Why is my deployment time high this week?" could be addressed through dynamic tool selection and orchestration. This flexibility represents a significant advancement in LLMOps implementation, where the AI system can adapt to new scenarios without explicit programming. However, this phase also introduced new challenges typical of agentic LLM systems. Tool chaining became fragile, as outputs from one tool needed to perfectly match the expected inputs of subsequent tools. When any tool in the chain failed or produced unexpected results, the entire workflow would break. This highlighted a critical LLMOps consideration - the need for robust error handling and recovery mechanisms in production AI systems. ### Phase 3: Resilience and Recovery Mechanisms Recognizing that production LLM systems must handle failures gracefully, Qovery implemented comprehensive resilience layers and retry logic. This phase addresses one of the most critical aspects of LLMOps - ensuring system reliability in the face of unpredictable AI behavior and external system failures. The recovery system operates by analyzing failures when they occur, updating the execution plan accordingly, and retrying with corrected approaches. This required implementing sophisticated state tracking between tool steps, validation mechanisms to catch errors early, and re-planning capabilities when executions fail. Such recovery mechanisms are essential for production LLMOps as they prevent minor failures from cascading into complete system breakdowns. The implementation of intermediate state tracking represents an important LLMOps pattern. By maintaining visibility into each step of the execution process, the system can identify where failures occur and make informed decisions about recovery strategies. This approach enables successful completion of multi-step workflows that weren't anticipated during initial development, demonstrating the adaptability that production LLM systems require. ### Phase 4: Conversation Memory and Context Management The final phase addressed a fundamental limitation of stateless AI interactions by implementing conversation memory. This enhancement allows the system to maintain context across multiple interactions within a session, enabling more natural and efficient user experiences. Context management is a critical LLMOps capability that significantly impacts user adoption and system effectiveness. Without conversation memory, each user request was treated in isolation, forcing users to provide complete context for every interaction. With memory implementation, the system can now understand references to previous conversations, reuse earlier analysis, and maintain continuity across complex, multi-turn interactions. For example, a follow-up question like "What about the staging cluster?" can now be properly contextualized based on previous discussions about production clusters. This capability opens new possibilities for deeper optimization and monitoring tasks that require building upon previous analysis. The conversation memory implementation represents sophisticated LLMOps engineering, requiring careful management of context windows, relevant information extraction, and persistence across user sessions. ## Current Production Implementation The current production system utilizes Claude Sonnet 3.7 as the underlying language model, representing a strategic choice for enterprise-grade LLM deployment. However, Qovery acknowledges important LLMOps considerations around model selection and deployment strategies. While they don't currently send sensitive information to external models, they recognize the need for self-hosted model options to meet enterprise compliance requirements. The system demonstrates impressive capabilities in handling complex DevOps tasks. Examples include generating detailed usage statistics over specified time periods, optimizing Dockerfiles based on best practices, and automating environment management with team notifications. These capabilities showcase the practical value of well-implemented LLMOps in production scenarios. Performance considerations remain a focus area, with planning tasks currently taking up to 10 seconds for complex workflows. This latency is acceptable during alpha testing but requires optimization for production deployment. This highlights a common LLMOps challenge - balancing model capability with response time requirements in user-facing applications. ## LLMOps Challenges and Lessons Learned The evolution of Qovery's DevOps copilot illustrates several critical LLMOps lessons. The progression from simple rule-based systems to sophisticated agentic AI demonstrates the importance of iterative development in LLM applications. Each phase addressed specific limitations while introducing new challenges, emphasizing the need for careful architectural planning in production LLM systems. Tool interface design emerged as a crucial factor for success. The requirement for clear input/output specifications, stateless operation, and independent testability reflects best practices in LLMOps. These design principles enable reliable tool chaining and facilitate system debugging when issues arise. Error handling and recovery mechanisms proved essential for production deployment. The fragility of tool chaining in complex workflows necessitated sophisticated failure analysis and retry logic. This experience underscores the importance of resilience engineering in LLMOps, where AI unpredictability must be managed through robust system design. ## Future Directions and LLMOps Evolution Qovery's roadmap addresses several key areas of LLMOps advancement. Performance optimization focuses on reducing planning latency, which is critical for user experience in production systems. The move toward self-hosted models reflects growing enterprise requirements for data sovereignty and compliance in LLM deployments. The planned implementation of long-term memory across sessions represents an advanced LLMOps capability that could enable personalized user experiences and continuous learning from previous interactions. Such features require sophisticated memory management and user modeling capabilities that push the boundaries of current LLMOps practices. The emphasis on building "DevOps automation with a brain" rather than another chatbot reflects a mature understanding of LLMOps value proposition. Success in this domain requires deep integration with existing infrastructure tools and workflows, not just conversational interfaces. ## Production Deployment Considerations The alpha deployment strategy demonstrates prudent LLMOps practices for introducing AI capabilities to existing user bases. By initially targeting existing Qovery users through controlled channels like Slack workspaces, the company can gather feedback and refine the system before broader release. The integration with Qovery's existing DevOps platform provides crucial context and tool access that enables the AI system to perform meaningful work. This tight integration represents an important LLMOps pattern where AI capabilities enhance existing platforms rather than operating in isolation. The system's ability to handle diverse tasks from infrastructure optimization to compliance reporting demonstrates the versatility possible with well-implemented agentic AI systems. Such capabilities require extensive tool development and careful orchestration logic that represents significant LLMOps engineering investment. This case study illustrates the practical challenges and solutions involved in deploying sophisticated LLM systems in production environments, providing valuable insights for organizations considering similar agentic AI implementations in their own infrastructure automation efforts.