Motorola Solutions deployed a Model Context Protocol (MCP) based AI-powered observability system to reduce incident resolution time for their mission-critical public safety platforms that handle 911 emergency dispatch services. The platform engineering team built custom MCPs for Kubernetes and Grafana that enable AI agents to autonomously investigate alerts, analyze logs, correlate metrics, and provide root cause analysis. This implementation reduced their Mean Time To Resolution (MTTR) from 45 minutes to under 4 minutes while maintaining 99.99% availability requirements. The solution integrates human-in-the-loop approval gates and uses LangFuse for LLM observability and evaluation frameworks to ensure the AI-generated recommendations are validated before execution in their production environment.
Motorola Solutions operates mission-critical public safety infrastructure that powers 911 emergency response systems across the United States. Their platform handles the entire emergency workflow from citizen calls to computer-aided dispatch (CAD) systems to personnel deployment. Given the life-critical nature of these services, the company cannot afford extended outages, making rapid incident resolution paramount. The platform engineering team developed an MCP-powered AI observability solution that fundamentally transformed how they handle incidents, dropping resolution times from 45 minutes to approximately 4 minutes while maintaining the 99.99% availability required for public safety operations.
The implementation represents a production deployment of AI agents for SRE operations, with multiple custom MCPs serving as the interface layer between LLMs and their observability infrastructure. The team built this on AWS Bedrock and integrated it with their existing Kubernetes clusters running across hybrid environments including AKS, EKS, GKE, and Red Hat OpenShift, as well as edge devices handling IoT telemetry.
Before implementing the MCP-powered solution, the SRE team faced several chronic challenges in their incident response workflow. When Grafana alerts triggered, engineers had to manually navigate multiple dashboards to understand what was happening. Often metrics would be missing or incomplete, forcing them to pivot to checking Kubernetes logs directly. They would manually create Jira tickets and investigate pod crashes, node failures, and performance degradation through a time-consuming process of context switching between different monitoring tools.
The team identified specific pain points including alert fatigue with too many non-actionable alerts, excessive context switching between Grafana dashboards and Kibana dashboards without clear correlation, knowledge silos where observability specialists familiar with Elastic lacked expertise in Grafana and vice versa, and junior engineers who struggled to navigate the complex debugging process effectively. The typical incident investigation could take 45 minutes or more, with engineers manually correlating data across systems to find root causes.
The architecture centers on a master AI agent referred to as an SRE or AI operator that resides on AWS Bedrock. This agent communicates with three different MCP servers, each providing specialized capabilities for different aspects of the observability stack. The MCPs are exposed as external endpoints via ingress controllers and integrated with Amazon Q for the conversational interface.
The MCP layer serves as a standardized interface between the AI agent and the underlying infrastructure tools. Based on the intent detected in user queries, the LLM determines which MCP to invoke. Each MCP has well-defined tool definitions that describe the operations it can perform. This architectural pattern evolved from earlier custom API wrapper approaches, with the team recognizing that MCP provides not just an interface standard but an entire ecosystem with governance and gateway capabilities.
The deployment runs live in production, handling real incidents for the observability team. The MCPs are deployed with role-based access control (RBAC) to ensure security boundaries are maintained, preventing the AI agent from accessing resources outside its designated scope.
The Kubernetes MCP provides the AI agent with visibility into cluster operations across their multi-cluster hybrid environment. When a developer reports a service failure, the traditional SRE workflow involves manually checking pod logs, running kubectl describe commands, examining dashboards for specific timeframes, and analyzing resource utilization patterns. The Kubernetes MCP automates this investigation process through several key capabilities.
The MCP performs pod inspection, checking deployment status, analyzing Kubernetes events, and assessing resource utilization. When the agent receives a query about service failures, it automatically invokes the appropriate tools to gather this information. For example, in the demonstration shown, when investigating pods in a non-ready state in the default namespace, the MCP identified an incorrect image name in an nginx pod configuration and reported the specific error to the SRE.
Importantly, the Kubernetes MCP is deliberately restricted from performing write operations. When the agent attempted to fix the nginx pod by deleting and recreating it, the MCP correctly returned a permissions error. This illustrates the human-in-the-loop design where the AI provides diagnosis and recommendations but cannot execute changes without human approval. For partner deployments where the SRE team lacks direct code access, the system generates appropriate output for Jira tickets explaining the issue to the owning team.
The team emphasized security considerations in their implementation. The MCP does not have node-level access, preventing it from accessing sensitive cluster infrastructure. Namespace isolation is enforced through RBAC, ensuring the agent cannot query logs from namespaces it should not access. This prevents data leakage concerns that would arise from giving broad cluster access to a system that ultimately sends data to public LLMs.
The Grafana MCP provides the agent with access to metrics, dashboards, and alerting data. When incidents trigger, the MCP can query specific dashboards and analyze metrics for particular timeframes to understand what went wrong and when recovery began. This eliminates the manual dashboard navigation and metric correlation that previously consumed significant SRE time.
The MCP checks dashboard states, identifies when metrics transitioned from healthy to unhealthy states, and correlates these transitions with other events in the system. This correlation capability represents what the team calls “30-second correlation magic” where the AI can identify patterns across multiple data sources that would take human engineers 20-30 minutes to manually correlate.
For example, when investigating high latency alerts where response times exceed 5 seconds, the traditional approach requires checking CPU utilization, memory utilization at both pod and node levels, pod scheduling capacity, and correlating all these signals. The Grafana MCP automates this multidimensional analysis, providing the SRE with a synthesized view of the contributing factors.
The end-to-end incident workflow begins when an incident triggers in the monitoring system. The AI agent automatically activates and begins calling the appropriate MCP tools based on the nature of the alert. The system intelligently selects which tools to invoke, then analyzes historical data to understand patterns and context around the current issue.
After gathering data from the MCPs, the agent performs root cause analysis and generates recommendations. At this point, the human-in-the-loop gate activates. The system does not execute automated remediation actions, instead presenting the analysis and recommendations to the SRE engineer who makes the final decision on whether to implement the suggested solution or investigate further.
The team demonstrated this with a node failure scenario, a common Kubernetes problem where nodes enter a not-ready state. When this happens during peak traffic, pods evict from the failing nodes, potentially causing service degradation. The MCP-powered system rapidly identifies the node issue, analyzes the impact on pod scheduling, and provides actionable recommendations to the SRE within minutes rather than the traditional extended investigation period.
Critical to the production deployment is the evaluation framework built on LangFuse, which the team uses as their observability platform for agentic AI workloads. The presenters repeatedly emphasized that they cannot and do not rely blindly on MCP outputs due to the mission-critical nature of their services. LLM hallucinations pose an unacceptable risk when incorrect information could endanger lives in emergency response scenarios.
LangFuse provides built-in evaluation frameworks that assess LLM behavior over time. Before any MCP output reaches customer-facing systems, it passes through evaluation metrics that validate the quality and accuracy of the response. The team maintains detailed audit logs of agent decisions, traces of reasoning processes, and evaluation scores that enable them to understand on what basis the agent made specific recommendations.
This evaluation layer represents their answer to the fundamental tension between rapid AI-powered operations and the 99.99% availability requirement. They achieve speed improvements while maintaining reliability through continuous evaluation and validation. The team stated explicitly that without evaluation frameworks in place, teams should not deploy MCP in production environments, particularly for mission-critical applications.
Security received significant emphasis throughout the implementation. The RBAC controls ensure the AI agent cannot access resources beyond its authorized scope. Zero-trust principles govern the deployment, with the team leveraging MCP gateway security features discussed in the broader MCP ecosystem.
Approval gates prevent automated execution of potentially impactful changes. The team explicitly does not grant the MCP permission to push code changes to Git repositories or modify production configurations, as this would create unacceptable data breach and operational risks. Detailed audit logging through LangFuse ensures compliance teams can review agent actions and verify appropriate security controls.
The team also highlighted infrastructure scaling concerns related to security. Deploying MCP effectively opens the floodgates for AI-driven requests to the platform. Without proper monitoring and scaling policies, this could overwhelm infrastructure or create vectors for abuse. They use Kubernetes Event-Driven Autoscaling (KEDA) with the agent to intelligently distinguish genuine traffic from malicious requests before scaling infrastructure.
For multi-cluster environments spanning self-managed and cloud-managed clusters across AKS, EKS, GKE, Red Hat OpenShift, and edge devices, maintaining consistent security posture requires careful governance. The team does not expose sensitive cluster internals or node-level access to the MCP layer, minimizing the attack surface and data exposure risk.
The quantitative results demonstrate dramatic improvement in operational efficiency. Mean Time To Resolution dropped from 45 minutes to under 4 minutes, representing over a 90% reduction in incident response time. The system maintains the required 99.99% availability while enabling this faster response capability.
Beyond raw numbers, the implementation improved operational risk profiles by reducing the window of vulnerability during incidents. Customer trust increased through more reliable service delivery. The platform benefits junior engineers by providing immediate operational context and intelligence that previously required senior SRE knowledge. This democratization of expertise addresses knowledge silos and enables more team members to effectively handle incidents.
The team characterizes this as moving beyond traditional MLOps and AIOps to “AI-powered ops” where AI agents actively participate in operational workflows. They report real production gains from the MCP implementation and encourage other teams to adopt similar approaches, with the caveat that evaluation frameworks must accompany any production deployment.
Despite the success, the presenters were transparent about limitations and concerns. LLMs hallucinate, and the system cannot be trusted to always provide correct outputs. The mission-critical nature of public safety infrastructure means incorrect information presented as truth could endanger lives. This drove their conservative human-in-the-loop design where humans retain final decision authority.
They acknowledge that achieving critical availability at 99.99% comes at a cost. The evaluation frameworks, security controls, and human approval gates add complexity and potentially slow down the theoretical speed AI could achieve with full automation. However, they view this as an acceptable tradeoff given their operational requirements.
The team also noted that MCP is still evolving, and earlier skepticism about it being merely a custom API wrapper gave way to recognition of the broader ecosystem developing around the protocol. They position their implementation as production-ready while acknowledging ongoing evolution in tooling, governance, and best practices.
The presenters offered several strong recommendations based on their experience. Teams should not experiment with AI in production without evaluation frameworks in place. The ability to audit agent decisions, validate outputs, and track reasoning is essential for any mission-critical deployment. LangFuse or similar LLM observability platforms should be considered non-negotiable components of the architecture.
Security and compliance must be designed into the system from the start, not retrofitted. RBAC controls, approval gates, and granular permissions prevent unauthorized actions while audit logs ensure accountability. Teams should adopt zero-trust principles and leverage MCP gateway security features as they mature.
Infrastructure scaling needs to account for the increased request volume from AI agents. Event-driven autoscaling policies should distinguish between legitimate AI workload and potential abuse. Multi-cluster support requires consistent governance across diverse environments from cloud providers to edge devices.
Finally, while MCP enables powerful automation, human judgment remains essential. The team deliberately restricts write permissions and automated remediation in favor of AI-assisted diagnosis with human-approved actions. This balanced approach captures efficiency gains while maintaining the control necessary for mission-critical operations.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
This panel discussion features engineering leaders from Abridge, Replit, and Hebbia discussing their experiences building sophisticated AI agent systems at production scale. Abridge tackles clinical documentation by recording and summarizing doctor-patient conversations for over 250 healthcare systems, addressing challenges around clinical compliance and trust. Replit builds autonomous coding agents that can plan, design, write, test, and debug software with increasingly long-running capabilities. Hebbia creates AI tooling for major financial institutions like KKR and Morgan Stanley, managing extremely spiky workloads with hundreds of thousands of agents processing high-value questions worth hundreds of millions of dollars. All three companies leverage Temporal for durable execution, have moved beyond proof-of-concept to production systems with high stakes, and share common challenges around reliability, cost optimization, model selection, and the evolving balance between agent autonomy and human control.
Two organizations operating in highly regulated industries—Sicoob, a Brazilian cooperative financial institution, and Holland Casino, a government-mandated Dutch gaming operator—share their approaches to deploying generative AI workloads while maintaining strict compliance requirements. Sicoob built a scalable infrastructure using Amazon EKS with GPU instances, leveraging open-source tools like Karpenter, KEDA, vLLM, and Open WebUI to run multiple open-source LLMs (Llama, Mistral, DeepSeek, Granite) for code generation, robotic process automation, investment advisory, and document interaction use cases, achieving cost efficiency through spot instances and auto-scaling. Holland Casino took a different path, using Anthropic's Claude models via Amazon Bedrock and developing lightweight AI agents using the Strands framework, later deploying them through Bedrock Agent Core to provide management stakeholders with self-service access to cost, security, and operational insights. Both organizations emphasized the importance of security, governance, compliance frameworks (including ISO 42001 for AI), and responsible AI practices while demonstrating that regulatory requirements need not inhibit AI adoption when proper architectural patterns and AWS services are employed.
