AI Building Agents with Model Context Protocol

Shopkick’s North Star Goal

The goal of the platform is to turn fragmented operational data into immediate clarity during outages. By combining code changes, infrastructure configurations, and live performance metrics into a single AI-driven workflow, the system gives teams a faster path from detection to resolution. Instead of relying on manual investigation across multiple dashboards, engineers receive a consolidated diagnosis, rollback strategy, and verified supporting data in one place.

‍

Over time, the North Star is to reduce the average Mean Time to Resolution (MTTR) across critical incidents by automating the investigative process and giving leadership a clear understanding of what happened, why it happened, and how it should be fixed.

‍

Evolution of Client's stack with Analyze Agency

The transformation of the client's technology stack led by Analyze Agency represented a paradigm shift from a limited retrieval system to an intelligent orchestration engine. Previously, ShopBack Singapore had utilized a basic Retrieval Augmented Generation application built on top of siloed internal databases. The front-end interface could only answer basic statistical questions based on available data, completely lacking the intelligence required to combine proprietary internal data with live external APIs and unstructured sources. This static approach prevented the generation of real-time diagnostic outputs and required decision making to depend heavily on manual analysis and static reporting.

‍

By implementing a multi agent framework, Analyze Agency transformed the system into a continuously evolving, context-aware decision support engine. Once deployed, it dynamically ingested live system metrics, code commits, cloud configurations, and network routing events. With this comprehensive data ingestion, the platform provided stakeholders with a massive variety of strategic assets. It successfully generated complete incident summary reports, developed detailed rollback strategies, and suggested code fixes perfectly aligned with current system errors. Ultimately, it outputted a finalized executive dashboard that told the internal leadership team exactly why an outage occurred and the optimal steps to resolve it.

‍

Architecture

The revised architecture we built was entirely driven by a LangGraph Supervisor Node acting as the central brain to trigger workflows and manage multiple specialized agents. We utilized the powerful Claude 3.5 Sonnet model to drive the core reasoning, synthesis, and iterative prompting capabilities required by the client. A key architectural decision was to build heavily on standardized Model Context Protocol tools to ensure seamless data pipelines.

‍

We divided the architecture into five core operational areas:

• The Data Origins: The physical reality of software engineers deploying code, global traffic hitting the network, and application servers generating operational metrics.

• Live Enterprise Tools: The recording layer where GitHub tracks code changes, Datadog APM logs CPU spikes, AWS EC2 hosts the active cloud, and Cloudflare Edge monitors web traffic.

• Standardized Connectors: This is the critical Model Context Protocol layer. Instead of custom APIs, we deployed a GitHub MCP Server, Datadog MCP Server, Terraform MCP Server, and Cloudflare MCP Server. These act as universal translators for the AI.

• Multi Agent Orchestration: Hosted on a FastAPI and LangServe backend, a LangGraph Supervisor Node acts as the Chief Incident Agent. It delegates tasks to specialized LangChain ReAct Agents. A Code Specialist Agent accesses the GitHub and Datadog MCPs, while an Infrastructure Specialist Agent queries the Terraform and Cloudflare MCPs.

• The Executive Dashboard: A React Web Application that allows the executive user in Singapore to ask plain English questions. The dashboard displays the AI's final root cause analysis alongside clickable, verified data citations directly from the MCP servers.

Implementation  

Because the client operations were centered in Singapore, all AWS infrastructure resources were hosted in the ap-southeast-1 region to ensure low latency. We leveraged AWS EC2 instances, Lambda, and complex step functions to ensure scalability and secure integration with the client's existing virtual private clouds. The MCP servers were individually Dockerized and orchestrated using Kubernetes.

‍

A primary challenge was managing the rate limits of external third-party feeds and APIs. We implemented asynchronous queuing and exponential backoff strategies to prevent throttling. Additionally, keeping the latency low while the backend consolidated insights from the Datadog MCP and live streaming performance required aggressive query optimization and caching mechanisms.

‍

This best-in-class implementation required a cross functional team of 9 professionals. The team consisted of 2 Data Engineers to build pipelines, 3 AI/ML Engineers focusing on LangGraph orchestration and MCP integrations, 1 AWS Solutions Architect, 1 React Frontend Developer, 1 Quality Assurance Tester, and 1 Technical Project Manager.

‍

Evaluation of the Model  

To ensure the model functioned as a reliable enterprise tool rather than a novelty, we established rigorous evaluation protocols using industry standard frameworks for Large Language Model applications. We utilized the RAGAS framework to quantitatively measure Context Relevance, Groundedness, and Answer Relevance.

‍

To measure Groundedness, we employed an LLM as a Judge pipeline. A secondary evaluation model automatically audited the Chief Agent's finalized incident reports to detect hallucination rates, ensuring every server status or code citation could be traced directly back to the Datadog or GitHub MCP servers. Operational performance was continuously tracked using Mean Time to Resolution metrics to measure the precision of the AI's diagnostics. Because the solution supports iterative prompting and constraint-based reasoning, we heavily tracked the approval rate at the Human in the Loop checkpoint. By monitoring how often the human approver in the React dashboard had to request an AI revision versus granting immediate approval for the diagnostics package, we established a clear baseline for the model's practical utility.

‍

Future Priorities  

The current implementation is just a minor piece in a major puzzle. While the multi agent system is fully capable of working independently, it is fundamentally designed to integrate with larger plans that are going to be implemented in the future. To support this expanded vision and ensure the system can scale reliably, the technical plan must prioritize the following foundational pillars:

‍

• Robust API Management and Scaling: As more enterprise applications leverage the REST API feature, establishing an enterprise grade API Gateway will be mandatory. This will manage heavy traffic, enforce strict authentication protocols, and provide necessary load balancing to prevent the core system from being bottlenecked by requests from other internal tools.

• Modular Agent Expansion: Future iterations will likely require new specialized components, such as a Security Compliance Agent or a Billing Agent. The architecture must maintain strict decoupling, so the LangGraph Supervisor can dynamically register new agents without disrupting foundational data fetchers via MCP servers.

• Advanced LLMOps Pipelines: Implementing continuous integration and deployment for prompt templates will be crucial. Establishing an LLMOps framework will allow the engineering team to version control the AI's reasoning logic, ensuring that updates to the underlying Claude model do not degrade the contextual accuracy of the system.

• Zero Trust Data Governance: Opening the orchestration engine to other internal tools via the REST API necessitates rigorous, granular access controls. We must ensure that downstream applications only receive the data they are authorized to view, safeguard sensitive metrics handled by components like the internal agents and ensure strict compliance with enterprise security policies.

‍

Why Choose Us?

At Analyze Agency, we focus on solving operational problems that slow companies down. In the case of Shopkick Singapore, the challenge wasn’t a lack of data  it was the inability to connect everything quickly enough during incidents. Engineering teams had to move between dashboards, logs, and internal tools just to understand what had gone wrong.

‍

Our role was to bring those pieces together into a system that could actually assist in decision making. Instead of relying on manual investigation, the platform now pulls information from code repositories, infrastructure logs, and network traffic in real time. The system analyzes those signals and produces a clear explanation of what happened, along with suggested steps to fix it.

Our team has worked across both startups and enterprise environments, so we understand how costly outages and slow diagnostics can be. That experience shapes how we design systems: practical, reliable, and built to operate under real production workloads.

‍

Most importantly, we work as part of your team. Our goal isn’t to hand over software and disappear,  it’s to build tools that your engineers and leadership can rely on every day.

‍

Our Success Framework

At Analyze Agency, we believe technology only matters if it produces real operational improvements. Our framework is designed to align directly with a client’s North Star, whether that means reducing incident resolution time, improving system reliability, or giving engineering teams clearer visibility into their infrastructure. In the case of Shopkick Singapore, the focus was on eliminating the slow, manual process of diagnosing system failures. Instead of forcing engineers to move between dashboards and logs, we built a unified architecture that connects critical operational tools such as GitHub, Datadog, AWS infrastructure, and Cloudflare traffic data through standardized Model Context Protocol servers. This structure allows AI agents to analyze activity across the entire system in real time and assemble a complete picture of what is happening during an incident. The platform can correlate code deployments, infrastructure changes, and performance signals to generate clear incident summaries and suggest practical recovery steps. The objective is not to replace engineers but to give them immediate context so they can respond faster and make better decisions under pressure. By structuring the system around clarity, speed, and long-term scalability, the framework helps organizations move away from reactive troubleshooting and toward a more proactive approach to operating and maintaining complex production systems.

‍

Get In Touch

Building reliable systems requires more than adding another tool to the stack. It requires connecting the tools you already use and turning the data they produce into something engineers and leadership can act on.

‍

At Analyze Agency, we specialize in building AI-driven infrastructure and operational systems that help companies understand their platforms in real time. Whether the goal is faster incident response, better system visibility, or a scalable AI operations layer, we work closely with teams to design solutions that fit their environment.

‍

If you’re interested in exploring what this could look like for your organization, we’d be happy to talk.

‍

Contact us at Discovery@analyze.agency or visit Analyze.Agency to start the conversation.