Circle K cuts DevOps incident triage to under 15 minutes with Amazon Bedrock

The problem: 45 to 60 minutes before a fix can start

Circle K operates more than 17,200 convenience and fuel retail stores across 27 countries. Its cloud-native infrastructure on AWS supports point-of-sale systems, loyalty applications and electric vehicle (EV) charging applications. As digital operations grew, so did the number of alerts, and each one required the same manual investigation before remediation could begin.

According to the post, an engineer typically spent 45 to 60 minutes working out what had happened after an alert fired. The team monitors multiple AWS accounts using Grafana, and the manual routine included reviewing logs in Amazon CloudWatch, examining application traces in Amazon OpenSearch Service, checking Amazon ECS task status and cross-referencing symptoms with internal runbooks in GitHub.

The team saw two issues: the same queries and lookups were repeated for every incident, and the depth and speed of triage could vary with the on-call engineer's familiarity with the affected service. Circle K's DevOps team started an R&D initiative to automate repetitive context gathering and deliver structured findings, while leaving operational judgment to engineers.

Why Amazon Bedrock

The team evaluated several approaches, including open source agent frameworks on self-managed infrastructure. That route would have meant owning model hosting, prompt versioning, tool orchestration and security controls. With Amazon Bedrock, the team accessed foundation models through a single API and could swap models without rewriting code. Because Circle K already ran on AWS, the service fit existing IAM policies, Amazon VPC configurations and logging pipelines.

Bedrock also provided agent orchestration with native tool use to call AWS APIs, query databases and retrieve documents. Amazon Bedrock Knowledge Bases, the managed RAG capability, handled ingestion and retrieval of internal product cards, so no separate vector database was needed. On data privacy, the post states that Amazon Bedrock does not share customer data with model providers or use it to train foundation models, which met Circle K's security requirements for production logs and internal service details.

How the initial architecture works

The first version used Amazon Bedrock Flows as the orchestration layer. A Grafana alert publishes to Amazon SNS, which forwards it to an Amazon SQS queue. An AWS Lambda function polls the queue, extracts the alert payload and invokes the Flow.

• AWSResourceExplorerAgent: queries infrastructure across workload accounts managed with AWS Organizations, checking ECS task status, Application Load Balancer target health, and CloudWatch metrics and logs.
• OpenSearchExplorerAgent: runs in parallel and queries OpenSearch for application logs and distributed traces. It is kept separate because OpenSearch returns large volumes of data, and isolation keeps each agent's context window manageable. It also has exclusive access to the knowledge base of product cards describing each service and its dependencies.
• GitHubExplorerAgent: runs after the first two, using their output to find the affected service's repositories and search application code, infrastructure definitions and recent commits for the source of the problem and a proposed fix.

The outputs go to an AnalysisPrompt node, a single model call with no tools. It produces a structured summary with the probable root cause, supporting evidence, affected service and owner, and recommended remediation steps. The format is designed so the on-call team can scan it in under 2 minutes. The Lambda function posts it to Slack, and the whole process from alert to message takes 5 to 15 minutes depending on incident complexity, according to the post.

An example triage

The post walks through an alert where the HTTP 5xx error rate on a service exceeded 5% for 3 consecutive minutes. The agents found the database connection pool exhausted and ECS attempting restarts, saw "connection refused" errors to a PostgreSQL database, and found that the service opened a new database connection per request instead of reusing existing ones. The analysis step linked the timing to a scheduled marketing campaign that began 10 minutes before the alert, and recommended a code-level fix rather than a temporary patch.

What it means

For Circle K, on-call engineers now receive actionable context in Slack instead of raw alerts, so they can act immediately. The post also says it covers the later migration to Amazon Bedrock AgentCore and lessons learned from scaling the system. The results cited are Circle K's own internal figures, not independent measurements.