The DevOps AI Agent That Can Fix Production: But Who Watches the Agent?
Autonomous remediation can reduce MTTR. The real test is whether an agent can act without creating a bigger incident than the one it was asked to fix.
Picture a common production alert: a Kubernetes node is under disk pressure, pods are being evicted, and application latency is climbing. Traditionally, the path is familiar:
Alert → Engineer → Investigation → Root Cause → Remediation → Verification
Now imagine a different operating model.
The alert reaches an AI-powered DevOps agent. The agent checks node conditions, container runtime statistics, application logs, recent deployments, disk utilization, Kubernetes events, and historical incidents. It correlates the signals, identifies the likely cause, selects a remediation runbook, validates that the action is permitted, executes the remediation, and watches the system recover.
A few minutes later, the application is healthy. No engineer was paged. No human typed the remediation command.
This is the promise of Agentic DevOps.But it immediately raises a much harder question:
If an AI agent can change production, who watches the agent?
That question is becoming more important as enterprises move AI beyond copilots and recommendations into systems that can actually take operational actions.
At Crest Data, our work building DevOps solutions across cloud operations, observability, automation, AIOps, and enterprise AI points toward a simple principle:
Autonomous operations should not mean uncontrolled operations.
The objective is not to build an AI agent that can execute the most commands.
The objective is to build an AI agent that can make the right operational decision, within defined boundaries, with evidence, and with a verifiable outcome.
Automation is familiar. Independent decisions are not.
DevOps didn’t begin with AI agents. Teams have spent years moving manual work into scripts, Infrastructure as Code, CI/CD pipelines and GitOps workflows, often through DevOps automation services built around those practices. Observability made production behaviour easier to examine, and AIOps helped make sense of large volumes of alerts. None of that is going away. Agentic DevOps sits on top of those practices and introduces something different: software that can choose what to investigate next, which tool to call and, potentially, which change to make.
A script usually follows a path somebody designed in advance. An agent has more freedom. Faced with a failing service, it might inspect an error spike, notice a deployment from ten minutes earlier, check Kubernetes events, abandon its first theory and try another line of investigation. That flexibility is useful because production failures rarely arrive with a neat label attached. It is also where the risk begins. Missing logs, stale runbooks and misleading correlations are part of normal operations; an agent is no less vulnerable to them than a person.
The question, then, isn’t whether the agent is clever enough to use kubectl or Terraform. It’s whether the surrounding system is disciplined enough to keep an imperfect decision from turning into an uncontrolled production change.
The dangerous gap between “can execute” and “should execute”
Consider a production database.
An AI agent detects high disk utilization. It searches its knowledge base and finds several previous incidents where old log files were consuming disk space. It decides to clean the logs. The command succeeds. Disk utilization drops.
The agent reports:
Incident resolved.
But suppose the disk was actually filling because of an active database process. The agent removed files that were still operationally important. The immediate disk alert disappeared. The underlying problem did not.
Or consider a Kubernetes cluster.
A pod is repeatedly crashing. The agent observes the crash loop and decides to restart the workload. The pod starts. The health check turns green. The agent reports success. But the application is still failing because the actual root cause is a bad configuration deployed several minutes earlier.
The agent fixed the symptom, not the system.
This is one of the most important distinctions in Agentic DevOps:
A successful action is not necessarily a successful remediation.
Why a green check mark can be misleading
Take a disk-space incident on a database host. An agent searches previous tickets and finds that old logs were responsible for several similar alerts. It removes what it believes are unnecessary files. The command returns successfully and disk usage drops. From the agent’s point of view, the job may look finished. Yet the current problem could be a database process generating data at an unusual rate, and the cleanup may have removed evidence engineers need for the real investigation.
Restarting a Kubernetes workload creates a similar trap. A pod leaves CrashLoopBackOff, starts successfully and passes its first readiness check. Five minutes later, customers are still seeing errors because the latest release contains a bad configuration. The restart worked; the service did not recover in any meaningful sense.
I would be cautious of any remediation system that treats command completion as the final answer. A proper recovery check looks at the application as well as the infrastructure:
- Error rates and restart counts
- Health of dependent services
- Traffic behaviour
- Customer-facing SLOs
It also needs a sensible observation window. A dashboard turning green for thirty seconds is not the same as a stable recovery.
The agent should not approve its own changes
One architectural choice matters more than it first appears: keep the agent’s reasoning separate from the authority to execute.
- Agent: investigates and proposes an action.
- Policy service: decides whether that action is allowed in this environment, against this resource, at this time. This is a separate service, not the language model itself.
- Execution tool: narrowly scoped, and receives only approved requests.
- Verification step: independently checks what changed and whether it helped.
That boundary makes it possible to enforce identity, action allowlists, environment restrictions, approval rules and limits on the number of resources affected. It also leaves behind a record that another engineer can inspect. Without it, an agent with powerful tools can quietly become an alternative route around the controls a team spent years building.
Signals
Metrics, logs, events
Reasoning
Context + hypothesis
Policy gate
Scope, risk, approval
Execution
Approved runbook
Verification
SLOs + dependencies
Give it enough access to help, but not enough to roam
Imagine giving a pod-troubleshooting agent cluster-admin because it is convenient. That permission may solve the access problem during a demo, but it creates a very different problem in production. An agent checking logs should not be able to alter unrelated workloads. Likewise, a tool that evaluates Terraform drift should not automatically inherit the right to apply every proposed change. Scope access by action and resource, use short-lived credentials where possible, and prefer tested execution paths over arbitrary shell commands. This is the same least-privilege discipline that DevOps security services apply to human and pipeline identities.
Permissions are only half the story. An authorized change can still affect far too much. If an agent spots the same configuration issue on thousands of servers, its first move should not be a fleet-wide update. Start with a small canary group, check whether the expected improvement appears, and expand gradually. If the metrics move in the wrong direction, stop. This is routine release-engineering discipline; it deserves a place in autonomous remediation too.
Some actions should still wait for a person.
There is no useful reason to make every task fully autonomous on day one. A practical starting split looks like this:
- Good first candidates: reading logs, comparing configurations, gathering diagnostics.
- Possible within a defined runbook: service restarts and deployment rollbacks, with known dependencies and recovery checks.
- Higher bar, often explicit approval: changes to IAM permissions, network security policies, production data or broad infrastructure resources.
Of course, those labels are not absolute. Restarting a payment service at the wrong time can be more disruptive than changing a harmless setting. Risk comes from context: which service is involved, who depends on it, whether rollback is possible and how many customers might be affected. An effective approval model reflects those details instead of treating every restart as safe and every configuration edit as dangerous.
Low risk
- Read logs • collect diagnostics
- Compare configurations
Conditional
- Restart a service • scale a workload
- Only within tested boundaries
High risk
- Modify IAM / network controls
- Delete infrastructure
- Alter production data
The agent becomes another system you need to monitor
We have mature habits around application and infrastructure monitoring. The agent operating those systems needs comparable visibility, but not just another dashboard of CPU and memory. Teams should be able to reconstruct its decisions: which alerts it saw, which tools it used, what data it relied on, what approvals were applied and why it stopped or kept going.
It is worth measuring decision latency, but speed on its own can be a poor target. An agent that reaches a weak conclusion in three seconds is not necessarily better than one that spends thirty seconds checking a deployment history. Track these alongside decision speed:
- Failed remediations and repeat incidents
- Policy blocks and unnecessary retries
- Human overrides
- Actual time to recovery
Those measures tell you whether the agent is improving operations, rather than merely generating more activity.
Agent activity
- Tool calls and retries
- Decision latency
- Escalations and overrides
Decision quality
- Grounded evidence
- Incorrect remediations
- Policy violations / rollbacks
Service outcomes
- Time to recovery
- Repeat incidents
- SLO and error-rate recovery
What if the first fix is wrong?
It will be, eventually. No policy engine can guarantee that the proposed root cause is correct; it can only decide whether a particular action is permitted. That is why the control loop needs a failure path, not just a happy path. If the expected recovery does not happen, the agent should avoid repeating the same intervention indefinitely. It may collect more evidence, use a safe predefined rollback or hand the incident to an engineer with the investigation already documented.
A few unglamorous controls do real work here:
- Limits on retries and concurrent actions
- Limits on run duration and affected resources
- A reliable kill switch
The aim is to make a bad decision visible while its consequences are still manageable. Some changes can be reversed quickly; others cannot. The latter deserve tighter permissions and stronger approval requirements from the start.
Good memory needs good maintenance
There is another less visible part of this design: operational memory. An agent needs access to approved runbooks, recent changes, dependency maps, incident history and the outcomes of previous remediations. Otherwise, each alert becomes a fresh investigation with little understanding of the environment. That depends on reliable DevOps integrations with CI/CD, ticketing, observability and configuration management systems. Retrieval-augmented generation can help surface useful context, but connecting an agent to a large knowledge base does not automatically make that knowledge trustworthy.
Runbooks get outdated. Services change owners. A workaround that was safe last quarter may no longer be acceptable after an architecture change. Someone must own the material the agent uses, review it and retire instructions that no longer apply. Keeping that context current may sound less exciting than building the agent, yet it is often what determines whether its recommendations are dependable. In managed DevOps services engagements, this ownership should be written into the operating model, not left implicit.
Start small, then earn more autonomy
For a team adopting agentic operations, I would sequence autonomy in four stages:
- Read-only investigations and incident summaries.
- Recommended actions with evidence, while engineers remain responsible for execution.
- Approval-gated changes, once the patterns are well understood.
- Automatic execution for a few low-risk, reversible runbooks with clearly defined boundaries.
This is also where DevOps consulting services add value: defining which runbooks qualify, which actions need approval, and what evidence counts as recovery.
Expansion should depend on what happens in practice. Did the agent reduce repetitive pages? Did the service stay healthy after its interventions? Were risky changes escalated? Can an engineer trace the decision afterward? A measured record of safe behaviour is a stronger reason to grant more autonomy than an impressive demo or a high number of tool calls.
Assist
Summarize incidents
Recommend
Suggest tested remediations
Approve
Human-reviewed actions
Automate
Low-risk, scoped execution
Expand
Bounded autonomy with verification
The future is not autonomous DevOps. It is governed autonomous DevOps.
AI agents are moving from experimentation into real operational environments. They can investigate incidents. They can analyze telemetry. They can interpret logs. They can reason over the infrastructure state. They can generate Terraform changes. They can interact with Kubernetes. They can trigger CI/CD workflows. They can execute remediation.
And eventually, they will manage increasingly complex operational decisions at machine speed. That future is powerful. But speed without control is not resilience. Automation without verification is not reliable. AI without observability is not operational intelligence. And autonomy without governance is simply risky at scale.
At Crest Data, we see the opportunity at the intersection of DevOps, CloudOps, SRE, observability, AIOps, security, and agentic AI: not just to automate individual operational tasks, but to build intelligent operating models where machines handle the repetitive work while engineers retain control over the decisions that matter. Our enterprise DevOps services span CI/CD, IaC, GitOps, Kubernetes, SRE, observability, AIOps, agentic automation, and production AI, providing the engineering foundation for this convergence.
The question enterprises should be asking is therefore not
“Can an AI agent fix production?”
It can.
The better question is
“Can we give an AI agent enough authority to fix production, while retaining enough control to trust what it does?”
That is the real engineering challenge of Agentic DevOps.
And perhaps the most important component of the entire architecture is the one sitting between the agent and production:
The system that watches the agent.
Planning to bring AI agents into production operations? Crest Data’s DevOps services help enterprises design the policy controls, guardrails and observability that let automation act safely. Explore Crest Data DevOps Services
Written by Binal Kagathara | Thought Leader: Jay Patel
Frequently Asked Questions
-
DevOps services help organisations build, release and operate software faster and more reliably. They typically cover CI/CD pipelines, Infrastructure as Code, Kubernetes operations, observability, SRE and security automation. With agentic AI, they increasingly include the guardrails that govern what automated systems may change in production.
-
Traditional DevOps automation follows paths engineers define in advance, such as a pipeline stage or a scripted runbook. Agentic DevOps adds software that decides what to investigate and which action to take. That flexibility helps with messy incidents, but it needs policy checks, scoped permissions and independent verification.
-
DevOps consulting services focus on strategy and design: assessing maturity, defining the operating model and setting up tooling and guardrails. Managed DevOps services take on the ongoing operation of pipelines, infrastructure and incident response. Many enterprises use consulting to design an agentic operations model and a managed service to run it.
-
Here is how managed DevOps services work in that setup: the provider operates the environment under agreed service levels, while agents handle defined tasks such as diagnostics, incident summaries and approved runbooks. A policy service controls what agents may execute, engineers approve higher-risk changes, and every action is logged.
-
DevOps services for enterprises should cover least-privilege access for agents, an action allowlist enforced outside the model, approval rules for high-risk changes, canary rollouts, SLO-based recovery checks, retry limits, a kill switch and full decision logging. Current runbooks matter as much as the agent itself.
-
For a narrow set of low-risk, reversible actions with clear runbooks, yes. For changes to IAM, network policy, production data or broad infrastructure, most teams should keep a human approval step and expand autonomy only after a measured record of safe outcomes.
