Why AI agents fail in production

A demo is a controlled environment

A demo puts boundaries around a problem. It selects inputs, makes tools available, and lets a team observe a sequence of behavior. It is useful for exploring a capability and deciding whether further investigation is worthwhile.

The difficulty begins when the conditions that made that sequence possible never become part of the architecture. In production, inputs may be incomplete, systems respond late, and the state of an operation can change while the agent decides what to do next. Other people and processes are acting on the same environment.

This creates a different requirement: the system has to behave acceptably as conditions vary. It needs to recognize what it knows, what is no longer valid, and when a decision requires more information. It must also be able to stop without losing the work already done or leaving its effects unaccounted for.

Moving into production therefore means making assumptions explicit. Which dependencies must be available? Which information must be current? What limits on time and cost are acceptable? Which parts of the work can continue if a dependency fails?

A system that completes a sequence once has demonstrated a possibility. Operational confidence requires knowing when it can repeat that performance, how it behaves outside those conditions, and who takes responsibility when it cannot finish. The architecture needs to account for what happens beyond the expected path.

Capability is not authority

Capability describes what an agent can do. Authority defines what it is allowed to do in a particular situation. Confusing the two turns access to a tool into a delegation of power that nobody may have intended to grant.

An agent's mandate needs a goal, a scope, and recognizable limits. It must be clear what the agent may read, what it may change, and which decisions remain with a person. That distinction must also hold in the integrations and permissions that actually govern access. A written instruction is no substitute for an access control.

Autonomy is not uniform across tasks. It can be broader where actions are reversible and more constrained where a decision commits resources, affects other people, or is difficult to correct. Its scope should reflect the consequences of the decision and the organization's actual ability to supervise it.

Adding human approval does not automatically resolve that responsibility. Meaningful review requires enough information, time, and authority. The reviewer needs to understand what they are approving and what will happen as a result. Otherwise, the system preserves the appearance of oversight while handing someone a decision they can barely assess.

Authority remains with the organization that delegates it. Designing an agent means expressing that delegation precisely enough to enforce, review, and revoke it. As the agent becomes more capable, that precision becomes more important.

Context needs an owner

An agent works from the representation of its environment that it receives. That representation brings together instructions, documents, tool results, history, and task state. Its quality shapes what the agent can interpret and which decisions it can make.

Having information is not the same as having the right context. A document may explain a policy that is no longer current. A conversation may preserve a decision that was later revoked. A correct result may describe an earlier point in time rather than the operation as it stands now.

Context needs provenance, currency, and a hierarchy of authority. The system must distinguish the rules governing its behavior from the data it is processing. It also needs criteria for resolving contradictions and recognizing when the available information is insufficient to proceed.

This responsibility spans technology and the organization. Someone maintains the sources. Someone decides which information takes precedence. Someone determines which changes must be reflected in the systems and how exceptions are communicated. If these responsibilities are undefined, the agent inherits an ambiguity that reasoning alone cannot resolve.

Time adds another difficulty. During longer tasks, the system has to retain relevant decisions without treating the entire history as equally valid. Memory requires selection and updating. Remembering a statement does not make it a current fact.

Anthropic documented a specific problem while developing its SWE-bench agent: relative file paths caused errors when the working directory changed. Requiring absolute paths in the tool corrected that behavior. The example shows how much execution can depend on an unambiguous representation of the environment.

Designing context is therefore part of operational design. Its purpose is to maintain a sufficiently reliable view of the work to support the next decision, with clear ownership of the information on which that decision depends.

Taking an action does not mean the work is complete

A tool may accept a request before it has finished processing it. An operation may execute only in part. An agent may give a coherent account of its work while the external system is in a different state altogether.

Assessing the outcome requires separating intent, attempted execution, and verified effect. A record of the agent's actions provides evidence, but it does not replace checking whether the required state was actually reached.

That means defining what completion looks like. Completion criteria belong to the process: what must have changed, which conditions must still hold, and what evidence establishes that the work is resolved. If those criteria are ambiguous, the agent may stop too early or continue acting after it should have finished.

The state of an operation must also survive beyond a conversation. Pending work remains pending when a session changes, another person steps in, or a component restarts. Continuity depends on preserving decisions and effects so that the system can recover an intelligible account of the situation.

Observability makes that continuity possible to examine. What information was available? Which action was requested? What did the tool return? Which outcome was confirmed? Collecting more data without relating it to the work can produce plenty of traces and little ability to diagnose a problem.

The evidence needed to establish completion should be proportionate to the consequences of the operation. When that evidence is missing, uncertainty must remain visible. A convincing response should not close a task whose outcome is still unknown.

Exceptions belong in the design

How a system handles exceptions is part of its normal operation. Dependencies stop responding, permissions change, and some operations remain partly complete. The architecture needs to represent those situations and provide a clear way forward.

Different failures call for different recovery strategies. A temporary interruption may justify a retry. A permissions rejection requires a different decision. An uncertain result calls for establishing what happened before repeating the operation. Retrying an action with external effects without making those distinctions can duplicate them. Idempotency and state reconciliation are relevant mechanisms for controlling that risk.

Stripe's API uses idempotency keys to recognize retries of the same request and prevent repeated effects within the service's documented conditions. An integration used by an agent has to respect that contract. The model's ability to decide to retry cannot replace the guarantee the operation itself needs to provide.

Recovery also needs limits: a budget for attempts, time, and resources, as well as conditions for stopping. If the system keeps searching for a way forward after exhausting those limits, its autonomy can prolong the problem.

The design must account for partial effects too. Some actions can be reversed; others need a compensating action or later intervention. It is unsafe to assume that every sequence can be cleanly undone.

When work passes to a person, the handoff must preserve its context: what was attempted, what changed, what remains uncertain, and which decision is still needed. An exception with no owner, priority, or adequate information becomes work abandoned between systems.

The ability to stop, preserve state, and transfer responsibility is part of well-defined autonomy. It lets the process continue under another form of control without forcing someone to reconstruct events from the beginning.

Evaluation starts before deployment and continues afterward

Evaluating an agent means examining the complete system. Model quality influences the outcome, but so do tools, context, orchestration, and the rules that constrain its actions. Improving one component does not guarantee that the system as a whole will improve.

Tests before deployment help define acceptable behavior and check it under known conditions. Production monitoring provides evidence about actual use and situations those tests did not cover. Both need human review and interpretation.

Success criteria must cover the correctness of the outcome and the conditions under which it was achieved. Did the system respect permissions? Did it recognize its limits? Did it leave an operation in a recoverable state? Completing more tasks by acting outside its mandate is not an improvement.

The burden surrounding the system matters too. Time to resolution, execution cost, and the human intervention required help establish whether it is serving its purpose. An automation that shifts effort into constant review may reduce the agent's visible workload while increasing the organization's.

Evaluation must continue because the system changes. Models, instructions, tools, and information sources are updated. Each meaningful change needs evidence that behavior remains acceptable and that previously working capabilities have not been lost.

No test suite eliminates all uncertainty. Its value lies in making clear what has been checked and where the evidence ends. Combined with observation of the live service, it supports informed decisions about what to keep, what to correct, and how much autonomy to grant.

Not every workflow needs an agent

The case for autonomy depends on the nature of the work. Some processes follow paths that can be specified through clear rules. Others require interpreting open-ended situations, selecting tools, or adapting their steps to information that emerges during execution.

An architecture can combine both approaches. It can assign interpretation to the model while keeping operations that need greater predictability under explicit logic. It can also allow dynamic decisions within a bounded part of the process without extending that freedom to the whole workflow.

The design question is how much discretion the system needs to fulfill its purpose. More autonomy introduces more behavior to observe, evaluate, and govern. That complexity should be justified by value the organization can recognize.

Before expanding the agent's role, it is worth establishing whether the obstacle is actually a lack of decision-making capability. An unassigned responsibility, a neglected data source, or a contradictory process still requires an organizational decision. Automating it does not resolve the ambiguity.

A design becomes more robust when each component has a clear role and the system retains understandable limits. Architectural sophistication should serve the work the system needs to sustain.