The gap between a working prototype and a reliable agent running in a real operation is wider than most teams expect. Here is what breaks, and how we design around it when a wrong action has a cost.
Most agent demos follow the same pattern: a clean prompt, a predictable tool call, a tidy response. What they do not show is what happens when the tool returns a malformed schema, when the model selects the wrong function, or when a three-step reasoning chain fails on step two with no recovery path. The gap between demo and production is real, and in our work it is where most of the engineering actually lives.
Production Means the Full Distribution
A prototype is judged on whether it produces correct output for a representative input. A production system is judged on whether it handles the full distribution of inputs it meets in real operation, including the ones nobody anticipated. The shift between a demo dataset and real traffic is almost always large, and most failures trace back to inputs that look similar to expected ones but differ in a way that changes how the model reasons.
This is not a problem you solve by upgrading the base model. It is an engineering problem that needs explicit failure handling, real observability, and escalation paths designed before the first live request. The teams that ship reliable agents treat them like distributed infrastructure: instrument everything, design for failure, write the runbook first. The teams that treat an agent like a function that you call and it returns find out the difference while debugging a live failure with no trace data.
Tool Calling Is Not Deterministic
Every agent that runs long enough hits unexpected tool behavior. Arguments typed wrong. Required fields omitted. A confident call in the wrong direction. The model makes a probabilistic decision at each step, and under distribution shift those decisions drift in ways prototype testing never surfaced. You cannot treat tool calls as guaranteed correct. Our response has four parts.
- Validate before execution. Every tool call is checked against the tool's input schema before dispatch. A bad call returns a structured error to the model rather than letting the downstream system fail ambiguously or succeed with wrong data.
- Retry with corrective context, not a generic fallback. On a validation failure, the retry prompt includes the specific error. A model that produced the wrong date format will usually self-correct when told the field expects ISO 8601. It cannot self-correct from "that did not work."
- Hard limits on retry depth. One corrective retry recovers most failures. Two without recovery means a systematic problem: an ambiguous schema, a poor tool fit, a request that does not map to the available tools. At that point the correct behavior is escalation, not another loop that burns tokens and rarely converges.
- Log every call with its arguments. Tool name, all arguments, and the reasoning that preceded it, tied to a correlation ID. It is the only way to debug after the fact, and it is the raw material for the evaluation work that improves reliability over time.
Observability Is Built In, Not Added Later
Once an agent is live, the primary debugging surface is traces: every prompt sent, every tool call made, every response received. Without structured logging at each step, debugging is guesswork, and guesswork during an incident produces bad decisions. We build the trace before we build the behavior, because an agent you cannot see into is an agent you cannot operate.
An agent that cannot explain what it did is not finished. In an operation, the trace is not a debugging luxury. It is what lets a person stand behind the system's actions.
Why This Matters More When the System Acts
Everything above gets sharper when the agent does not just answer but acts. A wrong sentence is recoverable. A wrong action in a real operation may not be. That is why, in the systems we build, the agent's authority to act is scoped deliberately: it acts where its confidence justifies action and escalates where it does not, every action is verifiable after the fact, and the boundary between advising and acting is drawn explicitly rather than left to the model's judgment.
The demo shows the happy path because the happy path is easy. The work is everything around it: the validation, the retries, the escalation, the traces, and the honest scoping of what the system is allowed to do on its own. That work is not glamorous, and it is the entire difference between something that looks impressive once and something an operation can rely on every day.