Why AI Agents Fail in Production (And How to Build Ones That Don't)
The demo worked. Then real users arrived and it fell apart. Six failure modes that kill production agents — and what actually prevents each one.
Ritik MakhijaTL;DR
AI agents rarely fail because the model is bad. They fail because of silent errors, unbounded input, missing evaluation, no tracing, brittle integrations, and full autonomy granted on day one. Every one of these is an engineering problem with a known fix — and none of them show up in a demo, which is exactly why demos are a poor way to judge an agent.
Key takeaways
- Agents fail quietly: they keep running and return wrong answers instead of erroring.
- A demo tests the happy path. Production is almost entirely edge cases.
- Without an evaluation set you cannot know whether a change made the agent better or worse.
- Grant autonomy incrementally, gated on evaluation data — never on day one.
- Most 'the model isn't good enough' diagnoses are really retrieval or data problems.
There's a specific moment in agent projects that I've now watched enough times to predict. The demo goes well. Everyone's impressed. It ships. And within a week it's quietly producing nonsense, nobody noticed for four days, and the conclusion in the retro is "the model wasn't good enough."
The model was almost never the problem. Here are the six things that actually were.
1. Silent failure
Conventional software fails loudly — an exception, a stack trace, an alert. Agents fail quietly. An upstream API renames a field, retrieval returns nothing useful, and the agent doesn't crash. It answers anyway, from whatever it has, with total confidence. Everything looks fine on the dashboard because nothing threw.
The fix: treat empty or low-confidence retrieval as an error condition, not a normal state. If the agent can't ground an answer, it should say so or escalate — never improvise. Assert on the shape of what comes back from every tool call, and alert when the assertion fails.
2. Unbounded input
Your demo used five tickets you picked. Production sends you a ticket in Portuguese, a ticket that's just a screenshot, a ticket that's three unrelated questions, and a ticket that's someone's out-of-office reply. The distribution of real input is much stranger than anything you'd think to test with.
The fix: build the test set from real historical data, not invented examples. Pull two hundred actual tickets, including the weird ones. The weird ones are the entire point — they're what production is mostly made of.
3. No evaluation set
Ask a team how they know their agent works and you'll usually hear "we tried it and it seemed good." That's not a measurement, and it means you cannot tell whether a prompt change improved things or quietly broke a case that used to work.
The fix: a set of real cases with known-good answers, scored automatically, run on every change. It doesn't need to be sophisticated — a hundred cases and a pass rate beats intuition comprehensively. Without this, every change is a guess and regressions are invisible.
4. No tracing
An agent gives a wrong answer. Why? Was retrieval bad? Did it pick the wrong tool? Did the tool return garbage? Did it reason badly over good data? Without traces you're guessing, and the usual response to guessing is to fiddle with the prompt until the symptom disappears — which fixes nothing.
The fix: log every step — the retrieval hits and their scores, each tool call and its raw response, the model's reasoning, the final output. When something's wrong you want to read the trace and point at the step, not theorise.
5. Brittle integrations
Agents live or die on the systems around them, and those systems are not stable. Rate limits get hit. Schemas change without notice. Auth tokens expire at 2am on a Sunday. A demo runs once, so none of this surfaces; production runs continuously, so all of it does.
The fix: the unglamorous stuff. Retries with backoff, timeouts on every call, schema validation on responses, circuit breakers, and alerting when a tool starts failing. This is ordinary distributed-systems engineering — and it's the part that separates an agent that survives from one that doesn't.
6. Autonomy granted too early
The most expensive failures happen when an agent takes consequential actions unsupervised before anyone knows its real error rate. Refunds issued wrongly. Emails sent to the wrong segment. Records overwritten. The agent was probably 90% accurate, which sounds fine until you notice what the other 10% did.
The fix: autonomy is a destination, not a launch setting. Start with human approval on anything consequential. Watch the evaluation data. Loosen the gate one category at a time, only where the numbers earn it. This is slower and it is the entire difference between an agent people trust and an agent people switch off.
The pattern underneath
Every one of these is an engineering problem, not a model problem. And every one is invisible in a demo — which is why demos are a terrible way to evaluate agents, and why "can you show me one running in production?" is the single best question you can ask a prospective builder.
The difference between a demo and a system that lasts isn't the model. It's retrieval over real data, guardrails, evaluation and monitoring — the parts nobody demos.
If you've got an agent that worked in testing and is misbehaving in production, send me the symptoms. Hardening other people's agents is a large share of what I do, and it's usually one of the six above. More on how I build them on the AI agents development services page.
Frequently asked questions
Six failure modes account for most of it: silent failure (the agent answers wrongly instead of erroring), unbounded real-world input, no evaluation set, no tracing, brittle integrations with upstream systems, and full autonomy granted before the error rate is known. All six are engineering problems rather than model problems, and none of them appear in a demo.
Because a demo tests the happy path and production is almost entirely edge cases. Test sets built from invented examples miss the real distribution of input — tickets in other languages, screenshots, multi-part questions, auto-replies. Build your evaluation set from real historical data, including the weird cases, because those are what production is mostly made of.
You need an evaluation set: real cases with known-good answers, scored automatically and run on every change. 'We tried it and it seemed good' is not a measurement and cannot detect regressions. A hundred real cases and a pass rate beats intuition comprehensively.
Not at launch. Start with human approval gates on any consequential action, watch the evaluation data, and loosen gates one category at a time only where the numbers justify it. An agent that is 90% accurate sounds acceptable until you look closely at what the other 10% did unsupervised.
Rarely. Most 'the model isn't good enough' diagnoses turn out to be retrieval or data problems — the agent was never given the right context to answer with. Check retrieval quality and data cleanliness before switching models, because a better model reasoning over the wrong documents still gives you the wrong answer.
About the author
Ritik Makhija
Founder & Product Lead · AI Kaptan
I build AI agents and automation that run in production — and I've open-sourced 5,000+ workflows so you can read the work rather than take my word for it. I run outreach infrastructure sending 6,000 emails a day on this stack, and I've mentored 700+ builders 1:1.
Got a process you're trying to automate?
Tell me what it is and I'll say straight whether it needs an agent, a plain workflow, or nothing at all. Free 30 minutes, no pitch.