Before you fix anything, you need to know which step is failing, and whether it fails at random or always the same way. Those two are repaired differently, and getting the diagnosis wrong costs days.
Copy these four requests into your assistant, one after the other, on the project concerned. Give it about fifteen minutes.
The diagnosis runs your task about ten times. If it sends email, moves money or publishes anything, it will do so ten times for real.
Before any test, add a dry-run mode: when it is on, no action that touches the outside world actually happens (email, payment, final write, publishing, deletion). It is only recorded in the log, together with what would have been done. Show me how to turn this mode on.
Most people underestimate the number of steps in their task by half. Start by seeing it as it is.
List the real steps of this task, one per line and numbered, as you actually execute them and not as they are described in the documentation. For each one, say what it produces as output, and how we would know it had failed. Do not modify any file, just show me the list.
What you will see. Often twice what you expected. That is already information: every step is one more chance to fail.
Without a trace, no diagnosis is possible. This is the only code change in the whole process.
Add an execution log: on every run, record for each step whether it succeeded or failed, the exact error message on failure, and how long it took. Write this log to a dated file, one line per step. Change nothing else about the program's behaviour.
Run the task ten times, in dry-run mode, on ten different real cases and not on the same case repeated. Then give me a table with, for each step: the step number, the number of failures out of ten, and the most frequent error message.
Then sort every failing step into one of two categories: random failure if it fails sometimes and for different reasons, systematic failure if it fails always, or nearly always in the same way. Justify each call in one sentence, quoting the error messages.
How to read the table. A step that fails twice out of ten with two different errors is a random failure, and a retry repairs it. A step that fails eight times out of ten with the same error every time is a systematic failure, and retrying changes nothing: that is where your days were going.
The limit, stated honestly. Ten runs are enough to spot a step that is systematically broken. They are far too few to measure a failure rate precisely. We are looking for the culprit, not a statistic.
From this table, propose a fix plan, step by step: for each random failure, add an automatic retry and a check that detects the failure; for each systematic failure, propose either splitting the step into two simpler steps, or taking it out of the model and rewriting it in ordinary code if it is predictable.
Show me this plan before changing anything, and tell me for each fix how many failures it is supposed to remove.
If your assistant tells you everything is fine and there is nothing to fix, ask again with different test cases. A diagnosis run on the same document ten times never finds anything: that is exactly the easy case the chain was written for.
This diagnosis finds the steps that break. It says nothing about steps that succeed with a wrong result, an amount misread but plausible, one supplier confused with another. Those only show up when you compare the output against the truth, on cases where you know the right answer. That is a different job, and it is worth doing as soon as a mistake costs money.
World News AI, the newsletter for people building agents.
Twice a week, Tuesday and Friday, in four minutes: what shipped in AI and what you can actually try the same day.
Double opt-in by email. One-click unsubscribe, link in plain sight in every issue.
Geneva, Switzerland. Write to hello@newsworldai.xyz.