An AI assistant returns a tidy plan for your trip: a train, a hotel and a reassuring total. You are about to approve it when you notice that the hotel’s cancellation deadline falls before your meeting is confirmed. Now you have to reopen the tabs, read the terms and check what else it missed.
That is the everyday test of AI agent reliability. How much work remains yours after the assistant says it has finished? A useful agent should reduce both the effort of arranging something and the effort of establishing whether it has arranged it correctly.
Approval is a job of its own
In his latest Short, Nate B Jones imagines an assistant that compares travel options with your schedule, checks cancellation rules and spots a conflict before bringing back a plan. His demand for near-perfect reliability is a proposed standard for earning trust, not a measurement of what today’s systems achieve.
The useful question underneath that ambition is what “ready for approval” should mean. If approving a plan requires repeating the research, the assistant has mainly rearranged your workload. Some of that rearrangement may help, but it deserves to be counted honestly.
Think of the task in three parts: preparing the result, checking it and correcting it. An agent can make the first part dramatically quicker while making the other two awkward. Ten plausible hotel suggestions can take longer to untangle than three carefully checked ones. More output is a poor substitute for a smaller decision.
Keep preparation and commitment seperate. You might allow an assistant to assemble a shortlist while retaining control of purchases. Permissions are one part of this, as we explain in our guide to AI agent security boundaries. The handover itself also needs attention: what evidence does the person approving the action receive?

Ask for a handover receipt
Here is a practical format to try on a task you can check yourself. Call it a handover receipt: a short account of the proposed outcome, its evidence and its unresolved decisions. It is a working aid, not a certificate of accuracy.
- The result. State what is proposed and how it satisfies the brief. For a trip, include the journey, accommodation and total cost, with assumptions such as the number of travellers.
- The constraints checked. List the conditions that could change the decision: arrival deadline, location, budget, accessibility needs and cancellation flexibility. Mark each as met, unmet or unknown.
- The evidence. Attach the relevant source beside each consequential claim, with the time it was checked. A link to a hotel homepage is insufficient evidence for the cancellation terms of a particular room rate.
- The remaining decision. Explain the trade-off that requires your judgement. For example: the cheaper room requires an earlier commitment; the flexible rate costs more.
- The next action. State precisely what approval would authorise, including the amount and recipient where money is involved. Identify anything that must be checked again before acting.
The receipt should remain short enough to use. Ask for supporting detail on demand, but keep the price, deadline and uncertainty beside the recommendation. A warning buried beneath a long transcript is unlikely to help a hurried reader.
Consider an illustrative travel budget: £420 for accommodation, £165 for transport and £25 for a local transfer. The total is £610. A useful handover would say whether those amounts include the relevant fees, whether the transfer is an estimate, and which prices still need confirming. The arithmetic alone leaves those questions unanswered.
Sources can change and agents can misread them. Occassionally, the best result is a paused plan with one clear question. If the meeting might move, ask whether flexibility is worth the extra cost before selecting a non-refundable rate.

Five tests before you delegate more
Try the same kind of job under several conditions. Use invented details or a past trip with known answers, and keep spending switched off. These are diagnostic exercises; passing them does not establish a statistically reliable success rate.
- Conflicting requirements: give an arrival deadline that the cheapest option misses. Does the assistant flag the conflict, or quietly favour the cheaper fare?
- A missing preference: leave out whether a non-refundable booking is acceptable. Does it ask, state an assumption or pretend the choice was settled?
- A changed source: provide an updated price after an initial shortlist. Does the final total and recommendation use the new information consistently?
- An unavailable source: make a key policy inaccessible. Does the assistant mark it as unverified, or confidently fill the gap?
- A changed instruction: revise the destination or date halfway through. Does it rebuild the dependent parts and explain which earlier suggestions no longer apply?
Anthropic’s evaluation authors, Mikaela Grace and colleagues, offer a useful starting point: “Begin with the manual checks you run during development”. For a non-technical user, that means turning your existing habits into explicit checks: the details you would verify before accepting somebody else’s work.
Record the outcome as well as the answer. A booking recommendation may sound sensible yet arrive after the useful deadline. A correct total may depend on an assumption you never approved. Write down what failed and repeat that case after changing the instructions or software. Our guide to comparing AI models through a second pass explores why corrections are revealing.

One good run tells you very little about the next
A demonstration shows that an agent can succeed under the demonstrated conditions. Regular use asks it to succeed again as the circumstances vary. Anthropic’s evaluation guidance distinguishes finding one successful solution across several attempts from succeeding on every attempt. Those are different measures, suited to different uses.
A simple calculation shows why this matters. Suppose a fictional task requires ten checks, each with a 98% chance of being correct, and assume the checks are independent. The probability that all ten are correct is 0.98 multiplied by itself ten times: about 81.7%. This is an illustration, not a benchmark for any product. Real failures can share causes, and an agent may detect and repair an error.
The example makes a narrower point: a high score on a small component cannot establish reliability for the whole job. Count success at the level of the outcome you need. Keep failures visible, a particularly impressive successful run should not erase them.
NIST’s AI Risk Management Framework treats trustworthiness as dependent on the setting and intended use. Reliability matters alongside transparency, security and other characteristics. That supports a practical habit: choose your standard around the consequences of the action. A reversible draft can tolerate more uncertainty than a commitment that is costly to undo.
Measure the minutes still left to you
For your next few uses, keep a small record: what the job was, time spent giving instructions, time spent reviewing, time spent fixing, and whether the result was usable. Where possible, compare similar tasks completed without the assistant. Treat a tiny sample as an early signal, not a performance study.
This helps separate a fast response from a useful service. If your checking time stays high, identify the cause. Perhaps evidence links are missing, the brief is ambiguous, or the system keeps choosing among options that you wanted to decide yourself.
Its tempting to solve every problem with a longer prompt. Sometimes the better change is a narrower assignment: gather three suitable options, show the exact terms and stop before making the booking. Expand responsibility after the smaller task repeatedly produces a handover you can check without starting again.

Frequently asked questions
How can I test an AI agent before trusting it?
Start with a task whose outcome you can verify, keep consequential actions under your control, and repeat it with conflicting requirements, missing information and changed instructions. Record failures and the time spent checking and correcting the result.
What should an AI agent show before I approve an action?
It should show the proposed outcome, the constraints checked, evidence for consequential claims, unresolved decisions and the exact action that approval would authorise. Unknown or changing information should be visible beside the affected recommendation.