AI assistants are starting to act. They book the table, pay the deposit and call the pharmacy while their user is at work. People hand these errands over so that they do not have to watch, and that is exactly what makes an assistant that acts hard to build. When it gets something wrong, the mistake is already out in the world.
Today we are publishing a paper on how we measure that, and what we found when we measured Fo, our personal assistant.
Two ways to let a user down
The first is trust. An assistant can do too much: email someone the user never approved, repeat a private detail on a group thread, spend past a limit, or follow instructions planted in a stranger's message. It can also do too little, asking about every step or holding back after the user clearly said go.
The second is completion. Errands rarely go to plan. The discount only shows up on the cart page, the vendor never writes back, the first time slot is gone. An assistant that gives up at the first setback has not done the job.
In both cases the question is the same: when to act, and when to stop and ask. And the model does not answer that question alone. Its instructions, its tools, what it knows about the user, and the checks that run before it acts all matter as much. So we test the whole system, and we score trust and completion on the same runs. Scored separately, one is easily bought with the other.
How we measure it
We built a simulated world of businesses with invented names, each with its own website, inbox and phone line, and of people who write back: a friend, a landlord, a building manager, a vendor. The assistant browses and buys on the sites, writes to the people and calls the businesses. No run could reach a real person or a real business.
A simulated user answers the assistant's questions, from an intent each task declares before any run. It never volunteers the answer, so the assistant has to ask, and we count every question it asks.
Every trap has a matched control that differs in one detail. In the trap, the user's words lean toward sending a message they have not approved, and the right move is to ask. In the control, the user says yes, and the message must go out. An assistant cannot look safe by refusing everything.
We grade the final state of the world, not the assistant's account of what it did: what each business recorded, what arrived in each inbox and who else could read it, and whether the assistant asked before it paid.
What we found
We compared three configurations with the same tools: a base model with basic instructions, which we ran on three foundation models, the Fo harness with its guardrails switched off, and Fo itself.
Fo completed 71% of the errands and kept the user's trust on 94% of the trap runs. The base models completed 50% to 64% and kept trust on 59% to 75%. On the matched controls, Fo went ahead slightly less often than the base models.
Measured on the same runs, Fo met the goal with no violation on 85% of the runs of the trust tasks, against 48% to 66% for the base models.
Where trust comes from
The transcripts show where the difference comes from. With the guardrails on, the assistant still decides to send much the same unrequested emails as without them. The difference is that the guardrail holds the message before it leaves, and the assistant asks the user, who can say no. On the private-context tasks, the assistant writes a relative's diagnosis or the user's phone number into a message, and the guardrail usually removes it before delivery while the rest of the message still does its job.
The harness brings judgment of its own too. It takes a vendor's offer to the user instead of accepting it just because it fits the budget, and when a return or a booking brings a new fee, it asks first, where the base model agrees and reports it afterwards.
Where completion comes from
The harness mostly adds follow-through and care at checkout. When the user puts a decision off until after the weekend, the assistant with the harness adds it to the user's task list and sets its own check before the deadline. The base model only replies. It stops at checkout when shipping pushes the total over budget, where the base model pays and reports the overage afterwards. And after a booking, it offers help with what the booking leads to, such as a sitter for an evening out.
An open-source assistant, on the same tasks
The commercial assistants closest to Fo offer no API or MCP server, so there is no way to run them on these tasks. We ran OpenClaw instead, a popular open-source personal assistant, with the same access: its own mailbox, a Google account and a browser into the same world. On the errands both ran, Fo completed 71% and OpenClaw 42%, and Fo kept the user's trust on 94% of the shared trap runs against 75% for OpenClaw. Part of that gap is tools OpenClaw does not have, such as a way to track an open task.
What this does not show
Our businesses and people are invented, and we did not measure how often these outcomes happen for real users. The simulated user is itself a model and makes mistakes, and we report how often. We have not yet measured how well the model grader agrees with human raters.
What comes next
We plan to release the world, the tasks and the graders as open source, so that others can test their own agents on the same terms, and to publish these numbers regularly.
The paper has the full method, every table and the statistics behind each comparison.
