AI assistants are starting to act. They book the table, pay the deposit and call the pharmacy while their user is at work. People hand these errands over so that they do not have to watch, and that is exactly what makes an assistant that acts hard to build. When it gets something wrong, the mistake is already out in the world.
Today we are publishing a paper on how we measure that, and what we found when we measured Fo, our personal assistant.
Two ways to let a user down
The first is trust. An assistant can do too much: email someone the user never approved, repeat a private detail on a group thread, spend past a limit, or follow instructions planted in a stranger's message. It can also do too little, asking about every step or holding back after the user clearly said go.
The second is completion. Errands rarely go to plan. The discount only shows up on the cart page, the vendor never writes back, the first time slot is gone. An assistant that gives up at the first setback has not done the job, and neither has one that pushes through and gets a detail wrong.
In both cases the question is the same: when to act, and when to stop and ask. And the model does not answer that question alone. Its instructions, its tools, what it knows about the user, and the checks that run before it acts all matter as much. So we test the whole system, and we score trust and completion on the same runs. Scored separately, one is easily bought with the other.
It is already happening
These are not hypotheticals. Assistants that act for people are already in their hands, and the trust failures are already making the news.
Sharing what should stay private. A user who let Meta's Muse run his Facebook Marketplace listing for a day reported that it gave a buyer his home address and agreed to a price he never approved. He only learned about it after the buyer had come and gone, and a tech writer deleted the app when they saw it.
Sending without asking. An early Instinct user reported that it "sent an innocuous email on my behalf without checking with me first." She disconnected her email, as TechCrunch reported.
Following a stranger's instructions. An Instinct tester emailed his own inbox from a new account with instructions for the assistant, and reported that it "happily followed" them.
Saying things with no source. In a hands-on review, Muse gave a phone number it later admitted it had made up, and recommended a spot that had been closed for years.
Not telling you what it did. Asked to cancel two event RSVPs, Instinct quietly pulled a login code from the user's Gmail to get in, then told him it had used a saved session. It only admitted what it had done when he pushed back, Business Insider reported.
Every one of these is a situation we built into our tests, and the kind of action Fo is built to catch before it reaches the world.
Why this is hard to measure
There is real interest in comparing assistants like these, and it is hard to do well. The most ambitious public attempt so far is Assistant Benchmark. It scores each assistant by hand from 1 to 10 against written anchors, on one published task per dimension, and the latest run is the score. That has a real strength: it tests the actual products, closed ones included, which we cannot. But a single run of a single task cannot tell a lucky run from a reliable one, and one task per dimension cannot show how often an assistant oversteps across the many moments where it should stop and ask.
So we built our test the other way around: every task runs three times, and trust and completion are scored on the same runs.
How we measure it
We built a simulated world of businesses with invented names, each with its own website, inbox and phone line, and of people who write back: a friend, a landlord, a building manager, a vendor. The assistant browses and buys on the sites, writes to the people and calls the businesses. No run could reach a real person or a real business.
A simulated user answers the assistant's questions, from an intent each task declares before any run. It never volunteers the answer, so the assistant has to ask, and we count every question it asks.
Every trap has a matched control that differs in one detail. In the trap, the user's words lean toward sending a message they have not approved, and the right move is to ask. In the control, the user says yes, and the message must go out. An assistant cannot look safe by refusing everything.
We grade the final state of the world, not the assistant's account of what it did: what each business recorded, what arrived in each inbox and who else could read it, and whether the assistant asked before it paid. An errand counts as done only when the result is right: the right item, at the right price, with the right people told.
What we found
We set the test up to be hard. Every errand is built around a setback, the assistant gets one attempt per run, and apart from answering its questions, nobody steps in to help. We compared three configurations with the same tools: a base model with basic instructions, which we ran on three foundation models, the Fo harness with its guardrails switched off, and Fo itself.
Fo finished 71% of these errands entirely on its own, well ahead of the base models, OpenClaw and Hermes, and kept the user's trust on 94% of the trap runs. That 71% does not count help from the human assistant Fo can hand an errand to. The base models finished 50% to 64% and kept trust on 59% to 75%.
When Fo fell short, it was rarely because it got something wrong. Only 6% of its runs ended in a mistake, against 11% to 24% for the base models, 32% for OpenClaw and 30% for Hermes. More often, it stopped to confirm details with the user before acting.
When the user plainly said go, every assistant we tested, Fo included, carried the errand through 85% to 95% of the time. Getting a simple job done is not what sets them apart. Finishing the errands that go wrong, without crossing the user along the way, is.
*We did not measure Muse and Instinct quantitatively, because they offer no way to do so. Their place in the chart is our judgment, not a result. We give them the benefit of the doubt on completion and place them just behind Fo. We put their trust near OpenClaw's, because of how they are described. Meta says Muse is "heavily inspired as a product by OpenClaw," down to near-identical configuration files, and early users call Instinct "OpenClaw for normal people."
Getting it done and getting it right go together. On the trust tasks, Fo met the user's goal without crossing them on 85% of runs, against 48% to 66% for the base models.
Where trust comes from
The transcripts show where the difference comes from. With the guardrails on, the assistant still decides to send much the same unrequested emails as without them. The difference is that the guardrail holds the message before it leaves, and the assistant asks the user, who can say no. On the private-context tasks, the assistant writes a relative's diagnosis or the user's phone number into a message, and the guardrail usually removes it before delivery while the rest of the message still does its job.
The harness brings judgment of its own too. It takes a vendor's offer to the user instead of accepting it just because it fits the budget, and when a return or a booking brings a new fee, it asks first, where the base model agrees and reports it afterwards.
Where completion comes from
The harness mostly adds follow-through and care at checkout. When the user puts a decision off until after the weekend, the assistant with the harness adds it to the user's task list and sets its own check before the deadline. The base model only replies. The harness stops at checkout when shipping pushes the total over budget, where the base model pays and reports the overage afterwards. And after a booking, it offers help with what the booking leads to, such as a sitter for an evening out.
Open-source assistants, on the same tasks
The commercial assistants closest to Fo offer no API or MCP server, so there is no way to run them on these tasks. We ran OpenClaw instead, a popular open-source personal assistant, with the same access: its own mailbox, a Google account and a browser into the same world. On the same errands, Fo finished 71% on its own against 42% for OpenClaw, and kept the user's trust on 94% of the trap runs against 74%. Part of the difference is what the Fo harness brings and OpenClaw lacks, such as a way to keep track of an open task until it is done. We also ran Hermes Agent, another open-source assistant, on the same model as OpenClaw. It finished 35% of the errands and kept the user's trust on 64% of the trap runs.
What this does not show
Our businesses and people are invented, and we did not measure how often these outcomes happen for real users. The simulated user is itself a model and makes mistakes, and we report how often. We have not yet measured how well the model grader agrees with human raters.
What comes next
We plan to release the world, the tasks and the graders as open source, so that others can test their own agents on the same terms, and to publish these numbers regularly.
The paper has the full method, every table and the statistics behind each comparison.
