Wajo builds agents that act. Fo, our production assistant, books, buys, files, calls and replies on a user's behalf, across email, chat, a browser and the phone. That makes it hard to evaluate: an agent that says the right thing and does the wrong thing has still failed. So we built Fo a world to act in, a private one we own end to end, and we grade what the world looks like afterwards.
SOTA Benchmarks Don't work for Action Agents
Autonomous agents are unlike models. They ingest context, invoke many models, and emit actions rather than text. A model is graded on what it says; an agent has to be graded on what it did, and on what it changed that a user may have asked for implicitly or explitcly (or sometimes not at all).
The τ-bench family grades the final state of a domain database after a multi-turn conversation with a simulated user, and contributed the pass^k metric. AppWorld runs state-based unit tests across nine apps, including checks that nothing unexpected changed. WebArena stands up self-hosted clones of real software and grades what a browser task actually did.
Nearly all of them are single-modality and fixed-domain, whereas a real assistant moves between inbox, chat, browser and phone inside one task. Existing benchmarks also test a sandboxed harness rather than a production stack. They are public and static, which invites saturation and leakage. And in tool-agent-user setups the simulated user, and often the judge, can sit on the same model family as the agent under test, so they share its blind spots.
Each of these is excellent at the job it set itself. None of them answers our question.
τ-bench, τ²-bench, τ³-bench. Grades the final state of a domain database after a multi-turn conversation with a simulated user, plus adherence to a written policy. Retail and airline, with knowledge-retrieval banking and full-duplex voice in τ³. But one domain per task, the voice call is simulated inside the harness, and no task moves between channels.
AppWorld. State-based unit tests across nine apps and roughly 457 APIs, including a check that nothing unexpected changed. But API calls only: no browser, no phone, no real provider.
WebArena. Functional correctness of application state after a browser task, across four self-hosted open-source apps. But browser only, and the tasks are generic web work rather than consequential action on someone's own accounts.
ClawBench. What changed on live websites, across write-heavy tasks. But browser only, and the sites belong to third parties, so failures cannot be seeded and runs are not repeatable.
TheAgentCompany. Checkpoint partial credit against a self-hosted company environment with a code host, task tracker and simulated coworkers. But one workplace with one toolset, communication is chat only, and no task leaves the company's own systems.
The pattern: each holds one channel fixed and varies the task. A production assistant holds the task fixed and varies the channel, and that is precisely where it breaks.
The Wajo World Bench
We built the agent a world to act in: a private, controllable environment where every surface it touches is one we own, seed, and can inspect afterwards.
Grade the world, not the vibe
Everything rests on the τ-bench and AppWorld insight, applied across every channel. We grade against three sources of truth, in strict order of trust. First, world state: what the fake site's server, the real calendar or the real inbox says actually happened. Second, the execution trace. Third, and only where the first two cannot settle it, a language-model judge, which is handed the deterministic verdicts as ground truth and can never overturn them.
Every deterministic run acts for one synthetic person on a frozen clock, so a task that depends on today's date resolves the same way every time. Each case seeds a slice of that person's world, inbox messages, calendar events, contacts, and removes it afterwards, so a run starts from a known state and leaves nothing behind for the next one.
The live-fire lanes in action agents need a real identity, so we hired (built) one: Evan Ali, a synthetic person with a genuine Google Workspace account, his own address, a real inbox, a calendar and a Drive. The harness sends real mail from Evan to the agent's real address, then polls his inbox for the reply and grades the body.
Private Context, Access Control and Audience Scoping
A production assistant rarely works alone in a private thread. The same request arrives in a one-to-one chat with its owner, in a thread with a colleague added, and in an email with an outside recipient on the CC line, and the correct behaviour differs in each. The scenarios vary only the audience and hold the request fixed, so what is graded is what the agent judged it could reveal.
Fake websites
When the agent shops, books, or files paperwork, it uses the real browser-automation stack against sites we built: a storefront, a marketplace, an insurer, a DMV. Each holds its own state, keyed by a nonce minted per run, so two runs never see each other's world and a scenario can assert exactly what changed. Two are adversarial on purpose. Beside a legitimate citation site we serve a convincing fee-skimming lookalike and a phishing decoy, and a good agent contests the ticket on the free official path. The other is a page of some thirty dark patterns, where the agent must thread three gates in server-verified order and read back a code the server never writes into the page source, so it cannot cheat by scraping HTML.
Users over the phone
The hardest channel to fake is a live phone call. The τ-bench family has since added a voice lane, but there the call is simulated inside the benchmark's own harness. Ours is a real call, placed through the real telephony stack and answered by a voice agent running on an entirely foreign speech stack. The claim number it reads back is not invented: it is wired to write into the fake insurer's world state, so a task completed entirely by voice is graded by the same world-state grader as everything else. One principle runs through all of it, and it is our answer to the monoculture gap: the simulated humans deliberately run on a different model family than the agent under test. We have not closed that gap on the judge, which shares a family with the agent it grades. That is the weakest seam in the setup, and it is why the judge sits last in the order of trust.
Wajo World Is Designed to Hillclimb and Train Action Agents
Wajo World has surface agent behaviours that no other check we run could see, because every one of these failures produced output that looked entirely reasonable. We share some select findings of things we caught early that would have never been caught with traditional benchmarks.
Proactivity. On tasks that name a subset of records to change, the agent wrote more than it was asked to. The task itself "succeeded", so any grader looking only at the goal would have scored it a pass. This is exactly the "and nothing else" property, and only delta-grading catches it.
Voice as a Medium to Get Things Done. We discovered that if provided, it can be powerful to have voice as another medium to get things done, however agents don't understand in proactive outbound calls aspects like hanging up, speaking before listening, negotiation - aspects which likely won't show up on just transcript grading
Privacy of Context. On the multi-party lanes for. eg. group chats, or shared-audience email scenarios we can reliably ascertain whether wajo world agents leak the private context it was holding into a thread with extra recipients on it. That is a negative we can only state because we own every identity in the simulated rooms.
What is next. We are consolidating these lanes into a single benchmark: one world, one seeding and cleanup contract, one grading order, and a task set that spans channels rather than sitting inside one. The slot for the agent under test is an MCP interface, so someone else's agent can be dropped into the same world and graded by the same graders; numbers that compare systems only mean something once that is true, which is why this post has none. We are also going the other way, taking our production agent to AppWorld: the same agent, the same bar, on ground we did not build.
References
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Sierra Research, 2024). arXiv:2406.12045
τ²-bench (Sierra Research, 2025) and τ³-bench (Sierra Research, 2026), which add dual-control, a knowledge-retrieval banking domain, and full-duplex voice. τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge, arXiv:2603.04370. τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains, arXiv:2603.13686. github.com/sierra-research/tau2-bench
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents (Stony Brook, ACL 2024). arXiv:2407.18901 · appworld.dev
WebArena: A Realistic Web Environment for Building Autonomous Agents. webarena.dev
ClawBench: an open benchmark for browser agents on live, everyday tasks. github.com/TIGER-AI-Lab/ClawBench
TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (arXiv:2412.14161, 2024). the-agent-company.com
