One character can reverse an agent leaderboard

Take two hypothetical agents with the same 50% success rate. Give each four attempts at the same task. Under one scoring rule, agent A beats agent B by 93.75% to 50%. Under another, B beats A by 50% to 6.25%.

The scoring rules are pass@4 and pass^4. Replace the @ with a caret and the ranking reverses. Both calculations are correct. They describe different behavior, and neither tells you which answer your product will actually return.

That's the part I want next to an agent benchmark score. Show me how the system chooses what to ship.

Two agents that average out the same

This is an invented probability model, not a measurement of any commercial agent. Assume the success labels are correct. We can get into trouble before arguing about whether the tests are any good.

Agent A has a 50% chance of succeeding on every fixed task. Each repeat is independent. Agent B always succeeds on one half of the task types and always fails on the other half. Both agents face the same equally weighted task set.

Run either agent once on a randomly chosen task and its success probability is 50%. Now keep the task fixed and repeat the run four times, starting each run from the same initial state.

pass@4 asks whether at least one of those four runs succeeds. For A, the only losing outcome is four failures. That happens with probability 0.5 to the fourth power, or 6.25%, leaving a 93.75% chance of finding at least one success. B stays at 50%. More attempts cannot help with its impossible half.

pass^4 asks whether all four runs succeed. A manages that 6.25% of the time. B manages it on the same half of tasks it always solves, so it stays at 50%.

These are exact probabilities under the assumptions, not estimates from four observed runs. The Python example enumerates the possible outcomes using fractions. It makes no model calls and requires no packages.

Calling B the more reliable agent would leave out something fairly consequential. It never solves half the tasks. Calling A the better agent would also need a qualification. Its advantage comes from collecting multiple attempts and recognizing a successful one.

Suppose the product generates four answers but always returns the first. Its success rate remains 50% for both agents. Generating four candidates has bought A a larger collection of answers, including more chances to contain a correct one. The product still needs a way to choose it.

The chooser was in the original papers

This distinction predates the current enthusiasm for agents. In the 2021 HumanEval paper, Chen and colleagues explicitly distinguished an oracle that knows the unit tests from a heuristic that selects an answer without them. In the reported Codex-S comparison, generating 100 samples at temperature 0.8 found a test-passing function for 77.5% of the 164 problems. Selecting one from the same 100-sample pool by highest mean log-probability solved 44.5%. Those are results for that model and code-synthesis setup, not current agent performance.

The authors measured both quantities. The trouble starts when a reader takes the larger number into a product discussion and leaves the selection mechanism out.

AlphaCode made the constraint explicit with n@k. Generate k programs, select n submissions using information available to a competitor, then see whether any selected submission passes the hidden tests. Its system filtered candidates with public examples and clustered them by behavior before selecting submissions. A correct program hidden somewhere in a large pool was useful only if the selection procedure could get it into the much smaller submission budget.

There is a good reason to build this way. If you can generate isolated candidates, safely run a sufficient verifier, and return one that passes, sampling can improve the product. In our toy example, A plus a perfect chooser really does succeed on 93.75% of tasks with four candidates. That improvement is real within the model. It depends on having the chooser.

The verification available during evaluation can be stronger than the information available when making the choice. A hidden answer key can tell an evaluator which candidate was right after the fact. A deployed ranker has to make its choice before that answer is revealed. Measure the ranker on the same candidate pools and the gap becomes visible.

The caret doesn't mean power the average

The 2024 tau-bench paper introduced pass^k for repeated agent tasks. It asks for the probability that all k independent, identically distributed trials of the same task succeed, averaged across tasks. A trial is a complete interaction, potentially containing many tool calls. Four trials are not four steps within one conversation.

For a fixed task with success probability p, the independent-repeat calculations are simple:

Any of k succeeds: 1 - (1 - p)^k
All k succeed:     p^k

Calculate those quantities for each task, then average. Raising the benchmark's average success rate to the fourth power would assign both of our agents a 6.25% all-four score. That is wrong for B by a factor of eight. Averaging first erased the difference between being intermittently successful everywhere and being consistently successful on only half the tasks.

This is why I'd want the distribution behind the average. How many tasks never succeeded in the observed runs? Which tasks succeeded intermittently? Do failures concentrate in a particular task class? Zero observed successes does not prove a task is impossible, but it tells you where another identical attempt has not yet helped.

Real evaluations also have finite observations. With n runs and c successes for one task, the reference implementation uses combinations to estimate all-k success. The corresponding any-k estimate counts subsets containing at least one success:

pass@k estimate = 1 - C(n-c, k) / C(n, k)
pass^k estimate =     C(c, k) / C(n, k)

Here C counts combinations and n must be at least k. These are per-task estimates, averaged afterward. Under independent, identically distributed trials within each task, they estimate the probabilities above without the bias introduced by simply substituting the observed success fraction for p and taking powers. The downloadable example checks the subset calculations too.

Independence is an assumption with consequences. A shared outage can make several trials fail together. An adaptive second attempt can use information the first attempt lacked. A run against a database already changed by an earlier run is a different experiment. In those cases, report the actual joint outcomes and protocol instead of manufacturing a probability from the single-run average.

Evaluate the policy that returns the answer

Anthropic's evaluation guide makes room for both metrics because the relevant question depends on the product. That seems more useful than declaring one punctuation mark the honest one.

For a concrete evaluation, imagine a coding assistant allowed to generate up to four patches in isolated checkouts. Its actual tests select a patch, or it returns the task for human review. Run that whole policy as one trial, including the selection and the handoff. Grade the selected patch with a separate evaluation suite. If an acceptable patch existed in the pool but the selector rejected it, record that. If the selector accepted a wrong patch, record that too.

Now you can distinguish a generation failure from a selection failure. You can also compare the policy with a single-attempt baseline, including generation and verifier cost, latency, and the fraction of tasks handed back to a person. An escalation should remain visible rather than quietly becoming an autonomous success.

If you then want repeatability, rerun that complete policy from the defined initial state. Its internal retries belong inside each trial. Its pass^4 describes four executions of the product policy, not four chances for a user to recover from an already released mistake.

Even that leaves other questions. Rabanser and colleagues' ICML 2026 study evaluated 15 models on two benchmarks using twelve metrics spanning consistency, robustness, predictability, and safety. Repeatability is one part of that picture. An all-four score cannot tell you how severe a failure was or what happens when tools misbehave.

For a benchmark report, I want the attempt budget, task-level outcomes, and success rate of the actual selection policy beside the candidate-pool score. If the system cannot identify its successful attempt, some of the improvement exists only in the evaluator's knowledge. The product has to earn it by choosing what the user receives.