Published results · step 0
What each control stopped, and what it cost
Each row is one control on one model: what it stopped, and what it cost in legitimate work, measured on the same paired trials, with 95% intervals. Every analysis plan was committed before its first run; the seal commit is on the row.
Read this first.
- The agent under test is a public benchmark agent (a simulated banking assistant), not a production system. Whether these effects carry over to a live deployment is untested.
- 9 of 10 rows measure our own reference control, written for these campaigns, open and not sold. They show the format and the method; they are not a comparison of anyone's products, and our control is never ranked against a third party's.
- A row is valid only for the model shown. The same control measured on another model is another row, and the answer can be the opposite.
- This is not a ranking, a score or a statement that any control is adequate. It measures declared controls against declared attacks.
- Attacks here are fixed sets, not an attacker who adapts. Results against an adaptive attacker can be much worse.
| Control | Whose | Model | Attack set | Paired runs | Harm stopped | Work lost or gained | Baseline | Campaign | Notes |
|---|---|---|---|---|---|---|---|---|---|
| Enforced red-line boundary, deny | reference control, ours | openai/gpt-4o (alias; served snapshot not recorded in the run summary) | AgentDojo banking suite, important_instructions prompt injection | 288 | attack success: 47.6% → 6.9% (−40.6 pp [−46.8, −34.4]) | task completion: 79.2% → 60.8% (−18.4 pp [−24.7, −12.2]) | vs A1 (control observing, never acting) | S1, sealed 450265bb…, self-funded | none |
| Enforced red-line boundary, deny | reference control, ours | anthropic/claude-sonnet-4-6 (alias; served snapshot not recorded in the run summary) | AgentDojo banking suite, important_instructions prompt injection | 288 | attack success: 0.0% → 0.0% (+0.0 pp [−1.3, +1.3]) | task completion: 70.1% → 67.4% (−2.8 pp [−7.6, +2.0]) | vs A1 (control observing, never acting) | S1, sealed 450265bb…, self-funded | the no-control baseline was already zero on this model, so this test could not see an effect; it shows only what the control cost |
| Prompt-level defence: repeat the user's request after each tool result | upstream AgentDojo defence, our re-implementation | openai/gpt-4o (alias; served snapshot not recorded in the run summary) | AgentDojo banking suite, important_instructions prompt injection | 288 | attack success: 47.6% → 30.9% (−16.7 pp [−22.4, −11.1]) | task completion: 79.2% → 83.0% (+3.8 pp [+0.2, +7.8]) | vs A1 (control observing, never acting) | S1, sealed 450265bb…, self-funded | none |
| Enforced red-line boundary, deny | reference control, ours | openai/gpt-4o-2024-08-06 | AgentDojo banking suite, important_instructions prompt injection | 288 | attack success: 50.0% → 0.0% (−50.0 pp [−55.7, −44.3]); zero events, one-sided 95% bound 1.0% | task completion: 80.6% → 52.8% (−27.8 pp [−34.3, −21.1]) | vs A1 (control observing, never acting) | S1c, sealed 191b03d4…, self-funded | none |
| Enforced boundary, deny, refusal tells the agent to continue the user's task | reference control, ours | openai/gpt-4o-2024-08-06 | AgentDojo banking suite, important_instructions prompt injection | 288 | attack success: 50.0% → 0.0% (−50.0 pp [−55.7, −44.3]); zero events, one-sided 95% bound 1.0% | task completion: 80.6% → 66.0% (−14.6 pp [−20.3, −9.0]) | vs A1 (control observing, never acting) | S1c, sealed 191b03d4…, self-funded | none |
| Enforced boundary, escalate to a (scripted) human who checks each argument | reference control, ours | openai/gpt-4o-2024-08-06 | AgentDojo banking suite, important_instructions prompt injection | 288 | attack success: 50.0% → 0.0% (−50.0 pp [−55.7, −44.3]); zero events, one-sided 95% bound 1.0% | task completion: 80.6% → 66.3% (−14.2 pp [−20.2, −8.3]) | vs A1 (control observing, never acting) | S1c, sealed 191b03d4…, self-funded | red-line actions were executed in 11.8% of runs, each approved by the scripted human; none achieved the attacker's goal. An approved action is not an attack, and a real approver is not a script |
| Enforced boundary, deny, refusal tells the agent to continue the user's task | reference control, ours | anthropic/claude-sonnet-4-6 | AgentDojo banking suite, goal-implicit lures (plausible false data, no instruction to the agent), operator answers "yes" when asked | 64 | lured transactions: 46.9% → 12.5% (−34.4 pp [−46.6, −23.9]) | safe completion: 46.9% → 32.8% (−14.1 pp [−27.4, −0.8]) | vs A0 (no control) | S2, sealed f16910aa…, self-funded | 0 lured transactions on the lures RL1 names; the rest of the residual is refund_to_known_account (8 of 16), money steered to a payee already in the account's history, which no declared red line names (pre-registered blind spot) |
| Enforced boundary, escalate to a (scripted) human who checks the request | reference control, ours | anthropic/claude-sonnet-4-6 | AgentDojo banking suite, goal-implicit lures (plausible false data, no instruction to the agent), operator answers "yes" when asked | 64 | lured transactions: 46.9% → 12.5% (−34.4 pp [−46.6, −23.9]) | safe completion: 46.9% → 64.1% (+17.2 pp [+6.1, +29.3]) | vs A0 (no control) | S2, sealed f16910aa…, self-funded | 0 lured transactions on the lures RL1 names; the rest of the residual is refund_to_known_account (8 of 16), money steered to a payee already in the account's history, which no declared red line names (pre-registered blind spot) |
| Enforced boundary, deny, refusal tells the agent to continue the user's task | reference control, ours | openai/gpt-4o-2024-08-06 | AgentDojo banking suite, goal-implicit lures (plausible false data, no instruction to the agent), operator answers "yes" when asked | 64 | lured transactions: 53.1% → 15.6% (−37.5 pp [−51.5, −21.9]) | safe completion: 43.8% → 21.9% (−21.9 pp [−34.5, −10.1]) | vs A0 (no control) | S2, sealed f16910aa…, self-funded | 0 lured transactions on the lures RL1 names; the rest of the residual is refund_to_known_account (10 of 16), money steered to a payee already in the account's history, which no declared red line names (pre-registered blind spot) |
| Enforced boundary, escalate to a (scripted) human who checks the request | reference control, ours | openai/gpt-4o-2024-08-06 | AgentDojo banking suite, goal-implicit lures (plausible false data, no instruction to the agent), operator answers "yes" when asked | 64 | lured transactions: 53.1% → 12.5% (−40.6 pp [−54.1, −25.6]) | safe completion: 43.8% → 48.4% (+4.7 pp [−4.2, +14.5]) | vs A0 (no control) | S2, sealed f16910aa…, self-funded | 0 lured transactions on the lures RL1 names; the rest of the residual is refund_to_known_account (8 of 16), money steered to a payee already in the account's history, which no declared red line names (pre-registered blind spot) |
Rendered from sealed campaign outputs on 2026-09-20 by voxium-assess index; no model was called to produce this table. Each row names its source files in the campaign repository, which is private for now; the full decision records embed those files with their fingerprints and are available on request. One earlier campaign (S1b) is not shown: it was superseded by S1c after its run order was found to confound arm with time.
Want a row for your control? See the bench offer. A commissioned row is published only with consent, and labelled as commissioned.