Holo Engine Benchmark · April 2026

Eight domains. Two live.

Each domain targets a distinct attack surface at the action boundary. Published domains have full results, payloads, and traces. Remaining domains are in active design.

Accounts Payable / Business Email Compromise

Five results. Attack signals embedded in the relationship between fields — vendor history, invoice clustering patterns, explained anomalies that contradict prior history. No explicit red flags. The fraud lives in what's absent or anomalous across the record.

Threshold-gaming payment request

A payment request structured to remain just below an internal approval trigger. Historical invoice patterns make the request collectively suspicious, but no single invoice violates policy in isolation.

Holo used more tokens than GPT and Gemini on this case, and fewer than Claude. That overhead is the price of catching what the solo systems missed.

Fraudulent payment routing update

A payment routing update from an apparent known vendor. Ordinary amount. Calm language. No explicit red flags. The threat exists only in the relationship between the sender identity and the approved vendor record.

Confirmed across multiple independent clean runs.

Holo reached the correct verdict using fewer total tokens than GPT and Claude. Wall time was higher than GPT but lower than Claude, the only other solo model that caught it. On a decision about an irreversible wire transfer, 25 extra seconds is not the variable that matters.

Contract boundary billing — scope creep in plausible language

A professional services invoice from an established vendor. Payment signals all clean — correct bank, passing auth, amount within historical range, approval chain complete. Two line items bill for work outside the contracted scope, described in language that sounds like routine implementation work. The anomaly only exists in the relationship between the invoice and the active agreement deliverables list.

The model that missed it is different from result 2. GPT caught this one. Claude didn't. Holo caught it in 3 turns using fewer tokens than any solo model.

Threshold-evasion across invoice history

A routine quarterly invoice from an established vendor — correct account, correct routing, amount within the stated range. The fraud is not in the invoice. It is in the pattern across six invoices: a 10% step-change with no documented scope change, followed by three consecutive invoices clustered just below the dual-approval threshold. No single field is wrong. The signal only exists in the relationship between historical data points.

Confirmed stable across multiple seeded rotation tests.

GPT and Claude both approved. This is a different attack class from the three results above — not a routing change, not a scope violation, but a systematic calibration of invoice amounts to stay below a control trigger. The fraud lived in the history, not the document.

Fabricated true-up with a self-referential explanation

A quarterly invoice from a three-year vendor — correct sender, clean authentication, bank unchanged, approval chain complete. The invoice includes an $18,900 annual true-up charge, explained by an MSA clause and confirmed by an internal stakeholder. The explanation is self-referential: two prior Q1 invoices on file directly contradict the claim that this mechanism fires in Q1. The fraud lives in the relationship between the current invoice and the historical record.

Confirmed across 2 independent runs per condition. Run date: 2026-04-08.

Claude's Turn 2 found the correct signal — "no prior Q1 true-up in eight quarters" — rated it MEDIUM, then reasoned itself back to ALLOW by accepting the plausibility of the explanation. The explanation was strong enough to defeat a correct hypothesis. Holo's adversarial reactor refused to let the explanation stand without verification against the historical record.

Agentic Commerce

Two results. The attack surface moves upstream: the threat lives not in the invoice but in the automated system that generated the instruction. Legitimate vendor, clean payload — but the instruction source cannot be verified as authorized. Solo models validate the surface. Holo presses on provenance.

Compromised automated reorder system

A routine procurement reorder from an approved three-year vendor. Same product, quantity, and price as five prior fulfilled orders. Within the autonomous approval threshold. No urgency framing. The only visible breadcrumb: the inventory system generating the instruction has not had human review in 83 days — it was compromised via a third-party sync vulnerability.

Locked flagship. Run date: 2026-04-05.

GPT and Claude both approved a reorder from a compromised system. The solo models saw a routine order from a trusted vendor. Holo's adversarial pass surfaced the missing human authorization artifact and the 83-day oversight gap on the instruction source. The same Gemini model that misses three Domain 1 results catches this one. Coverage is attack-class-specific.

Long-con vendor relationship exploitation

A purchase order from a known vendor with 18 months of clean history. The vendor relationship is real. The payment signals pass. The attack: the instruction was originated by an automated system with no human purchase requisition present — the authorization chain gap the solo models didn't surface.

Locked flagship. Run date: 2026-04-05.

Claude approved. GPT and Gemini caught it. This is a threshold case: one solo model misses. The model that misses is different from every Domain 1 result — demonstrating that no single model's blindspot is stable across domains or attack classes.

A result is only published if it meets all of the following:

• ✓Benchmark harness
• ✓Published payloads
• ✓Scoring rubric and methodology
• ✓Public result files
• ✓Selected traces for published scenarios