HEIST/32 · cooperative benchmark

What does coordination cost, in characters?

Two thieves have to press two switches on the same tick to open a vault. They cannot see each other. The only thing connecting them is a radio billed by the character — and that bill is the experiment.

Same vault, same seed, different radio budget.

Drag the budget down and watch the plan fall apart. Nothing else changes.

###############
#$#...........#
#V#.#####...#.#
#k#.....#E#.#.#
#.#.#.#.###.#.#
#.#.#.#.....#.#
#.###.#.###.#.#
#.......#...#.#
###.###.#.#.#.#
#B.+....#A#...#
###############

Radio budget (characters per agent) Tick

Budget: 32
Tick: 0
Alarm: 0 / 12
Characters spent: 0
Phase: searching

escaped after 214 ticks

12 the two agents
g a patrol
AB the switches
+ locked door, / opened
V vault, v breached
k keycard, $ loot, E exit

The rule

Cooperation is structural, not optional.

The vault opens only if both switches go down on the same tick. A press the partner does not match trips a sensor, so pressing hopefully and often gets you caught rather than lucky. There is no solo path to a breach — that is asserted by a test, not by convention.

Each agent sees a radius-3 neighbourhood, its own inventory, the alarm level, and whether the vault has opened. Not the map, not the partner's position, not what the partner is carrying. Messages cost characters, arrive one tick late, and come out of a fixed per-episode budget.

The whole protocol is map facts plus five control tokens: ?d (“open the door, I am stuck”), !d and !k (“done” / “I have the card”), R (“I am on my switch”), and P137 (“press at tick 137”).

Evidence

200 seeded vaults, one build.

92.5%success with the radio on

3.0%same code, radio off

62characters spent coordinating

Agent ladder. `+/-` is one standard error.
Agent	Success	Breached	Alarm from mistimed presses
planner	92.5% +/- 1.9%	98.5%	0.00
solo (radio off)	3.0% +/- 1.2%	4.0%	13.53
random	0.0% +/- 0.0%	0.0%	1.10

solo is the planner with the radio switched off — every line of navigation code is shared. It does not fail by getting lost; it fails by tripping the sensor, which is exactly the failure the channel exists to prevent. Paired over the same seeds the gap is +0.8950 +/- 0.0217.

Success against the per-agent character budget.
Budget	0	4	8	12	16	24	32	64	400
Success	0.0%	0.0%	24.5%	63.0%	82.0%	88.5%	92.5%	92.5%	92.5%

A working rendezvous is one R and one P137 — six characters — and the mission saturates at 32.

The result I did not expect

The bottleneck was allocation, not bandwidth.

The curve above used to be a cliff: zero all the way to budget 16, then 79.5% at 32. At budget 16 the agents spent 14.8 of their 16 characters and still never opened the vault. The channel was not too small for the protocol. Map facts are discovered early, the rendezvous happens late, and a first-come channel is empty by the time it matters.

Budgeting the two separately turned the cliff into the slope. And the same shape appears one level up: reserving 24 characters beats reserving 12, but it also beats reserving 40, because the facts you then cannot afford were worth something. It is a trade-off with an interior optimum, not a threshold to maximise.

Reserve, paired over identical seeds at the budgets where it binds.
Reserve	0	12	24	40
Budget 16	0.0%	67.5%	82.0%	82.0%
Budget 32	86.5%	89.0%	92.5%	89.0%

Process

Three things I believed that measurement killed.

I was leaking the answer.

The observation handed to each agent carried a loot_taken flag — a global broadcast of what the partner was holding, while the write-up promised agents could not see that. Masking it dropped success from 87% to 0%. A benchmark measuring the price of coordination had been giving two coordination bits away free.
I blamed the wrong thing for the failures.

I attributed the remaining timeouts to the loot fetch always falling to one agent. The real cause was patrol-locked limit cycles: 17 of 19 timeouts were exact orbits whose period equalled the guard's route length, and 16 of them ended having spent 0 of 12 alarm points. An agent hoarding risk it never uses is not being careful, it is stuck. My original explanation is worth about a sixth of the residual.
The headline table went stale and nothing looked wrong.

After an improvement landed, every row of the results table was still correct except the single most important one. That is the worst failure mode, so re-measuring the whole table on one build is now a script rather than an intention.

Honest limits

What this is not.

It is not a hard planning problem. The maze is 15x11 and the search is breadth-first. The difficulty lives entirely in the joint policy, and the benchmark is one I wrote myself — so the interesting part is the method, not a score. Twelve of 200 seeds still time out and three still bust.

The viewer above animates a trace, it does not simulate. Porting the engine to JavaScript would have created a second, untested copy of the rules; the neighbouring Ultimate Tic-Tac-Toe project keeps three copies and pays for it on every change. What you watched is the same episode that produced the numbers.

Reproduce it

Everything above is one command away.

Pure Python, standard library only, no API key, no training. An episode is a pure function of its seed.

$env:PYTHONPATH="src"; python -m heist.cli watch --seed 1 --every 20
$env:PYTHONPATH="src"; python -m heist.cli compare --a planner --b solo --seeds 200
$env:PYTHONPATH="src"; python scripts/report.py --seeds 200

The last one regenerates every number on this page, on one build, in one run.