Zork I benchmark — brain × ZorkGPT
Can the TerranSoul brain + harness make a weak local model play Zork I? All runs are on the
same Jericho zork1.z5 engine as zorkgpt.com (0–350), and all are
real agent runs (the agent's commands, the game's responses) — no engine auto-solver.
| Agent | Model | Score | Note |
|---|---|---|---|
| Opus 4.8 (recall) | frontier | 350 | reproduces the solution it knows; real Jericho run, winning game |
| Opus 4.8 (no recall, cold) | frontier | 50 | from-scratch reasoning — below deepseek's 94 |
| TaughtLocalLLM | gemma4:e4b · 4B | 350 | brain taught the solution (from Opus 4.8's 350 run); orchestrator-fork serves it every turn |
| ZorkGPT ep120 | deepseek-v3.2+27B | 115 | frontier, best of 122 public episodes |
| ZorkGPT typical | deepseek-v3.2+27B | 88–102 | public SOTA-ish |
| TerranSoul brain | gemma4:e4b · 4B | 10–20 | brain + harness, no taught solution |
| TerranSoul brain · self-improve stack (campaign) | gemma4:12b-it-qat · 12B | 10 → 50 | Modal-10 ceiling BROKEN — via across-run accumulation. Self-improvement-stack re-validation, 9 sequential 3-episode × 300-turn runs on a persistent brain (2026-06-18→21): peak single episode 45 — the campaign later peaked at 50 via the log-grounded Fix M/N loop (consistent 30/30/30, then 20/50/35, 2026-06-22) (full underground-descent chain closed AND survived), per-episode floor lifted to 35, best mean 36.7. AGI-pure (zero game tokens, memory-seed.sql unchanged); the cap-breaking prerequisite (move-the-rug → reveal the trap door) discovered by the agent at runtime and replayed by the brain (state-change-replay). Honest caveat (de-confounding): the persistent brain conflated within-run cross-episode learning with across-run accumulation of that one-time lucky discovery. Re-running on a fresh task-naïve brain per run gives the clean cross-episode result 10/10/15 then 10/20/15 (mean 11.7 → 15.0, peak 20 over six episodes): the agent reliably reaches the interior but never re-discovers the full rug→trap-door→lamp→cellar chain from a clean start. So the peak-45/50 required across-run accumulation; the pure per-run cross-episode result (~15) is consistent with the modal-10 / actor-bound thesis — the frozen 12B's discovery is the bottleneck, not the memory layer. This sharpens, not overturns: externalised memory is still a genuine performance axis. Frontier reached the Troll Room. See paper §4.3d. (Supersedes the earlier "self-improve r4" 10→20 single-lucky-episode.) |
| TerranSoul brain · cross-model | qwen2.5:7b · 7B | 0 → 5 | Memory-lift generalises across architectures. qwen-7B (a different model family): unaided 0/0/0, reading the same accumulated brain 5/5/5 — a real lift, but its magnitude is bounded by the reading actor's planning depth (qwen reaches 5 where the 12B reaches 45 from the identical store). Paper §4.3d. 2026-06-20 |
| zorkgpt-default / none | gemma4:e4b · 4B | 0 | baselines, never enter the house |
Claude Opus 4.8 →
Recall-based: reproduces the solution it knows (real run, 350). From-scratch reasoning (no recall) is 50 — honest breakdown inside.
TaughtLocalLLM →
Teach the brain the solution; the weak 4B follows it step-by-step (396/396 moves served 1:1, 0 errors). Demonstrates context → capability.
12B self-improve stack →
Modal-10 ceiling broken — via across-run accumulation. 9 sequential 3-ep × 300-turn runs on a persistent brain (2026-06-18→21): peak 45, later 50 via the Fix M/N loop (full underground-descent chain closed and survived), floor lifted to 35, best mean 36.7. AGI-pure; the cap-breaking prerequisite discovered at runtime and replayed by the brain (state-change-replay). *De-confounded: on a fresh task-naïve brain per run the clean cross-episode result is ~15 (10/10/15 then 10/20/15, peak 20) — the agent reaches the interior but never re-discovers the full chain from a clean start, so peak-45/50 required across-run accumulation of a one-time lucky discovery. The bound that remains is the frozen model's planning, not memory. Paper §4.3d. (Supersedes the earlier r4 10→20.)
Cross-game generalisation
A different Infocom game — Detective, not Zork I — run to test whether the brain-mediated self-improvement above generalises beyond a single game. This score is not on the 0–350 Zork scale used everywhere else on this page, so it's kept in its own table and card rather than mixed into the Zork rows/cards above.
| Agent | Model | Score (Detective, not Zork) | Note |
|---|---|---|---|
| TerranSoul brain · cross-game (harness v2) | gemma4:12b-it-qat · 12B | 20 (lift); climb 20→60 n=1 only | Cross-game self-improvement (harness v2 = 3 generic brain-routing fixes, no game seeds, zorkgpt agent frozen). The memory-lift to 20 is consistent (20/20/20 at n=2), but the deeper death-aversion climb to 60 did NOT reproduce at n=2 — that 20→60 was a single favorable episode (same high-variance regime as Zork's 10→20), planning-bound not memory-bound. 9:05 reproduced its null (0/0/0) — a robust parser/planner-bound result. Paper §4.3c. 2026-06-13 (n=1) / 2026-06-21 (n=2) |