What 5,535 DolphinBench Runs Taught Us About Local AI Memory

1. Why DolphinBench Breaks Toy Haystacks & Our 5-Gate Clean-Room

Toy needle-in-a-haystack benchmarks hide a self-contained fact inside filler text and query the exact same nouns. Real workspace agents fail on stateful action execution under evolving rules. Released by Mem0 (mem0ai/dolphinbench), DolphinBench tests 600 live mock_mcp workspace tasks across 13,539 chronological sessions (1,500,265 o200k_base tokens) and three personas—Morgan (200 Founder/Ops tasks), Alex (200 Staff SWE tasks with 18 KB incident logs and deprecated flags), and Riley (200 Growth PM tasks)—plus 15 pure-deterministic counterfactuals. Passing requires retrieving standing policies that share zero nouns with today's terse prompt, resolving [Date: YYYY-MM-DD] temporal supersession, chaining list_* state lookups before mutating records, and withholding prohibited side effects.

Every run across our 9 × 615 = 5,535-trace local ablation suite and our 14 × 600 = 8,400-trace official openai/gpt-5.6-sol scoreboard is enforced by verify_zero_leakage.py across five gates:

1. Zero-LLM 0o444 SQLite Immutability: Ingests all 13,539 sessions in 153.4s with 0 LLM calls ($0.00), truncates WAL, applies chmod 0o444 + file:...?mode=ro URIs, and verifies pre/post SHA-256 digests across every 600-task run.
2. Static AST Clean-Room Scan: Verifies zero imports from grading rubrics, zero persona proper nouns, and zero test-ID branching.
3. Official 5-Method BenchmarkHarness Replay: Passes upstream python -m harness.export_submission --release upstream --check offline with 0 errors across all 14 archives.
4. Mechanical Check Suite (599 Tasks / 1,310 Assertions): Audits deterministic tool arguments and mock_mcp state mutations without an LLM judge.
5. Counterfactual Causality Gate (15 Tasks, memory_enabled=True vs. False): Confirms causal lift from retrieved memory (+53.3 pp on both 30B and 120B).

2. Axis A: Zero-LLM Hybrid Memory vs. Raw Cosine (+12.50 pp SWE)

Cloud memory layers run paid LLM extraction over every message during ingestion, whereas naive vector RAG embeds whole turns and fetches top-k cosine neighbors. Holding Stock gpt-oss:120b, 1024d snowflake-arctic-embed2, the 4,200-token budget, and judges fixed, swapping single-query cosine (VanillaSnowflakeHarness) for SovereignMemoryStore lifts overall pass rate from 48.50% (291/600) → 55.83% (335/600, +7.33 pp).

• Morgan Ties (65.0% Vanilla vs. 64.0% Sovereign): Short, self-contained executive notes (flight seating, wire caps) already rank cleanly under dense cosine similarity.
• Alex (32.0% → 44.5%, +12.50 pp) & Riley (48.5% → 59.0%, +10.50 pp) Jump: SWE and PM logs mix 14 KB pasted incident packets with terse channel rules. SovereignMemoryStore fixes three failure modes deterministically: (1) 256/32 Raw-RAG windows (arXiv:2610.30755) prevent long turns from diluting pooled vectors or starving token budgets; (2) Porter-FTS5 BM25 + 4-Way RRF (k = 60) anchors exact symbols like dedup-v2 and #release-ops; and (3) directive/transition regex priors + chronological turn_index packing keep later-dated policy overrides ordered last, backed by active search_persona_memory calls.

# sovereign_memory.py — Read-only 0o444 URI guard + 4-Way Hybrid RRF + Raw-RAG window projection
def retrieve_persona_briefing(path: Path, cache: FrozenCache, query: str, q_vec: list[float], rrf_k: int = 60):
    if (path.stat().st_mode & 0o222) != 0:
        raise PermissionError(f"Refusing benchmark retrieval on writable SQLite store: {path}")

    fts_expr = build_fts5_or_query(query)  # Quoted Porter-stemmed token OR-clause
    conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True, timeout=10.0)
    try:
        top_turn_bm25 = [int(r[0]) for r in conn.execute(
            "SELECT rowid - 1 FROM turns_fts WHERE turns_fts MATCH ? ORDER BY bm25(turns_fts) ASC LIMIT 128",
            (fts_expr,),
        ).fetchall()]
        top_win_bm25 = [int(r[0]) for r in conn.execute(
            "SELECT rowid - 1 FROM windows_fts WHERE windows_fts MATCH ? ORDER BY bm25(windows_fts) ASC LIMIT 128",
            (fts_expr,),
        ).fetchall()]
    finally:
        conn.close()

    # 4-Way RRF: Turn-Dense (1.0) + Turn-BM25 (1.15) + Win-Dense (0.85) + Win-BM25 (0.95)
    turn_rrf: dict[int, float] = {}
    for rank, (t_idx, _) in enumerate(top_turn_dense):
        turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 1.00 / (rrf_k + rank + 1)
    for rank, t_idx in enumerate(top_turn_bm25):
        turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 1.15 / (rrf_k + rank + 1)
    for rank, (w_idx, _) in enumerate(top_win_dense):
        for t_idx in cache.windows[w_idx]["turn_indices"]:
            turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 0.85 / (rrf_k + rank + 1)
    for rank, w_idx in enumerate(top_win_bm25):
        for t_idx in cache.windows[w_idx]["turn_indices"]:
            turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 0.95 / (rrf_k + rank + 1)
    # Cross-encoder rerank + salience boost -> knapsack-pack 4,200 tokens in chronological order.

3. Axis B: Parameter Scale Phase Transition (8B → 30B MoE → 120B MoE)

With retrieval frozen, every model evaluated on SovereignMemoryStore receives the byte-identical 4,200-token turn-1 briefing across all 600 tasks, isolating parameter scale across our 6-run Phase-1 matrix (3,690 traces across Node-02 Dual RTX 2080 Ti 22 GB NVLink, Node-01 Dual RTX 3060 24 GB, and 96 GB Blackwell G4 burst):

Stock hermes3:latest (8B Q4_0) did not fail on JSON tool syntax—it emitted valid tool calls on 593/600 tasks (66.44% mechanical task gate), yet collapsed to 4.50% (0/15 pure deterministic) when binding multiple policy constraints simultaneously. Stepping up to local Qwen3-Coder-30B-A3B (only 3.3B active parameters on Dual RTX 2080 Ti at $0.00) crosses that reasoning threshold immediately, hitting 80.33% under hermes3 and a suite-high 88.81% (532/599) mechanical compliance.

4. Axis C: Wave-3 Refusal Postmortem → Wave-4 Unified QLoRA + DPO

Our most useful discovery came from Run 3's failure, where applying our Wave-3 SRE Code-Auditor adapter to gpt-oss:120b dropped generalist accuracy from 55.83% → 36.50%, alongside a hidden Ollama template bug on 8B:

• Wave-3 Auditor Role Bias (30.0% Side-Effect Withholding): Hardcoding a zero-trust SRE Auditor SYSTEM prompt into the fused Modelfile plus bug-hunting SFT caused 15.6x over-calling of read-only search_persona_memory (701 vs. 45 calls) and 180/600 (30.0%) zero-mutation turns where the model asked for human approval instead of executing routine calendar/email tasks.
• Ollama's Silent Hermes-3 Go Template Bug ({{- else if .System }}): Inspecting ollama show --modelfile hermes3:latest explained why Stock 8B scored near zero—attaching MCP tools silently discarded the entire runtime SYSTEM prompt while duplicating <tool_call> branches on assistant turns:

{{- if .Tools }}
<|im_start|>system
You are a function calling AI model... <tools>{{ .Tools }}</tools><|im_end>
{{- else if .System }}   <-- BUG: .System is silently dropped whenever .Tools is non-empty!
<|im_start|>system
{{ .System }}<|im_end>
{{- end }}

5. Official 14-Run openai/gpt-5.6-sol Scoreboard (8,400 Traces · 8 Queued)

Building on our 9-run Wave-3/Wave-4 local ablation (5,535 verified traces), we evaluated 14 clean-room 600-task configurations (8,400 official traces across 5 model families: Google Gemini, Google Gemma 4, Alibaba Qwen, OpenAI GPT-OSS, and Nous Hermes) on the byte-identical frozen 0o444 SovereignMemoryStore SQLite checkpoints. Every archive was graded with the official openai/gpt-5.6-sol (reasoning_effort="medium") judge, verified offline with 0 errors via harness.export_submission --check (14/14 passed), and 8 flagship archives are queued on dolphinbench.ai alongside 6 clean-room 2×2 search/thinking and dense-vs-MoE scaling controls:

*Official grades_official_gpt56_sol evaluates 1,324–1,468 strict state/field sub-checks per run (parentheses show our zero-LLM 1,310-check inner-loop tool-argument gate).

• Wave-4 Beat Stock at Every Fine-Tuned Scale: Local 30B climbed 42.00% → 42.67% (252 → 256/600, 1,123/1,310 = 85.73% local mechanical checks, +11 tasks on Riley) at $0.00 and 15.72s—outperforming MiniMax M3 + Hindsight (41.00% @ $158.33) and Claude Sonnet 5 + Honcho/Mem0 (35.83% / 32.33% @ $1,566–$1,831). 120B eliminated Wave-3 prose refusals (39.17% → 39.50%, 235 → 237/600), and 8B Q4_0 jumped 15.7x (7/600 → 110/600 = 18.33%, beating MiniMax M3 + Supermemory on Morgan 52 vs. 45).
• Why Local 30B (42.67%) Beats Sparse 120B (39.50%) & 80B (38.50%): Qwen3-Coder-30B-A3B chains multi-step list_* state discovery before writes on 320+ tasks (only 2 zero-tool turns), whereas scaling inactive MoE experts alone (80B / 120B at 3B–5B active params) leaves multi-constraint synthesis bounded near ~40%.

6. 5-Family Synthesis: Gemma 4 31B, Frontier Argon 91.17% & 2PC Action Guardrails

6.1 What Kind of Local Model Excels at Multi-App Agentic Memory? (Gemma 4 31B 54.17% vs. Coder-32B 35.33%)

Before testing dense non-coder models, it looked plausible that open-weights agents hit a hard ~42% wall on DolphinBench. Evaluating Apache-2.0 Google Gemma 4 31B (30.7B 100% active dense Q4_K_M) disproved that hypothesis immediately: stock Gemma 4 31B reaches 54.17% (325/600) with native <|thought|> thinking enabled and 52.00% (312/600) with thinking disabled (4.9x fewer output tokens: 174,226 vs. 853,604; Oracle@2 = 61.50% [369/600]) at $0.00 total cost—beating all 8 non-GPT-5.6-Luna commercial cloud baselines on the official leaderboard up to $1,830.57 (ranking #6 vs. all 13 public cloud stacks) and tying GPT 5.6 Luna + Hindsight/Honcho (144/200 = 72.0%) on Morgan.

Holding active parameter count fixed near ~31B dense, general-domain multi-app tool post-training (Gemma 4 31B: 54.17%) outperforms code-specialized Qwen2.5-Coder-32B (35.33%, 212/600, only 0.03 memory calls/task) by +18.83 pp (+113 tasks, McNemar p = 1.35 × 10⁻¹⁶) and sparse 3.3B-active Qwen3-Coder-30B-A3B Wave-4 (42.67%, 256/600) by +11.50 pp (+69 tasks, p = 4.96 × 10⁻⁷). Even 14.8B dense generalist Nous Hermes-4-14B (34.50%, 207/600, 1.99 memory calls/task) statistically ties 2.2x-larger Qwen2.5-Coder-32B (p = 0.729) and beats Claude Sonnet 5 + Mem0 (32.33% @ $1,830.57)—confirming that workplace assistants need dense general-domain social/calendar/email post-training alongside coder backbones.

6.2 Frontier Upper Bound & Retrieval-Sufficiency Proof (Argon Low 91.17% · Oracle@14 = 96.17% · Oracle@10 = 78.83%)

Pairing the byte-identical 0o444 frozen SovereignMemoryStore with Gemini 4 Argon (Low Thinking) (1.84 memory calls/task) achieves 91.17% (547/600, #1 Overall) at $24.54 total cost and 35.38s median latency—beating Official #1 Hermes + GPT 5.6 Luna + Mem0 (70.67% @ $96.21) by +20.50 pp (+123 tasks) at ~3.9x lower cost (Argon Medium scores 89.17% / 535/600 @ $24.57). Across our 2×2 Argon factorial (2,400 traces), Thinking × Agentic Search exhibits a statistically significant interaction (ΔΔ = +3.17 pp, 95% CI [+0.27, +6.06], p = 0.0320): lighter per-turn thinking avoids over-committing to partial Turn-0 context and reinvests budget into iterative search_persona_memory calls (+13.17 pp lift on Argon Low vs. +10.00 pp on Medium), while even Turn-0 Briefing-Only Argon Low (78.00% @ $15.76) beats #1 Mem0 by +7.33 pp at 6.1x lower cost.

Across all 14 runs, union pass rate reaches Oracle@14 = 96.17% (577/600)—and Oracle@10 = 78.83% (473/600) across $0.00 open-weights runs alone (+8.17 pp above #1 Mem0!), with a 4.19x superlinear multiplier at compositional depth LBF=4+ (89.33% [67/75] vs. 21.33% [16/75] on 30B Stock & Gemma 4 31B) and flatter corpus-entropy decay (-3.50 pp Argon Med / -7.00 pp Argon Low vs. -10.00 pp Morgan → Riley [78.0% → 68.0%] and -12.00 pp Morgan → Alex [78.0% → 66.0%] on Mem0). See companion Paper #27 (Return on Memory) for the full 4-chart scaling derivation and @krusch/toolkit/cascade for L1 (8B) → L2 (Gemma 4 31B / 30B @ $0.00) → L3 (Argon Low) routing.

6.3 Closing Execution Faults Without Retraining (krusch-stack INV-STACK-10 2PC Action Harness & Tradeoffs)

• Deterministic 2PC Tool-Action Guardrails (sovereign_harness_krusch_stack.py): Open-weights trace forensics exposed mechanical execution faults orthogonal to memory retrieval—3x identical read spin loops, duplicate state-mutating MCP writes, blind ID mutations without prior list_* inspection, and premature chat-only stops. Subclassing BenchmarkHarness over krusch-stack Tier 3 (POST /action-harness/{start,intercept,outcome,verify-stop,summary}, 63/63 Node + 6/6 Python tests) enforces BLOCK_SPIN_LOOP, BLOCK_DUPLICATE_WRITE, HOLD_FOR_REVISION, and single-user-turn role: "developer" CONTINUE_VERIFY steers (preserving len(user_turns) == 1). On a live 10-task Qwen3-Coder-30B failure slice at $0.00, it cut duplicate identical MCP calls 10 → 0 (-100%), reduced zero-app chat-only stops 3 → 1 (-66.7%), and recovered +20.0 pp (0/10 → 2/10) mechanical task passes (riley/096 & morgan/058) with zero weight retraining.
• Judge Calibration & 8B Q4_0 Triage Boundary: Local Referees (hermes3 80.33%, gemma4:e4b 53.00%) preserved identical rank order to gpt-5.6-sol (42.67%) for $0.00 inner-loop iteration, while 8B Q4_0's 18.33% ceiling (14.5% SWE/PM) confirms 4-bit 8B models belong in L1 triage rather than multi-app state mutation, and role-neutral Modelfiles prevent auditor refusal drift.

7. Reproducibility, Compute Ledger & Public Artifacts

# 1. Offline unit suites (including 6/6 krusch-stack 2PC harness tests) & 5-gate clean-room AST verifier
.venv/bin/python -m unittest discover -s tests -p "test_*.py" -v && .venv/bin/python verify_zero_leakage.py --static-only

# 2. Local 600-task run or clean-room Tier 3 krusch-stack 2PC Action Harness adapter ($0.00)
.venv/bin/python run_homelab_models_600.py --model qwen3-coder-30b-moe:latest --workers 4
PYTHONPATH=.:upstream .venv/bin/python -m harness.runner --release upstream --adapter sovereign_harness_krusch_stack:BenchmarkHarness

# 3. Official openai/gpt-5.6-sol grading + offline upstream archive check across all 14 archives
.venv/bin/python package_official_submissions.py --only gemma4_31b_stock --workers 12
PYTHONPATH=upstream .venv/bin/python -m harness.export_submission --release upstream --check submissions/sovereign_gemini_4_argon_low.zip

• Compute & Cost Ledger: $0.00 API spend across 153.4s zero-LLM SQLite ingestion and all 10 official 600-task open-weights archives (6,000 official open-weights traces + counterfactuals across Node-02 Dual RTX 2080 Ti, Node-01 Dual RTX 3060, Node-03 RTX 2080 Ti, and on-demand 96 GB Blackwell G4 burst with 0 active sessions post-run), plus $81.78 total frontier API spend across all four Gemini 4 Argon 2×2 factorial runs ($24.54 Argon Low #1 Overall, $24.57 Argon Medium, $16.91 Med Briefing-Only, and $15.76 Low Briefing-Only).
• Public Datasets & Open Weights: krush420/sovereign-g4-benchmarks (14 dolphinbench_600 rows + 36 sanitized raw JSON audit logs) · krush420/sovereign-router-arena (Tab 5) · krush420/sovereign-code-audit-sft (18,131 rows) · Wave-4 adapters (30B, 120B, 8B) · github.com/kruschdev/krusch-stack.