How an Open-Weights Sovereign Stack Stacks Up Across 317,000+ Benchmark Queries

1. Condensed 6-Pillar External Scorecard (11 Suites)

Rather than inventing custom micro-benchmarks or grading models with themselves, we ran each subsystem of krusch-stack against eleven established academic and industry benchmarks inside an isolated 96 GB NVIDIA RTX PRO 6000 Blackwell (sm_120) clean-room VM and local 1024d snowflake-arctic-embed2 loopback (zero outbound cloud API calls on every local open-weights inference run; 14-run, 5-family reader & Thinking × Search ablation on byte-identical 0o444 SQLite stores & official upstream gpt-5.6-sol rubric judge on DolphinBench).

2. How the Stack Compares Against Three Architecture Archetypes

A. vs. Standard Open-Source "Wrapper" Stacks

Typical open-source agent stacks glue LangChain or LlamaIndex splitters (fixed 500-token windows) to a single vector store, stuff every MCP tool schema into the system prompt, pass raw untrusted tool outputs straight back into the model, and allow coding or SQL agents to execute unverified 1-shot generations. Our controlled ablations isolate why each of those defaults breaks at scale:

• Out-of-Distribution Routing Collapse & Preflight Latency: Supervised routers fit in-distribution benchmarks well, then degrade sharply when domains shift. On ACL 2026 LLMRouterBench, RouteLLM (8.75%) and MLP (9.25%) collapse on unseen ARC-AGI reasoning tasks while zero-shot structural complexity gating + isotropic 8K snowflake-arctic-embed2 centroids reach 25.25% and #1 overall Pareto distance (0.4269). Independently on Martian RouterBench (arXiv:2403.12031 Sec. 5.2, 36,497 0-shot + 36,483 5-shot rows), KNN (0.6648), SVM (0.6363), and MLP (0.5572) all collapse below non-predictive ZeroRouter (0.6941) under Leave-One-Suite-Out OOD, whereas KruschZeroShotRouter (0.7268 0-shot / 0.7455 5-shot) and KruschCascadeRouter (0.7083 / 0.7389) rank #1 and #2 above ZeroRouter while also taking #1 In-Distribution Micro AIQ (0.7488 / 0.7695). On multi-step Aider Polyglot (n=60), replacing a 569.51ms 7B LLM preflight classifier with deterministic compiler/test failure routing (0.395ms, 1,442x faster) lifts Pass@2 from 21.67% to 51.67% (2.38x).
• Structure-Preserving Indexing vs. Fixed Token Windows: Holding snowflake-arctic-embed2 (1024d) constant on LegalBench-RAG, replacing fixed 160-token windows (11.70% F1) with KruschNexus hierarchical Section → Clause parsing + contextual section-path headers + local mxbai-rerank-large-v2 cross-encoder reranking lifts span-exact Macro F1 by +128.5% (26.74%). Across all 10/10 ACL 2025 CoIR tasks (2,389,123 docs & 162,216 queries), krusch-git AST-spine + identifier subtoken hybrid RRF reaches 56.40% Macro NDCG@10 (+7.42 pp over BM25 48.98%, lifting codefeedback-mt +8.00 pp to 60.23% and synthetic-text2sql +4.22 pp to 48.71%) while beating pure dense vectors across all 8/8 RepoBench-R splits.
• 2PC State Grounding, Generalized Action Harness, Deferred MCP Discovery & Taint-Tracked Tool Guards: Staging code patches behind TruncationGuard + Verify-on-Stop heals 72 failed Aider Polyglot suites (+32.00 pp), while read-only node:sqlite FK/cell-value introspection + 2PC SQL self-healing lifts BIRD-SQL Mini-Dev from 26.00% to 40.20% EX (100.00% valid SQL across 500/500 queries) at just 0.09 extra calls/query. Extending that pattern to arbitrary MCP tool workflows via krusch-stack's INV-STACK-10 Generalized 2PC Tool-Action Harness (action-harness-engine.js) eliminates 100% of duplicate MCP calls (10 → 0), cuts premature zero-action chat stops -66.7% (3 → 1), and recovers +20.0 pp (0/10 → 2/10) on a live Qwen3-Coder-30B failure slice at $0.00. Mounting a 64-tool Enterprise MCP hub on all 2,251 UC Berkeley BFCL v3 Live queries, createDeferredToolBridge + schema-repair guard cuts prompt schema tokens by -80.67% (7,894 → 1,526 tok/turn) while lifting Micro accuracy from 59.88% to 66.28% (+144 queries) and irrelevance abstention from 67.23% to 84.69% (+17.46 pp). And on ETH SpyLab AgentDojo v1.2.2 (1,046 episodes), taint-tracked untrusted boundary delimiting + deterministic sink-provenance policy floors drive indirect prompt-injection ASR from 19.28% (183/949) down to 0.00% (0/949) with zero benign utility degradation.

B. vs. Commercial Cloud Router, Memory, Code & Legal Stacks

• Cloud Routing Economics: In official RouteWorks/RouterArena GitHub CI, krusch-cascade-router achieves 0.7883 (80.72% accuracy) at $0.2117 / 1K queries—surpassing commercial routers such as Azure-Model-Router (0.7042), Martian (0.5756 @ $4.83/1K, 22.8x costlier), and NotDiamond (0.5729 @ $4.10/1K, 19.4x costlier). On Martian RouterBench (0-shot ID, n=10,953 eval), KruschCascadeRouter retains 98% of GPT-4 accuracy at $9.89 vs. $36.15 (-72.63% cost, 504.7 µs/q—2.83x faster and half the cost of KNN $18.71) and 95% of GPT-4 at $6.08 (-83.18%). On live multi-GPU generation, speculative SSE logprob/loop stream abort (Path B) cuts 120B TTFT by -48.0% (1,694.5ms → 880.7ms) and compute cost by -37.4% even with all 325 aborted streams (58,743 wasted tokens) strictly billed.
• Contradiction, 10M Multi-Session & 14-Run Official DolphinBench Memory: Packing a fixed ~8K-token slice (0.66% of a 10M-token chat) on zero-Docker node:sqlite, krusch-context-mcp v2.0.0 shows zero E2E degradation from 100K → 500K → 1M (70.60% → 71.82% → 72.86% on hermes3) and beats cloud past.dev (openai/gpt-5.6-luna) on Contradiction Resolution across all 4/4 splits (92.50% Stock / 98.13% SRE-QLoRA vs. 78.13% at 100K; 89.64% vs. 78.57% at 500K; 77.68% vs. 75.36% at 1M; 78.75% vs. 73.13% at 10M) and on 10M Multi-Session Reasoning (29.54% Qwen / 38.33% Hermes vs. 28.17%). Across 14 Official 600-Task DolphinBench runs (8,400 traces, 5 model families) reading the byte-identical 0o444 frozen $0.00 SovereignMemoryStore SQLite checkpoints under the official gpt-5.6-sol rubric judge, open-weights Google Gemma 4 31B reaches 54.17% (325/600, #1 Open-Weights @ $0.00; 52.00% No-Think, 61.50% Oracle@2) and local Qwen3-Coder-30B-A3B Wave-4 hits 42.67% (256/600, 85.73% mechanical accuracy) at $0.00 (alongside 34.50% Hermes-4-14B and a 15.7x Wave-4 lift on Hermes-3-8B from 1.17% → 18.33%)—shattering the 50% open-weights barrier to beat all 8 non-GPT-5.6-Luna commercial cloud baselines (MiniMax M3 + Mem0 47.83% @ $148.02 through Claude Sonnet 5 + Mem0 32.33% @ $1,830.57, ranking #6 Overall at $0.00; local open-weights study →). Pairing those exact same 0o444 frozen $0.00 SQLite stores with Gemini 4 Argon (Low) scales to 91.17% (547/600, #1 Overall @ $24.54, 35.38s median latency; plus 89.17% Argon Medium @ $24.57 and 78.00% Turn-0 Briefing-Only @ $15.76)—topping Official #1 Hermes + GPT 5.6 Luna + Mem0 (70.67% @ $96.21, 37.69s) by +20.50 pp (+123 tasks) at ~4x lower cost, establishing 96.17% (577/600) Oracle@14 retrieval sufficiency (78.83% Oracle@10 across $0.00 open-weights alone), a 4.19x LBF=4+ multiplier (89.33% vs. 21.33%), a significant 2×2 Thinking × Search interaction (ΔΔ = +3.17 pp, p = 0.0320), and flatter corpus-entropy decay (-7.00 pp Argon Low / -3.50 pp Argon Medium vs. -10.00 pp on Mem0 [156 → 136] to -32.50 pp on MiniMax M3 + Hindsight; Return on Memory scaling study →).
• Air-Gapped Code & Legal Retrieval Precision: On MAUD merger agreements (1,676 queries), KruschNexus hits 33.95% character-exact Recall@2 locally—exceeding published OpenAI text-embedding-3-large + Cohere Rerank Recall@64 (28.28%) while retrieving 32x fewer spans. On ACL 2025 CoIR (10/10 tasks, 2.39M docs), krusch-git's local 1024d hybrid RRF (56.40% NDCG@10) tops proprietary Voyage-Code-002 (56.26%) and OpenAI-ada-002 (53.66%; see §3 context-window disclosure).

C. Clean-Room Auditability vs. Unverified Leaderboard Claims

Every run in our benchmark archive passes 73 / 73 automated integrity gates: (1) loopback air-gap verification (127.0.0.1:11434, cloud API keys unset during sovereign model inference), (2) runtime JavaScript Proxy traps, opaque RPC session tokens, frozen 0o444 pre-populated SQLite memory stores on DolphinBench, and Python pre-eval column stripping on ground-truth answer/SQL/qrel/injection fields that block oracle access before inference, (3) RestrictedUnpickler + SafeZipExtractor guards, read-only DatabaseSync SQL enforcement, full-corpus 2,389,123-doc anti-shrinkage checks, and zero hardcoded attacker literals on AgentDojo, (4) mandatory Exercism test un-skipping + pristine test SHA-256 checks, and (5) zero LLM self-judging (deterministic compiler, SQL set-equivalence, AST bijection, and math verifiers on 9 benchmarks; dual cross-family open-weights referees on BEAM; official gpt-5.6-sol rubric judge on DolphinBench).

3. Where Frontier Cloud Stacks Still Beat Us (Honest Gaps & Open PRs)

• 1. Open-Weights Multi-Fact Synthesis Ceiling vs. Frontier Reasoning on BEAM & DolphinBench (70.6%–73.5% BEAM & 34.50%–54.17% Open-Weights DolphinBench vs. 91.17% Argon / 92.1% Cloud): Our 14-run (8,400-trace, 5-family) Official gpt-5.6-sol DolphinBench ablation isolated the open-weights gap strictly to multi-fact reader synthesis and post-training specialization rather than retrieval: holding the $0.00 0o444 SovereignMemoryStore byte-for-byte constant (96.17% Oracle@14 overall and 78.83% Oracle@10 across $0.00 open-weights runs alone—already +8.17 pp above #1 Mem0!), general-domain tool-trained Google Gemma 4 31B reaches 54.17% (325/600, #1 Open-Weights @ $0.00)—beating matched-size code-specialized Qwen2.5-Coder-32B (35.33%) by +18.83 pp (p = 1.35×10⁻¹⁶) and all 8 non-GPT-5.6-Luna cloud stacks—while pairing the exact same SQLite checkpoints with Gemini 4 Argon (Low) reaches 91.17% (547/600, #1 Overall @ $24.54; 89.17% Medium), surpassing #1 GPT 5.6 Luna + Mem0 (70.67% @ $96.21) by +20.50 pp. Across purely open-weights readers, four distinct factors explain the remaining gap:
  • Cross-family referees vs. cloud self-judging on BEAM: We grade open-weights gpt-oss:120b (5.1B active MoE) with two independent cross-family open-weights judges (qwen2.5:32b and hermes3:latest, 81%–88% binary agreement). By contrast, past.dev's 92.1% uses openai/gpt-5.6-luna as both reader and judge.
  • Exact-position ordering rubrics: On Event Ordering (100K), krusch-context-mcp v2.0.0 retrieves 81.19% of gold turns, yet qwen2.5:32b scores 9.68% while hermes3:latest scores 43.17% (vs. 100% past.dev) because qwen2.5:32b awards 0.0 whenever a 10-event timeline shifts by one slot.
  • Passive supersedes_id = NULL ingestion on Knowledge Update: Despite 97.50% retrieval recall at 100K, E2E QA lands at 37.5%–52.5% (vs. 86%–92.5% past.dev) because our zero-heuristic benchmark loader ingests raw chat turns with supersedes_id = NULL, leaving outdated historical turns active in the 8K window. In production IDE sessions, coding agents explicitly invoke krusch_context_revise({ action: "supersede" }) to mark superseded facts inactive at the SQLite/Postgres layer.
  • 4+-fact exhaustive compositional synthesis in open-weights decoders: While $0.00 open-weights readers (54.17% Gemma 4 31B, 52.00% Gemma 4 31B No-Think, 42.67% 30B Wave-4, 39.50% 120B Wave-4, 38.50% 80B Stock, 35.33% 32B Dense, and 34.50% Hermes-4-14B) beat commercial cloud stacks up to $1,830.57, open-weights decoders still trail frontier reasoning models on 4+-fact exhaustive synthesis (21.33% [16/75] on Gemma 4 31B & 30B Stock vs. 89.33% [67/75] on Argon Low at LBF=4+, a 4.19x multiplier and 16.75x over 8B 5.33%) where open-weights models omit secondary constraint clauses despite retrieved facts sitting in context.
• 2. Multi-Turn Frontier Reasoning Upper Bound on Coding & Text-to-SQL (51.56% Polyglot & 40.20% BIRD-SQL vs. 65%–88%+ Cloud): Lifting open-weights gpt-oss:120b at $0 API cost from 19.56% → 51.56% (116/225) on Aider Polyglot matches Claude 3.5 Sonnet (51.6%) and beats GPT-4o (23.1%), and lifting BIRD-SQL Mini-Dev from 26.00% → 40.20% EX (100% valid SQL at 1.09 calls/query) recovers +14.20 pp without fine-tuning—yet 109/225 hard algorithmic C++/Rust/Go/Java suites and complex multi-join SQL aggregates still separate single-retry open-weights MoEs from multi-turn/multi-candidate frontier reasoning pipelines (o3, gpt-5, Claude 4, Gemini 4 Argon at 75%–88%+ coding and 65%–75%+ multi-candidate SQL). That gap is why our Cascade Router retains an L3 frontier escalation tier for the hard tail.
• 3. Monolingual English Feature Variance on Held-Out CJK OOD (0.0811 leave-chinese-out 0-Shot): On Martian RouterBench 0-Shot Leave-One-Suite-Out OOD, learned KruschCascadeRouter ranks #1 among learned routers on 7-suite Macro AIQ (0.6873) and #2 overall Micro AIQ (0.7083, behind zero-shot KruschZeroShotRouter at 0.7268), yet drops to AIQ = 0.0811 on held-out chinese (n=236, pulling Macro-9 to 0.6156) because training exclusively on English 0-shot suites produces std(cjk_ratio) ≈ 0 against the 1e-4 z-score variance floor. In-distribution 0-shot chinese ranks #1 (0.4880) and 5-shot OOD chinese recovers to 0.3059; zero post-hoc test tuning was applied.
• 4. ACL 2025 CoIR Context-Length Protocol Difference (8,192-Token Production Window vs. Table 3 512-Token Truncation): In the ACL 2025 CoIR paper's Table 3, all 10 baseline retrievers (including Voyage-Code-002 56.26% and E5-Mistral 7B 55.18%) were evaluated under a uniform 512-token truncation cap so legacy 512-token encoders could be compared side by side. Our krusch-git run (56.40% 10-task Macro NDCG@10 across all 2,389,123 unshrunk docs) evaluates snowflake-arctic-embed2:latest at its native 8,192-token production context window + AST spine headers—reflecting real repository indexing while accounting for part of the margin on long-document tasks such as apps and stackoverflow-qa.
• 5. Open Upstream Leaderboard Submissions (Explicit Status):
  • DolphinBench (dolphinbench.ai): All 8/8 queued official submission archives—sovereign_gemini_4_argon_low.zip (91.17%, 547/600 @ $24.54, UUID c047b063-822c-49fb-8b36-27c3763d6ca7, SHA-256 80a199a60aef65f0d963d0402a2e3b0af99b82ed63e2f422001234b6ed770f0b), sovereign_gemini_4_argon.zip (89.17%, 535/600 @ $24.57, UUID 29389157-b01b-42b5-8193-17d9a496f580, SHA-256 9a7fa41fb3d1cf5f71f0c3cf74a0c97f8f39d50e26a7dd484c323f7719ecb7a0), sovereign_gemma4_31b_stock.zip (54.17%, 325/600 #1 Open-Weights @ $0.00, UUID 288671ad-afe0-40a3-93af-b834c9bbd306, SHA-256 2a23c2a023473a6642f5a61f1d99ff569f3d50ac53e60576f8195eef659c254b), plus local $0.00 30B Wave-4 42.67% (fd45f16c-fd3c-471c-b93d-413f61f1dbcf), 30B Stock 42.00% (9cac5ab1-ecba-40a6-ad84-55ec2d67bfce), 120B Wave-4 39.50% (83a4c085-0e44-42af-a625-b496e63c8949), 120B Stock 39.17% (7b5073fd-e1d9-458b-a607-7c55f76b6df1), and 8B Wave-4 18.33% (19552654-be39-4fe0-8043-a49158582356)—are verified (submission.validate / export_submission --check, error_code: null) and uploaded to dolphinbench.ai with status: "queued" awaiting maintainer leaderboard publication, alongside 6 additional clean-room ablation & scaling archives (Gemma 4 31B 52.00% No-Think, Argon Turn-0 Briefing-Only 79.17% & 78.00%, 80B Stock 38.50%, Qwen2.5-Coder-32B 35.33%, and Hermes-4-14B 34.50%; 14/14 verified 600-task runs = 8,400 official gpt-5.6-sol traces).
  • RouteWorks/RouterArena (PR #169): Scored 0.7883 by the official RouteWorks github-actions[bot] CI workflow across all 8,400 queries + 420 robustness perturbations; PR #169 is open and awaiting maintainer merge onto the upstream leaderboard.
  • withmartian/routerbench (kruschdev/routerbench:feat/krusch-cascade-router): Drop-in AbstractRouter implementation (KruschCascadeRouter + KruschZeroShotRouter) and 4-regime ID/OOD harness published on our public fork branch and ready for upstream PR submission.
  • ETH SpyLab AgentDojo, UC Berkeley BFCL & Aider Polyglot: Our full 1,046-episode AgentDojo v1.2.2 (0.00% ASR), 2,251-query BFCL v3 Live, and 225/225 Aider Polyglot runs execute the official upstream suites inside clean-room harnesses; upstream Python adapter PR playbooks for ethz-spylab/agentdojo and Aider-AI/polyglot-benchmark remain unsubmitted next steps.
• 6. Upstream Dataset Hygiene Notes: Two upstream rows in RepoBench-R ([5522, 7591] in python_cff/test_easy) carry out-of-bounds gold_snippet_index pointers and are deterministically quarantined (47,998 / 48,000 evaluated). In Martian RouterBench routerbench_5shot.pkl, 28 upstream rows have missing model cells across the 11-model matrix and are deterministically filtered so all 6 routers evaluate the identical 36,483 / 36,511 complete rows. In UC Berkeley BFCL v3 BFCL_v3_live_relevance.json, upstream ships a duplicate row ID (live_relevance_3-3-0 at lines 3 and 4 with distinct prompts), deterministically disambiguated as live_relevance_3-3-0__dup2 so all 18 / 18 rows (2,251 / 2,251 Live queries) execute without overwrite. In ICLR 2026 BEAM 100K, upstream Conversation 14 Knowledge Update Q1 asks about a nonexistent "running shoe" topic; our verifier asserts score == 0.00 (canary_conv14_ku_q1_zero: true).

4. Raw Artifacts & 60-Second Cryptographic Verification

Every raw per-query JSON audit log (36 raw JSON audit logs, SHA-256 pinned, including all 14 dolphinbench_600 official gpt-5.6-sol runs / 8,400 traces) and 14 Hugging Face Dataset Viewer splits are published under CC-BY-4.0 at krush420/sovereign-g4-benchmarks, alongside the interactive Pareto & trajectory explorer at krush420/sovereign-router-arena and repository-level reproduction docs in kruschdev/krusch-stack.

# Clone the raw clean-room audit logs & verify all SHA-256 digests in <60s
git clone https://huggingface.co/datasets/krush420/sovereign-g4-benchmarks
cd sovereign-g4-benchmarks

python3 -c '
import hashlib, json, pathlib
rows = [json.loads(l) for l in pathlib.Path("data/scorecard_summary.jsonl").read_text().splitlines()]
for r in rows:
    if r["audit_log_file"].startswith("raw_audit_logs/"):
        digest = hashlib.sha256(pathlib.Path(r["audit_log_file"]).read_bytes()).hexdigest()
        assert digest == r["audit_log_sha256"], f"Mismatch: {r[\"audit_log_file\"]}"
        print("VERIFIED:", r["audit_log_file"], digest[:16])
'