arXiv:2611.24576 · 2026 · Looped LLM Inference Systems
In looped Transformers, deep cached context substitutes for token-local early passes. Our converged decoder verifies candidate blocks through only the final K < T passes on the critical path, completes exact 1 → T KV caches as piggybacked batch rows at zero extra serial calls, and projects finished wake states inside the same output GEMM to unify greedy-exact (τ = 0) and high-speed lossy (τ = ∞) decoding in one CUDA graph.
Interactive Architecture Walkthrough
Watch how tentative late-pass block decoding, piggybacked 1 → T KV cache completion, and same-GEMM Wake verification operate together at T = 4, K = 2. Switch scenarios or drag the live tolerance knob τ below to see how a single engine unifies greedy-exact (τ = 0), margin-gated (τ > 0), and non-rollback lossy (τ = ∞) decoding.
lm_head Output GEMM
81 rows
Whereas LoopSpec, WaveFront, DAS, and LoopAhead accelerate drafting via shallow initial passes while still verifying every token through all T passes, our (1+B)2 block compresses critical-path verification itself to the final K < T passes.
Because looped layers share weights across passes, tentative groups advance K true passes per step inside ongoing forward calls. Chronological ordering guarantees every group reads exact earlier-group pass-k KV states.
Projecting finished T-pass wake rows inside the existing lm_head GEMM guarantees bit-for-bit T-pass greedy equivalence in exact arithmetic at τ = 0 (verified on 258/258 adversarial fp64 tests) and strict prefix progress i*+1 > u.
Core Scientific Finding · Section 2
Raw late entry fails on unadapted checkpoints only because pass 4 expects layer-normalized states (norms 37–51) rather than raw embeddings (norm 0.76). Once lightweight Stage-2 fine-tuning aligns that interface, coherent pass-aligned (K, V) context substitutes for skipped local passes on >98% of positions, isolating residual discrepancies to the 6% uncertain tail.
GSM8K teacher-forced next-token agreement with full T-pass decoding
Stage-2 aligns K = 1 queries onto the pass-4 manifold (linear CKA .961 vs. .273–.365 unadapted); tokens >64 positions back supply 53% of IFEval's total context-depth gain.
Ouro-2.6B fp32 agreement stratified by full-depth confidence pmax (30,736 positions)
On the 68% of tokens where full depth is confident (≥.95), adapted K = 2 matches at 1.000. Wake verification only needs to intervene inside the rare 6% uncertain tail (≤2.2 rollbacks / 100 tokens).
Unified Spectrum · Section 4.5 (Table 7)
Across every benchmark on both Ouro-2.6B (T = 4) and Huginn-3.5B (T = 32), Exact Wake (τ = 0) accelerates decoding well beyond full-depth K = T block verification while preserving greedy-exact T-pass predictions, and Margin Wake (τ > 0) filters benign near-tie flips to unlock further speedups with zero accuracy loss.
| Benchmark | Budget | Seq. T (tok/s) | K = T Block | Exact Wake (τ = 0) | rb / 100t | Margin Wake (τ > 0) | rb / 100t | Lossy (τ = ∞) |
|---|---|---|---|---|---|---|---|---|
| Ouro-2.6B (T = 4 passes · teacher-forced τ = 0 audit diff: 0.007% GSM8K, 0.09% MATH500, 0.025% MBPP) | ||||||||
| GSM8K | K = 3 | .868 (62.3) | .867 · 4.13× | .867 · 4.19× (+2%) | 1.14 | .870 (τ=.25) · 4.55× (+10%) | 0.29 | .853 · 5.18× |
| GSM8K | K = 2 | .868 (62.3) | .867 · 4.13× | .867 · 5.20× (+26%) | 2.02 | .877 (τ=.15) · 5.52× (+34%) | 1.26 | .842 · 6.82× |
| MATH500 | K = 2 | .532 (62.7) | .526 · 4.08× | .526 · 5.23× (+28%) | 2.11 | .548 (τ=.30) · 5.83× (+43%) | 0.96 | .492 · 6.61× |
| IFEval | K = 2 | .732 (63.8) | .739 · 2.15× | .739 · 2.66× (+24%) | 5.98 | .719 (τ=.15) · 2.94× (+36%) | 4.08 | .699 · 3.76× |
| MBPP | K = 2 | .616 (63.4) | .626 · 3.79× | .626 · 4.61× (+22%) | 2.00 | .618 (τ=.30) · 5.13× (+35%) | 0.70 | .602 · 6.20× |
| Huginn-3.5B (T = 32 Core passes · unified CUDA-graphed paged FlashAttention-3 engine) | ||||||||
| GSM8K | K = 16 | .391 (21.9) | .394 · 3.35× | .396 · 5.21× (+56%) | 0.48 | .404 (τ=.25) · 5.42× (+62%) | 0.05 | .399 · 5.80× |
| GSM8K | K = 8 | .391 (21.9) | .394 · 3.35× | .392 · 6.09× (+82%) | 2.22 | .397 (τ=.25) · 7.15× (+114%) | 0.88 | .353 · 9.13× |
| IFEval | K = 16 | .560 (21.5) | .547 · 2.91× | .547 · 4.85× (+66%) | 0.37 | .560 (τ=.25) · 5.10× (+75%) | 0.02 | .562 · 5.21× |
| IFEval | K = 8 | .560 (21.5) | .547 · 2.91× | .556 · 6.46× (+122%) | 2.03 | .543 (τ=.25) · 7.59× (+161%) | 0.59 | .542 · 8.85× |
| MBPP | K = 16 | .234 (21.9) | .236 · 4.47× | .236 · 6.64× (+49%) | 0.25 | .234 (τ=.25) · 6.79× (+52%) | 0.03 | .232 · 7.80× |
| MBPP | K = 8 | .234 (21.9) | .236 · 4.47× | .236 · 7.69× (+72%) | 1.41 | .236 (τ=.25) · 8.93× (+100%) | 0.56 | .228 · 12.50× |
Drafter-Verifier Factorization: Replacing trained mask-placeholder proposals with a parameter-free 4/3/2-gram context-suffix drafter preserves or strengthens the multiplicative speedup from K < T Wake verification (1.37× at τ = 0 and 1.48× at τ = 0.25 on Ouro GSM8K; 1.89× on Huginn MBPP; 2.25× on Huginn IFEval).
Depth Scaling & Hardware Law · Sections 4.6 – 4.7
Evaluating the exact same Huginn-3.5B checkpoint across T ∈ {8, 16, 32, 64} (36 → 260 effective layers) shows that while full-depth K = T block speedup saturates flatly near 3.3×, Wake rollback rates drop 21.5× with depth, driving τ = 0 speedups to 8.42× (+153% over K = 64 block) and τ = ∞ to 10.57×.
Speedup climbs with T as K = T/4 rollbacks fall 12.46 → 0.58 per 100 tokens
At shallow T = 8, K = 2 is below the readout horizon (.030 lossy), yet τ = 0 Wake catches every slip to restore .262 (+23.2 pts) while throttling speed gracefully; for T ≥ 16, τ = 0 climbs past the K = T wall up to 8.42× (9.17× at τ = 0.25).
tpass(M, C) = ((t0 + αkvC)p + (βtcM)p)1/p calibrated strictly on K = 4
Skipping T − K passes on rejected draft branches cuts per-step row-passes by 39% (198 vs. 324 at B = 8), sustaining 2.31× decode (1.67× end-to-end with online prefills) at C = 32 via pre-captured B = 8 → 4 → 2 graphs.
Comprehensive Evaluation · Sections 4.1 – 4.4
Before enabling Wake rollbacks, Sections 4.1–4.4 isolate pure pass-placement speedups (τ = ∞) across all six benchmarks on Ouro-2.6B (against matched trained early exit, LoopSpec, and LoopAhead) and across four capability-active benchmarks on 32-pass Huginn-3.5B.
Normalized by fastest sequential engine per benchmark (62.4–67.2 tok/s; Table 5)
| Task (Seq.) | LoopSpec | LoopAhead | Ours K=4 | Lossy K=3 | Lossy K=2 |
|---|---|---|---|---|---|
| GSM8K (.873) | 2.84× | 2.94× | 3.82× .867 | 4.80× .851 | 6.35× .839 |
| IFEval (.732) | 2.34× | 2.67× | 2.12× .739 | 2.75× .734 | 3.68× .699 |
| MATH500 (.532) | 2.85× | 2.59× | 4.11× .526 | 5.07× .536 | 6.63× .492 |
| MBPP (.616) | 2.86× | 2.90× | 3.64× .626 | 4.51× .618 | 5.95× .602 |
| BBH (.577) | 2.62× | 3.25× | 3.29× .577 | 4.35× .564 | 6.33× .549 |
| MMLU-Pro (.504) | 2.75× | 2.44× | 3.56× .504 | 4.56× .501 | 6.12× .488 |
8,871 items at B = 8; late entry drives +74% to +206% over K = 32 block (Table 6)
| Task | Seq. T=32 | K = 32 Block | K = 16 Late | K = 8 Late |
|---|---|---|---|---|
| GSM8K | .393 (21.7) | .397 · 3.37× | .405 · 5.84× (+74%) | .354 · 9.13× (.369 @ 9.93×*) |
| IFEval | .551 (21.6) | .551 · 2.93× | .566 · 5.24× (+79%) | .542 · 8.96× (+206%) |
| BBH | .283 (22.6) | .286 · 3.79× | .287 · 6.65× (+75%) | .274 · 10.86× (+186%) |
| MBPP | .234 (22.4) | .236 · 4.46× | .232 · 7.76× (+74%) | .228 · 12.41× (+178%) |
*2-stage partial-cache K = 8 (passes 17→32) reaches 9.93× (+9.7 pts vs. trained early exit at ~220 tok/s); enabling τ = 0 Wake in Table 7 restores full .392 GSM8K accuracy at 6.09× (.397 at 7.15× for τ = 0.25).
Xalphinions/ouro-2.6b-late-entry
Main purely looped Ouro-2.6B (T = 4, 48 layers) late-entry & Wake model
Xalphinions/huginn-3.5b-late-entry
Main Prelude-Core-Coda Huginn-3.5B (T ∈ {8..64}) late-entry & Wake model
Xalphinions/ouro-1.4b-late-entry
Scale-replication Ouro-1.4B (T = 4, 24 layers) late-entry checkpoint
Xalphinions/ouro-2.6b-early-exit
Matched Stage-2 early-exit control for Ouro-2.6B (+ Huginn control in collection)
Citation
@article{jin2026lateentry,
title = {Late Entry in Looped Language Models},
author = {Jin, Ziqi and Li, Jiaxi and Xu, Weiwen and Sun, Aixin},
journal = {arXiv preprint arXiv:2611.24576},
year = {2026},
url = {https://arxiv.org/abs/2611.24576}
}