arXiv:2611.24576 · 2026 · Looped LLM Inference Systems

Late Entry in Looped Language Models

In looped Transformers, deep cached context substitutes for token-local early passes. Our converged decoder verifies candidate blocks through only the final K < T passes on the critical path, completes exact 1 → T KV caches as piggybacked batch rows at zero extra serial calls, and projects finished wake states inside the same output GEMM to unify greedy-exact (τ = 0) and high-speed lossy (τ = ∞) decoding in one CUDA graph.

Ziqi Jin1, Jiaxi Li2, Weiwen Xu3,†, Aixin Sun1,†

1Nanyang Technological University  ·  2Singapore University of Technology and Design  ·  3The Chinese University of Hong Kong
†Co-corresponding authors: weiwen.xuu@gmail.com, axsun@ntu.edu.sg

(a) Full-Depth Block (K = T) 4 serial calls / step
P4P3P2P1 2+3 new block
Deep KV, but pays all T passes Flat ~3.3× wall as T → 64
(b) Early Exit (First K) Truncates context KV
P4P3P2P1 DEEP KV MISSING 2+3 new block
2 serial calls, shallow prefix −16.9 pts GSM8K drop
(c) Late Entry + Wake (Ours) 2 calls + Wake GEMM ✓
P4P3P2P1 skip P1, P2 verif. wake 2 wake 1 enter P3→P4
0 extra serial calls · same GEMM 5.2× to 8.4× greedy-exact
5.20× / 6.82×
Ouro-2.6B (T = 4, GSM8K)
5.20× (+26%) greedy-exact at τ = 0 (.867 acc, 0.007% diff); 5.52–5.83× on math at τ ∈ [.15, .30]; 6.82× at τ = ∞ (+16.9 pts vs. early exit).
6.09× → 8.42×
Huginn-3.5B Greedy-Exact (τ = 0)
+82% to +153% faster than full-depth K = T block across T = 32 → 64 (7.69× on MBPP at T = 32; 10.57×–12.50× at τ = ∞).
21.5× fewer
Wake Rollbacks as T = 8 → 64
Holding Huginn-3.5B weights fixed, K = T/4 rollbacks drop 12.46 → 0.58 per 100 tokens as deeper loops stabilize late states.
8.3% MAPE
Zero-Shot H100 Roofline Law
Calibrated strictly on K = 4; explains free wake rows at batch 1 (99 ≪ 348 ridge rows) and matches 11/12 optimal B*(C) across C ∈ [1, 32].

Interactive Architecture Walkthrough

How Our Converged Method Works Inside One Static CUDA Graph

Watch how tentative late-pass block decoding, piggybacked 1 → T KV cache completion, and same-GEMM Wake verification operate together at T = 4, K = 2. Switch scenarios or drag the live tolerance knob τ below to see how a single engine unifies greedy-exact (τ = 0), margin-gated (τ > 0), and non-rollback lossy (τ = ∞) decoding.

Wake Tolerance τ:
A. Sequence Partition: Verified Prefix (< u) vs. Tentative Frontier
Verified T-pass Tentative (in wake) Active block (P3, P4) Rolled back
B. Pass-Aligned KV Cache Matrix (P1 → P4 Across Token Positions) P3*, P4* = provisional embedding KV
C. Inside One H100 Step (K = 2 Serial Calls) 0 extra layer calls
Packed Per-Call Query Batch 81 / 348 ridge rows
81 block
Call 1: P3 (block) · Call 2: P4 (block) Inside H100 HBM weight-streaming floor (≪ 348 ridge rows/pass)
Shared Single lm_head Output GEMM 81 rows
Waiting for first wake group to finish P1→P4...
No finished wake group this step
Step 1

Serial Calls 2 / step
Verified (< u) 3 toks
Active Regime Greedy-Exact
Critical Path · Sec. 3.2

1. Late-Pass Block Verification

Whereas LoopSpec, WaveFront, DAS, and LoopAhead accelerate drafting via shallow initial passes while still verifying every token through all T passes, our (1+B)2 block compresses critical-path verification itself to the final K < T passes.

Background Wake · Sec. 3.3

2. Causal ⌈T / K⌉ Cache Completion

Because looped layers share weights across passes, tentative groups advance K true passes per step inside ongoing forward calls. Chronological ordering guarantees every group reads exact earlier-group pass-k KV states.

Proposition 1 · Sec. 3.4

3. Exactness & Strict Progress

Projecting finished T-pass wake rows inside the existing lm_head GEMM guarantees bit-for-bit T-pass greedy equivalence in exact arithmetic at τ = 0 (verified on 258/258 adversarial fp64 tests) and strict prefix progress i*+1 > u.

Core Scientific Finding · Section 2

Why Final Passes Beat Initial Passes, and Where Errors Hide

Raw late entry fails on unadapted checkpoints only because pass 4 expects layer-normalized states (norms 37–51) rather than raw embeddings (norm 0.76). Once lightweight Stage-2 fine-tuning aligns that interface, coherent pass-aligned (K, V) context substitutes for skipped local passes on >98% of positions, isolating residual discrepancies to the 6% uncertain tail.

Pass-Coherent (K, V) & Cross-Scale Asymmetry

GSM8K teacher-forced next-token agreement with full T-pass decoding

Adapted K = 2 (ours)Ouro-2.6B · 1.000 on confident ≥.95
.984
Adapted K = 1 over D = 41 late pass > 2 early passes (.954)
.962
Early Exit K = 2 over D = 2Ouro-2.6B · shallow 2-pass context
.954
Pass-4 K + Pass-1 V swapAligned query, shallow values
.296
Pass-1 K + Pass-4 V swapCross-pass mismatch → 100% EOS
.000
Huginn-3.5B: 4 late passesBeats 8 trained early passes (.944)
.954

Stage-2 aligns K = 1 queries onto the pass-4 manifold (linear CKA .961 vs. .273–.365 unadapted); tokens >64 positions back supply 53% of IFEval's total context-depth gain.

Discrepancies Localize to the 6% Uncertain Tail

Ouro-2.6B fp32 agreement stratified by full-depth confidence pmax (30,736 positions)

Wake target zone (6% uncertain tail) 1.00.80.60.40 <.50(6%) .50–.80(14%) .80–.95(12%) ≥.95(68%) Adapted K=2 (1.000 @ ≥.95) Adapted K=1 (.998) 1-pass early exit Unadapted K=1

On the 68% of tokens where full depth is confident (≥.95), adapted K = 2 matches at 1.000. Wake verification only needs to intervene inside the rare 6% uncertain tail (≤2.2 rollbacks / 100 tokens).

Unified Spectrum · Section 4.5 (Table 7)

Breaking Past the K = T Block Ceiling With Greedy-Exact Fidelity

Across every benchmark on both Ouro-2.6B (T = 4) and Huginn-3.5B (T = 32), Exact Wake (τ = 0) accelerates decoding well beyond full-depth K = T block verification while preserving greedy-exact T-pass predictions, and Margin Wake (τ > 0) filters benign near-tie flips to unlock further speedups with zero accuracy loss.

Benchmark Budget Seq. T (tok/s) K = T Block Exact Wake (τ = 0) rb / 100t Margin Wake (τ > 0) rb / 100t Lossy (τ = ∞)
Ouro-2.6B (T = 4 passes · teacher-forced τ = 0 audit diff: 0.007% GSM8K, 0.09% MATH500, 0.025% MBPP)
GSM8KK = 3.868 (62.3).867 · 4.13×.867 · 4.19× (+2%)1.14.870 (τ=.25) · 4.55× (+10%)0.29.853 · 5.18×
GSM8KK = 2.868 (62.3).867 · 4.13×.867 · 5.20× (+26%)2.02.877 (τ=.15) · 5.52× (+34%)1.26.842 · 6.82×
MATH500K = 2.532 (62.7).526 · 4.08×.526 · 5.23× (+28%)2.11.548 (τ=.30) · 5.83× (+43%)0.96.492 · 6.61×
IFEvalK = 2.732 (63.8).739 · 2.15×.739 · 2.66× (+24%)5.98.719 (τ=.15) · 2.94× (+36%)4.08.699 · 3.76×
MBPPK = 2.616 (63.4).626 · 3.79×.626 · 4.61× (+22%)2.00.618 (τ=.30) · 5.13× (+35%)0.70.602 · 6.20×
Huginn-3.5B (T = 32 Core passes · unified CUDA-graphed paged FlashAttention-3 engine)
GSM8KK = 16.391 (21.9).394 · 3.35×.396 · 5.21× (+56%)0.48.404 (τ=.25) · 5.42× (+62%)0.05.399 · 5.80×
GSM8KK = 8.391 (21.9).394 · 3.35×.392 · 6.09× (+82%)2.22.397 (τ=.25) · 7.15× (+114%)0.88.353 · 9.13×
IFEvalK = 16.560 (21.5).547 · 2.91×.547 · 4.85× (+66%)0.37.560 (τ=.25) · 5.10× (+75%)0.02.562 · 5.21×
IFEvalK = 8.560 (21.5).547 · 2.91×.556 · 6.46× (+122%)2.03.543 (τ=.25) · 7.59× (+161%)0.59.542 · 8.85×
MBPPK = 16.234 (21.9).236 · 4.47×.236 · 6.64× (+49%)0.25.234 (τ=.25) · 6.79× (+52%)0.03.232 · 7.80×
MBPPK = 8.234 (21.9).236 · 4.47×.236 · 7.69× (+72%)1.41.236 (τ=.25) · 8.93× (+100%)0.56.228 · 12.50×

Drafter-Verifier Factorization: Replacing trained mask-placeholder proposals with a parameter-free 4/3/2-gram context-suffix drafter preserves or strengthens the multiplicative speedup from K < T Wake verification (1.37× at τ = 0 and 1.48× at τ = 0.25 on Ouro GSM8K; 1.89× on Huginn MBPP; 2.25× on Huginn IFEval).

Depth Scaling & Hardware Law · Sections 4.6 – 4.7

Deeper Recurrence Makes Wake Faster, and Roofline Explains Why

Evaluating the exact same Huginn-3.5B checkpoint across T ∈ {8, 16, 32, 64} (36 → 260 effective layers) shows that while full-depth K = T block speedup saturates flatly near 3.3×, Wake rollback rates drop 21.5× with depth, driving τ = 0 speedups to 8.42× (+153% over K = 64 block) and τ = ∞ to 10.57×.

Huginn-3.5B Across T ∈ {8, 16, 32, 64} (GSM8K, n = 1,319)

Speedup climbs with T as K = T/4 rollbacks fall 12.46 → 0.58 per 100 tokens

12×9×6×3×0× T=8 (36L)rb 12.5 T=16 (68L)rb 6.25 T=32 (132L)rb 2.22 T=64 (260L)rb 0.58 10.57× 8.42× 5.42× 3.34×
• Exact Wake K=T/4 (τ=0) ┄ Exact Wake K=T/2 ■ Lossy K=T/4 (τ=∞) ─ Full K=T block

At shallow T = 8, K = 2 is below the readout horizon (.030 lossy), yet τ = 0 Wake catches every slip to restore .262 (+23.2 pts) while throttling speed gracefully; for T ≥ 16, τ = 0 climbs past the K = T wall up to 8.42× (9.17× at τ = 0.25).

Physical H100 Roofline & Multi-Request Serving (C ∈ [1, 32])

tpass(M, C) = ((t0 + αkvC)p + (βtcM)p)1/p calibrated strictly on K = 4

6.5×4.5×2.5×0.5× C=1C=2C=4 C=8C=16C=32 Concurrent active requests C on single H100 (Ouro-2.6B GSM8K) 1.0× continuous-batching AR parity 2.31× 1.71× 0.77×
• K=2 adaptive B∈{8,4,2} ─ K=3 adaptive B∈{8,4,2} ┄ Fixed K=4, B=8

Skipping T − K passes on rejected draft branches cuts per-step row-passes by 39% (198 vs. 324 at B = 8), sustaining 2.31× decode (1.67× end-to-end with online prefills) at C = 32 via pre-captured B = 8 → 4 → 2 graphs.

Comprehensive Evaluation · Sections 4.1 – 4.4

21,403 Test Examples Across Six Benchmarks and Full-Depth Baselines

Before enabling Wake rollbacks, Sections 4.1–4.4 isolate pure pass-placement speedups (τ = ∞) across all six benchmarks on Ouro-2.6B (against matched trained early exit, LoopSpec, and LoopAhead) and across four capability-active benchmarks on 32-pass Huginn-3.5B.

Ouro-2.6B (T = 4) vs. Full-Depth Speculative Decoders

Normalized by fastest sequential engine per benchmark (62.4–67.2 tok/s; Table 5)

Task (Seq.)LoopSpecLoopAheadOurs K=4Lossy K=3Lossy K=2
GSM8K (.873)2.84×2.94×3.82× .8674.80× .8516.35× .839
IFEval (.732)2.34×2.67×2.12× .7392.75× .7343.68× .699
MATH500 (.532)2.85×2.59×4.11× .5265.07× .5366.63× .492
MBPP (.616)2.86×2.90×3.64× .6264.51× .6185.95× .602
BBH (.577)2.62×3.25×3.29× .5774.35× .5646.33× .549
MMLU-Pro (.504)2.75×2.44×3.56× .5044.56× .5016.12× .488

Huginn-3.5B (T = 32) Middle-Recurrent Scaling

8,871 items at B = 8; late entry drives +74% to +206% over K = 32 block (Table 6)

TaskSeq. T=32K = 32 BlockK = 16 LateK = 8 Late
GSM8K.393 (21.7).397 · 3.37×.405 · 5.84× (+74%).354 · 9.13× (.369 @ 9.93×*)
IFEval.551 (21.6).551 · 2.93×.566 · 5.24× (+79%).542 · 8.96× (+206%)
BBH.283 (22.6).286 · 3.79×.287 · 6.65× (+75%).274 · 10.86× (+186%)
MBPP.234 (22.4).236 · 4.46×.232 · 7.76× (+74%).228 · 12.41× (+178%)

*2-stage partial-cache K = 8 (passes 17→32) reaches 9.93× (+9.7 pts vs. trained early exit at ~220 tok/s); enabling τ = 0 Wake in Table 7 restores full .392 GSM8K accuracy at 6.09× (.397 at 7.15× for τ = 0.25).

Released Hugging Face Checkpoints

Citation

BibTeX

@article{jin2026lateentry,
  title   = {Late Entry in Looped Language Models},
  author  = {Jin, Ziqi and Li, Jiaxi and Xu, Weiwen and Sun, Aixin},
  journal = {arXiv preprint arXiv:2611.24576},
  year    = {2026},
  url     = {https://arxiv.org/abs/2611.24576}
}