ProVe: Proposer-Verifier Decoding

Two model calls per decoding cycle, spent three ways. Blue is the large model, orange the small one.

2.5×ProVe-losslessAverage over six benchmarks with a 32B proposer. Output identical to greedy AR. No draft head to train.

2.7×ProVe with a 3B verifierSame six benchmarks, 2.6 accuracy points below AR. A 7B verifier gives 2.3× within 0.3 points.

6.6 vs 3.0Tokens per blockOn GSM8K, 6.6 of the 16 proposed tokens match greedy AR on average. The proposer's own confidence commits only 3.0.

How it works

One cycle, with real tokens

The proposer sees the context plus 16 placeholders and fills all of them in one forward pass. A verifier then reads the whole block in one more pass and scores every token after seeing the tokens before it. The longest run that passes the gate is committed.

position	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
context ends …So in total it+ 16 placeholders	[M]	[M]	[M]	[M]	[M]	[M]	[M]	[M]	[M]	[M]	[M]	[M]	[M]	[M]	[M]	[M]
proposed blockproposer · one forward pass	takes	␣	2	+	␣	1	=	␣	3	bolts	.	The	:	answer	␣	3
confidence p̃from the same pass	.92	.98	.95	.93	.95	.74	.96	.97	.94	.99	.80	.19	.29	.31	.50	.40
verifier q3B model · one more pass	.95	1.0	1.0	.77	1.0	1.0	1.0	1.0	1.0	1.0	.12	.01	.00	.00	1.0	1.0
p̃ · q ≥ 0.10commit the leading run	✓.87	✓.98	✓.94	✓.72	✓.95	✓.74	✓.96	✓.97	✓.94	✓.99	✗.098	–	–	–	–	–

Commit 10 tokens, takes 2 + 1 = 3 bolts. Position 11 fails (q = 0.12), so the next cycle starts there. Deeper positions were already guesses.

ProVe

A smaller stock AR model verifies. Gate on p̃ · q ≥ θ: the proposer's confidence times the verifier's probability. Lower θ commits longer runs and trades a little accuracy for speed.

lossy · tunable · one extra small pass

ProVe-lossless

The proposer verifies itself. Its second pass yields the greedy next token after every prefix; at the first disagreement that token is committed instead. Every committed token lies on the greedy AR path.

exact · no threshold · no second model

Why verify

Confidence cannot see the block it judges

The proposer's confidence is computed from placeholders, before the proposed tokens exist. It is free, but it leaves most of the usable block on the table. A verifier reads the realized tokens, and one teacher-forced pass scores all 16.

6.6tokens match greedy AR, on average

decreasingly reliable guesses

3.0committed by the confidence rule (τ = 0.95)

A proposed block of 16 along GSM8K greedy continuations. Lowering τ commits more tokens, but also commits tokens that leave the AR path; the verifier separates the two better (AUC 0.960 vs 0.910; 0.964 for their product).

96%1

86%

79%

71%4

67%

59%

53%

51%8

47%

44%

39%

34%12

34%

30%

28%

27%16

Proposal quality decays with depth: how often position i of the block equals the greedy token (437 blocks, 64 GSM8K continuations). Position 1 is the ordinary next-token prediction.

Cost

The small model runs once per block, not once per token

Count one proposer pass as 1 and a small-model pass as ρ (0.34 for a 3B next to a 32B). Speculative decoding pays ρ for every drafted token. ProVe pays it once.

Speculative decoding, k = 5

1ρρρρρ

1 + kρ = 2.70

ProVe

1ρ

1 + ρ = 1.34

ProVe-lossless

2 = 2.00

Per-cycle cost in proposer-pass units; the speedup is tokens committed per cycle divided by this. Most of a verifier pass is fixed per-call overhead, so a 7B verifier costs little more than a 3B.

Results

Six benchmarks, one 32B proposer

HumanEval, GSM8K, GPQA, BBH, MMLU-Pro and IFEval. Greedy decoding, one request in flight, up to 1024 new tokens. Throughput is relative to greedy AR decoding of the same model on the same hardware.

Average throughput

Exact · output equals AR

ProVe-lossless2.54×

EAGLE-3, head fine-tuned for this proposer tree of 602.58×

EAGLE-3, public head tree of 602.32×

Speculative decoding 3B draft, k = 51.63×

AR1.00×

Lossy · Δ accuracy in points

ProVe 3B verifier, θ = 0.012.72×−2.6

ProVe 7B verifier, θ = 0.052.30×−0.3

ProVe 3B verifier, θ = 0.102.28×−1.1

Confidence rule MARS, τ = 0.95, no verifier1.48×−0.9

ProVe-lossless by benchmark

BBH3.98×

GSM8K2.86×

HumanEval2.42×

MMLU-Pro2.37×

GPQA2.16×

IFEval1.43×

Speedup follows the continuation: templated chains of thought on BBH reach 4.0×, short constrained answers on IFEval 1.4×.

A reasoning proposer

Qwen3-32B in thinking mode as proposer; stock Qwen3-0.6B as verifier and as the speculative draft.

Configuration	GSM8K	MATH-500
Speculative decoding, k = 5	2.06×	2.28×
ProVe-lossless	2.47×	2.79×
ProVe, θ = 0.10	2.34× −0.4	3.16× −0.2

1.9×→5.5×

ProVe-lossless speedup across the 27 BBH subtasks, from the shortest responses to the longest.

4.2→8.9

Tokens per block that match greedy AR, from the start to the end of a GSM8K response. Later text is more predictable, so more of each block survives.

Recipe

What it takes

Proposer: Qwen2.5-32B-Instruct, fine-tuned with MARS: one epoch of ordinary SFT, then one epoch of block training with 16 placeholders, both on Dolci. Same transformer, same output head, no extra module. It costs 0.6 points of ordinary AR accuracy.
Verifier: Any stock Qwen2.5-Instruct model that shares the tokenizer (3B or 7B), or the proposer itself for ProVe-lossless. Fine-tuning the verifier on the proposer's outputs commits 6–10% more tokens per cycle at the same error rate.
Serving: One SGLang-based engine. Each cycle is two passes. Tokens that are not committed are dropped from the KV cache, so the cache after a cycle equals what AR decoding would have produced.

Citation

BibTeX

@article{jin2026prove,
  title   = {ProVe: Proposer-Verifier Decoding},
  author  = {Jin, Ziqi and Xu, Weiwen and Li, Jiaxi and Zhong, Yujie and She, Qi and Sun, Aixin},
  journal = {arXiv preprint arXiv:2610.27815},
  year    = {2026}
}