Multi-byte decoding for tokenizer-free language models

Problem

Tokenizer-free language models operate directly on bytes rather than subword tokens. Models such as H-Net and BLT use variable-length chunks to reduce the cost of processing long sequences, but generation remains sequential: one byte at a time. This makes inference significantly slower than in conventional language models.

Method

Speculative decoding: a cheap drafter proposes several bytes, the frozen model verifies them in one pass, and the longest prefix that agrees with greedy decoding is kept. The main contributions are:

• Drafting head. A one-layer transformer (about 15M parameters) that drafts several bytes in a single denoising step. It is distilled from the frozen model and then fine-tuned on the model’s own greedy outputs, with greater weight assigned to the first drafted bytes, since acceptance stops at the first error.
• Incremental verification cache. Exact cached decoding for both models, particularly for parallel byte drafting.
• Gated verification. The main network runs only for chunks whose opening byte has been accepted. Drafts that fail early never reach it.

Every speedup is measured in wall-clock time against each model’s own cached greedy decoder.

Results

End-to-end wall-clock time, batch size 1, bf16, four drafted bytes, on a single NVIDIA GB10:

Model	Greedy, ms per byte	Ours, ms per byte	Speedup
H-Net (679M)	4.25	3.30	1.29×
BLT-1b	9.06	6.77	1.34×

In fp32, the output is bit-identical to greedy decoding. Without the gating mechanism, the same drafts would make our approach slower than the baseline (~0.88×), because the main network runs 0.441 times per generated byte instead of 0.185.