Hierarchical Reasoning Models with Iterative Revision

Question

Large models get better mostly by getting bigger. Can a small model trade depth for repeated computation, running the same blocks several times, and what does it gain or lose in the exchange?

Models

• HR-LLM, 288.9M parameters: a hierarchical model built on an anchored dual-module architecture.
• RSN (Recursive Small Network), 580.2M parameters: a recursive fusion architecture inspired by recent work on tiny reasoning networks.

Both were pre-trained from scratch on FineWeb-Edu, about 9.4B tokens for HR-LLM and 10.5B for RSN, with gradient accumulation and deep supervision. The baseline is NanoChat, a standard transformer of comparable size.

Results

• RSN scored higher than NanoChat on reasoning benchmarks (Winogrande, PIQA, ARC-Challenge) and lower on knowledge retrieval. Recursion traded stored knowledge for reasoning.
• Fine-tuned on Banking77, a 77-class intent classification task, RSN reached 88.2% accuracy, competitive for a decoder-only model of this size.

Written up as Hierarchical Reasoning Language Models with Iterative Revision.