把少数几层反复用，效果追平12层大模型
现在的语言模型想变聪明，习惯性靠堆层数，但每一层都得单独存一份参数，内存开销巨大。这篇反过来：只保留开头、结尾两组固定层，中间只留一个共享核心，让它被反复迭代使用——关键在于每次迭代都加一个受隐藏状态、固定层输出和随机噪声共同控制的门，决定这一轮该强化哪些信息。结果是，3层反复迭代的模型匹配了12层GPT-2小模型的精度，FLOPs相当；在同样参数预算下，循环越深，验证损失越低(2.76对2.84)。它不是你明天能拿来改自己项目的，但提示了一个方向：模型的深度不一定要物理堆出来，动态复用共享层是省内存换质量的有效策略，大规模下只需37%的参数就能逼近密集模型质量。
📄 原文摘要(英文)
Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality. We introduce Gated Recurrent Transformer, a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times. Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate---conditioned on the hidden state, the fixed prelude output, and noise resampled at every step---to modulate the recurrent update. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity. Under an isoFLOPS constraint, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs, and leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled. Under an isoPARAMS constraint, deeper recurrence achieves a 2.76 validation loss versus 2.84 for a non-recurrent counterpart at matched parameter and data budget. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality: at large scale, 63% fewer parameters and 59% less peak decoding memory for a 10% increase in compiled generation latency.