LSTM 翻红：新架构在复杂任务上反超 Mamba
当所有人都在追逐 Transformer 替代品时，一篇新论文给了一个反直觉的结论：老架构 LSTM 的升级版 xLSTM，在代码预训练、大模型蒸馏和时间序列建模上，全面击败了更受关注的 Mamba-2 和 Gated DeltaNet。研究者发现，xLSTM 的优势来自其门控机制带来的更灵活、稳定的记忆修正能力——它不像 Mamba 那样依赖固定状态，而是能动态调整哪些信息该记住、哪些该忘。在需要长程依赖的复杂任务上，这种能力至关重要。这不是一篇教你明天就能用的论文，但它提醒我们：在架构竞赛中，经典设计的潜力可能被低估了。
📄 原文摘要(英文)
Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. However, it remains unclear which designs yield the most effective sequence models. We compare three leading approaches: xLSTM, Mamba-2, and Gated DeltaNet. We evaluate these models on tasks with complex dependencies: (1) code-model pre-training, (2) distillation of code models from large language models, and (3) pre-training of time-series foundation models. Across these settings, xLSTM delivers the strongest overall performance. To explain xLSTM's advantage, we present a unified formulation and analyze the underlying architectural mechanisms, focusing on state tracking and memory dynamics. Our results show that xLSTM enables more flexible and stable memory correction via its gating scheme. We corroborate these findings on controlled synthetic length-generalization tasks. Overall, our findings indicate that xLSTM's gains on complex tasks stem from robust state tracking and accumulation.