AI学会双向思考：新模型不按顺序也能写文章
主流大模型（如GPT）都是按顺序一个字一个字生成，像从左到右读句子。但新模型iLLaDA打破了这一规则：它用“双向注意力”同时看到所有位置，像拼图一样随机遮住部分词，再根据上下文猜出被遮的词。这种训练方式让它在数学推理（MATH提升14.5分）和代码生成（HumanEval提升16.5分）上大幅超越同类模型，甚至与Qwen2.5 7B不相上下。虽然它不是你明天就能用的工具，但证明了“不按顺序思考”的AI也能很强，这可能会改变未来模型的设计方向。
📄 原文摘要(英文)
Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present iLLaDA, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.