训练大模型，数据混合比过滤更重要
训练视觉语言模型（如能看图说话的AI）时，大家总在纠结怎么筛选数据——去掉低质量、重复的。但这篇大规模实验告诉你：关键不是过滤，而是怎么把不同类型的数据混在一起。研究者建了一个包含160个数据集、6万亿token的基准，让不同模型在固定预算下测试各种策略。结果发现，指令数据（比如问答、对话）占比高的混合方案，比纯图片-文字描述的效果好得多，而且模型越大优势越明显。他们用这个策略训练出的8B模型，在33项任务上平均准确率63.6%，比目前最好的开源数据集高出5.4个百分点。这不是你明天能直接用的技巧，但它给所有做AI训练的人指了个方向：别只盯着数据干净不干净，多想想怎么配比。
📄 原文摘要(英文)
Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we collect 160 datasets spanning four data types -- image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data -- into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B-8B models and 6.25B-200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-Baseline, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FineVision, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available at https://www.datacomp.ai/dcvlm/.