Yonggan Fu

3 weeks ago | research.nvidia.com | Yu Yang |Yonggan Fu |Xin Dong |Dan Su

Abstract Pre-training datasets are typically collected from web content and lack inherent domain divisions. For instance, widely used datasets like Common Crawl do not include explicit domain labels, while manually curating labeled datasets such as The Pile is labor-intensive. Consequently, identifying an optimal pre-training data mixture remains a challenging problem, despite its significant benefits for pre-training performance.