QuRating: Selecting High-Quality Data for Training Language Models

1 code implementation • 15 Feb 2024 • Alexander Wettig, Aatmik Gupta, Saumya Malik, Danqi Chen

Selecting high-quality pre-training data is important for creating capable language models, but existing methods rely on simple heuristics.

In-Context Learning