Data Preparation for Large Language Models

Hao Liang

doi:10.1007/s11390-026-5948-8

Data Preparation for Large Language Models

2026-03-01

0

PRE

AI

DOI：10.1007/s11390-026-5948-8

Original

Original request for help

Save

Abstract

En 中文

Large language models (LLMs) have demonstrated remarkable generalization capabilities across diverse domains, largely attributed to the availability of massive amounts of high-quality training data. Recently, the development paradigm of LLMs has been shifting from a model-centric to a data-centric perspective. In this paper, we provide a comprehensive survey of data preparation algorithms and workflows for LLMs, categorized into three stages: pre-training, continual pre-training, and post-training. We further summarize widely used datasets along with their associated data preparation method, offering a practical reference for researchers who may lack extensive experience in the field of data preparation. Finally, we outline potential directions for future work, highlighting open challenges and opportunities in advancing data preparation for LLMs.

Keywords:

data-centric artificial intelligence (AI)

data management

large language model (LLM)

Data Preparation for Large Language Models

Abstract

Journal

Organization