Definition
What it means
Pretraining optimizes a model on large and diverse collections of text, code, images, audio, or other data. It builds general capabilities before the model is adapted to specific behaviors or tasks.
Why it matters
Role in the system
It supplies the general knowledge and representations that later training stages refine.
Example
In practice
A language model predicts the next token across a large mixture of web, book, and code data.