How much data do I need?
Pick a model to see how much training data it needs, from too little to more than enough.
Data needed for pre-training
For a 7.00B-parameter model. Aim for
the Optimal range.
Each bar shows the token range for one level, based on
7.00B parameters.
In examples
The same ranges as a number of texts, images or clips.
520 tokens per text example.
- Too little
- < 13.46M
700.00M–7.00B tokens
- Bare minimum
- 13.46M–134.62M
7.00B–70.00B tokens
- Optimal
- 134.62M–403.85M
70.00B–210.00B tokens
- Extended
- 403.85M–2.69B
210.00B–1.40T tokens
Text examples needed for each range.