Estimate what it costs to train an ML model.

Cached prices · updated just now

How much data do I need?

Pick a model to see how much training data it needs, from too little to more than enough.

7,000,000,000

Data needed for pre-training

For a 7.00B-parameter model. Aim for the Optimal range.

1.00B10.00B100.00B1.00TToo littleToo little — Under 1 token per parameter. Fewer tokens than parameters. Training will likely fail.700.00M–7.00BBare minimumBare minimum — 1–10 tokens per parameter. Trains, but the model will be weak.7.00B–70.00BOptimalOptimal — 10–30 tokens per parameter. The best model for the compute you spend.70.00B–210.00BExtendedExtended — 30–200 tokens per parameter. Costs more to train, but gives a smaller model that is cheaper to run.210.00B–1.40T
Each bar shows the token range for one level, based on 7.00B parameters.

In examples

The same ranges as a number of texts, images or clips.

About 1.3 tokens per word.

520 tokens per text example.

Too little
< 13.46M

700.00M–7.00B tokens

Bare minimum
13.46M–134.62M

7.00B–70.00B tokens

Optimal
134.62M–403.85M

70.00B–210.00B tokens

Extended
403.85M–2.69B

210.00B–1.40T tokens

Text examples needed for each range.