Estimate what it costs to train an ML model.

Cached prices · updated just now

How it works

The formulas behind every estimate, and what they assume. Prices come live from AWS for the region you pick.

Overview

Floply builds an estimate in five steps: dataset, model, hardware, experiments, then total cost. Each step needs the one before it.

These are calculations, not benchmarks. Use them to judge whether a project is affordable, not to predict a run to the dollar.

Tokens per sample

Each data type is converted into tokens and an estimated file size.

Text

tokens_per_sample = words × 1.3   # average for English
bytes_per_sample  = tokens × 4    # plain text, ~4 bytes per token

Image

tokens_per_sample = (resolution ÷ patch_size)²
bytes_per_sample  = resolution² × 3 ÷ 10   # JPEG, ~10:1 compression

A 224px image with 16px patches is 196 tokens. Partial patches don't count: at 336px with 32px patches, that's 10 per side, so 100 tokens, not 110.

Audio

TokenizerRate
Whisper50 tokens/sec
EnCodec 24 kHz75 tokens/sec per codebook
SoundStream50 tokens/sec per codebook

File size is always clip_duration × 32,000 bytes (16 kHz, mono, 16-bit).

Video

num_frames        = clip_duration × sampled_fps
tokens_per_frame  = (resolution ÷ patch_size)²
tokens_per_sample = tokens_per_frame × num_frames
bytes_per_sample  = (resolution² × 3 ÷ 10) × num_frames

Video is stored as JPEG frames, the way most video datasets are kept on disk.

How much data is enough

The health check compares your dataset to the number of parameters being trained. The ranges depend on how you train.

Pre-training

The ideal is about 20 tokens per parameter .

Tokens per parameterWhat it means
Under 1Far too little. Training will likely fail.
1–10Not enough. The model will be weak.
10–30Optimal. The best model for the compute spent.
30–200More than optimal. Costs more to train, but the smaller model is cheaper to run.
Over 200More than the model can use. Consider a bigger model.

Full fine-tuning

The model has already learned a lot, so it needs much less data. Too much data can make it forget what it knew.

Tokens per parameterWhat it means
Under 0.1Too little to change the model.
0.1–1Probably not enough for stable results.
1–5Optimal. The typical range.
5–20Plenty. Watch that the model doesn't forget what it knew.
Over 20Too much. Use LoRA instead.

LoRA and QLoRA

LoRA freezes the model and trains small add-on layers called adapters. The ranges are based on adapter parameters , not the full model.

Tokens per adapter parameterWhat it means
Under 10Possibly too little.
10–100Optimal. The typical range.
100–1,000Plenty. A higher rank would help.
Over 1,000More than the adapter can use. Consider full fine-tuning.

Training compute

Total compute is measured in floating-point operations (FLOPs):

C = 6 × N × D
  • C — total FLOPs
  • N — parameters being trained
  • D — training tokens (dataset tokens × epochs)
  • 6 — 2 for the forward pass, 4 for the backward pass

Other architectures use a different multiplier: CNN 4, RNN 8, ViT 6, diffusion 6.5. Gradient checkpointing adds 2.

Mixture-of-experts models

These models only use some of their parameters for each token. Compute is based on the active parameters; memory and checkpoints use the total. DeepSeek V3 is costed at 37B active parameters, not 671B.

LoRA parameters

For models in the list, adapter size is calculated layer by layer from the real architecture.

MODULE_DIMS = {
    "q_proj":    (d_model, q_dim),    # q_dim  = num_heads × head_dim
    "k_proj":    (d_model, kv_dim),   # kv_dim = num_kv_heads × head_dim
    "v_proj":    (d_model, kv_dim),
    "o_proj":    (q_dim, d_model),
    "up_proj":   (d_model, ffn_intermediate),
    "down_proj": (ffn_intermediate, d_model),
}

trainable_params = Σ[rank × (in_dim + out_dim)] × num_layers

Custom models use an approximation:

trainable_params ≈ num_modules × 2 × rank × d_model × num_layers

LoRA vs. QLoRA

Both train the same number of parameters. QLoRA stores the frozen model in 4-bit, so it uses less memory.

MethodFrozen model stored asMemory (bytes)
Full fine-tuningfp32/bf16params × 16
LoRAbf16base × 2 + adapters × 16
QLoRAint4base × 0.5 + adapters × 16

Time and cost

cluster_flops_per_s = peak_flops × utilisation × total_GPUs
wall_clock_hours    = total_flops ÷ cluster_flops_per_s ÷ 3600
gpu_hours           = wall_clock_hours × total_GPUs
compute_cost        = wall_clock_hours × hourly_rate × num_instances

Speed by precision, compared to FP16

PrecisionV100A100H100 / H200B200
fp4———4×
fp8——2×2×
int80.5×2×2×2×
bf16—1×1×1×
fp161×1×1×1×
tf32—0.5×0.5×0.5×
fp32uses the GPU's listed fp32 speed

A dash means the GPU has no hardware support for that format, so it isn't offered for that instance.

GPU utilisation

No training run reaches a GPU's peak speed. Utilisation is the share you actually get, and it depends on your model, batch size, networking and code. The 30% default is cautious; well-tuned large runs reach 40–55%.

GPU memory

Model

QLoRA:  base × 0.5 + adapters × 16   # 4-bit base
LoRA:   base × 2   + adapters × 16   # bf16 base
Full:   params × 16                  # weights, gradients and optimizer state

DPO and PPO

DPO keeps a second copy of the model for reference ( 2× model memory). PPO adds a reward model and a critic ( 4× ).

Activations

without checkpointing: batch × seq_len × d_model × num_layers × 4 × 2
with checkpointing:    batch × seq_len × d_model × 4 × 2

The 4 covers the main intermediate results in each layer; the 2 is bytes per value in bf16. With gradient checkpointing, only one layer is held in memory at a time.

memory_per_GPU_GB = (model_bytes + activation_bytes) ÷ total_GPUs ÷ 1e9

Storage

dataset_TB   = bytes_per_sample × num_samples ÷ 1e12
dataset_cost = price_per_TB_month × dataset_TB × months

Each checkpoint saves 14 bytes per trained parameter.

checkpoint_TB = params × 14 × (
      checkpoints_per_run × full_runs
    + tuning_runs      # one checkpoint each
    + ablation_runs    # one checkpoint each
  ) ÷ 1e12

Tuning runs add up fast: dozens of runs, each leaving a checkpoint, can outweigh the main runs.

Total project cost

single_run    = wall_clock_hours × hourly_rate × num_instances

total_compute = single_run × full_runs
              + single_run × tuning_run_length × tuning_runs
              + single_run × ablation_run_length × ablation_runs

total_project = total_compute + dataset_storage + checkpoint_storage

Instances

On-demand prices for the selected region.

InstanceGPUGPUsMemoryFP16Cost/hr
p6-b200.48xlargeB2008179 GB2250 TFLOPS$113.93
p5.48xlargeH100880 GB989 TFLOPS$55.04
p5en.48xlargeH2008141 GB989 TFLOPS$63.30
p4d.24xlargeA100840 GB312 TFLOPS$21.96
p4de.24xlargeA100880 GB312 TFLOPS$27.45
p3dn.24xlargeV100832 GB125 TFLOPS$31.21

Adding more GPUs or instances is assumed to speed training up perfectly, with no networking slowdown.

Limitations

Where the estimate is simplified. Read this before using a number to justify a budget.

AreaAssumption
ScalingMore GPUs always means proportionally faster. No networking overhead.
PricingOn-demand only. Spot can be 60–90% cheaper, reserved 30–40%.
CloudAWS only. Google Cloud, Azure and others aren't included.
OptimizerAdam, at 16 bytes per parameter. Others may use less memory.
StartupNo time for setup, data loading or compilation.
Data loadingAssumed never to slow training. If it does, utilisation drops.
PrecisionSpeedups are best-case. Real code rarely reaches them.
LoRAExact for listed models; custom models use an approximation.
StorageS3 only. No other file systems or local disks.