Methodology

Overview

Floply builds a bottom-up estimate in five stages: dataset dimensions, model configuration, cluster, experiment plan, and cost rollup. Each stage gates the next, so compute settings only appear once a model and dataset exist.

Nothing here is a benchmark. These are analytical estimates from published scaling relationships and vendor datasheets — useful for deciding whether a project is affordable, not for predicting a specific run to the dollar.

Tokenisation

Token counts and raw storage are derived per modality.

Text

tokens_per_sample = words × 1.3   # BPE subword average
bytes_per_sample  = tokens × 4    # raw UTF-8, ~4 bytes/token

1.3 is the empirical BPE inflation ratio for English.

Image (ViT patch tokenisation)

tokens_per_sample = (resolution ÷ patch_size)²
bytes_per_sample  = resolution² × 3 ÷ 10   # JPEG ~10:1

A 224px image at 16px patches gives 196 tokens. Note the division floors: 336 ÷ 32 is 10 patches per side, not 10.5, so the count is 100 rather than 110.

Audio

Raw storage is always clip_duration × 32,000 bytes (16 kHz mono 16-bit PCM).

Video

num_frames        = clip_duration × sampled_fps
tokens_per_frame  = (resolution ÷ patch_size)²
tokens_per_sample = tokens_per_frame × num_frames
bytes_per_sample  = (resolution² × 3 ÷ 10) × num_frames

Storage is modelled as JPEG-encoded extracted frames, matching how datasets like Kinetics are stored on disk.

Scaling-law thresholds

The health check adapts its thresholds to the training method.

Pre-training

Hoffmann et al. 2022 (Chinchilla): D* ≈ 20 × N .

Full fine-tuning

Starts from a converged base, so far less data is needed. Chinchilla does not apply; the risk is catastrophic forgetting rather than underfitting.

LoRA and QLoRA

The frozen base provides strong priors, so the relevant count is the adapter size , not the full model.

Training FLOPs

Kaplan et al. 2020:

C = 6 × N × D

• C — total floating-point operations
• N — trainable parameters
• D — training tokens (dataset tokens × epochs)
• 6 — forward pass (2N) plus backward pass (4N)

Other architectures use different multipliers: CNN 4, RNN 8, ViT 6, diffusion 6.5. Gradient checkpointing recomputes activations, adding 2 to the multiplier.

Mixture-of-Experts

MoE models route each token through a subset of experts, so FLOPs scale with active parameters while checkpoints and VRAM scale with the total. DeepSeek V3 is costed at its 37B active parameters, not its 671B total.

LoRA parameter counts

For models in the library, trainable parameters are computed per-module from the real architecture, handling Grouped Query Attention correctly.

MODULE_DIMS = {
    "q_proj":    (d_model, d_model),
    "k_proj":    (d_model, kv_dim),   # kv_dim = num_kv_heads × head_dim
    "v_proj":    (d_model, kv_dim),
    "o_proj":    (d_model, d_model),
    "up_proj":   (d_model, ffn_intermediate),
    "down_proj": (ffn_intermediate, d_model),
}

trainable_params = Σ[rank × (in_dim + out_dim)] × num_layers

Custom models fall back to a uniform approximation:

trainable_params ≈ num_modules × 2 × rank × d_model × num_layers

LoRA versus QLoRA

Both train an identical number of parameters. They differ only in how the frozen base is stored.

GPU time and cost

cluster_flops_per_s = peak_flops × MFU × total_GPUs
wall_clock_hours    = total_flops ÷ cluster_flops_per_s ÷ 3600
gpu_hours           = wall_clock_hours × total_GPUs
compute_cost        = wall_clock_hours × hourly_rate × num_instances

Precision multipliers, relative to FP16

FP4 is Blackwell-only (B100/B200/GB200), so none of the instances above accelerate it, and FP8 arrived with Hopper. A precision the GPU cannot accelerate is costed at its FP16 rate — inventing a speedup for absent hardware would understate cost, which is the wrong direction to be wrong in.

Model FLOPs Utilisation

MFU is the share of theoretical peak actually achieved, and it is adjustable because it depends on your setup: architecture and batch size, multi-node communication, data-loading bubbles, and framework overhead. The 30% default is deliberately conservative; well-tuned production LLM training reaches 40–55%.

GPU memory

Weights

QLoRA:  base × 0.5 + adapters × 16   # int4 base
LoRA:   base × 2   + adapters × 16   # bf16 base
Full:   params × 16                  # weights + grads + Adam states

Alignment overhead

DPO holds a frozen reference model alongside the policy ( 2× ). PPO adds a reward model and a critic ( 4× ). The multiplier applies to weight memory before dividing across GPUs.

Activations

without checkpointing: batch × seq_len × d_model × num_layers × 4 × 2
with checkpointing:    batch × seq_len × d_model × 4 × 2

The 4 covers QKV projections, attention scores and MLP intermediates; the 2 is bf16. With checkpointing only one layer of activations is live at a time.

memory_per_GPU_GB = (weight_bytes + activation_bytes) ÷ total_GPUs ÷ 1e9

Storage costs

dataset_TB   = bytes_per_sample × num_samples ÷ 1e12
dataset_cost = price_per_TB_month × dataset_TB × months

Each checkpoint stores trainable parameters in mixed precision, 14 bytes per parameter — bf16 weights plus fp32 optimiser state.

checkpoint_TB = params × 14 × (
      checkpoints_per_run × full_runs
    + hp_trials        # one final checkpoint each
    + ablations        # one final checkpoint each
  ) ÷ 1e12

Sweeps are where this bites: dozens of trials each leaving a final checkpoint adds up faster than the runs themselves.

Experiment rollup

single_run   = wall_clock_hours × hourly_rate × num_instances

total_compute = single_run × full_runs
              + single_run × hp_fraction × hp_trials
              + single_run × ablation_fraction × ablations

total_project = total_compute + dataset_storage + checkpoint_storage

Instance reference

On-demand averages across US regions.

All use NVLink or NVSwitch within a node. Multi-node scaling is modelled as linear, with no communication penalty.

Limitations

Where the model is knowingly simplified. Read this before using an estimate to justify a budget.