bf16 vs fp16: range or precision in the same 16 bits

fp16 and bf16 both store a number in 16 bits: 1 sign bit, exponent bits that set how large or small it can be, and mantissa bits that set how finely it is rounded. fp16 spends 5 bits on the exponent and 10 on the mantissa. bf16 spends 8 and 7, the same exponent as 32-bit fp32.

2 min read · Lensa ML
  1. The top chart shows each format's range, from its smallest value above zero to its largest. fp16 runs from about 6e-8 to 65,504, while bf16 covers fp32's range, about 9e-41 to 3.4e38. The amber band is 400 sample gradients, the slopes training uses to update weights, spread around the slider value with a standard deviation of one power of ten. The samples are generated, not taken from a real model.
  2. As the band slides below 1e-5, fp16 starts rounding gradients to zero: 27% at 1e-7, 68% at 1e-8 and all of them at 1e-12. bf16 rounds none to zero across the whole sweep, and its mean relative error stays near 0.14%. This is why fp16 training needs loss scaling, multiplying the loss by a large power of 2 so gradients land back in range, while bf16 training usually does not.
  3. At the top of the sweep the trade reverses. Centered at 1e-3, 92% of the gradients are normal fp16 numbers, and fp16's 3 extra mantissa bits give a mean error of 0.025% against bf16's 0.13%. The lower chart shows why: between 1 and 1.0625, fp16 has 64 steps of 2^-10 and bf16 has 8 steps of 2^-7. On GPUs that support both, such as the A100 and H100, they run matrix math at the same speed.

Takeaway

For training, pick bf16 when the hardware has it, since it keeps small gradients without loss scaling. fp16 is more precise for values inside its range, and it remains the choice on older GPUs such as the V100 and T4, which have no bf16 tensor cores.

Go deeper · interactive

Mixed Precision Training

On step 5, drag the typical gradient size and watch fp16 lose small gradients while bf16 keeps them.

Open the explainer

References