Research & models Nov 17, 2026
According to the paper's abstract, two authors report that pretraining from scratch in NVFP4, NVIDIA's 4-bit format, leaves validation loss only 0.5% above BF16, but the model needs about 1.32 times as many exposures to memorize the same rare facts.

In a controlled experiment on synthetic biographies, judging FP4 pretraining by the loss curve alone missed the cost of slower learning of rare knowledge. The 1.32 times is for training on synthetic biographies alone; when web text is mixed in, the loss gap is 2% and the factor rises to 1.52 times. The authors' main reading compares the two at equal loss, where the factor is about 1.2 times. The abstract doesn't give the model size (arXiv 2611.12832, 2026-11-09). For comparison, Four Over Six, a method from an MIT and NVIDIA team, changes NVFP4's block scaling and defines success as exactly that: training loss that tracks BF16 more closely; its abstract has no figure for whether models retain facts (arXiv 2512.02010, v5 2026-05-09). How we checked →
Source and checks
This item ran in the brief of Nov 17, 2026, alongside 13 other items. How we checked it, the quote checks and what was still missing that day are on that issue's How we checked page.
Cite this item
SecondSource (2026). "According to the paper's abstract, two authors report that pretraining from scratch in NVFP4, NVIDIA's 4-bit format, leaves validation loss only 0.5% above BF16, but the model needs about 1.32 times as many exposures to memorize the same rare facts.." https://secondsource.io/en/research/2026-11-17-1/. Accessed access date.