SecondSourceJudgment rebuilt from primary sources

According to the paper's abstract, two authors report that pretraining from scratch in NVFP4, NVIDIA's 4-bit format, leaves validation loss only 0.5% above BF16, but the model needs about 1.32 times as many exposures to memorize the same rare facts.

Concept illustration: a sheet of paper taped by its four corners to a light wall, partly in front of a beige panel, with a black bar across its top lettered ARXIV and two short black lines beneath it. On the left half a laptop with a black screen lettered BF16 sits beside a short stack of white papers, and on the right half a laptop whose screen is lettered NVFP4 and NVIDIA with a small green square icon sits beside a taller stack of brick-red papers; above each laptop a rounded speech bubble, its tail pointing down to the laptop, holds a cartoon head-and-shoulders figure of a person with short dark hair and three grey lines. Printed on the image: About 1.32x Exposures to memorize the same rare facts, NVFP4 vs BF16, per two authors' synthetic-biography test.

In a controlled experiment on synthetic biographies, judging FP4 pretraining by the loss curve alone missed the cost of slower learning of rare knowledge. The 1.32 times is for training on synthetic biographies alone; when web text is mixed in, the loss gap is 2% and the factor rises to 1.52 times. The authors' main reading compares the two at equal loss, where the factor is about 1.2 times. The abstract doesn't give the model size (arXiv 2611.12832, 2026-11-09). For comparison, Four Over Six, a method from an MIT and NVIDIA team, changes NVFP4's block scaling and defines success as exactly that: training loss that tracks BF16 more closely; its abstract has no figure for whether models retain facts (arXiv 2512.02010, v5 2026-05-09). How we checked →

Source and checks

This item ran in the brief of Nov 17, 2026, alongside 13 other items. How we checked it, the quote checks and what was still missing that day are on that issue's How we checked page.

Cite this item

SecondSource (2026). "According to the paper's abstract, two authors report that pretraining from scratch in NVFP4, NVIDIA's 4-bit format, leaves validation loss only 0.5% above BF16, but the model needs about 1.32 times as many exposures to memorize the same rare facts.." https://secondsource.io/en/research/2026-11-17-1/. Accessed access date.