Research & models Oct 24, 2026

Trend · preprint October 22

Barron flags softmax gradient errors; no figures are given

A preprint by Google researcher Jonathan Barron and two co-authors says standard softmax and cross-entropy implementations have two gradient errors, and says that with the fix, student models in distillation no longer drift away from the teacher model. Softmax and cross-entropy are the output layer and loss function most commonly used in training classifiers and language models. Distillation trains a smaller student model to imitate a larger teacher model. The abstract names two errors. The first, which the abstract calls Softmax Collapse: when one output value is far larger than the rest, the gradient for the correct answer is rounded to 0. The second, Softmax Cancellation: when the student model's output is close to the teacher model's, equality should leave no adjustment signal, yet standard implementations still produce a nonzero gradient (arXiv 2610.31642, 2026-10-22). The abstract says the errors are in "standard implementations" but gives no quantified figures and doesn't say what the effect is on large language model pretraining; the full paper wasn't read. Our reading: the work most directly affected is research on distillation and long training runs, rather than model training in general. Whether mainstream frameworks adopt the fix depends on their coming releases.