Looking back · SecondSource call, October 25

Shrinking the data AI models hold during a conversation hasn't cut AI chips' memory demand; SecondSource's reading is that the demand has moved.

Concept illustration: a dark-haired person seen from behind at the lower right holds up with both hands a large light-grey board with a white label lettered SECONDSOURCE at its top and two side-by-side diagrams below, each showing, on one baseline from left to right, a black block, a low beige strip, a tall four-shelf beige tower and a grey cabinet. In the left diagram an HBM label has a line to the top of the tower, the strip and every shelf hold large red blocks with one more red block on top of the tower, and the cabinet's two slots are plain grey; in the right diagram a KV CACHE label has a line to the strip, which holds many thin vertical red bars, every shelf is filled with stacks of thin horizontal red bars, and the cabinet's two slots also hold thin red bars. Printed on the image: 288 GB Current Rubin's HBM capacity per GPU.

Our inference: the demand has shifted from how much high-bandwidth memory next to the GPU can hold to how much data it can move per second, and to memory off the GPU. HBM is the high-bandwidth memory stacked beside the GPU. The KV cache is the intermediate data a model holds during a conversation, and KV compression shrinks it. The reasoning in the October 25 deep dive was that once that cache shrinks, one GPU can serve more conversations at once. All of those conversations read from the same memory stack, so per-second data movement tightens first. Whatever doesn't fit moves off the GPU (card). The test is one spec: SecondSource expects the officially published HBM capacity per GPU of NVIDIA's next-generation Rubin Ultra to be no higher than the current Rubin's 288 GB. Industry reports say NVIDIA is weighing fewer HBM stacks for Rubin Ultra, and the final spec hasn't been published. If NVIDIA publishes a capacity above 288 GB before March 31, 2027, this call has to be rewritten; that deadline is one SecondSource set. How we checked →

Source and checks

This item ran in the brief of Nov 4, 2026, alongside 10 other items. How we checked it, the quote checks and what was still missing that day are on that issue's How we checked page.

Cite this item

SecondSource (2026). "Shrinking the data AI models hold during a conversation hasn't cut AI chips' memory demand; SecondSource's reading is that the demand has moved.." https://secondsource.io/en/news/2026-11-04-7/. Accessed access date.