Research & models Oct 11, 2026

This week · paper submitted October 8

Internalizer bakes text into a 284B LLM; it's self-reported

The Internalizer preprint turns a document, in one pass, into a small set of add-on weights for a 284-billion-parameter model (DeepSeek V4 Flash); earlier methods of this kind went no larger than 14 billion. A small model built to generate weights reads the document and outputs a LoRA: an add-on layer that changes only a small number of parameters and hangs outside the original model, leaving the original model untouched. The abstract says the test documents were up to 4,096 tokens long and unseen by the model, and the model's input contained only a three-word instruction, not the document. A token is the chunk of text, a few characters long, that AI models count in, and usage is billed per token. The measurement: given the correct preceding text, have the model rank the tokens it thinks most likely to come next, and check whether the right answer comes first or in the top five. With the add-on layer, it came first 84.9% of the time and was in the top five 97.8% of the time; the same original model, also without the document in its input, scored 63.4% and 83.5% (arXiv 2610.11715). The abstract doesn't compare it with simply putting the document in the input. The paper says the scale is two orders of magnitude beyond earlier work; by our own arithmetic, 14 billion against 284 billion is only about 20 times, and the abstract doesn't explain the gap. Today, to make a model remember a document, you either put the document into the input every time or build a separate retrieval system. Our reading: once the add-on layer exists, you don't have to send the whole document again with every question; if generating a layer is cheap and fast enough, one layer per customer or per document becomes an option. What to watch: whether the authors release code and weights, whether anyone else reproduces the result, and what it costs, in money and time, to generate a layer. ⚠️ The numbers are the paper's own, and we read only the abstract; predicting the next token correctly is not the same as answering questions correctly, only short documents were tested, and the abstract gives no cost or latency for generating a layer.