Research & models Oct 19, 2026

Trend · preprint October 15

All 14 LLM deployments vary on rerun; it's not peer-reviewed

A single-author preprint tests 14 commercial LLM deployments on compliance rulings for two kinds of medical-device documents: with each vendor's deterministic settings on and the same request resent, all 14 changed at least one result, and 12 flipped even the "compliant / non-compliant" label. The 14 deployments span 3 vendors, 7 models and 6 routes; the abstract doesn't name them (arXiv 2610.21300, 2026-10-15). The abstract lists neither how many questions each deployment was tested on nor how many flipped, and doesn't define what a route is. The author reports that lowering the models' reasoning effort didn't help. In the author's view, whether the same open-weight model reproduces depends only on how it is served. On a self-hosted pipeline, the author pinned each stage in turn: document parsing, chunking, embedding, indexing, retrieval and inference. With all of them pinned, reruns on other machines and other days produced bit-identical output. The author argues that reproducibility is a property of the serving layer, so audit records should include the GPU model, the inference engine's scheduling and the retrieval service. Only the abstract is available. The preprint has one author and hasn't been peer-reviewed.