Research & models Oct 25, 2026

Trend · preprint, October 19

Claude Code mostly mislabels kernel bugs; it's self-reported

A preprint by Haoran Xue and three co-authors finds that Claude Code, fitted with off-the-shelf security skills to hunt real Linux kernel vulnerabilities, classified the vulnerability type correctly in only 15.7% of runs. A security skill is a written procedure and toolset that tells an agent how to hunt for vulnerabilities. The study ran Claude Code with four underlying models and 37 security skills on 100 real Linux kernel vulnerabilities, 14,800 runs in all (arXiv 2610.25705, 2026-10-19). Across those runs, 24.4% pointed to the right function and 15.7% got the vulnerability type right. The authors' own method, SKILL-SIF, rewrites skills automatically. With it, aggregate accuracy across the four models rose from 6.05% to 11.20%. The abstract computes that aggregate differently from the 24.4% and 15.7% and doesn't say how, so the figures can't be compared directly. These results are the authors' own, and the readings come from the abstract. The abstract doesn't name the four underlying models. Our read: these readings measure off-the-shelf skills with no tuning. In that setup, installing a security skill alone doesn't yet make an agent reliable at finding kernel vulnerabilities.