Research & models Oct 11, 2026

This week · paper, October

Paper finds rulings may lag new rules; it's one preprint

The preprint "Verdict Without the Rule" tests 5 models across 20 regulatory and platform-policy areas and finds that when the rules are substantially rewritten, a model's rulings may not change to match. A guard model built for content moderation (the abstract doesn't name it) had 51% accuracy on an evaluation that rewrites its original classification scheme as custom rules, which the abstract calls close to chance. General-purpose models scored 90% to 92% on the same evaluation, but the guard model was tested in a different way, so this is not a straight ranking (arXiv 2610.12313). The paper's claim is that high accuracy can't prove a model is actually ruling by the rules you gave it. ⚠️ We read only the abstract of this one too, and it gives no sample sizes per area. For engineers who own compliance and content-moderation systems, here's one more acceptance test: change one rule and see whether the rulings change with it.