Open-ended tasks: under 2 teams will replicate synthetic-data students beating teachers — resolves 2027-07-14

Synthetic data is material generated by an AI and used to train another model. The model generating the data is the teacher; the model learning from it is the student. This card concerns writing and open-ended reasoning, where judging correctness is less straightforward than checking a calculation or executing code.

The question is whether a student trained this way can outperform its teacher on a named benchmark, with independent teams reproducing the result.

Through 14 July 2027, SecondSource expects fewer than two independent teams to reproduce a student model trained on synthetic data outperforming its teacher on a named benchmark in these domains. The estimated probability is 50%.

When this forecast was recorded, SecondSource had not verified whether such reproductions already existed. This is a forward-looking count to be checked at the deadline, not a claim that no example exists today.

The 50% estimate gives equal odds to the event and its opposite. “Ceiling” below means a limit to learning from synthetic data in these domains. The original rule excludes internal comparisons within the same paper at the same institution, but does not fully define other independence boundaries, eligible benchmarks, or whether examples predating this card count. These require clarification at assessment. If new benchmarks remove the agreed boundary between verifiable and unverifiable domains, the original rule voids the forecast.

The original material behind this call has not yet been linked here in a form readers can open and check.

Check published papers and technical reports for synthetic-data training results in unverifiable domains, looking for 2 or more independent teams each reproducing a student model outscoring its teacher on a named benchmark.

What would overturn it

Two or more independent reproductions would flip the synthetic-data ceiling reading in unverifiable domains.