investigation
When the test-difficulty examples are replaced with a mixture of easier examples, Qwen jumps to 91.6% and Llama to 98.6%, while the test remains at exactly the same difficulty level.
A investigation by המחברים. Statements are as reported by the cited publishers — not verified by DecisionGateway.
אבל כשהם מחליפים את הדוגמאות ברמת הקושי של הטסט בתערובת של דוגמאות קלות יותר, Qwen קופץ ל-91.6% ו-Llama ל-98.6%.
Discussion
Related claims
The models trained on the easier examples are tested on problems harder than those they saw in post-training and still succeed much more than models trained directly on the test difficulty level, without receiving full solutions or feedback on intermediate steps.המחברים · investigationWhen the authors train the models only on problems at the test difficulty level, the models reach about 50% accuracy, almost like random guessing.המחברים · investigationThe researchers confirm their claims in experiments: they post-train models such as Qwen and Llama on a task of solving systems of equations, where the final answer is 0 or 1 so about half the cases can be solved by guessing.החוקרים · investigationRaising reasoning effort improves average benchmark scores but produces painful performance drops at the individual task level; tasks solved easily and cheaply at Low fail at High or xHigh.החוקרים · investigationOn algorithmic tasks (CLRS-Text), ten such layers raised an 86M-parameter model from 55% to 84%, after regular training got stuck at 55% and even training twice as long did not move it.post author · statementThe paper proves that when training reasoning models with RL, it is not always true that training data should match the test-time distribution; sometimes data different from what the model will see at test time yields much better test-time results.המאמר · investigation