Endgame-60 dataset
Task
The puzzle used for this dataset is a noisy variation of Mastermind - the goal is to guess a secret code by making repeated guesses. Simul setup requires a model to solve multiple puzzles at a time, helping push model for longer reasoning and longer context while keeping individual task complexity unchanged.
Dataset
This is a fixed dataset built on top of simul puzzle. It has 60 endgames with following restrictions:
- Position is solvable in a single turn by greedy player.
- Top probability option is unique - there are no ties. Player which correctly identifies most likely answer is guaranteed to solve the puzzle.
- There’s more than one valid option at the final step - just keeping track of what’s possible and what’s not is not going to be enough - player needs to estimate posterior of candidates.
- The dataset has diverse complexity - some samples have obvious top option, others have p(top) and p(second) pretty close, but always different.
Demo benchmark setup
Full history of guesses and noisy answers is provided. Model is instructed to make a single guess - we explicitly tell model to be greedy.
The 60 endgames are shuffled and organized in batches of sizes 2-30 (every divisor of 60). Each batch becomes a single task for the model: check history for K games, and make a single guess for each in one tool call.
Two models are used:
The new version claims higher reasoning efficiency, which is exactly what this dataset can help validate. Old model only had data for batch size = 2,3,4. For the new model all batch sizes were run across reasoning settings, but to verify a claim that new model is better, we can use the 2,3,4 only.
| Batch size | Solved (old → new) | Δ accuracy | Tokens/solve (old → new) | Δ tokens/solve |
|---|---|---|---|---|
| 2 | 88% → 97% | +8 pp [−2, +18] | 42.7k → 30.8k | −28% [−43, −10] |
| 3 | 80% → 93% | +13 pp [0, +27] | 35.5k → 25.6k | −28% [−43, −11] |
| 4 | 77% → 83% | +7 pp [−8, +22] | 30.7k → 24.8k | −19% [−36, −1] |
| All | 82% → 91% | +9 pp [+3, +16] | 36.6k → 27.2k | −26% [−35, −15] |
Brackets are 95% bootstrap confidence intervals. Bold means the interval excludes zero. As we can see, both number of solved puzzles and tokens/solved game have improved significantly, which validates the claim.
For the next step, we’ll take a look at new version performance across batch sizes and reasoning budget settings.