Specific endgames

Let’s look at specific games to see if we can find any interesting patterns. We are working on endgame version of simul.

First, thing to fix: in some samples, the probability for top choice on the last step is equal for several options. For example, greedy player solved the puzzle on the next turn with these probabilities:

'2543': 0.00215
'3045': 0.01076
'3145': 0.02152
'3245': 0.01291
'3645': 0.03229
'3742': 0.05382
'3745': 0.02152
'3845': 0.01614
'3945': 0.05382
'4507': 0.13455
'4517': 0.13455
'4537': 0.02152
'6537': 0.13455
'7540': 0.13455
'7541': 0.13455
'7542': 0.08073

Multiple options are 0.13455, so in this case, the fact that greedy player solved it in one move is partially luck, and it will be better to filter such cases out. Having dataset where there’s a single most correct answer, yet with different complexity is likely to help us find more interesting patterns.

For the next steps we could compare model answer against entire greedy output vector and compute regret. We can make it linear or logistic, very similar to log loss. Or, ask model to produce multiple candidates/probabilities, not pick a single answer. This might be a good metric, we shall look into that later.

For now let’s pick several samples, where p(top choice) will be unique, with different count of valid candidates:

python replay.py --sample samples_unique/6203-022c42.json
{'6203': 0.8928571428571429, '6213': 0.10714285714285714}

python replay.py --sample samples_unique/2786-c190c1.json
{'2783': 0.016393442622950824,
 '2784': 0.0819672131147541,
 '2785': 0.0819672131147541,
 '2786': 0.8196721311475409}

python replay.py --sample samples_unique/9436-e92541.json
{'2359': 0.15873015873015878,                                                          
 '2396': 0.15873015873015878,                                                          
 '2536': 0.03174603174603175,                                                          
 '2596': 0.07936507936507937,                                                          
 '2653': 0.015873015873015876,                                                         
 '2659': 0.07936507936507937,                                                          
 '5436': 0.07936507936507937,                                                          
 '9436': 0.3968253968253968}                                                           

python replay.py --sample samples_unique/5740-d944a6.json  
{ 60+ entries, top @ 9%}

python replay.py --sample samples_unique/9583-95e6f1.json
{ 200+ entries, top @ 3%}

The way to read the output: before the last step where greedy player solved the puzzle, this was the set of possible candidates, with probabilities assigned to each.

Let’s run each sample individually, one game at a time.

model	path	n_games	pct_solved	n_tokens	n_sessions
Qwen3.8-27B-Q8	samples_unique/2786-c190c1.json	1	90.00	7779.95	20
Qwen3.8-27B-Q8	samples_unique/5740-d944a6.json	1	25.00	32142.60	20
Qwen3.8-27B-Q8	samples_unique/6203-022c42.json	1	85.00	23495.65	20
Qwen3.8-27B-Q8	samples_unique/9436-e92541.json	1	75.00	24007.40	20
Qwen3.8-27B-Q8	samples_unique/9583-95e6f1.json	1	15.00	31799.75	20
deepseek-flash	samples_unique/2786-c190c1.json	1	100.00	10209.10	20
deepseek-flash	samples_unique/5740-d944a6.json	1	95.00	54921.05	20
deepseek-flash	samples_unique/6203-022c42.json	1	100.00	28070.10	20
deepseek-flash	samples_unique/9436-e92541.json	1	100.00	33646.65	20
deepseek-flash	samples_unique/9583-95e6f1.json	1	95.00	46750.40	20

Deepseek is still doing great, but more complicated samples have 1 mistake each and need visibly more tokens. Local Qwen 3.8-27B model shows very clear drop from cases which look easy to cases which look hard.

Carefully constructing the dataset with a mix of easy/medium/hard individual problems and simul session size can allow cheap and efficient evaluation of models up to the current (Oct 2026) frontier. We can also easily have very similar public/hidden datasets and see if there’ll be a drift as models pick next iteration of web data.