Recap:
- We ask LLM to solve one or more puzzles in a single turn;
- The puzzles require very little knowledge but quite some reasoning effort;
- At this point it is less about comparing models to each other - more about measuring reasoning quality and efficiency for each model and building dataset;
Previous notes with more context:
Variance and sample complexity
Let’s pick several endgame samples, run models multiple times on each. The models used here are strong enough to solve the individual puzzle (n_games = 1), so we use token cost, not success rate as an estimate for complexity.
We can also try make a guess about ‘what makes sample complex for the model’. First, as we saw, different samples have different valid sizes with different probabilities for top choice. It might be that the samples which have more valid choices and/or less extreme probability distributions would be more challenging for the model.
Here we can see running Sol-6.1 and deepseek flash on the same 6 samples multiple times.
Observations:
- Samples are different in complexity
- We can see clear staircase pattern - models mostly agreeing with each other on relative complexity
- The variance between runs for same (model, sample) is significant, but not huge - we can see non-overlapping boxes
- The valid set size doesn’t seem to be a good predictor of complexity, while p_top is stronger. We probably don’t need to dig into this too much.