Reasoning study for Deepseek Flash
Reasoning study for DeepSeek-V4.1-Flash-1017 as a function of the reasoning budget and task size (not complexity).
Setup
The model needs to solve 60 puzzles. For more details on what the puzzles are, please refer to dataset description.
For every batch size bsz in
{2, 3, 4, 5, 6, 10, 12, 15, 20, 30} we:
- Shuffle the 60 puzzles
- Arrange them into batches of size
bsz - For each batch, model is requested to produce solutions for all puzzles in a single turn.
So, for every batch size same 60 puzzles are solved. This allows to observe ‘larger tasks with more reasoning’ while keeping average individual puzzle complexity constant. The puzzle inputs themselves are tiny, ~3K tokens total for largest batch sizes, so even if ‘long context’ is the issue, it is self-imposed by long reasoning, not a property of the input.
Deepseek API supports
low/high/max settings: link
Results
Correctness
We can clearly see quality drop as batch size grows. Despite individual puzzles having same complexity, asking to solve more in a single turn is much harder. Higher reasoning effort shows smaller, but still dramatic drop. Smaller tasks are easier to manage, so, practically, splitting work into different sessions is helpful. ‘Do 20 things from TODO list’ might end up suboptimal.
Token usage
Looking at token usage per batch, we can see that large batches are getting ~250K tokens. Deepseek Flash has 384K token generation limit, no individual batch exceeded this. At low/high level the token budget plateued, so likely model is trained to allocate not more than certain total budget per reasoning effort parameter.
Tokens per puzzle
As batch size grows, model allocates less tokens for each puzzle in a batch. There’s plenty of headroom for medium sizes, yet model limits itself. Some flexible/adaptive reasoning approach might help.
Tokens vs Accuracy tradeoff
There’s a clear but non-linear relationship between accuracy and tokens per puzzle, which looks very similar across effort levels.
Next steps:
- Run on more samples to get higher confidence
- Try dynamic adjustment of reasoning effort