Option-Independent
Retrieve, then locate
The answer can be derived before inspecting the candidates.
“What is the capital of Italy?”
knowledge→answer→scan
EMNLP 2026 · Findings
Lost in Choice:
Evaluating LLMs Under Extreme Multiple-Choice Scaling
Large context windows can hold thousands of alternatives. That does not mean models can reliably compare them.
University of Bologna, Italy
Code and analysis notebooks are public; MMLU-XT will be released with the paper.
Multiple-choice task
N = 1,000
Option-Independent example
Which city is the capital of Italy?
Accuracy reaches its minimum at late-middle positions—and often fails to recover at the end.
1,000core questions
31,000test instances
6reasoning LLMs
1,000options at maximum scale
In brief
A structural failure,
not just a scaling curve.
Existing multiple-choice evaluations use small option sets and can conflate distinct sources of failure as candidate spaces grow. We decompose large-choice degradation across comparison dependency, cardinality, spatial layout, and gold-answer position. Across six reasoning LLMs, comparative questions and dispersed candidates degrade most sharply, while accuracy often collapses at late-middle and terminal positions instead of recovering in the classic U-shape. A ToolBench replication and a reasoning-trace audit show that the pattern extends beyond the synthetic benchmark and reshapes how models approach selection.
The core question
Is the model failing to find the answer—or to compare the choices?
Existing multiple-choice evaluations use small option sets and can blur together different sources of failure. We separate them.
Option-Dependent
Compare, then decide
The correct answer emerges only through comparative analysis.
“Which algorithm is most efficient?”
inspect↔compare→decide
Reasoning vs. Context Paradox
For four of six models, 1,000 localized options on an Option-Independent question are less damaging than the native 10-option version of an Option-Dependent question.
MMLU-XT
A controlled four-axis decomposition
Rather than another single scaling curve, we vary the factors independently to reveal how they interact.
Comparison load
Option-Independent vs. Option-Dependent questions.
Cardinality
Candidate sets with 10, 100, 500, or 1,000 options.
Layout
Original candidates remain local or are dispersed.
Position
The gold answer moves across five normalized windows.
Spatial disposition
Same candidates.
Different geometry.
The ten original MMLU-Pro options are embedded in a fixed pool of random in-domain distractors.
Why random distractors? Their deliberately weak, exchangeable distribution holds semantic hardness approximately fixed, isolating structural effects rather than hard-negative composition.
Blocklocalized signal
Sparsedispersed signal
original option distractor gold
Main results
The context can fit the choices.
The model still loses them.
Accuracy is shaped jointly by task dependency, scale, spatial layout, and target position.
Comparison is the bottleneck
Option-Dependent questions degrade substantially more than retrieval-driven questions.
Dispersion amplifies failure
Sparse placement consistently hurts more than keeping the original candidates together.
No classic U-shape
Accuracy generally does not recover when the answer reaches the end of the list.
Explore by model
Six architectures, one structural pattern
Real-trajectory replication
The same collapse appears in ToolBench.
We repeat the positional test on real solved tool-use trajectories, retaining ten task-relevant tools and defining the first recorded tool call as the target.
Both models reach their minimum at window 0.75, matching MMLU-XT’s hardest position. The controlled expansion tests transfer of the structural effect, not its prevalence in deployed systems.
Qwen3.5-9BGemma-4-31B
50%60%70%80%
Beyond accuracy
How reasoning changes
as choice expands
An audit of generated traces reveals recurring strategies—and a sharp difference between retrieval and comparison.
Independent derivation then scan
Solve first, then search the list for the internally derived answer.
Sequential review
Evaluate candidates linearly, one option at a time.
Eliminative filtering
Discard implausible candidates to reduce the active set.
Candidate-led shortlisting
Match hypotheses against promising items in the list.
The operational takeaway
Report performance by target rank, preserve locality among mutually relevant candidates, and do not assume that robust lookup implies robust comparison.
Citation
If this work helps your research, please cite it.
Copy the camera-ready BibTeX entry below; page numbers can be added once the proceedings metadata is final.
@inproceedings{DBLP:conf/emnlp/CocchieriRTM26,
author = {Alessio Cocchieri and
Luca Ragazzi and
Giuseppe Tagliavini and
Gianluca Moro},
title = {Lost in Choice: Evaluating LLMs Under Extreme Multiple-Choice Scaling},
booktitle = {Findings of the Association for Computational Linguistics: {EMNLP}
2026, Budapest, Hungary, October 24-29, 2026},
publisher = {Association for Computational Linguistics},
year = {2026},
}