Are We Done with UMLS Injection into Language Models for Medical Question Answering?
Department of Computer Science and Engineering, University of Bologna, Italy
All authors contributed equally.
To answer medical exam questions more reliably, language models are often given UMLS, the largest curated map of medical terminology. The reported gains are small, and nobody had checked whether they come from UMLS at all. We reviewed every study from 2020 to 2025, 25 in total, and rebuilt their choices one at a time.
How a language model answers a medical question with UMLS
Follow one exam question through the pipeline that every reviewed system builds. Click the steps or use the buttons.
Each step hides construction choices: which UMLS release and vocabularies, which linker, which relations and how many hops, what happens when nothing links, and how the facts are shown to the model. The findings below show how much each choice changes, and how rarely papers report it.
Inside UMLS
UMLS, the Unified Medical Language System of the US National Library of Medicine, is a database, not a model. It joins the main medical vocabularies under shared concept IDs. Pick a table to see the rows that describe one disease, esophageal achalasia.
What we did
We searched Scopus for work on knowledge graphs and language models for question answering and reasoning, published from 2020 to 2025. We kept the studies where UMLS plays a substantive role, the task is medical question answering or a closely related reasoning task, and the method is described well enough to compare. Two medical NLP researchers annotated every study independently, with 90.2% agreement.
Then we tested the choices these studies make, one at a time and with everything else fixed: two entity linkers, five UMLS releases and vocabulary subsets, retrieval depth and connectivity, retraining of QA-GNN and GreaseLM, and ten modern language models reading UMLS evidence as text.
Nine skeletons in the closet
Each finding is a choice that changes the UMLS evidence a model receives. Each one stays hidden until you answer its question; switch to Read all to open them at once.
What to report with every UMLS result
Each of these choices changes the UMLS evidence that reaches the model. Reporting all nine is the minimum for a gain to be traceable to UMLS.
Prefer to play?
The same story as an 8-bit game: the four authors learn how UMLS works, open the literature closet, and catch the nine skeletons one by one. About ten minutes, on desktop or on a phone held sideways.
Play the 8-bit gameCitation
The repository contains the annotations of all 25 studies, the linker comparisons, the retrieval diagnostics and the reader experiments. Experiments use licensed UMLS releases supplied as local inputs; no UMLS data is redistributed.