Mention extraction
Transcribe the PHI surface form exactly as printed.
"ELEANOR GRACE"
EMNLP 2026 · Main Conference
Paint It, Black
Evaluating LLMs for De-Identification of Multimodal Clinical Documents
Reading sensitive information is not enough. A safe system must identify what it is—and locate exactly where it appears on the page.
A joint research collaboration
WESTSHOREREGIONAL MEDICAL CENTER
LABORATORYSERVICES
Complete Blood Count
AnalyteValueRange
White Blood Cell Count6.43.5–10.5
Hemoglobin11.411.6–15.5
Platelet Count312150–450
Validated: 05/15/2024 09:22 AM
Detect labels PHI and predicts its bounding box. Redact shows the privacy-critical end result.
414page instances
2,579PHI annotations
23HIPAA-aligned entity types
8open multimodal LLMs
The missing evaluation
A model can read the right PHI
and still redact the wrong pixels.
Text-only benchmarks measure whether a model recognizes a sensitive span. Clinical PDFs add a second requirement: the model must ground that span precisely in a heterogeneous visual layout.
Spatial grounding
Predict normalized coordinates that cover the correct region.
[0.178, 0.096, 0.327, 0.188]
Joint pass rate
Text, label, and location must all be correct together.
PHI safely redacted
The central finding
Mention extraction is within reach. Spatial grounding remains the decisive bottleneck—and text-level evaluation alone conceals it.
The PIB benchmark
Clinical PDFs,
evaluated end to end.
PIB casts de-identification as visually grounded NER: from a raw page image directly to HIPAA-aligned entity labels, exact strings, and bounding boxes—with no OCR, parser, or retrieval model in between.
Clinical document types
Laboratory, CT, MRI, and gynecological reports.
Difficulty tiers
Clean source pages plus controlled medium and hard visual degradation.
PHI macro-categories
Names, dates, age, addresses, contacts, identifiers, and date of birth.
Real patient records
Every document is synthetic and reviewed for clinical plausibility.
NameDatetimeDate of birthAgeAddressContactID
Controlled visual decay
From clean render
to hospital reality.
Every source page is replicated across three difficulty tiers while labels and bounding boxes remain fixed, isolating robustness to physical acquisition artifacts.
01
Base
Clean PDF rendered at 144 DPI.
02
Medium
Seven scan, print, and paper artifacts.
03
Hard
Eleven artifacts, fax noise, stains, shadows, and markup.
All names and records shown above are synthetic. No real patient, hospital, or clinician data was used to construct PIB.
Main results
Strong readers.
Unreliable grounders.
Switch metrics to see how a compact text-level ranking becomes a sharp capability divide once correct localization is required.
Exact PHI span extraction, ignoring bounding-box placement.
Average across Base, Medium, and Hard splitsHigher is better · %
Best Text F1
Gemma-4-31B shows that current MLLMs can often recognize and transcribe PHI.
Best joint Pass Rate
Even the leading model remains far from the reliability required for clinical deployment.
Spatial F1 for two models
InternVL3.5-14B and MedGemma-27B fail to produce usable document-level grounding.
A model that correctly reads John Smith while placing its redaction rectangle on the wrong line provides no privacy protection.
Privacy by construction
A benchmark for sensitive data—without sensitive data.
Real medical records cannot be openly released. PIB uses synthetic clinical documents, a physician-guided generation pipeline, manual bounding-box annotation, and controlled physical degradation to create a reproducible evaluation without re-identification risk.
No real patientsPhysician validatedLocally deployable modelsHIPAA-aligned taxonomy
Clinical research partnership
PIB was developed through a collaboration between UniBoNLP and the IRCCS Policlinico di Sant’Orsola, bringing multimodal NLP and clinical expertise into the same evaluation pipeline.
Citation
If PIB helps your research, please cite it.
The page range and official Anthology identifier can be added when the proceedings metadata becomes available.
@inproceedings{zangrillo2026paint,
author = {Francesco Zangrillo and Alessio Cocchieri and
Giacomo Frisoni and Davide Freddi and Luca Ragazzi and
Alberto Aguzzi and Diego Raimondo and Renato Seracchioli and
Gianluca Moro},
title = {Paint It, Black: Evaluating LLMs for De-Identification
of Multimodal Clinical Documents},
booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural
Language Processing, {EMNLP} 2026, Budapest, Hungary, October 24-29, 2026},
publisher = {Association for Computational Linguistics},
year = {2026}
}