Published: Aug 20, 2026
We are delighted to announce that we will be at EMNLP 2026 with 2 long papers in the Main and Findings Track! Catch us in Budapest, Hungary to learn more about medical deindefication and multimodal hate speech detection.
"Paint It, Black: Evaluating LLMs for De-Identification of Multimodal Clinical Documents
by F. Zangrillo, A. Cocchieri, G. Frisoni, D. Freddi, L. Ragazzi, A. Aguzzi, D. Raimondo, R. Seracchioli, and G. Moro
Secondary use of clinical data is strictly conditioned on prior removal of Protected Health Information (PHI) under regulatory frameworks such as HIPAA. While the NLP community has made considerable progress on text-based de-identification, clinical records in practice arrive as structured PDF documents, whose PHI is distributed across heterogeneous visual layouts, multi-column tables, and letterheads. Current benchmarks are exclusively textual, and the few multimodal pipelines proposed to date rely on intermediate OCR stages rather than evaluating the raw spatial-grounding capability that modern Multimodal Large Language Models (MLLMs) now claim to possess. We introduce Paint-It-Black (PIB), the first benchmark designed to evaluate MLLMs on end-to-end, pixel-level PHI de-identification in clinical PDF documents. Because real medical records cannot be shared and no synthetic substitute currently exists, we develop a corpus of physician-validated clinical PDFs spanning multiple document types, diverse visual layouts, and controlled visual degradation tiers, with manually curated bounding-box annotations following a HIPAA-aligned taxonomy. On PIB, we conduct the first systematic evaluation of privacy-safe, open-source MLLMs, revealing that even the strongest models fall short of the reliability threshold required for clinical deployment.
-
The paper will be available soon!
Lost in Choice: Evaluating LLMs Under Extreme Multiple-Choice Scaling
by A. Cocchieri, L. Ragazzi, G. Tagliavini, and G. Moro
While LLMs are increasingly analyzed in multiple-choice scenarios, current studies rely on small option sets, failing to capture their real-world behavior. This paper provides the first systematic investigation of LLM selection dynamics under extreme candidate cardinality. We stress-test models by scaling the candidate space up to 1,000 options while manipulating their layout within the context window. Importantly, to distinguish model failures, we categorize questions solvable via internal knowledge from those requiring exhaustive option comparison. Our results uncover a systematic, layout-driven degradation of global comparative reasoning—a failure mode we define as the Lost-in-Choice phenomenon. By auditing Chain-of-Thought paths, we establish a taxonomy of the heuristics and shortcuts models adopt as the candidate space expands.
-
The paper will be available soon!