Zur Hauptnavigation wechseln Zur Suche wechseln Zum Hauptinhalt wechseln

Robustness Evaluation of Entity Disambiguation Using Prior Probes: the Case of Entity Overshadowing

Publikation: Beitrag in Buch/Konferenzband › Beitrag in Konferenzband

Abstract

Entity disambiguation (ED) is the last step of entity linking (EL), when candidate entities are reranked according to the context they appear in. All datasets for training and evaluating models for EL consist of convenience samples, such as news articles and tweets, that propagate the prior probability bias of the entity distribution towards more frequently occurring entities. It was previously shown that the performance of the EL systems on such datasets is overestimated since it is possible to obtain higher accuracy scores by merely learning the prior. To provide a more adequate evaluation benchmark, we introduce the ShadowLink dataset, which includes 16K short text snippets annotated with entity mentions. We evaluate and report the performance of popular EL systems on the ShadowLink benchmark. The results show a considerable difference in accuracy between more and less common entities for all of the EL systems under evaluation, demonstrating the effects of prior probability bias and entity overshadowing.
OriginalspracheEnglisch
Titel des SammelwerksProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Untertitel des Sammelwerks EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021
Herausgeber*innenMarie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
VerlagAssociation for Computational Linguistics (ACL)
Seitenumfang10
DOIs
PublikationsstatusVeröffentlicht - 2021
Extern publiziertJa
VeranstaltungProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing - Punta Cana, Dominikanische Republik
Dauer: 7 Nov. 2021 → …

Konferenz

KonferenzProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
KurztitelEMNLP
Land/GebietDominikanische Republik
OrtPunta Cana
Zeitraum7/11/21 → …

Zitat