Human-centric Evaluation of Semantic Resources: A Systematic Mapping Study

Semantic resources such as ontologies and knowledge graphs enable intelligent applications in a broad range of application areas. As ensuring their quality is important, evaluation activities are an integral part of their life-cycle. Yet, several aspects of these resources, such as domain or modelling correctness, are not amenable to automatic evaluation and require assessment through human participation. Despite such human-centric evaluation of semantic resources (HESR) is essential, there is a lack of a systematic understanding of this area in terms of a theoretical framework as well as an overview of concrete HESR approaches and trends. We address this gap by, firstly, integrating existing literature into a theoretical framework for defining and characterising HESR thus developing a new perspective on this area. Second, we perform a Systematic Mapping Study of 15 years of work published in this area covering 144 papers in order to empirically ground and extend the theoretical framework and identify trends and best practices. These theoretical foundations, insights into current trends and practices as well as our openly available data and analysis scripts aim to enable new research in this area and offer practical guidelines for performing HESR to researchers and practitioners alike.