Empirical Software Engineering Experimentation with Human Computation

Empirical software engineering (ESE) focuses on gathering evidence through measurements and experiments involving humans and software systems (software products, processes, and resources). While empirical studies often include considerable human effort for study planning, execution, and data analysis, human computation (HC) methods, such as crowdsourcing, are increasingly used to address human input intensive tasks in software engineering and beyond. Therefore, in this chapter, we explore the use of HC techniques to support ESE experiments. We address researchers from both research communities and provide (1) introductory notions into both fields, (2) an analysis of ESE experiment requirements and HC capabilities that could match those, and (3) a concrete example of an ESE experiment that compares the effects of using HC in software inspection with respect to a traditional inspection process preformed using pen and paper. Our focus is on software inspection for detecting defects in software engineering models (namely, extended entity relationship models). This chapter will enable ESE researchers to apply HC in their work and HC researchers to explore ESE as a new application area to further improve their methods and tools.