An online statistical quality control framework for performance management in crowdsourcing

No Thumbnail Available
File version
Author(s)
Saberi, Morteza
Hussain, Omar K
Chang, Elizabeth
Griffith University Author(s)
Primary Supervisor
Other Supervisors
Editor(s)
Date
2017
Size
File type(s)
Location

Leipzig, Germany

License
Abstract

The big data research topic has grown rapidly for the past decade due to the advent of the "data deluge". Recent advancements in the literature leverage human computing power known as crowdsourcing to manage and harness big data for various applications. However, human involvement in the completion of crowdsourcing tasks is an error-prone process that affects the overall performance of the crowd. Thus, controlling the quality of workers is an essential step for crowdsourcing systems, which due to unavailability of ground-truth data for any task at hand becomes increasingly challenging. To propose a solution to this problem, in this study, we propose OSQC (Online Statistical Quality Control Framework) for managing the performance of workers in crowdsourcing. OSQC ascertains the worker's performance by using a statistical model and then leverages the traditional statistical control techniques to decide whether to retain a worker for crowdsourcing or to evict him. We evaluate our proposed framework on a real dataset and demonstrate how OSQC assists crowdsourcing to maintain its accuracy.

Journal Title
Conference Title

WI '17: Proceedings of the International Conference on Web Intelligence

Book Title
Edition
Volume
Issue
Thesis Type
Degree Program
School
Publisher link
Patent number
Funder(s)
Grant identifier(s)
Rights Statement
Rights Statement
Item Access Status
Note
Access the data
Related item(s)
Subject

Science & Technology

Technology

Computer Science, Artificial Intelligence

Computer Science, Information Systems

Computer Science

Persistent link to this record
Citation

Saberi, M; Hussain, OK; Chang, E, An online statistical quality control framework for performance management in crowdsourcing, WI '17: Proceedings of the International Conference on Web Intelligence, 2017, pp. 476-482