Crowdsourcing label noise simulation on image classification tasks

Tanguy Lefort (1, 2) , Benjamin Charlier (1) , Alexis Joly (2) , Joseph Salmon (1, 3)

1 IMAG - Institut Montpelliérain Alexander Grothendieck
2 ZENITH - Scientific Data Management
3 IUF - Institut universitaire de France

It is common to collect labelled datasets using crowdsourcing. Yet, labels quality depends deeply on the task difficulty and on the workers abilities. With such datasets, the lack of ground truth makes it hard to assess the quality of annotations. There are few open-access crowdsourced datasets, and even fewer that provide both heterogeneous tasks in difficulty and all workers answers before the aggregation. We propose a new crowdsourcing simulation framework with quality control. This allows us to evaluate different empirical learning strategies empirically from the obtained labels. Our goal is to separate different sources of noise: workers that do not provide any information on the true label against poorly performing workers, useful on easy tasks.

Keywords: Crowdsourcing

Keywords: Apprentissage participatif

Domains: Human-Computer Interaction [cs.HC]

Tanguy Lefort, Benjamin Charlier, Alexis Joly, Joseph Salmon. Crowdsourcing label noise simulation on image classification tasks. JDS 2022 - 53es Journées de Statistique, Jun 2022, Lyon, France. ⟨hal-04562503⟩