Engineering and Information Technology Collected Works - Research Publications

Permanent URI for this collection

Now showing 1 - 1 of 1

Exploiting Worker Correlation for Label Aggregation in Crowdsourcing

LI, Y ; Rubinstein, B ; Cohn, T ; Chaudhuri, K ; Salakhutdinov, R (International Machine Learning Society, 2019)

Crowdsourcing has emerged as a core component of data science pipelines. From collected noisy worker labels, aggregation models that incorporate worker reliability parameters aim to infer a latent true annotation. In this paper, we argue that existing crowdsourcing approaches do not sufficiently model worker correlations observed in practical settings; we propose in response an enhanced Bayesian classifier combination (EBCC) model, with inference based on a mean-field variational approach. An introduced mixture of intra-class reliabilities---connected to tensor decomposition and item clustering---induces inter-worker correlation. EBCC does not suffer the limitations of existing correlation models: intractable marginalisation of missing labels and poor scaling to large worker cohorts. Extensive empirical comparison on 17 real-world datasets sees EBCC achieving the highest mean accuracy across 10 benchmark crowdsourcing methods.