Discrimination-aware task assignment in crowdsourcing.

Abstract

Algorithmic bias has been identified as a key challenge in many AI applications. One major source of bias is the data used to build these applications. For instance, many AI applications rely on crowdsourcing to generate training data. The generated data might be biased if the task assignment function is skewed towards certain groups of workers based on say gender, ethnicity or location. This typically happens as a result of a hidden association between the workers’ qualifications for the task and the workers’ attributes. Even in the case where such bias is intentional, e.g., in the case of positive discrimination, other biases may be hidden and can thus unintentionally favor acquiring data from certain groups of workers over others. In this thesis, we propose to quantify and address discrimination in crowdsourcing task assignment. We define discrimination as the unbalanced targeting of workers by the task assignment function. To quantify discrimination, we formulate an optimization problem that partitions workers based on their attributes, computes the qualifications of workers in each partition, and finds the partitioning that exhibits the highest discrimination in task assignment decisions. Due to the combinatorial nature of our problem, we devise heuristics to navigate in the space of partitions. We also propose a way to address discrimination to achieve discrimination-free task assignment. Our experimental results on real and simulated data show that our approach can effectively unveil, quantify and address discrimination in crowdsourcing task assignment.