Task design in complex crowdsourcing experiments: Item assignment optimization

Crowdsourcing is the practice of outsourcing a task that would otherwise be performed by one or a few experts to a crowd of individuals. It is often used to collect large amounts of manually created labels that form datasets for training and evaluating supervised machine learning models. When designing a (micro-task) crowdsourcing experiment, it is often desirable to combine different types of items (e.g., images of both cats and dogs for an image classification task) together into the same task and to have diverse contributors (e.g., male and female crowd workers) providing labels for the same item so as to mitigating existing bias in the collected labels. Setting up such a crowdsourcing experiment, i.e., constructing a human intelligence task containing the right mix of items and assigning it to the right mix of crowd workers, becomes a complex problem. In this paper we address the problem of optimizing human intelligence task construction by providing its formal definition and by applying a local search method initialized by a greedy construction to solve it. We experimentally show the flexibility of the proposed solution in addressing different type of constraints, both on the item and on the worker side, and demonstrate its scalability to complex crowdsourcing setups. The proposed solution enables the design of tools for crowdsourcing requesters to set up complex task deployments in platforms like Amazon MTurk or Prolific.