Combining human intelligence and machine learning for fact-checking: Towards a hybrid human-in-the-loop framework

Online misinformation is posing a serious threat for the modern society. Assessing the veracity of online information is a complex problem which nowadays is addressed by heavily relying on trained fact-checking experts. This solution is not scalable, and due to the importance of the problem the issue gained the attention of the scientific community, which proposed many based on Artificial Intelligence and Machine Learning methods. Despite the efforts made, the effectiveness of such approaches is not yet enough to allow them to be used without supervision. In this position paper, we propose a hybrid human-in-the-loop framework for fact-checking: we address the misinformation issue by relying on a combination of automatic Artificial Intelligence methods, crowdsourcing ones, and experts. We study the single components of the framework as well as their interactions, and we propose an interleaving of the different components which we believe will serve as a useful starting point for the future research towards effective and scalable fact-checking.