Located latent class modelling for expert fact-checking from many non-expert checks

Fact-checking is increasingly more critical as false news spreads across social networks. Experts cannot check all the news as fast as needed, so crowd-sourcing is on the rise as a way to distribute tasks to many non-experts. This topic is widely studied in the information retrieval literature. A Bayesian-located latent class model is proposed to surrogate expert judgment by aggregating ratings from multiple workers. The approach combines a prior distribution for the expert’s rating with conditional distributions for each worker’s rating. Ratings are ordinal variables with levels ranging, e.g., from "false" to "true". Monotonic effects for ordinal predictors and hierarchical priors for workers’ parameters help regularize the model. The approach is evaluated on a dataset of fact-checks by PolitiFact, with each statement also rated independently by 10 workers. Patterns in the workers’ parameters motivate grouping the workers according to their political orientation. A closed-form but less structured alternative utilizes only the workers’ misclassification probabilities and treats all variables as categorical. The hierarchical model outperforms the less expensive alternative and achieves a reasonable misclassification rate.