Conference Proceedings

Modelling Uncertainty in Collaborative Document Quality Assessment

Aili Shen, Daniel Beck, Bahar Salehi, Jianzhong Qi, Timothy Baldwin

Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019) | Association for Computational Linguistics | Published : 2019


In the context of document quality assessment, previous work has mainly focused on predicting the quality of a document relative to a putative gold standard, without paying attention to the subjectivity of this task. To imitate people’s disagreement over inherently subjective tasks such as rating the quality of a Wikipedia article, a document quality assessment system should provide not only a prediction of the article quality but also the uncertainty over its predictions. This motivates us to measure the uncertainty in document quality predictions, in addition to making the label prediction. Experimental results show that both Gaussian processes (GPs) and random forests (RFs) can yield comp..

View full abstract