Unsupervised prediction of acceptability judgements

Abstract

In this paper we present the task of unsupervised prediction of speakers' acceptability judgements. We use a test set generated from the British National Corpus (BNC) containing both grammatical sentences and sentences containing a variety of syntactic infelicities introduced by round trip machine translation. This set was annotated for acceptability judgements through crowd sourcing. We trained a variety of unsupervised language models on the original BNC, and tested them to see the extent to which they could predict mean speakers' judgements on the test set. To map probability to acceptability, we experimented with several normalisation functions to neutralise the effects of sentence lengt..