Multitask learning for query segmentation in job search

Abstract

© 2018 Association for Computing Machinery. In this paper, we present the first attempt to use multitask learning for query segmentation. We use the semantic category of the words as an auxiliary task and show that segmentation improves when the model is also trained to predict the semantic category of the query terms, outperforming benchmark methods over a novel dataset from a popular job search engine. Our further experiments show that the task of modeling the query term semantics performs better as a standalone task, without adding segmentation as an auxiliary task.