Conference Proceedings

An improved system for sentence-level novelty detection in textual streams

Xinyu Fu, Eugene Ch'ng, Uwe Aickelin, Lanyun Zhang

International Conference on Smart Sustainable City and Big Data (ICSSC) | The Institution of Engineering and Technology | Published : 2015


Novelty detection in news events has long been a difficult problem. A number of models performed well on specific data streams but certain issues are far from being solved, particularly in large data streams from the WWW where unpredictability of new terms requires adaptation in the vector space model. We present a novel event detection system based on the Incremental Term Frequency-Inverse Document Frequency (TF-IDF) weighting incorporated with Locality Sensitive Hashing (LSH). Our system could efficiently and effectively adapt to the changes within the data streams of any new terms with continual updates to the vector space model. Regarding miss probability, our proposed novelty detection ..

View full abstract

University of Melbourne Researchers

Citation metrics