Conference Proceedings

Compact features for detection of near-duplicates in distributed retrieval

Yaniv Bernstein, Milad Shokouhi, Justin Zobel, F Crestani (ed.), P Ferragina (ed.), M Sanderson (ed.)

STRING PROCESSING AND INFORMATION RETRIEVAL, PROCEEDINGS | SPRINGER-VERLAG BERLIN | Published : 2006

Abstract

In distributed information retrieval, answers from separate collections are combined into a single result set. However, the collections may overlap. The fact that the collections are distributed means that it is not in general feasible to prune duplicate and near-duplicate documents at index time. In this paper we introduce and analyze the grainy hash vector, a compact document representation that can be used to efficiently prune duplicate and near-duplicate documents from result lists. We demonstrate that, for a modest bandwidth and computational cost, many near-duplicates can be accurately removed from result lists produced by a cooperative distributed information retrieval system. © Sprin..

View full abstract

University of Melbourne Researchers