Conference Proceedings

A scalable system for identifying co-derivative documents

Y Bernstein, J Zobel, A Apostolico (ed.), M Melucci (ed.)

STRING PROCESSING AND INFORMATION RETRIEVAL, PROCEEDINGS | SPRINGER-VERLAG BERLIN | Published : 2004

Abstract

Documents are co-derivative if they share content: for two documents to be co-derived, some portion of one must be derived from the other or some portion of both must be derived from a third document. The current technique for concurrently detecting all co-derivatives in a collection is document fingerprinting, which matches documents based on the hash values of selected document subsequences, or chunks. Fingerprinting is currently hampered by an inability to accurately isolate information that is useful in identifying co-derivatives. In this paper we present spex, a novel hash-based algorithm for extracting duplicated chunks from a document collection. We discuss how information about share..

View full abstract

University of Melbourne Researchers