Computing maximized effectiveness distance for recall-based metrics

Abstract

Given an effectiveness metric M(·), two ordered document rankings X1 and X2 generated by a score-based information retrieval activity, and relevance labels in regard to some subset (possibly empty) of the documents appearing in the two rankings, Tan and Clarke's Maximized Effectiveness Distance (MED) computes the greatest difference in metric score that can be achieved that is consistent with all provided information, crystallized via a set of relevance assignments to the unlabeled documents such that |M(X1)-M(X2)| is maximized. The closer the maximized effectiveness distance is to zero, the more similar X1 and X2 can be considered to be from the point of view of the metric M(·). Here, we co..