As a natural outgrowth of the research described above, we further (1) investigated the techniques of metasearch, pooling and pseudo-evaluation; (2) established formal connections between these seemingly disparate techniques; and (3) developed unified models for approaching all three tasks simultaneously. Individually, these techniques are used to either improve, or evaluate the effectiveness of, retrieval in the context of multiple available search strategies. By establishing formal connections between these techniques, we have developed formal frameworks for investigating all three techniques simultaneously with the end goals of (1) improving the quality of retrieval effectiveness in the context of multiple available search strategies, (2) improving the quality of the evaluation of search strategies, and (3) reducing the resources necessary to perform such evaluations.
As described above, metasearch is the process of fusing the ranked lists of documents returned by a collection of retrieval systems in response to a given query in order to obtain a combined list whose quality is better than any of the underlying lists. Many metasearch algorithms have been proposed and studied, and metasearch has proven to be an effective technique in improving retrieval quality.
Traditionally, retrieval systems are evaluated by constructing a test collection of documents, a set of query topics, and a set of relevance judgments for a subset of the collection (the pool) with respect to these topics. Much research has been conducted in how to best construct the pool of documents to be judged in order to effectively evaluate a collection of retrieval systems, and pooling techniques are often used to evaluate collections of retrieval systems in, for example, the annual TREC competitions.
Recently, Soboroff, Nicholas and Cahan surprisingly demonstrated that retrieval systems can be effectively ranked without relevance judgments by appropriately constructing a pool and assigning relevance judgments at random within the pool. This process of pseudo-evaluation holds out the promise of being able to efficiently extend the evaluation of retrieval systems to large collections of systems over large and possibly dynamic databases of information such as the World Wide Web.
While these techniques have generally been studied in isolation, we have shown there are deep connections between them; in effect, we have provided strong evidence that metasearch, pooling, and pseudo-evaluation are ``three sides'' of the same coin. To demonstrate this fact, we developed a general framework [APS03b,APS03a] which, given a user query and access to multiple retrieval systems, could (1) identify those documents returned by the underlying systems most likely to be relevant to the query (metasearch), (2) estimate the likely quality of each system's response to the query (pseudo-evaluation), and (3) identify those documents whose relevance, if known, would most likely influence the quality of the system evaluation (pooling). The end goal of this research is both theoretical and applied: first, to establish the formal connections among metasearch, pooling and pseudo-evaluation; and second, to develop highly effective meta-retrieval systems which could simultaneously perform metasearch, pseudo-evaluation and pooling, as well as incorporate user feedback in the form of relevance judgments for pooled documents to improve the quality of the above.