|
Archive@NYU >
Stern School of Business >
CeDER Published Papers >
Please use this identifier to cite or link to this item:
http://hdl.handle.net/2451/27824
|
| Title: | Classification-Aware Hidden-Web Text Database Selection, |
| Authors: | Ipeirotis, Panagiotis
Gravano, Luis |
| Keywords: | distributed information retrieval
web search
database selection |
| Issue Date: | Mar-2008 |
| Publisher: | ACM |
| Citation: | ACM Transactions on Information Systems (TOIS), vol. 26, no. 2, article
6, March 2008 |
| Series/Report no.: | CeDER-PP-2008-07 |
| Abstract: | Many valuable text databases on the web have noncrawlable contents that
are “hidden” behind search interfaces. Metasearchers are
helpful tools for searching over multiple such “hidden-web”
text databases at once through a unified query interface. An important
step in the metasearching process is database selection, or determining
which databases are the most relevant for a given user query. The
state-of-the-art database selection techniques rely on statistical
summaries of the database contents, generally including the database
vocabulary and associated word frequencies. Unfortunately, hidden-web
text databases typically do not export such summaries, so previous
research has developed algorithms for constructing approximate content
summaries from document samples extracted from the databases via
querying.We present a novel “focused-probing” sampling
algorithm that detects the topics covered in a database and adaptively
extracts documents that are representative of the topic coverage of the
database. Our algorithm is the first to construct content summaries that
include the frequencies of the words in the database. Unfortunately,
Zipf’s law practically guarantees that for any relatively large
database, content summaries built from moderately sized document samples
will fail to cover many low-frequency words; in turn, incomplete content
summaries might negatively affect the database selection process,
especially for short queries with infrequent words. To enhance the
sparse document samples and improve the database selection decisions, we
exploit the fact that topically similar databases tend to have similar
vocabularies, so samples extracted from databases with a similar topical
focus can complement each other. We have developed two database
selection algorithms that exploit this observation. The first algorithm
proceeds hierarchically and selects the best categories for a query, and
then sends the query to the appropriate databases in the chosen
categories. The second algorithm uses “shrinkage,” a
statistical technique for improving parameter estimation in the face of
sparse data, to enhance the database content summaries with
category-specific words.We describe how to modify existing database
selection algorithms to adaptively decide (at runtime) whether shrinkage
is beneficial for a query. A thorough evaluation over a variety of
databases, including 315 real web databases as well as TREC data,
suggests that the proposed sampling methods generate high-quality
content summaries and that the database selection algorithms produce
significantly more relevant database selection decisions and overall
search results than existing algorithms. |
| URI: | http://hdl.handle.net/2451/27824 |
| Appears in Collections: | CeDER Published Papers
|
All items in Faculty Digital Archive are protected by copyright, with all rights reserved.
|
