STUDIES IN THE EVALUATION OF A DOMAIN-INDEPENDENT NATURAL LANGUAGE QUERY SYSTEM

Abstract

There is growing consensus that some of the most crucial questions concerning the feasibility and desirability of natural language interfaces to databases can only be resolved by empirical research. This paper reports the results of several empirical studies which investigated the same domain-independent natural language query system, using various applications in two different natural languages - English and German. Taken together, these experiments involved about 100 subjects and over 12,000 queries, constituting the bulk of empirical evaluations of natural query language systems reported to date. Some definitive results are derived from the combined experience, and plans are outlined to resolve several of the remaining issues.