Introduction: The use of averaged topic-level scores can result in the loss of valuable data and can cause misinterpretation of the effectiveness of system performance. This study aims to use the scores of each document to evaluate document retrieval systems in a pairwise system evaluation. Method: The chosen evaluation metrics are document-level…

Competency education is student-centric, personalizing student progress so that every child has adequate time and support to reach proficiency every step of the way. Competency education fundamentally changes the way the educational enterprise is organized around student needs, and thus must have a dynamic IT system to support it. Following an…

Purpose: The purpose of this paper is to introduce the concept of human information behaviour and to explore the relationship between information behaviour of users and the existing approaches dominating design and evaluation of information retrieval (IR) systems and also to describe briefly new design and evaluation methods in which extensive…

Explores the properties of independent variables in information retrieval systems and defines a class of basic variables named M-factors. (Author/PF)

Demonstrates that if simple foundations of the cost structure are taken into consideration, high values of both precision and recall are not necessarily the goal when searching in an information system. Five figures and a reference list accompany the text. (Author/JL)

The history and significance of relevance as a concept in information retrieval is reviewed. The shift from the idea of relevance as an objective criterion for measures to the widespread acceptance of relevance as a manifestation of a highly subjective and inconsistent response is discussed. It is argued that relevance judgments can b e useful as…

The traditional concern with relevance or nonrelevance of unexamined documents is misplaced, and traditional measures of effectiveness should be replaced by estimates of direct utility of examined documents. (Author)

In part I (JASIS, March-April 1973) the author proposed a methodology for evaluating a retrieval system based on how much users are willing to pay to use the system. Part II sets forth steps in testing the methodology through the analysis of underlying assumptions and through experimentation. (3 references) (Author/RS)

Reports preliminary results of a study to develop and automate new metrics for assessment of information quality in text documents, particularly in news. Through focus group studies, quality judgment experiments, and textual feature extraction and analysis, nine quality aspects were generated and applied in human assessments. Experiments were…

In assessing information retrieval systems, it is important to compare the number of retrieved relevant items to the total number of relevant items. A variant of the statistical "capture-recapture" method can be used in large collections to estimate total number of relevant documents, providing the model supporting such an analysis can…

Examines the problems of developing a framework and testbed for quantitative assessment of image retrieval effectiveness. To better harness the extensive research on content-based retrieval and improve capabilities of image retrieval systems, this article advocates the establishment of common image retrieval testbeds consisting of standardized…

Describes a technique for automatic reformulation of Boolean queries which compares favorably with feedback as employed in a SMART system. Using patron relevance judgments, prevalence measures reflecting term distribution in relevant and nonrelevant documents are derived to guide the construction of a Boolean query for a subsequent retrieval.…

Introduces a novel approach to unsupervised text summarization. Proposes an "information-centric" approach to evaluation, where the quality of summaries is judged not in terms of how well they match human-created summaries but in terms of how well they represent their source documents in information retrieval tasks such as document…

The transfer of information through space and time in communication systems is often accompanied by significant delays which give rise to meaningful storage problems. Mathematical models have been developed for the study of these kinds of problems which are applicable to the design of manual, library-type, or mechanized information storage and…

Results from three studies examining 1295 relevance judgments by 36 information retrieval (IR) system end-users are reported. Both the region of the relevance judgments, from non-relevant to highly relevant, and motivations or levels for the relevance judgments are examined. Implications for relevance theory and IR systems evaluation are…