Discusses the problem of determining an adequate sample size for an information retrieval experiment comparing two systems on separate samples of requests. The application of statistical methods to information retrieval experiments is discussed, the Mann-Whitney U Test is used for determining minimum sample sizes, and variables and distributions…

Comparing variability of test scores across alternate forms, test conditions, or subpopulations is a fundamental problem in psychometrics. A confidence interval for a ratio of standard deviations is proposed that performs as well as the classic method with normal distributions and performs dramatically better with nonnormal distributions. A simple…