In this paper, we address the following questions: What misconceptions do high school students exhibit in their first encounter with significance test problems through a repeated sampling approach? Which theory or framework could explain the presence and features of such patterns? With brief prior instruction on the use of Fathom software to…

This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling…

In Taiwan, English language learning in senior high school has predominantly focused on reading, with a heavy emphasis on memorising vocabulary and grammar rules. English listening has been marginalised and is not officially taught until the first year of university. In 2012, the Joint Board of College Recruitment Commission in Taiwan passed…

This paper demonstrates that multiple comparison tests using a pooled error term are dependent on the circularity assumption and shows how to compute tests which are insensitive (robust) to this assumption. (Author/GK)

Texts often suggest running preliminary tests for homogeneity of variance prior to running an ANOVA. While it has been known for some time that most of the suggested tests are probably not appropriate, they are still being used. This paper is a review of the literature in terms of the implications involved in running preliminary tests in general…

Two psychological experiments--Anderson and Shanteau (1970), Berkowitz and LePage (1967)--are reanalyzed to present the problem of the relative importance of low Type 1 error probability and high power when answering a research question by testing several statistical hypotheses. (Author/PN)

This study focused on two questions: (1) whether the types of study-aid test, or the levels of test anxiety would affect the level of performance on the test under non-anxious conditions; and (2) whether there would be any interaction effect between the types of study-aid test and the levels of anxiety under non-anxious conditions. Twelve female…

Some examinees' test-taking behavior may be so idiosyncratic that their scores are not comparable to the scores of more typical examinees. Appropriateness indices, which provide quantitative measures of response-pattern atypicality, can be viewed as statistics for testing a null hypothesis of normal test-taking behavior against an alternative…

The known interval scale, referred to as the 7.8 scale, has been criticized as an invalid measuring instrument in the form of an attitude scale. It is the purpose of this paper to demonstrate that this scale can produce spuriously inflated correlation coefficients, high reliability, and false significance on statistical tests. The case will be…

The various costs of Type I and Type II errors of inference from data are discussed. Six methods for minimizing each error type are presented, which may be employed even after data collection for Type I and which minimizes Type II errors by a study design and analytical means combination. (Author/CM)

Efforts to study the fidelity of translation of attitudinal scales into foreign languages have faltered due to the lack of powerful statistical tests to assess such transformations. This study uses a maximum likelihood factor analysis procedure to compare multivariate factor structures across subpopulations. The results showed that inconsistent…

The study reported in this paper examined whether a disproportionate number of boys and girls in a given grade level at various schools might have an inequitable effect on the mean achievement test scores of those schools. The student population of the Palm Beach County Schools, Florida, reflecting a cross-section of socio-economic backgrounds,…

A random sample of 113 members of the American Vocational Education Research Association (AVERA) was surveyed to obtain baseline information regarding AVERA members' perceptions of statistical significance tests. The Psychometrics Group Instrument was used to collect data from participants. Of those surveyed, 67% were male, 93% had earned a…

To examine the validity of short forms of the Wechsler Intelligence Scale for Children-Revised (WISC-R), the WISC-R was first administered to 106 hospitalized psychiatric patients, aged 8-16. No subjects had a primary diagnosis of mental retardation or learning disability, and one-third were receiving psychotropic medication. WISC-R IQ scores…

The orientation of this report is that of identifying educational projects which can be considered truly exemplary. The bulk of the report consists of a 23-step procedure for validating the effectiveness of educational programs using existing evaluation data. It is not intended as a guide for conducting evaluations but rather for interpreting data…