The historical turn in social science has prompted scholars to engage with the work of historians on a large scale. Here, social scientists face two standard problems of selection bias: confirmation bias and convenience sampling. So far, the record of dealing with these problems has been poor, and little has been done to specify how social…

This research study aims to evaluate the graduate theses and articles conducted on the concept of instructional leadership over the period between 2002 and 2017 in Turkey by means of methodological and statistical analysis techniques. In the study, which is configured by using the case study design as one of the qualitative research methods, the…

This brief paper introduces a new approach to assessing the trustworthiness of research comparisons when expressed numerically. The 'number needed to disturb' a research finding would be the number of counterfactual values that can be added to the smallest arm of any comparison before the difference or 'effect' size disappears, minus the number of…

The purpose of this study was to examine the quality of quantitative articles published in the "Journal of Counseling & Development." Quality concerns arose in regard to omissions of psychometric information of instruments, effect sizes, and statistical power. Type VI and II errors were found. Strengths included stated research…

Monte Carlo methods are used to explore the accuracy of a method for establishing confidence intervals for variance component estimates in generalizability studies. Previous research has shown that variance component estimation errors due to sampling are often larger than suspected. (Author/CM)

Much of educational administration research suffers from lack of conformity between data analysis and research design, inattention to the test/retest stability of measurement instruments, and the use of statistics for nominal or interval data with ordinal data, random-sample statistics with nonrandom samples, and unnecessarily complex statistical…

A case study of a survey of new students made at a postsecondary vocational-technical school illustrates several sampling problems, including sample size, target population definition, determination of research purposes, costs, error tolerance, accuracy, and interpretation of research results for use by policymakers. (RW)

Discusses, in the context of library evaluation, the major sources of untrustworthy information and describes what can be done to control them. The emphasis is on assessing the impact of changed operations. (JD)

Reliability and validity of self-report data provided by 178 mentally ill homeless persons were generally favorable. Self-reports of service use also generally agreed with treatment staff estimates, providing further validity evidence. Researchers and administrators can be relatively confident in using such data. (SLD)

A study was conducted at Los Angeles Pierce College (LAPC) to test the relative reliability of small-sample telephone interviewing and large-sample mailings with postcard and telephone reminders for student follow-up. The study involved conducting a telephone survey of 245 former students who had enrolled at LAPC in four program areas, and a mail…

Until recently, norms of experimental reasoning have lacked systematic theoretical development. Thus, it has been easy for psychologists like Tversky and Kahneman to misclassify certain human reasoning processes as being Pascalian and invalid, rather than as being Baconian and valid. (CP)

In order to assess the methodological merit of the published research in educational administration based on surveys, the authors analyzed data from a random sample of 24 survey studies published in the "Education Administration Quarterly" and the "Journal of Educational Administration." Each article was evaluated according to…

This paper presents results of a generalizability assessment of two sets of scales designed to evaluate the quality of verbal and nonverbal interactions between parent and child. The Nursing Child Assessment Feeding Scale (NCAFS) and Nursing Child Assessment Teaching Scales (NCATS) were developed as part of a longitudinal study aimed at…

Five criteria are proposed for evaluating performance in research studies: the directness of sampling, the responsiveness of the target, the reliability of the observers, the frequent assessment of the target during the intervention period, and the critical nature of the target. A case study illustrates effectiveness of the criteria. (Author/CR)

Three common methodological problems in the analysis of observational data include: (1) failure to base degrees of freedom for the analysis on the appropriate sampling unit; (2) the validity of the design and data are inconsistent with the generalizations reached; (3) the statistic used to calculate coder reliability is often inappropriate and…