The fourth industrial revolution will bring extensive changes in the nature of work. While automation is likely to displace workers, new occupations will be created. Emerging occupations are likely to be disproportionately concentrated in the nonroutine and cognitive category, and require skills that cannot be easily automated. This paper argues…

In this research report, we describe the design and empirical findings for a large-scale study of essay writing ability with approximately 2,500 high school students in Germany and Switzerland on the basis of 2 tasks with 2 associated prompts, each from a standardized writing assessment whose scoring involved both human and automated components.…

Computer-graders have been in regular use in the context of MOOCs (Massive Open Online Courses). The automatic grading of programs presents an opportunity to assess and provide tailored feedback to large classes, while featuring at the same time a number of benefits like: immediate feedback, unlimited submissions, as well as low cost of feedback.…

Advanced software, smartphones, the Internet, and cognitive computing have already disrupted the retail, media, transportation, education, and health care industries. Widely different predictions have been offered about the scope of future workforce disruptions. Recent assessments about the impact of artificial intelligence on worker dislocation…

Massification is a reality facing universities around the world. While increased access to higher education has significant social and economic benefits, rapid growth in class sizes challenges institutions to maintain quality standards while teaching at scale, amidst ongoing cost pressure. This paper analyses this issue within the Australian…

COVID-19 is hastening the adoption of online learning and teaching worldwide, and across all levels of education. While many of the typical learning and teaching transactions such as lecturing and communicating are easily handled by contemporary online learning technologies, others, such as assessment of learning outcomes with closed book…

The Organization for Economic Co-operation and Development (OECD) estimated that approximately 9% of current jobs within OECD member states are threatened with automation and digitalization--all significant successes and advances in artificial intelligence, robotics, and computer science. With such global changes and forecasts, in the labor…

The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth graders were randomly assigned by classroom to an AWE condition that used "Project Essay Grade Writing" (n = 56) or a word-processing condition that used Google Docs (n = 58). Effectiveness was evaluated using multiple metrics:…

Ontario's 24 colleges will play a pivotal role in establishing Ontario as a global leader in higher education -- producing a workforce with the qualifications and expertise to help drive economic recovery in the aftermath of the COVID-19 lockdown. In its recently released white paper, The Future of Ontario's Workers, the StrategyCorp Institute of…

This report presents an overview of the "SpeechRater"? automated scoring engine model building and evaluation process for several item types with a focus on a low-English-proficiency test-taker population. We discuss each stage of speech scoring, including automatic speech recognition, filtering models for nonscorable responses, and…

The Washington State Department of Early Learning (DEL), in collaboration with the Department of Social and Health Services (DSHS), submits this preliminary report to detail quality control measures in the Working Connections Child Care (WCCC) program. This report highlights existing quality control measures and efforts in reducing overpayments…

Over two semesters short essay prompts were developed for use with the Graphical Interface for Knowledge Structure (GIKS), an automated essay scoring system. Participants were students in an undergraduate-level online introductory statistics course. The GIKS compares students' writing samples with an expert's to produce keyword occurrence and…

This research report provides an overview of the R&D efforts at Educational Testing Service related to its capability for automated scoring of nonnative spontaneous speech with the "SpeechRater"? automated scoring service since its initial version was deployed in 2006. While most aspects of this R&D work have been published in…

Automatic short answer grading (ASAG) is the task of assessing short natural language responses to objective questions using computational methods. The active research in this field has increased enormously of late with over 80 papers fitting a definition of ASAG. However, the past efforts have generally been ad-hoc and non-comparable until…

How can we automatically determine which skills must be mastered for the successful completion of an online course? Large-scale online courses (e.g., MOOCs) often contain a broad range of contents frequently intended to be a semester's worth of materials; this breadth often makes it difficult to articulate an accurate set of skills and knowledge…