When is fast too fast? Setting score and time thresholds for credentialing programs

Presented at NCME Annual Meeting 2019

Toronto, Canada

Are candidates responding too quickly to your credentialing exam items? When is being quick on an exam, too quick? When should the validity of the results be called into question? Candidate score by time analyses strive to address these questions. Fast response times at both the item and/or exam level can be indicative of item preknowledge. Credentialing programs frequently review candidate score by time for evaluating potential security concerns at the candidate, item, and exam levels. The analyses are appealing as they can be easily performed and interpreted, involve a visual element (score by time plots), and statistical probability (estimating the joint probability of each candidate’s score and time combination). Additionally, policy decisions can be reflected with differing flagging thresholds (conservative versus liberal rules). The current exploration crosses multiple credentialing programs, multiple exams within each of those programs, and exams at different lifecycle stages representing different rates and likelihoods of item exposure. This paper explores analyses of candidate score by candidate time in an attempt to define: 1) what is reasonable, 2) what is extreme, 3) what actions are defensible, and 4) what policies should be in place.

Speaker(s)

Casey Johnson, Ph.D., Alpine Testing Solutions

Diane Talley, Ph.D., Alpine Testing Solutions

Angelica Rankin, Ph.D., Alpine Testing Solutions

Jill R. van den Heuvel, Ph.D., Alpine Testing Solutions