2022 Conference on Test Security
Alpine Sessions: Thursday, October 27
A Comparison of an Approximation Score Similarity Index to Other Similarity Indices
Presented by:
- Amanda Wolkowitz and Russ Smith, Alpine Testing Solutions
- 9:45 – 10:45 am ET
This presentation will compare the results of multiple score similarity indices to an approximation score similarity index (aSSI) (Smith 2021, 2022). The goal of the study is to determine if aSSI produces similar results to other score similarity indices when applied to real data. Prior research has shown that aSSI does a reasonably good job of producing results similar to true score similar index (SSI) analyses (Smith 2021, 2022), the latter of which uses item response theory (IRT) and the generalized binomial model. However, research is lacking on the practical comparison of aSSI to other similarity indices on real data. The outcome of this study will be recommendations for practitioners on when to apply the aSSI method to real data and how it compares to other existing and more complicated methods.
Visualizing the Prevalence of Collusion/Content Exposure in the Testing Population
Presented by:
- Brett P. Foley, Alpine Testing Solutions
- 9:45 – 10:45 am ET
The development of score similarity and answer similarity indices in recent years has added to our psychometric arsenal for detecting candidate collusion and content exposure. These indices have been shown to be effective and often are calculated by comparing the expected and observed number of matching item scores/responses for each pair of candidates. These results are useful for identifying pairs of candidates with questionable levels of agreement. However, as candidate volume grows, the number of candidate pairs grows quadratically, resulting in very large numbers of pairs for even modest sample sizes. Given the extreme number of possible comparisons, it can be difficult for non-technical audiences to understand the extent of an exposure problem. In other words, is a security issue negligible, isolated, or widespread? In this session, the presenter will begin by briefly describing best practices for creating exam-security related visual displays. Next, we will present several examples of visual displays that use data from all candidate pairs to effectively communicate the prevalence and extent of collusion and content exposure issues in the testing population. The session is appropriate for both technical and non-technical audiences. Attendees will learn about important factors to consider when creating visual displays, be shown specific examples of effective tools for summarizing collusion/content exposure issues, and learn how scaling group frequencies can help highlight outliers or emphasize common patterns.
Approximation Score Similarity Index (aSSI) Analysis: An Analysis of its Effectiveness Compared to True Score Similarity Index
Presented by:
- Russ Smith, Alpine Testing Solutions
- 11:00 am – 12:00 pm ET
This presentation will describe continued research and development into a simple and effective approach to identifying pairs of test takers with an unusually high number of matching scores (Smith, 2021; 2022). True score similarity index (SSI) analyses leverage item response theory (IRT) and the generalized binomial model (van der Linden & Sotaridona, 2006; Zopluoglu, 2017) which require specialized software, are based on relatively strong assumptions, and, depending on the length of the exam and number of examinees, require substantial computational resources. The approximation score similarity index (aSSI) method is not intended to outperform true SSI. Rather, it provides reasonable estimates of pairwise probabilities without specialized software, without requiring the same computational resources, and (in many cases) without item-level data. The purpose of the presentation will be to describe the aSSI method, further explore the conditions in which it works reasonably sufficiently by comparing it to true SSI, and to make practical recommendations for its use.
Join experts from all types of organizations as they discuss the test security capabilities and enhancements that protect the validity of test results and brand integrity. This year’s conference will be hosted in partnership with ETS. We invite you to join us in-person in beautiful Princeton, NJ, October 26-28, 2022. We will also offer a virtual component November 9-11, 2022.
To learn more, please click here.