Prev Next
Close close-icon

2024 ATP Innovations in Testing Global Conference

March 3-6, 2024
Anaheim, CA

We are excited to learn about the possibilities at this year’s ATP conference. Stop by to say hello and start your day with Alpine + a custom hot beverage at Booth #125 in Anaheim. We look forward to seeing you all in March!

Be sure to add these sessions to your ATP agenda:

Monday, March 4

  • 10:00 am – 10:50 am PT
  • Session Track: Presentations

Presenters:

  • Jessica Marshall, Veeam Software
  • Lisa O’Leary, Alpine Testing Solutions

This presentation will highlight how credentialing program sponsors can leverage technology-enabled content review to efficiently, consistently, and accurately assess the psychometric quality of large quantities of existing content to minimize the need for additional content development. In addition to automatically generating customized feedback on exam items based on psychometric standards and exam development best practices, this approach can also apply program-specific branding and style requirements as well as identified deprecated content and/or terminology. The use of this methodology in conjunction with an item and form-level analysis can powerfully arm credentialing programs to make informed decisions about item dispositions, blueprint gap analyses, and future item development needs.

This presentation will discuss the technology utilized to generate the item feedback as well as how that information was incorporated with statistical results to recommend actions for the item bank and its health in an exam maintenance cycle. The impact of this information from the exam sponsor perspective will also be discussed in terms of how it shifted the traditional exam development cycle to include item alignment sessions in advance of item development, as well as reduced the overall level of SME burden associated with the content development and review process and shortened the development timeline.

  • 11:10 am – 12:00 pm PT
  • Session Track: Presentations

Presenters:

  • Jessica Marshall, Veeam Software
  • Julie Torres, Alpine Testing Solutions

Fiscal, human, and time-to-market constraints continue to limit many credentialing programs’ abilities to develop and maintain high-quality, rigorous credentialing programs. Simultaneously, shifts are occurring within many industries regarding the perceived value of traditional, voluntary, professional certifications, and individuals are moving away from job role-based certification paths toward more skills-based and learner-centric professional development journeys. In combination, these trends leave many credentialing organizations struggling to balance the business realities of program costs with the need to demonstrate return on investment for consumers of said programs’ certified populations. This changing atmosphere requires dynamic adjustments to conventional exam development and to its associated assumptions, one of which is the reconsideration of the traditional firewall between certification and training to help support these more individualized experiences and validations. This presentation will highlight one program’s attempts at leveraging joint involvement in certification domain analyses to springboard definition of and structure for associated training modules.

  • 1:20 pm – 2:30 pm PT
  • Session Track: Workshops

Presenters:

  • Lisa O’Leary and Corina Owens, Alpine Testing Solutions

The credentialing industry needs to evolve in its responsible incorporation of technological advancements into measurement processes. These integrated technology-enabled solutions should span exam development, delivery, and maintenance, with a focus on how to develop and maintain exams more effectively and with increased consistency while enhancing validity arguments. These efforts should focus on streamlining, not eliminating the involvement of subject matter experts (SMEs) to contribute at pivotal decision points to ensure the fidelity and accuracy of the outcomes. For many certification programs, SME availability is a major constraint, such that minimizing SME burden and level of effort while maximizing output is key to program success.

This workshop highlights a modernized approach to automating high-quality psychometric workflows across a full suite of exam development stages. This strategy augments rather than supplants these processes to increase efficiency, consistency, and accuracy through rule-based logic, natural language processing, machine learning, and artificial intelligence technology. This reduces the need for continual human production and generation but allows for directed input and review from measurement experts and SMEs. This strikes a balance between program practicalities and psychometric practices by accelerating development processes to generate more automated yet psychometrically-sound exam content.

  • 1:20 pm – 2:30 pm PT
  • Session Track: Workshops

Presenters:

  • Kelli Treadwell, American Board of Internal Medicine
  • Russ Smith, Alpine Testing Solutions
  • Amanda Wolkowitz, Data Recognition Corporation

Have a psychometric question that you want to ask, but are hesitant to ask? Now is the time! The purpose of this session is to answer operational psychometric questions from the audience in an open and welcoming environment. No question will be too big or small, and all questions are worth asking. With three psychometricians from three different organizations leading this workshop, audience members will hear answers from psychometricians with different perspectives and backgrounds, while learning from audience members who also wish to share.

The workshop will begin with a short self-assessment. The presenters will then lead a discussion of the assessment answers. Afterward, audience members will be invited to ask follow-up questions or other psychometric questions either anonymously via a QR code or by verbalizing their question in front of the group. While the presenters will answer questions to the best of their ability, they will also encourage the audience to share their knowledge and experiences.

  • 3:00 pm – 3:50 pm PT
  • Session Track: Sponsor Sessions

Presenters:

  • Keith Becraft, Tracey Hembry, Lisa O’Leary, Corina Owens, and Russ Smith, Alpine Testing Solutions

Creating and maintaining a robust testing program is a complex journey that involves meticulous attention to various facets, from item writing to psychometrics, data management, and test security. At this session, we will extend our assistance to those facing challenging decisions within their programs. Whether seeking advice on critical choices or aiming to enhance certification quality, participants are encouraged to bring their inquiries to a panel of trusted professionals boasting over 100 years of combined experience.

Our team of experts across domains such as test development, psychometrics, test security, and credential management, is prepared to field questions and provide valuable consultations. This session offers a unique opportunity for attendees to engage with seasoned professionals, and gain insights and solutions to propel their testing programs forward. Join us and unlock the information you’ve been searching for to elevate the standards of your program.

Tuesday, March 5

  • 9:40 am – 10:50 am PT
  • Session Track: Debates

Presenters:

  • MaryBeth Kurland, Commission for Case Manager Certification (CCMC)
  • Anne Richard, National Council for Therapeutic Recreation Certification (NCTRC) 
  • Tracey Hembry, Alpine Testing Solutions
  • John Dight, Surpass Assessments 
  • Samantha Sappington, Data Recognition Corporation

Certification and licensure boards have a myriad of options when it comes to choosing vendors in support of their examinations. These sponsor organizations can choose to work with one vendor or select multiple vendors with which to work. There are pros and cons to each solution, and no one solution is the right one for everyone. This session will uncover some of the benefits of working with multiple partners, each specializing in one or just a couple aspects of the assessment solution. We will also cover potential drawbacks of a multi-vendor arrangement and suggest ways in which to mitigate those. Additionally, we will touch on the potential challenges of switching vendors and some of the mitigation strategies to eliminate or lessen the impact of that change.

We expect the audience to come away from the session with a more comprehensive perspective on multi-vendor solutions, where organizations share solutions, embrace change, and are better together for it.

Wednesday, March 6

  • 8:30 am – 9:20 am PT
  • Session Track: Presentations

Presenters:

  • Nick Respecki, NCARB 
  • Natalie Flacco, Project Management Institute
  • Russ Smith, Alpine Testing Solutions

This session introduces a novel application of AI and NLP tools to detect enemy and variant items. While generative AI and natural language processing have gained attention for content generation, the identification of enemy and variant items is often overlooked. These items pose risks to validity, and traditional manual review processes by subject matter experts (SMEs) face limitations due to time and resource constraints—especially with long forms and large item banks. This session presents an approach that combines automated enemy and variant item detection using AI and NLP tools with SME-based item review. Presenters from licensure and certification programs will share their experiences using this approach and discuss the benefits and implications. The session aims to inspire discussions on the wider adoption of automated detection methods to improve the efficiency of the development processes and the validity of the use of the results.

BETTER TOGETHER. Embrace change. Share solutions.

To learn more about the 2024 ATP Innovations in Testing Global Conference, please click here.