PMI Embraces Virtual Exam Development Activities During Covid-19 Pandemic

Project Management Institute (PMI),the world’s leading association for the project management profession, has more than 600,000 global members and helps prepare more than three million professionals around the world, across industries, to be successful in project management. PMI sponsors several certifications, including their flagship product, the Project Management Professional (PMP)® certification, which is accredited by the American National Standards Institute (ANSI).   

As one would expect from a global leader in program management, PMI continually finds innovative ways to efficiently and proactively navigate in an ever-changing world, including the many challenges presented by COVID-19.

 “We were in the process of updating the exam content for the new PMP exam when the Covid-19 pandemic hit,” shared Sierra Hampton-Simmons, Director of Certifications at PMI. “Because we serve a global audience,” continued Hampton-Simmons, “our standard procedure is to conduct exam development workshops in-person with subject matter experts (SMEs) representative of our global population. The pilot for the updated exam was scheduled for March 2020 and the in-person Modified Angoff Standard Setting, which guides PMI in determining the final passing score, was planned for April in Croatia. Due to the pandemic, our plan to do this in person was no longer possible.”

“We considered postponing the pilot and standard setting,” said Hampton-Simmons. “However, we had already delayed our product due to market readiness concerns. In addition, we were unsure how long the pandemic would hinder travel and the ability to convene SMEs in-person. Finally, the work we had already completed in the exam update process indicated that the market changes addressed in the exam update were significant enough that it was crucial for us to avoid further delays in our schedule.”

Therefore, PMI moved forward with the pilot exam in March and requested that Alpine conduct a virtual standard setting workshop in April with the global SME committee that had already been recruited for the previously scheduled in-person event.

“Although we had concerns with conducting the standard setting virtually, we could hear the clock ticking so we were compelled to go with virtual facilitation,” stated Hampton-Simmons. “We formulated a solid plan with Alpine to alleviate our concerns and to help meet important requirements: (1) the panel represent our global audience, and (2) the workshop format accommodate the varied time zones of the panel members as much as possible to prevent SME fatigue and burnout.”

Global SME Audience & Time Zones

“We have strong markets around the world,” stated Hampton-Simmons. From a fairness and validity standpoint, we take great care to select SME panels that are representative of our global audience rather than just one region. And because we are ANSI-accredited, that same due diligence aligns with our efforts to continually do our best to ensure the credential is fair, valid, and reliable across the global market.”

To ensure SME representation for their large market segments, PMI created and used a matrix-structured spreadsheet to organize the information of the SMEs they were considering recruiting.

“When recruiting SMEs, we consider many factors including industry, region, and years of experience. Industry is important because the project management role is applied differently in each. For regional representation we selected SMEs from Asia-Pacific, the Middle East, Latin America, and North America,” shared Hampton-Simmons.

Meeting Format

The panel was divided into three subgroups to accommodate the various time zones of the SMEs. Group 1 was composed primarily of SMEs from the Asia‐Pacific region, Group 2 primarily of SMEs from the Middle East and Latin America, and Group 3 primarily of SMEs from North America.

The process for the standard setting workshop was the same for all three subgroups and utilized the Modified Angoff Method. The process for each group spanned two days and consisted of an initial 2‐hour virtual call followed by approximately 24 hours to submit the assigned tasks, and then a second 2‐hour virtual call with approximately 24 hours to submit the final assigned task.

To avoid item exposure, Alpine created three unique versions of the form. Each subgroup viewed 75 of the 175 items on the pilot exam. Of the 75 items, 50 items were unique to that particular subgroup, such that no one SME viewed the entire item pool. The remaining 25 items were common items across all three subgroups. Common items were used to assess severity or leniency across the subgroups.

Because the workshop was virtual instead of in-person, all parties (i.e., management and psychometric consultation) worked diligently to ensure everything was communicated as precisely as possible, including all verbal and written instructions between the Alpine facilitator and the SMEs. All players contributed, were supportive of the process, and put forth excellent effort.    

“We were worried about how the virtual format would work because of our preference to have workshop participants sequestered in the room together for a shared understanding. However, the Alpine facilitators made sure the message stayed the same in each subgroup so the SMEs all had the same understanding,” said Hampton-Simmons. “We were also pleased that there was not a lot of variation with ratings across subgroups.”

The Outcome

In spite of the pandemic, PMI was able to keep on schedule by leveraging a virtual solution to set the standard for the updated PMP exam. They were able to do so while remaining true to their requirements of having a SME panel representative of their global audience and a secure and conducive meeting format and process.

“Even though this exam version will not go live until January 2021, we are already finding supporting evidence that the standard setting suggestions are accurate,” shared Hampton-Simmons.

When asked if Hampton-Simmons would consider conducting a future virtual standard setting, she said, “Although I had initial concerns, we didn’t find any negative differences in the process or the results and I’m very pleased that we were able to do this. Alpine was great to work with and helped provide a timely and sound solution we can confidently consider for the future. Although we prefer the in-person option, we would go with the virtual option again.”