NVIDIANs refer to “speed of light” to describe their efficiency in getting things done. With this mindset, it’s not surprising that the COVID-19 lockdowns did not prevent Marco Alves, Technical Certification Lead at NVIDIA, and his team from meeting their goals to develop their initial certification pilot exam. Instead, they deployed innovative ways of completing test development in a distributed, virtual manner, including crowdsourcing for item development, while maintaining rigorous psychometric practices and standards.

The Goal – Educating and Breaking Down Barriers

NVIDIA is constantly looking to facilitate the adoption of its products and technologies, explained Alves. Not only are they the leaders in gaming (whether in the graphics card market for gaming PCs, or online, with their GeForce Now offering), but they also have considerable presence in many other industries such as supercomputers, professional visualization, auto enhancements, medical imaging, genomics and so on, where data centers are the prime infrastructure to host their full stack offerings. The goal with NVIDIA’s certification program is to empower the professionals that will deploy, maintain, and architect those solutions.

“At NVIDIA, our GPU deep learning is revolutionizing computing with modern artificial intelligence (AI) and natural language processing (NLP) by powering supercomputers, robots, and self-driving cars,” stated Alves. “Our AI and NLP technology is solving world problems in all walks of life, including auto, finance, retail, energy, Internet services, and healthcare.”

NVIDIA recognized the need to establish a certification program for their target audience of IT professionals who deploy, manage, and architect their technology with interest in focusing on the data center domain.

”Our Deep Learning Institute (DLI)has been doing a tremendous job of educating a vast audience of developers and software experts about our technology. We’ve placed a big emphasis on the app developer and software developer so they can use our hardware to get the associated performance advantage and adjust their apps to run on NVIDIA,” explained Alves. “While the DLI fills a crucial role in our education ecosystem, we knew there was even more we could do to provide career progression opportunities for our users in other domains.”

The Challenge – Building a Certification Program During Lockdowns

“Our utmost goal for our certification program is to provide fair and reliable exams to our candidates and give them an opportunity to showcase their credentials, knowledge, and skills,” stated Alves. “We knew it was critical to do things the right way from the beginning and have a valid program architecture and foundation established. Having worked with Alpine Testing Solutions (Alpine) while at a former company, I was aware of their validity-centered approach to test development and reached out to them knowing they could help us develop our program with professionalism and adherence to psychometric standards.”

Prior to the lockdowns, NVIDIA intended to have their stakeholders and subject matter experts (SMEs) meet in person for exam development workshops, knowing they would likely be more productive and have the advantage of the human connection in the room.

“As a culture, we prefer to meet face-to-face. That is the NVIDIA way,” said Alves. “But that was not possible due to the COVID-19 lockdowns. Fortunately, we had sharp, engaged, and dedicated stakeholders and SMEs with great facilitators guiding them through the distributed virtual work in an efficient manner. It was an effort on both sides of the partnership that helped make up the difference of not meeting in person.”

Program Design Review Workshop

Before engaging with Alpine, NVIDIA conducted an internal Program Design. Later, they requested that Alpine facilitate a Program Design Review to help them refine and validate their initial program architecture.

“Prior to the Program Design Review, we were told by some we were trying to boil the ocean,” stated Alves. “Getting top-level managers together was invaluable. We now have a foundational map for our program with full buy-in from upper management and are off to a great start with development of our pilot exam for the Data Center Associate. We’ve also captured support of other departments interested in building exams and getting their target audience(s) certified.” 

Overcoming Hurdles with a Distributed Virtual Process

In addition to the virtual Program Design review, NVIDIA requested that Alpine facilitate a Job Task Analysis (JTA) and Item Development Workshop (IDW) for their Data Center Associate Exam. However, it was at this point that Marco and his team began to encounter unexpected issues with SME availability. NVIDIA’s continued success and growth, coupled with the challenges of the lockdowns, made it very difficult for departments to allocate as much SME time for the exam development work as they had originally intended. 

“We were considering putting the exam development on hold until we could get more SMEs,” shared Alves. “The Alpine project team jumped in and began brainstorming creative ways to get us over our resource hurdles. They presented the idea of a distributed virtual process and we decided to team with them on that novel approach for both the JTA and IDW.”  

NVIDA soon discovered the upside and advantages of the distributed virtual process. In a typical JTA, there might be 8 to 10 SMEs. However, the distributed virtual model allows engagement of a much wider audience with a more diverse group to incorporate feedback from dozens of SMEs at an earlier stage than usual.  

The same is true with item development. SMEs can be trained, and content and perspectives can be gained from a much more diverse group of SMEs by engaging external folks and by crowd sourcing content. When more people are involved and engaged in the process stronger validity evidence is built.

Job Task Analysis

In the true spirit of the NVIDIA culture of cross-functional teamwork, which Alves attributes to the extraordinary leadership of Jensen Huang, NVIDIA’s CEO and founder, many SMEs were able to jump on board to participate in the revised Job Task Analysis (JTA) process which greatly reduced the time required for scheduled meetings and provided SMEs the flexibility to make contributions as their individual schedules allowed.

“We knew it was necessary to minimize the amount of time SMEs spent on calls and to maximize time for SMEs to perform homework tasks on their own. With Alpine’s guidance, we found creative ways to do this,” explained Alves. 

  • Prior to the initial virtual meeting, SMEs viewed an assessment literacy video regarding the importance of validity, reliability, and the role they would play in the JTA process.
  • Surveys were administered to SMEs for identifying critical tasks and themes and to establish the draft blueprint
  • Competency inventories were conducted using online collaboration tools concurrently with the facilitator to identify those conceptual competencies that are critical to the wide array of job roles that make up the minimally qualified candidate target audience.

The SMEs didn’t disappoint. They paid close attention to all steps in the process and completed their homework assignments with quality and timeliness. One facilitator relayed, “It was obvious that the SMEs were fully engaged. In fact, some were reciting steps of the process back to us, as facilitators, on the calls.”

Item Development Workshop Using Crowdsourcing

NVIDIA teams are known for accomplishing a lot with comparatively fewer employees than most organizations in the industry.

“Early on in the item writing phase, we, again, hit a hurdle with not having enough internal resources. Alpine recommended that we consider crowdsourcing by reaching out to external folks in the IT target audience who had taken our courses. We asked them to participate and got a great response,” said Alves.  

Prior to the initial virtual meeting, SMEs once again viewed an assessment literacy video regarding the importance of validity, reliability, and the role they would play in the IDW process. The SMEs were self-directed and used the video to self-train. The SMES were diligent in doing their homework and produced quality items that surpassed expectations, while Alpine professionals provided mentoring and editing feedback along the way. The standard process was initially scoped as a 26-hour effort for training, item writing and item review. The crowdsourcing approach resulted in cutting that projected time allocation by almost half.

Half of the items were built by NVIDIA internal SMEs, and the other half were built by external SMEs. From a management perspective, the Alpine facilitators and Program Manager devised an efficient system to keep up with multiple groups that were writing at the same time to different areas of the blueprint. Along with the item review, some new items were drafted as well.

“We had SMEs from across the globe contributing items and the quality of the items sourced in this way exceeded our expectations,” explained Alves. “Overall, the crowdsourcing process was a great success!”

What’s Ahead

“We have great plans for the future as we continue to build out the NVIDIA Certification Program! When travel is once again possible, we foresee employing a hybrid approach, where some workshops will take a distributed virtual approach and others will be in-person,” shared Alves. “We are in a happy place and that is due in large part to team efforts and flexibility among our stakeholders, SMEs, and Alpine,” shared Alves. “This synergistic team concept is a hallmark of our success that supports our ‘speed of light’ mentality and, in turn, is helping provide support for our certification program.”