What it really takes to run large-scale digital exams well

You can imagine how much this costs the Chinese government to run; the time and money spent distributing, collecting, shipping, marking and re-marking millions of test papers. Not to mention the legions of invigilators and other staff needed to run the exams.

For most companies and institutions that run large-scale exam events, paper tests usually thrust them into a dangerous waltz with their budgets – a partner whose green dress threatens to turn red at any moment. That’s why many companies turn to digital exams – they’re much more affordable.

However, digital exams or not, they still have to manage thousands of tests every year. Running large-scale digital exams is far from a breeze. It’s a hurricane that takes knowledge, skill, and support to tame.

In this article, we explore the crucial elements to running successful large-scale online exams; the infrastructure, processes, and employees that makes the process not difficult and expensive, but smooth and affordable – for staff and students alike.

Meticulous, ongoing data validation

As of 2023, if each byte of data in the world were a grain of rice, we could cover the Earth’s surface with a layer of rice multiple times over.

Even by themselves, large-scale digital exams contain an astonishing amount of data. There’s students’ personal information, biometric data, and test results; assessments are built from blueprints and may be made up of hundreds of questions drawn from vast item banks, with each question passing through a strict hierarchy of markers with their own painstaking processes; there’s test devices like laptops and tablets, each of which must be fit-for-purpose with the properly installed assessment software…it’s nonstop. And if important data is incorrect for any element, it can snowball and lead to big problems.

That’s why data validation is such a crucial part of successful large-scale testing. And it doesn’t just happen in a mad flurry leading up to a big exam event – it’s a meticulous, ongoing process that involves regular checking and refinement; a series of checks and balances, scripts and automations that lead to a symphony of data accuracy. And that takes real expertise and experience – a person or team that not only know the data intimately, but the complex processes behind hosting and updating and where things are liable to go wrong.

Despite your best efforts with data validation, there will always be something wrong – it’s just about making that something as small and scarce as possible, where you get the system to a state of readiness where students can access their digital exams and have a comfortable, frictionless experience – crucial for high-stakes tests.

Steady, reliable scaling

When it became clear that the mighty Titanic would sink, the ship’s officers had to direct over a thousand terrified people onto its 20 lifeboats. Each boat could hold a maximum of around 60 people. Too many, and it would flounder.

Similarly, if thousands of people try to access a digital assessment at once, and there are no “officers” in place to prevent its servers from being overwhelmed, the online exam also flounders – slightly lower stakes than the Titanic’s passengers faced, but important nonetheless.

While the Titanic had human officers to manage each boat’s capacity, an online exam system has someone to manage the exam’s scaling. It’s their job to determine how many servers are needed based on the number of students, and to have them “spun up” and ready to receive traffic. If they’d overseen the Titanic’s lifeboats, they’d have ordered double the capacity to ensure everyone had a seat.

As students start to access their online assessments, the person in charge of the scaling monitors the servers’ capacities, and if they’re in danger of being overloaded, they manually spin up more to ensure that the tests continue to work. They’re like a prodigy carpenter who can whip up extra boats when they need them.

They’re also helped by a load balancer: a program that directs arriving students to servers that are healthy and unbloated. It does this constantly, ensuring that loads are distributed evenly and servers working optimally.

These functions are essential for running stable large-scale digital exams. They make the entire system resilient. Without this crucial person to pre-plan and monitor capacities, supported by the load balancer, the servers can quickly become swamped with traffic and students may be unable to access their tests, or worse, be thrown out mid-way through. And by having the exact number of servers needed to handle traffic, the institution is paying just the right amount in hosting fees – an expense that can rapidly blow out.

Quality exam management

Whether large-scale online exams happen during one frenzied test window or are spread out over the course of the year, the work that goes into them is enormous. New tests need to be created and validated (potentially with fresh items), training provided to staff, trial runs completed, marking and re-marking…the work is colossal and usually has many contributors, including the team behind the digital exam platform (if you’re interested, see some of the work that went into digitising NAPLAN – Australia’s biggest annual exam event).

For large-scale exams to be successful, they require quality exam management where every process is properly considered; every aspect organised and executed on time. Often, for tests that are completed during a single window, the project starts months (or even years) in advance and is methodically executed until the day students plonk themselves down for their tests.

A crucial role in this whole process is played by event/exam managers, who may spend months ensuring that each student has details of the test’s location, a roof over their head, a desk to sit on, a device to access their test, and an invigilator to keep an eye on them.

When there are thousands of students, this task becomes huge. Every little detail needs to be correct, every element properly organised, and every individual exam event (there can be hundreds) administered with skill and integrity.  The Gaokao is the ultimate example of this, but it holds true for any institution or company that runs large-scale digital exams. For students to have a straightforward, positive experience, the institution’s event management must be a well-oiled machine. Every staff member at every level must have exactly what they need.

For remotely-proctored online exams that candidates sit at their homes, there’s less coordination, but it’s equally important. They need clear instructions on how to access the tests on their devices, what identification they need to provide, how the tests will be monitored, how their data will be used, and how to get support. Thousands of candidates can ask thousands of questions, so anticipating, planning, and providing key information upfront can save your support team countless hours. Also – who exactly is providing the support? Does your institution provide first level support and the exam platform’s team deal with escalations? This can get complex quickly, and clear processes must be in place so that candidates can get support from the right people.

Similarly, proctors need to know how to supervise the tests, how to answer common questions, how to tackle common problems, and anything else that enables them to run smooth tests for candidates. This requires not just quality support resources that proctors can access when they need, but a knowledgeable support team that can help when they’re stuck.

A usable, accessible and flexible digital exam platform

When an application or website does maddening, seemingly inexplicable things, and then politely corrects itself and starts behaving normally again, even the most stoic person might struggle to contain a swear word. For students trying to complete online tests that shape their very futures, blood can start to boil.

To prevent this from happening, digital exam applications need to be logical, consistent, and impeccably designed. The tests are just too important. For large-scale high-stakes exams, a cumbersome system that gets in the way might not only affect students’ test results, it can lead to a tsunami of panicked phone calls and emails, with students desperately trying to access their tests but with no clue how to do so. This reflects terribly on the institution or company running the exams and creates an ocean of ill will towards their brand. Similarly, for exams where students are spread out across large distances, with some people completing the test in a remote, slow (or no)-internet area, the exam system should still allow them to access and complete their test. Otherwise, it just isn’t equitable.

In addition to being easy to use, a digital assessment platform designed for large volumes of tests must cater to a variety of disabilities, or risk alienating a considerable number of people. Between 15 to 20 per cent of people live with a disability of some kind¹ – blindness, difficulty using a mouse, dyslexia, and other disabilities that need to be accommodated for in the exam platform’s design. This includes screen reader capabilities, colour contrast changes, being able to navigate the test with just a keyboard, adaptive testing options and many more features, each one improving the accessibility of the platform a little more, and providing an equitable experience for every student.

Rigorous identity verification

Unfortunately (but not surprisingly), tests with higher stakes tend to lead to more instances of proxy test-taking, where students hire others to complete their exams. In the U.S., a whopping 8 per cent of colleges reported cases of proxy test-taking from their students.² For companies that run large volumes of tests, this is a considerable risk as it can wreck their academic integrity.

When hundreds of students show up ready to begin their digital tests, you must have a fast but rigorous way to validate their identities. In the case of in-person tests, this can be a five-second review of a credible I.D. like a driver’s license. But it’s more complicated for remotely-proctored tests, which requires accurate facial recognition technology to scan both the candidate’s I.D. document (or a photo saved in the system), and their face peering into the device’s camera. In addition, a worthwhile system continues to scan the candidate’s face throughout the exam to catch any quick switches.

Real-time performance monitoring

With thousands of students to manage during an exam event (or hundreds spread across multiple events), your team needs a clear picture on how things are progressing. That means hard, up-to-date data on how many students are enrolled, how many have started the tests, how many are online, how many have dropped off, how many support queries you have…every key piece of information that tells you what is happening, and what problems you might need to fix. For remotely-proctored tests, this might include students with large numbers of flagged violations, prompting you to jump in and proctor them live for a short spell (if the tests are important enough).

Ideally, this exam data is fed through to an intuitive dashboard that gives you an instant glimpse of what’s going on, with options to delve into any piece of data you need. Without this, you’re basically in the dark with little idea if the online exam event is tracking as it should.

A digital exam partner, not a vendor

When you have thousands of tests to run every year, the team behind the digital exam platform needs to have your back – a partner that will do everything in their power to make your big exam events successful, not just a vendor who puts in the minimum amount of effort to keep you happy.

The sheer size and complexity of large-scale online exams almost demand a partnership – so much goes into organising them, and so much can go wrong. With both you and the exam platform fully invested in your success, with plenty of open communication and support in both directions, setting up and running the tests is so much more efficient. It’s a crucial ingredient to running successful large-scale exams.