CALL US TODAY 1300 857 687  

What are the five key questions towards improving exams by 2020?

Universities must aim to improve themselves over the next 3 years. There are many opportunities, challenges and threats they need to fend off to remain competitive and relevant.

Wait — Universities will always be relevant… won’t they?

Not without a committed plan to remain so. Historically, Universities have had a monopoly on education. There were no alternatives to learn advanced concepts but through formal education. With the rise of the Internet, Learning Management Systems and MOOCs, anyone with a mobile device has access to ever-increasing banks of knowledge. Universities are even starting to award credits from completed MOOCs towards degrees — how long until they compete directly?

The rise of self-taught professionals, entrepreneur & startup culture, and just-in-time learning resources mean there are more viable alternatives than going to University than ever before. More than simply viable; these alternatives reflect modern life more accurately, and evolve faster, than Universities do. People live, work and learn in collaboration with others on digital platforms. However, in a case of massive disconnect, students at Universities aren’t assessed on this ability – instead are graded on their ability to transfer knowledge through a pencil onto paper. After which, they go back to the ‘real world’ again.

Even the geographic advantage of Universities is disappearing. Students in Australia can now enroll online at American Institutions. Universities need to provide a compelling case for students to enroll locally. Improving exams by 2020 is one way Universities can fight towards remaining competitive and relevant for future students.

What are the five key questions towards improving exams by 2020?

 

1. How will we address the disconnect between pen on paper exams, and students’ everyday experiences of study, work and life?

One option is to switch the focus from ‘traditional assessment’ to being performance-based. This involves assessors supervising and marking group activities, conducting interviews, and setting up other real-world scenarios in which to determine a candidates ability. This approach is used heavily in medical and sporting fields, and could be extended to many others with some creative thinking.

Another option is to migrate pen-and-paper exams to a digital delivery platform. There are many benefits to this — for both the candidates and staff at all levels of the institution. Moving online is a journey — see this 3-minute video for an quick overview.

2. How will we leverage the proliferation of personal mobile devices?

 

A survey conducted by University of Queensland found that Mobile device ownership (excluding desktop computers) was an average of 2.3 devices per student ( Hillier, M (2015) “e-Exams: The story so far”). Students already have devices they are familiar and comfortable with, and use for study and work purposes. We are now at a point where these same devices can be leveraged for examinations, via bring-your-on-device policies There are at least two approaches here:

  1. Exams are run in ‘lock-down’ modes — preventing access to the internet. This itself can be achieved in a variety of ways, two of which are:
    -Device uses a bootable environment (USB) which has exam software and prevents internet access.
    -A lightweight client is installed (via download/app store), which either acts as a ‘safe-exam browser’ (no internet), or which goes on to download an encrypted exam, only unlockable to those in the exam center later on.
  2. Exam papers are designed to be ‘open book’ — allowing internet access (and tracking it). Personally I hope to see this as a future movement, where the ability to filter and find information online is part of a given assessment — better reflecting skills required in today’s workforce.

Leveraging the technology that students already have means there are less hurdles than ever for Universities to consider migrating their exams online.

3. How will we improve the quality and efficiency of our exam process?

Two key drivers for change that are sector agnostic are quality and efficiency. These often seem like opposing forces, however digital exams is a great example where both can be optimised.

Quality
Across an exam paper lifecycle, there are key points where quality can be improved:

  1. Authoring — item banks, review workflows, item analysis, item seeding (testing non-scoring items during exams to see how well they perform), and smart branching and adaptive tests all contribute to a better quality exam paper.
  2. Delivery — digital student responses are more legible and easier to format and edit than doing so on paper.
  3. Marking — digital marking systems can facilitate far greater transparency, process, quality controls and real time metrics and monitoring than paper based marking. Tolerances for double-marking and other quality controls can be fine-tuned mid-process, to ensure the marking activity is a well-oiled machine. It’s a key factor in why Singapore, #1 in PISA rankings, is moving their national O-level and A-levels online.

When marking quality is paramount, the best Education organisations in the world have gone digital.

Efficiency
As above, efficiency gains can be seen in a number of areas of an exam paper lifecycle. Authoring can leverage item banks of pre-seeded and pre-tagged questions, meaning writing new papers is less about starting from scratch, and more about curating the exam. It frees up exam-setters to spend more time thinking about the exam as a whole, instead of doing grunt work.

Digital marking again comes to the fore in terms of efficiency. Marking can begin as soon as the candidates have submitted their electronic responses, various levels of artificial intelligence can automark, semi-mark, and pre-score different question types, while the system handles all the response-management, metrics, quality controls and response distribution. Australia’s NAPLAN program is moving online in a large part due to the efficiency gains it will bring, enabling data to get flow back to the schools sooner.

4. How will we mitigate the risks inherent in high stakes exams?

There are many risks to think about when delivering exams. Let’s focus on four of them: exam paper leaks, poor exam quality, plagiarism, and impersonation.

Exam Paper Leaks: Last month, the UK’s national spelling test wasmistakenly published on a government website. While it was a low-stakes test, the public were urged not to ‘pass on’ the information (really?). Again in the UK (this month), a rogue marker published a SATS test online. What if these were high-stakes tests? The disruption to school’s schedules and reputational damage would be immense.

Technology can play a part in mitigating these risks in a number of ways:

  1. Increased security for login, such as two-factor authentication. This reduces the impact of a password-on-a-post-it-note, falling into the wrong hands.
  2. Limited access to the whole exam — item banks and randomisation mean only the most trusted sources can see the ‘full picture’ of a possible exam paper.
  3. Workflow statuses and tags reduce the chance of mistaken identity when it comes to differentiating practice and real exams.

Poor exam quality: Without digital systems that enforce process, reviews, quality and are backed by item analysis, mistakes are inevitable. This month, students took to Twitter on masse accusing Exexcel of setting the wrong paperafter the questions were seemed impossible without a calculator.

Plagiarism: Some students will go to extraordinar lengths to copy the work of others during an exam. Last month, three Thai students brought in smart-glasses linked to smart-watches to communicate the exam paper to outside tutors. While cell-phone blocking technology has been used in China to prevent exam cheating, most of that tech remains illegal to use in the West (so we don’t recommend it). However digital exams can play a smaller part here; enhanced randomisation and exam-branching reduce the chance of copying the work of any student around you, while linear navigation options mean you don’t have access to the whole exam upfront (only one question at a time), which could reduce efficiency of any back-and-forth comms.

Impersonation: A study from the University of Sydney found students are “increasingly paying impersonators to sit their exams”.Exam invigilators are often external contractors, and do not know the students personally.

Technology solutions such as facial recognition, audio recognition and keyboard signature profiling can alert invigilators to suspicious behavior . They can then investigate further to determine if the real candidate is present.

5. How will we maximise our capability to collect and analyse exam results?

Without an end-to-end online exam system, the process of measuring, calculating and analysing results is either lacking or inefficient. With online exams, you can achieve a closed-loop system which facilitates consistent and automated processes:

  • Data mining, available as soon as delivery and marking activities are complete.
  • Calculation and psychometric analysis that feeds back into your Item Bank.
  • Additional insights and trends that can be extracted from BI tools.

“We’re going to start answering these questions!” — Head Change Agents @ your University.

Here’s some more resources to help you get started on the journey:

  • Beg/borrow/steal/leverage this Business Case for Online Exams.
  • Get in contact around our maturity model, and how we can help you start realising business value now, without the need for a big bang approach.

Universities are great — let’s ensure the student experience remains great too (in addition to the toga parties).

Tweet about this on TwitterShare on LinkedIn

Published by

Stuart Dalrymple

Business Development Manager, Certified Scrum Master. My goal is to help organisations realise the benefits of online exams.

Leave a Reply

Your email address will not be published. Required fields are marked *

ABOUT US

We are at the forefront of innovation in the development and delivery of award-winning learning and assessment technologies that improve the performance of organisations and the learning experiences of employees and students, around the globe.

HELP & SUPPORT

Help: help.janison.com.au

Australia: 1300 857 687

International: +61 2 6652 9850

Fax: +61 2 6652 9855