Assessments are used in the Tertiary sector for formative and summative purposes. The economics and student demands are increasing the pressure for the Tertiary sector to provide online exams. The cost of Assessment can be ~$50 per high-stakes test per student, equating in a cost of $287 million dollars to the sector in Australia alone*.
The potential cost saving for a typical 50,000 person University are $6 million dollarsper annum.**
Schools are responding to these economic pressures and exams are shifting online with NAPLAN Online in 2018, and the State-accreditation bodies stating to move for budget approval.
What will school leavers expect from Tertiary institutions after having already used computers for exams for a number of years? It won’t be a pencil and piece of paper.
Before Tertiary can embrace Online Assessments, Australia’s leading Universities identified two key challenges: logistics and identity. Here’s what Janison is doing to tackle these head on.
How do you manage logistics?
When planning for a digital exam, you want a solution that can meet your needs for resiliency, security and bandwidth. For the NAPLAN Online project, we’re building three options: Online, School Server and USB technology. In this way, schools and University campuses can opt for the solution that best meets their level of readiness.
The online solution requires a reliable internet connection and the use of a lock-down browser. This can be done on Windows, Mac, Chromebook, iPad and Android tablet.
The School Server solution runs a Deployed Application on a server within the local network of the school. This can be via installation on a physical hardware or on a virtual machine. The school server is capable of delivering tests with no Internet connectivity. Afterwards, test attempts can be uploaded to the central server.
The USB solution is self-contained, designed to run on a Windows computer with no internet nor local network connectivity. Some installation is required, all school, test, student information, and student test attempts will remain stored on the USB drive.
We are extending on the above to develop hybrid solution of software and hardware. Janison physical mobile Exam Appliances can be setup in any location, which will provide a local high-density WiFi access point. Student devices connect will to the appliance via a mesh network. With battery backups and optional 3G/4G connectivity, this can convert even the most unlikely venue into a digitally-ready Exam Centre. These four options provide a broad range of delivery options to meet the unique challenges of Universities.
How do you manage identity?
The critical challenge in this decade for all institutions is ensuring that the identity of the person who is being assessed is the same as the person receiving the accreditation. This important issue is prevalent across essays, submissions, course work and exams.
Janison is conducting research and development in this area that allows institutions to improve their confidence in identity Forms of identity factors that can be developed include:
Facial recognition – Did you know algorithms have been shown to exceed human ability in terms of recognition? We’re keen to help Universities reduce any margin of error during the exam registration process.
Fingerprint biometrics – We are working with hardware partners such as Google to ensure we can integrate with the next generation of devices fingerprint capabilities as part of the authentication process of gaining access to an exam.
Editing and creative writing signatures.
What are your University’s next steps?
Open the discussion
Talk to us about your university’s unique challenges and opportunities. We’ll help you identify the approaches that help you realise business value sooner.
Request a demo
After identifying the features and needs that matter most to you, we can tailor a presentation that matches your context so you can make better decisions on what next. One size doesn’t fit all here.
Run a pilot
The fastest way to learn, is to do. We’re helping organisations run pilot programs so they can see first-hand how online exams will work in their environment. We then work with you to scale up when you’re ready.
*Based on $50 per test, 8 exams per year per student, 718,679 students taking Bachelor Degree in 2015, as per Education.gov.au. **Based on $50 per test, 8 exams per year per student, 30% savings in end-to-end costs.
Universities are now aware of the many benefits of moving to Online Exams. The challenge is how to get started. The delta between the ‘as is’ and ‘to be’ process can seem so large — all the while business-as-usual activities need to be supported. Like any summit, there are multiple paths to the top, and no-one makes it in a single bound. Here are three ways to get started on your journey…
1. Authoring Online
What does it involve?
Using a cloud-based authoring tool to create, modify, curate and collaborate your Item and Exam Bank. This often replaces MS Word and storing files on network drives / over email. Exams can be printed directly from the system (once approved), before traditional pencil-and-paper delivery.
Who is affected?
The people that write your exams (lecturers, tutors, etc). Existing processes for students taking the exam, exam scheduling and invigilation and marking are not affected.
What are the benefits?
A standardised process across all faculties to ensure consistency. Increased security via roles, workflow statuses, audit trails and two-factor authentication. Comments to facilitate collaboration. Time spent on curating the item bank becomes an investment in intellectual property. Once this is in place, you could move to Marking Online, by scanning the student papers.
2. Marking Online
What does it involve?
Using a cloud-based marking tool to perform marking of (scanned) student responses. This on-screen marking of scanned paper answers replaces marking the physical response sheets.
Who is affected?
The people that mark your exams (lecturers, tutors, contractors, etc). Existing processes for authoring the exam, and students taking the exam, exam scheduling and invigilation are not affected.
What are the benefits?
Markers need not be in the same physical location, with support for team management and real time monitoring .Increased efficiency, as the system handles the delegation and collection of responses to markers. Increased quality, with anonymisation, double-marking, tolerances, blind quality monitoring, escalation, commenting and reviews. Digital marker training can be delivered for professional development points. Next steps would involve expanding the use of the tool for full Authoring Online (if not already).
3. Run an Online Exams pilot
What does it involve?
Taking a couple of subjects through the end-to-end process of digital Authoring, Delivery & Marking. The lecturers would author these exams on a cloud-based authoring tool. It would then be delivered digitally; through an invigilated computer lab, a locked-down terminal in an exam hall or perhaps via BYOD with a safe-exam browser. The responses are then available in the marking system.
Who is affected?
Everyone confined to the chosen subject(s). By running a pilot, you limit the scope of total people affected, while gaining invaluable data from the few that are.
What are the benefits?
This approach is in line with Agile methodologies, to ‘experiment, inspect and adapt’. Lessons learned are taken into the next iteration, where more feedback is obtained, repeating the cycle. Organisations that have lead the pack on moving online have almost all taken this progressive ramp-up approach.
Not all subjects lend themselves to the digital medium yet — mathematics for example. One of the most powerful things about online exams is reducing the disconnect between how students learn and work, and how they are assessed. A pilot or phased approach can help ensure that this is maximised appropriately; for the arts, humanities and languages, business, and computing. For these, the digital experience is normal and exam engagement can be greatly enhanced.
Take your first step.
When taking the journey to online exams, look for a solutions provider with national and international experience. One that isn’t content giving you something out of the box, but will go the extra mile to ensure it’s the right solution for you. A company that has a strong view on online exams and strong product roadmap.
Moving online is a once-in-a-generation change. It’s your journey — make sure you take the right Sherpa.
Feel free to reach out to me directly for a confidential chat about your organisations assessment challenges via firstname.lastname@example.org.
Universities must aim to improve themselves over the next 3 years. There are many opportunities, challenges and threats they need to fend off to remain competitive and relevant.
Wait — Universities will always be relevant… won’t they?
Not without a committed plan to remain so. Historically, Universities have had a monopoly on education. There were no alternatives to learn advanced concepts but through formal education. With the rise of the Internet, Learning Management Systems and MOOCs, anyone with a mobile device has access to ever-increasing banks of knowledge. Universities are even starting to award credits from completed MOOCs towards degrees — how long until they compete directly?
The rise of self-taught professionals, entrepreneur & startup culture, and just-in-time learning resources mean there are more viable alternatives than going to University than ever before. More than simply viable; these alternatives reflect modern life more accurately, and evolve faster, than Universities do. People live, work and learn in collaboration with others on digital platforms. However, in a case of massive disconnect, students at Universities aren’t assessed on this ability – instead are graded on their ability to transfer knowledge through a pencil onto paper. After which, they go back to the ‘real world’ again.
Even the geographic advantage of Universities is disappearing. Students in Australia can now enroll online at American Institutions. Universities need to provide a compelling case for students to enroll locally. Improving exams by 2020 is one way Universities can fight towards remaining competitive and relevant for future students.
What are the five key questions towards improving exams by 2020?
1. How will we address the disconnect between pen on paper exams, and students’ everyday experiences of study, work and life?
One option is to switch the focus from ‘traditional assessment’ to being performance-based. This involves assessors supervising and marking group activities, conducting interviews, and setting up other real-world scenarios in which to determine a candidates ability. This approach is used heavily in medical and sporting fields, and could be extended to many others with some creative thinking.
Another option is to migrate pen-and-paper exams to a digital delivery platform. There are many benefits to this — for both the candidates and staff at all levels of the institution. Moving online is a journey — see this 3-minute video for an quick overview.
2. How will we leverage the proliferation of personal mobile devices?
A survey conducted by University of Queensland found that Mobile device ownership (excluding desktop computers) was an average of 2.3 devices per student ( Hillier, M (2015) “e-Exams: The story so far”). Students already have devices they are familiar and comfortable with, and use for study and work purposes. We are now at a point where these same devices can be leveraged for examinations, via bring-your-on-device policies There are at least two approaches here:
Exams are run in ‘lock-down’ modes — preventing access to the internet. This itself can be achieved in a variety of ways, two of which are:
-Device uses a bootable environment (USB) which has exam software and prevents internet access.
-A lightweight client is installed (via download/app store), which either acts as a ‘safe-exam browser’ (no internet), or which goes on to download an encrypted exam, only unlockable to those in the exam center later on.
Exam papers are designed to be ‘open book’ — allowing internet access (and tracking it). Personally I hope to see this as a future movement, where the ability to filter and find information online is part of a given assessment — better reflecting skills required in today’s workforce.
Leveraging the technology that students already have means there are less hurdles than ever for Universities to consider migrating their exams online.
3. How will we improve the quality and efficiency of our exam process?
Two key drivers for change that are sector agnostic are quality and efficiency. These often seem like opposing forces, however digital exams is a great example where both can be optimised.
Quality Across an exam paper lifecycle, there are key points where quality can be improved:
Authoring — item banks, review workflows, item analysis, item seeding (testing non-scoring items during exams to see how well they perform), and smart branching and adaptive tests all contribute to a better quality exam paper.
Delivery — digital student responses are more legible and easier to format and edit than doing so on paper.
Marking — digital marking systems can facilitate far greater transparency, process, quality controls and real time metrics and monitoring than paper based marking. Tolerances for double-marking and other quality controls can be fine-tuned mid-process, to ensure the marking activity is a well-oiled machine. It’s a key factor in why Singapore, #1 in PISA rankings, is moving their national O-level and A-levels online.
When marking quality is paramount, the best Education organisations in the world have gone digital.
Efficiency As above, efficiency gains can be seen in a number of areas of an exam paper lifecycle. Authoring can leverage item banks of pre-seeded and pre-tagged questions, meaning writing new papers is less about starting from scratch, and more about curating the exam. It frees up exam-setters to spend more time thinking about the exam as a whole, instead of doing grunt work.
Digital marking again comes to the fore in terms of efficiency. Marking can begin as soon as the candidates have submitted their electronic responses, various levels of artificial intelligence can automark, semi-mark, and pre-score different question types, while the system handles all the response-management, metrics, quality controls and response distribution. Australia’s NAPLAN program is moving online in a large part due to the efficiency gains it will bring, enabling data to get flow back to the schools sooner.
4. How will we mitigate the risks inherent in high stakes exams?
There are many risks to think about when delivering exams. Let’s focus on four of them: exam paper leaks, poor exam quality, plagiarism, and impersonation.
Technology can play a part in mitigating these risks in a number of ways:
Increased security for login, such as two-factor authentication. This reduces the impact of a password-on-a-post-it-note, falling into the wrong hands.
Limited access to the whole exam — item banks and randomisation mean only the most trusted sources can see the ‘full picture’ of a possible exam paper.
Workflow statuses and tags reduce the chance of mistaken identity when it comes to differentiating practice and real exams.
Poor exam quality: Without digital systems that enforce process, reviews, quality and are backed by item analysis, mistakes are inevitable. This month, students took to Twitter on masse accusing Exexcel of setting the wrong paperafter the questions were seemed impossible without a calculator.
Plagiarism: Some students will go to extraordinar lengths to copy the work of others during an exam. Last month, three Thai students brought in smart-glasses linked to smart-watches to communicate the exam paper to outside tutors. While cell-phone blocking technology has been used in China to prevent exam cheating, most of that tech remains illegal to use in the West (so we don’t recommend it). However digital exams can play a smaller part here; enhanced randomisation and exam-branching reduce the chance of copying the work of any student around you, while linear navigation options mean you don’t have access to the whole exam upfront (only one question at a time), which could reduce efficiency of any back-and-forth comms.
Technology solutions such as facial recognition, audio recognition and keyboard signature profiling can alert invigilators to suspicious behavior . They can then investigate further to determine if the real candidate is present.
5. How will we maximise our capability to collect and analyse exam results?
Without an end-to-end online exam system, the process of measuring, calculating and analysing results is either lacking or inefficient. With online exams, you can achieve a closed-loop system which facilitates consistent and automated processes:
Data mining, available as soon as delivery and marking activities are complete.
Calculation and psychometric analysis that feeds back into your Item Bank.
Additional insights and trends that can be extracted from BI tools.
“We’re going to start answering these questions!” — Head Change Agents @ your University.
Here’s some more resources to help you get started on the journey:
Did you ever sit an ‘open book’ exam growing up? You probably went through the three emotions I did.
Happiness — that we don’t need to study, since we’d have our text books.
Panic — as we sat down and read the exam, and realised the book was going to be useless.
Understanding — that the Teacher didn’t want us to recall facts and figures, but to display a deeper understanding.
These exams were actually more engaging as a student. You really had to use a combination of memory, logical thinking and creativity to get through them. Not to mention time management, as the brief flicker of hope that the text book might somehow help you never truly died out.
School curricula of the past have focused on classic disciplines; algebra, history, chemistry, geography. While these have merit, there is a change in the education zeitgeist; a focus on 21st century skills.
In 2012, a study called ATC21S identified 10 skills that people will need to master to be successful in today’s world.
For most of us, the notion of data storage is somewhat a distant and ‘cloudy’ concept (pun intended). In other words, we don’t really need to give it much thought because we don’t have to deal with it at all. Well, maybe only when we’re prompted to backup our iTunes purchases!
However, for those responsible for managing data within their workplace it’s pretty big business and a major consideration. Human resources is a sector that springs to mind when thinking about instances where the secure management and storage of data is paramount. Online learning is another area where data security is key.
Property is big business, not just in Australia, but in most countries around the world. And it makes sense really, humans have only a few requirements to ensure the survival of our species and that is water, food, sunlight and shelter. There are more I haven’t mentioned, but you get the gist.
This obvious truth makes the business of property – or real estate – a big deal that touches all of us in some way. In Australia you either board, rent or own, or ‘rent to own’ which is an evolving trend. And from this premise, most of us would have a ‘real estate agent’ story to tell, or two… or many!
Janison is always looking at new ways we can add value to the online learning and assessment space. One of the great things about going digital is the fresh new ways of engaging your Users. In version 1503 of the CLS – soon to be released – we’ve introduced new Question Types for Assessment Modules. All of the below can all be automatically marked, saving you time.
We are at the forefront of innovation in the development and delivery of award-winning learning and assessment technologies that improve the performance of organisations and the learning experiences of employees and students, around the globe.