Reviewing remote proctoring violations: a QA framework

However, to achieve academic integrity with this kind of testing, where the remote proctoring technology invigilates students for you, a rigorous Quality Assurance framework is needed for reviewing flagged violations.

This framework ensures that the review process is fair, consistent, and allows you to achieve academic integrity with remotely-proctored tests – high-stakes or otherwise.

To give you an idea of what goes into a QA framework for remote proctoring violations, we’ve created a template that includes every essential element. Feel free to download, extend, and adapt it as per your institution or organisations own rules and regulations.

We hope it forms a solid base for you, and saves time in creating an accurate QA process for your remote proctoring assessments.

Table of contents

1. Objectives
2. Staff training
3. Global violation settings
4. Violation flagging settings
5. Violations and penalties
6. Roles and responsibilities
7. Violation review process
8. Appeals process
9. Audits

1. Objectives

The key objectives of this framework are:

• To ensure all flagged violations are reviewed objectively and consistently, with fairness front of mind.
• To preserve and protect the academic integrity of our assessments.
• To make the review process efficient, and allow continuous improvement.
• To clearly outline the primary elements of achieving all of the above.

Note: this framework does not include the use live human proctors during exams, but can be amended if necessary.

2. Staff training

All staff members who are involved in the QA process for reviewing violations (listed in section 6, Roles and responsibilities) will complete training.

This training includes the following:

• How to complete their part of the review process, particularly how to adhere to consistent, fair standards.
• How the automatic flagging system works.
• How violations are categorised.
• How to document their justification for a review decision.
• Common false positive flags.
• How to avoid bias, including for cultural groups or those with disabilities.
• Broader guidance on how the remote proctoring software works, including how to log in, how to navigate to tests and violations, and how to get help.
• The best conditions for reviewing, such as wearing noise-cancelling headphones to identify illicit audio.
• General data privacy training, such as not sharing information with external sources, or making it public in any way.

3. Global violation settings

These scenarios will always be flagged as violations, by default:

• Constant background audio.
• Person in room.
• Imposter / proxy test-taker detected.
• Proctoring software manipulation.

4. Violation flagging settings

The proctoring system must be set up to automatically flag the correct violations. The following should be set:

1. No books or external notes.
2. No electronic devices like second monitors, mobile phones, smart watches, earphones, or scratchpads.
3. No computer applications or websites aside from the exam itself.
4. No breaks.
5. No food.
6. Drinks allowed.

5. Violations and penalties

These are common violations and their penalties, but the list isn’t exhaustive. If a Proctoring Analyst suspects foul play and the violation isn’t covered in this list, they should escalate to the Senior Proctoring Analyst with their concerns (more on this in section 7 – Violation review process).

High severity

Violation	Description	Violation setting(s) active	Penalty / action
Repeatedly staring away from the screen for prolonged periods.	Looking away from the screen 5 times or more during the entire exam, for periods of 3 seconds or longer, suggesting reading notes.	1. No books or external notes.   2. No electronic devices.	Exam fail.
Frequent talking to unidentified person.	Lips move and audio detected for 3 seconds or longer, with 3 or more instances over a 2-minute period, suggesting talking to someone else.	Person in room.	Exam fail.
Talking to another person.	Talks to another person in the room, with both voices detected. The other person may be on or off camera.	Person in room.	Exam fail.
Leaving the webcam’s view for a long period.	Leaves the webcam’s view for 30 seconds or longer.	4. No breaks.	Exam fail.
Long exam disconnection.	Candidates disconnects from the exam for a period of 2 minutes or longer, which may suggest cheating.	4. No breaks.	Exam fail.
Repeated, multiple short exam disconnections.	Candidate disconnects from the exam for a period of 2 minutes twice or more, which may suggest cheating.	4. No breaks.	Exam fail.
Using a prohibited external resource.	Looks at or interacts with a prohibited resource like a notebook or mobile phone.	1. No books or external notes. 2. No electronic devices.	Exam fail.
Using a prohibited software resource.	Looks at or interacts with a prohibited resource like a website or AI tool.	3. No computer applications or websites.	Exam fail.
Tampering with proctoring software.	Attempts to disable, bypass, or manipulate the proctoring software.	Proctoring software manipulation.	Exam fail.
Impersonation.	Attempts to use a proxy test-taker.	Imposter / proxy test-taker detected.	Exam fail.
Many mouse clicks outside of the exam window.	After already being warned, the candidate clicks outside of exam window (e.g. on another monitor) three times or more within 5 seconds.	3. No computer applications or websites.	Exam fail.
Pre-recorded or looping camera feed.	Uses software to show a fake video feed.	Proctoring software manipulation.	Exam fail.
Using earpiece.	Uses a hidden earpiece/earphones, likely to listen to audio notes.	2. No electronic devices.	Exam fail.
Constant background audio.	Constant background audio detected, which may be audio notes.	Constant background audio.	Exam fail.

Medium severity

Violation	Description	Violation setting(s) active	Action
Repeated, quick looks away from screen.	Looking away from the screen for a second or less, 3 or more times in quick succession, suggesting glancing at notes.	1. No books or external notes. 2. No electronic devices.	Consider alongside other flags and general behaviour.
Presence of a prohibited device.	A phone, tablet, smartwatch, earphones, or other device is in the immediate environment, which might be used for cheating.	2. No electronic devices.	Consider alongside other flags and general behaviour.
Repeatedly moves hands off camera	Places their hands off camera 3 times or more during the course of the exam, suggesting trying to access a cheating tool.	1. No books or external notes. 2. No electronic devices.	Consider alongside other flags and general behaviour.
Repeatedly leans off camera.	Leans off camera 3 times or more during the course of the exam, suggesting trying to access a cheating tool.	1. No books or external notes. 2. No electronic devices.	Consider alongside other flags and general behaviour.
Talking quietly, but infrequently.	Lips move and quiet audio detected for 3 seconds or longer, suggesting talking.	2. No electronic devices. Person in room.	Consider alongside other flags and general behaviour.
Intermittent background audio.	Intermittent background audio detected for 3 seconds or longer, with 3 or more instances over a 2-minute period, suggesting audio notes.	2. No electronic devices.	Consider alongside other flags and general behaviour.
Leaving the webcam’s view for a medium period.	Leaves the webcam’s view for 29 seconds or less.	4. No breaks.	Consider alongside other flags and general behaviour.
Repeated, limited short exam disconnections.	Candidate disconnects from the exam for a period of 2 minutes, twice. This may suggest cheating.	4. No breaks.	Consider alongside other flags and general behaviour.
Mouse clicks outside of the exam window.	Candidates clicks outside of exam window (e.g. on another monitor) three times or more within 5 seconds.	1. No books or external notes. 2. No electronic devices.	Consider alongside other flags and general behaviour.
Webcam obstruction.	The webcam is obstructed, either partially or fully.	Proctoring software manipulation.	Consider alongside other flags and general behaviour.
Using screen sharing or mirroring.	Uses a screen sharing or mirroring tool or feature.	2. No electronic devices.	Consider alongside other flags and general behaviour.

Low severity

Violation	Description	Violation setting(s) active	Action
Briefly looking away from the screen.	Looking away from the screen for 3 or more seconds.	1. No books or external notes. 2. No electronic devices.	Consider alongside other flags and general behaviour.
Temporary background noise.	Short, inconsistent bursts of sound (2 seconds or less) which may be other people, pets, traffic, etc.	2. No electronic devices. Person in room.	Consider alongside other flags and general behaviour.
One short exam disconnection.	Candidate disconnects from the exam for a period of 2 minutes or less, likely due to computer or internet issues.	4. No breaks.	Consider alongside other flags and general behaviour.
Talking to self.	Lips move but no audio detected, suggesting talking/thinking to self.	2. No electronic devices. Person in room.	Consider alongside other flags and general behaviour.
Leaving the webcam’s view for a short period.	Leaves the webcam’s view for 5 seconds or less.	4. No breaks.	Consider alongside other flags and general behaviour.

6. Roles and responsibilities

Proctoring Analyst

Role: reviews flagged violations first.
Responsibilities:

• Reviews flagged violations, and decides whether they’re true or false.
• Provides information on why they believe a violation is true.
• Re-classifies the severity of a violation, if needed.
• Escalates violations to senior reviewers, if needed.

Senior Proctoring Analyst

Role: reviews escalated flagged violations.
Responsibilities:

• Reviews escalated, typically borderline flagged violations, and decides whether they’re true or false.
• Provides information on why they believe a violation is true.
• Suggests a recommended penalty.
• Re-classifies the severity of a violation, if needed.

Academic Integrity Panel

Role: reviews and makes final decisions on flagged violations.
Responsibilities:

• Reviews final list of true violations as determined by Analysts, and decides whether they’re true or false.
• Provides information on why they believe a violation is true.
• Meets with students to get their perspectives on unclear/borderline violations.
• Decides a final penalty for true violations.

This panel should not include Analysts who completed prior reviews, to ensure fresh eyes.

QA & Compliance Officer

Role: ensures review process adheres to institutional and legal standards.
Responsibilities:

• Completes annual review of this framework, to identify improvements.
• Completes quarterly audits of 5% of true violations, to ensure accuracy and consistency.
• Ensures that compliance meets with all legal regulations.

7. Violation review process

Before we receive the flagged violations to review, the company providing our proctoring software will complete a preliminary review to remove any obvious false flags. This review will be based on our list of violations in this framework.

Once complete, and the final list of flagged violations is sent to us, our reviewers will evaluate each of them.

Each violation is located based on its timestamp on the recorded test session. The recording gives the reviewer video and audio for both the candidate taking the test, as well as what’s happening on the screen. In addition, they can see keystrokes and mouse activity.

Many remote proctoring tools classify flagged violations into levels, such as low, medium, and high, based on their severity. The violations should be ordered by priority level with the highest first.

This is the full review process:

1. Proctoring Analyst reviews incident

The Proctoring Analyst (reviewer) should examine the flagged incident, comparing what has happened with the explicit descriptions outlined for the suspected violation(s). They should check every relevant piece of data – particularly the video and audio – and use the helpful tools available in the software, like slow-motion and frame-by-frame scrubbing. In some instances, they may decide to reclassify the severity of a violation.

Some important factors that may indicate cheating:

• Suspicious movements are consistent and deliberate, like looking at the same off-screen area frequently.
• They occur alongside other violations, which the Analyst may want to review before making a decision.

2. Proctoring Analyst makes initial decision

After completing their evaluation, the Analyst can do two things:

1. Decide whether the violation is accurate

Confirm whether the flagged violation is true or false, and mark it as such. They can also add more information on why they believed the violation exists.

2. Escalate for re-review

If the Analyst is unsure whether a violation exists, they can escalate to the Senior  Proctoring Analyst for a second opinion. If this happens, the original Analyst should include notes on why they’re unsure.

3. Senior Proctoring Analyst reviews escalated violations

The Senior Proctoring Analyst reviews violations that have been escalated by the Proctoring Analyst, completing the same process while also considering the Proctoring Analyst’s notes.

When the review is complete for each violation, they mark it as true or false. For true violations, they will provide their recommended action based on the violation and penalties outlined in section 5.

4. Academic Integrity Panel makes final decision

The Academic Integrity Panel reviews the final list of violations from analysts, going through the same review process as a team.

Each violation can be marked as follows:

• False and dismissed.
• Unsure, requiring more information from the student. If this happens, two members of the panel arrange a meeting with the student to get their perspective. Once done, they re-meet as a team to make a final decision.
• In this case, the panel determines a penalty for the student, considering the recommended penalty outlined for the violation(s).

When true violations and penalties are determined, a member of the panel will meet with the student to explain the violation and reasoning behind the penalty, as well as their right to appeal. In addition, the information will be emailed as an official violation notice.

8. Appeals process

Students can appeal to decisions handed down from the Academic Integrity Panel. The process is as follows:

1. Student appeals

Students can respond to the official violation notice requesting an appeal, with the following information:

• Name.
• Name and date of exam.
• Reason for appeal / why they believe the violation is incorrect, or the penalty unfair.
• Supporting evidence like screenshots and medical notes.

This must happen within 1 week of receiving the violation notice.

2. Appeal reviewed by QA & Compliance Officer

The QA & Compliance Officer reviews the violation(s), following the same process as the Proctoring Analysts and Academic Integrity Panel.

If the violation is simple / clear-cut, they can maintain the original decision and notify the student via email within 7 days. If the violation is complex and difficult to judge, they can meet with the Academic Integrity Panel to re-review as a team, within 14 days.

If the violation and penalty is overturned, they will notify the student via email, and provide instructions on when the exam can be re-taken.

9. Audits

The QA & Compliance Officer will complete quarterly audits of true violations decided by the Academic Integrity Panel. This process is as follows:

1. Collect a random sample of 5% of violations

The Officer collates a random sample of 5% of all true violations for the quarter.

2. Review each case

The Officer reviews each case, following the same review process as Analysts and the Academic Integrity Panel. They determine whether they believe each violation is true, or false.

3. Investigate discrepancies

If discrepancies are identified, each questionable violation is investigated further, which may include meeting with the Academic Integrity Panel and the student.

4. Consider examples of bias

In addition to reviewing each case, bias is considered for the sampled cases as a whole. This includes biases for:

• Race / cultural.
• Age.
• Gender.
• Income.
• Disability.

5. Create final report with recommendations

Based on the any discrepancies and biases found through the audit, a final report is created which outlines any recommendations, particularly on how improvements can be made to the process. This may include additional staff training.