Translating global assessment in local languages: making it valid, reliable and fair

Translating an assessment into different languages has implications on fairness and validity

For an online test to be valid, reliable and fair it requires high quality test items that are fully translated and localised to meet the needs of the learners in their context. That work demands the expertise of skilled professionals who are qualified to provide linguistic quality control.

cApStAn, founded by Steve Dept, is a world-renowned organisation that’s recognised for excellence in providing linguistic quality control. Steve offered valuable insight into the importance of the work his organisation does to ensure valid and fair testing.

He advised that, “International Large-Scale Assessments (ILSAs) such as the OECD/PISA (Programme for International Student Assessment) use standardised instruments to collect data on the knowledge and skills of students, and on contextual variables.

If the results are used to make inferences and for benchmarking, the assumption is that the instruments are valid, reliable and fair. Translating an assessment into different languages has implications on fairness and validity.”

Validity, reliability and fairness are critical elements of all high-quality assessments:

1. Fairness affords every student equal opportunity to show what they know
2. Reliability ensures the repeatability of assessments, assuring us that different administrations of the assessment are equivalent to the original test
3. Validity attests to the fact the test measures what it intended to measure for its intended purpose, and nothing more

It’s crucial to purposefully translate and adapt assessment items into minority languages

Joanne Caddy from the OECD’s Directorate for Education and Skills explained the importance of making the PISA-based Test for Schools assessment available in local languages around the world: “We aim to empower schools with meaningful data on their student’s learning outcomes as well as their social and emotional skills.

We also give students a voice by asking them to share their perspectives on their classroom, their motivation for learning and experience of distance learning during the pandemic. An authentic assessment is one that uses the student’s language of instruction and reflects their daily lives, which is why we place such a strong emphasis on ensuring high quality translations.”

The preservation and continuation of local languages, especially minority ones, is a positive benefit of the work we do to ensure the online assessments we deliver are valid, reliable and fair. Steve Dept from cApStAn supports this.

He shared that, “It is crucial to purposefully translate and adapt assessment items into minority languages used as a medium of instruction: if a thorough methodology is applied and if due attention is given to both faithfulness to the source version and fluency in the target version, one creates a measurement tool that will provide a wealth of granular information on education outcomes.”

Welsh-medium schools can now measure their school performance on a global scale

In 2021, when Janison was accredited as the National Service Provider to deliver the OECD PISA-based Test for School across the UK, the translation of the assessment into Welsh quickly became a critical endeavour. This would ensure that Welsh-medium schools would be able to test in their local language of instruction. Thus, granting them access to localised data linked to the global PISA framework.

With the valuable support of the regional consortia of Wales, the OECD, Janison and cApStAn invested time and effort to produce a localised assessment that enables Welsh medium schools to engage with PISA for Schools in their language of instruction.

Reflecting on the project, Steve Dept attested to the high technical standards of the work completed: “In Wales, a Welsh language version of the PISA-based test for schools was prepared in adherence with the International Test Commission’s Guidelines for Translating and Adapting Tests (2018).

cApStAn has developed, tested, and improved linguistic quality assurance steps and corresponding metrics to ensure that the translated and adapted versions produced by PISA participants met the required comparability standards, while conforming to local usage and context.

In close collaboration with Janison, cApStAn applied the methodology implemented in PISA to the translation, adaptation and linguistic quality assurance of the Welsh-medium PISA-based Test for Schools, working with accredited linguists who have extensive experience in adapting educational materials.”

It speaks strongly to our deep-rooted commitment to equity of access for all learners

As Janison’s Chief Academic Officer, Sara Ratner couldn’t be more pleased that schools across Wales now have access to a localised, translated global assessment that can provide them with reliable data to inform school improvement plans whilst meeting the rigorous standards that cApStAn is renowned for.

“The work that the Janison team have done in partnership with the OECD, the Welsh regional consortia and cApStAn is to be celebrated. It speaks strongly to our deep-rooted commitment to equity of access for all. Despite the ongoing impacts of COVID-related challenges, many schools across Wales have already embraced the opportunity to undertake a global assessment in their local language or plan to do so in March 2022 when the next testing window is scheduled.”

Offering a global, digital assessment in a local language is an important feature of the PISA-based Test for Schools. Working in partnership with the OECD, cApStAn and with the support of National Service Providers in each country, the Janison Insights platform is currently able to offer the PISA-based Test for Schools in English (Australian, British and US), French, German, Spanish, Catalan, Arabic, Thai, Russian, Kazakh, Kazakh Russian, Portuguese, Brazilian Portuguese, Chinese (Mandarin), Japanese and now, Welsh!

Together we continue to advocate for the importance of providing global assessments that support local contexts and test using the language of instruction, whatever that may be.

In the meantime, to the schools of Wales we say, Croeso i PISA ar gyfer ysgolion Cymraeg!