IELTS Online: Overview

IELTS Online: Overview

IELTS Research Group (February 2023)

Dr Emma Bruce, Research and Validation of IELTS (British Council)

Dr Tony Clark, Head of IELTS Research (Cambridge University Press & Assessment)

Abstract

IELTS Online is the newest addition to the IELTS portfolio. This remote administration of the test complements the existing in-centre versions - IELTS on computer and IELTS on paper - and allows test-takers to take IELTS anywhere, providing they have a stable internet connection and a quiet, private space. Regardless of the delivery mode, the fundamental quality considerations remain the same. IELTS Online must maintain existing standards through accurate, fair and secure assessment practices, and avoid new irrelevant variance in scores or unintended consequences, thereby yielding reliable and meaningful measurement, without bias.

Here we provide an overview of the multiple sources of validity evidence currently available, relevant test security measures, and the upcoming research agenda for the IELTS Research Group.

Background

IELTS Online was launched in 2022, using a gradual approach to rollout based on the high-stakes nature of the test, and the potential impact on a large number of test-takers’ educational experiences worldwide. Although the substantial changes in assessment practices brought about by the pandemic necessitated an initial rapid response (in the form of IELTS Indicator, discussed below), a longer-term approach to valid and reliable high-stakes testing required a collation of existing research evidence combined with highly advanced security protocols. The result of this combined work is IELTS Online, a securely-delivered alternative to the trusted test centre version.

The particular challenge faced by the IELTS Partners was how to move an existing in-person test online, using an approach that allows test-takers the same opportunity that they would receive at a secure test centre. The following report describes how previous research was used to achieve this, some of the challenges faced as part of the process, and what the IELTS Research Group is currently investigating to continue to inform decision making.

Score and Construct Equivalence Studies (used to build on earlier research)

Existing research laid the foundations for the development decisions (Clark, Spiby, & Tasviri, 2021) surrounding at-home testing for IELTS required during the pandemic (Clark, Holland, & Spiby, 2022, Spiby & Clark, forthcoming 2023). Looking at previous research in more detail helps to clarify how this paved the way for IELTS Online (see Figure 1).

Figure 1: Validity evidence for IELTS Online

IELTS on Computer Studies Provided Strong Evidence for Technological Development

Previous studies into IELTS on computer (or Computer-delivered IELTS as it was previously known) provided a solid foundation for the technological development of IELTS Indicator, using validation research to connect the two delivery modes (IELTS on computer and IELTS Indicator).

It was not the first time that IELTS research had investigated ways to introduce new technological capabilities as part of innovative test delivery, but to combine these capabilities simultaneously in response to test centre closures during the pandemic was unprecedented.

Earlier internal IELTS research (2001-2005) provided evidence that the difficulty level of paper-based and computer-delivered modes were comparable, revealing strong score equivalence between the two for Listening, Reading and Writing (Blackhurst, 2007, Green, 2004, Maycock, 2004). Further large-scale trials followed, confirming earlier findings. External researchers commissioned to investigate this found that test-takers’ cognitive processing was similar across paper and computer IELTS tests (Weir, O’Sullivan, & Yin, 2007). Other independent work has since found that differences are minimal (e.g., Chan, Bax, & Weir, 2018).

Essentially, whether responses were handwritten or typed had little impact on scores. Continual monitoring of scores takes place at Cambridge to detect any issues that may arise, with extra vigilance since the international rollout of IELTS on computer, IELTS Indicator and now IELTS Online.

Extending Video-Call (Speaking) Technology Beyond its Original Purpose

As the comparability research on IELTS on computer provided a solid starting point for online test administration for Listening, Reading, and Writing, our attention turned to Speaking. Maintaining the face-to-face interaction with a human examiner in an online setting became the focus as this is a unique differentiator between IELTS and other computer-delivered tests. Existing recent research on Video-Call (VC) Speaking was central to this.

A four-stage trial (2016-2021) of the use of VC technology in a test centre to administer IELTS Speaking remotely was conducted and documented as part of the IELTS Research Reports and in academic journals (Nakatsuhara, Inoue, Berry, & Galaczi, 2021). Key findings which have informed online Speaking test delivery are:

• equivalence of scores between face-to-face and VC Speaking
• stable speaking construct (i.e. what was measured by the test) between the two modes
• examiner confidence in technology’s affordance for test-takers to demonstrate proficiency
• capability of available technology to address issues identified in earlier trials
• favourable test-taker perceptions and useful suggestions for change.

By early 2020 (and the beginning of the pandemic) VC Speaking was being used at a number of test centres globally. Switching to the VC mode for at-home testing was enabled by the extensive trialling noted above, and ultimately IELTS’ USP was able to remain as part of the IELTS Online test as a result.

Indicator Test-taker Surveys and Examiner Focus Groups

Although IELTS Indicator was decommissioned once test centres reopened, the experience provided us with a valuable opportunity to learn how each aspect of online delivery functioned, and what changes were required for the next stage.

In order to learn from this experience, large-scale surveys of test-takers (n = 2,444) were conducted, in addition to small-scale examiner focus groups This was to collect views on all aspects of the online testing process. Key insights gained from test-takers were:

• positive perceptions towards at-home testing overall, with the reduction of stress noted as a factor
• convenience was a clear benefit to many, as was less travel
• a quiet environment without distractions was possible for most test-takers (60%)
• anxiety around technology in the early stages of the pandemic led to many preferring the test centre experience
• communication between test-takers and examiners was an important issue for successful online test delivery (see Proctoring section below).

Key insights gained from examiner focus groups were:

• positive perceptions of IELTS Indicator and a desire for online testing to continue in some form
• due to the shift towards remote working, online testing may reflect real-life experiences better
• many test-takers (not all) seemed more relaxed
• less support available online in the initial rollout, supporting the test-taker feedback
• convenience for the test-taker (particularly choosing timeslots) and increased access to testing
• security was a concern for IELTS Indicator in 2020
• interruptions to the test due to technology issues were not uncommon.

Harnessing the benefits - while addressing the issues raised - was a necessity for subsequent online testing and essential for rollout of IELTS Online in 2022.

Our Multi-layered Approach to Test Security

Ensuring the validity of test scores relies on the requirement that each test is taken by the correct person, according to proper procedures and following security guidelines. A security failure can impact the validity of the scores and, therefore, the integrity of the test (Guidelines for Technology-Based Assessment, Association of Test Publishers, & International Test Commission, 2022).

Owing to the high-stakes decisions made on the basis of IELTS scores, test security is of paramount importance, regardless of the delivery mode. IELTS is a safe and trusted test and our approach to online delivery does not compromise the high standards we uphold for paper and computer tests. Instead, we enhance existing controls and leverage our experience in providing secure tests to implement specific checks for the online space.

According to the Guidelines for Technology-Based Assessment (2022), this involves three strategies: prevention, deterrence, and detection/response. We adopt an all-round, end-to-end approach to security encompassing these strategies to minimise opportunities for malpractice and deliver a testing experience that is comparable to in-centre tests.

Remote test administration carries an inherent risk and poses increased challenges, as evidenced by the increase in cheating reported by Meazure Learning (2022), the parent company of ProctorU. We translate our historical experience of high standards into the online environment by implementing appropriate pre-, during-, and post-test mitigations to create as level a playing field as possible regardless of the mode of test delivery. Figure 2 shows the range of security checks adopted throughout the whole testing process, from registration to results, to protect the integrity of the test and ensure fairness for all test-takers.

Figure 2: Multi-layered and comprehensive end-to-end approach to security (pre-, during and post-test)

Identity Document Verification

Identity Document (ID) verification procedures for IELTS Online are modelled on those for in-centre tests. ID checks happen at multiple points during the booking process and test administration, including a recheck if the test is interrupted for any reason. Test-takers must provide a current, valid ID to register for IELTS Online, and as part of the login process, the test-taker must take a photo of the same ID for verification. A trained greeter reviews the ID to ensure it matches that used at registration and the actual person sitting the test. The greeter also checks the ID photos taken during the Speaking test to ensure that the same person has attended both parts of the test. During this verification phase, the greeter performs a 360° room scan to check that the testing environment is compliant with the regulations.

The Purpose-built Platform

IELTS Online uses a bespoke solution for IELTS. A custom-built platform was necessary to meet the high expectations and strict specifications required to maintain our existing standards. The platform is a fully integrated solution which locks down the test-taker’s device at the operating system level, not just the browser level. This prevents test-takers accessing the internet or any other program for the duration of the test. Test-takers must download this platform before the test.

Speaking tests are administered live online with an IELTS-trained Examiner, and the VC Speaking software is integrated into this purpose-built test platform to streamline the experience for test-takers and Examiners.

Proctoring for Support and Security

Various options for remote proctoring exist. Solutions can be machine- or human-driven and can involve live invigilation or record and review (Isbell & Kremmel, 2020). IELTS Online adopts a combination of both. Test-takers are monitored in real time by specially trained IELTS proctors for the full duration of the test. An optimum ratio of proctors to test-takers not only facilitates continual monitoring capability but allows proctors to respond to test-takers who raise their hand to ask for help. Preliminary research shows favourable test-taker perceptions towards live human proctoring. Simply knowing that help is at hand eases test-taker anxiety (Bruce & Clark, forthcoming, 2023). In addition, the robust monitoring increases test-taker confidence and satisfaction owing to a trust that other test-takers can’t cheat.

Artificial Intelligence (AI) capabilities complement human proctoring to help ensure that no suspicious activity goes unnoticed, for example by identifying anomalous eye, head and face movements, unusual behaviour or background noise. Suspicious incidents are immediately investigated during the test session and the human proctor interacts to clarify instructions or deliver warnings to test-takers. If necessary, the proctor seeks a second opinion from the proctoring supervisor before taking appropriate action, which may involve further post-test investigation.

There are stringent criteria for aborting tests and banning test-takers if breaches are identified. To align with in-centre testing, the specific list of behaviours applied for banning is not publicised.

Unique Test Content

Each individual test is unique to the test-taker. In addition to remote proctoring and other mitigations, this lowers the threat of usable information being shared by test-takers after the test. Content harvesting is a very real threat to all high-stakes tests, and to mitigate this, we constantly monitor websites and social media to take down curated content and crib sheets. If such information is found, it forms part of the pre-results checks detailed below. Alongside these measures, we are able to refresh the item bank to introduce new content and add variety.

Pre-release Results Check

Regardless of which delivery mode the test-taker chooses, every IELTS result goes through a set of statistical analyses to ensure accuracy and validity. These stringent post-test analyses, honed over many years, involve mature solutions and algorithms to capture unusual scoring patterns or suspicious scores.

Test event recordings that are flagged for potential malpractice are also verified during this phase and validated before results are released. In addition, a plagiarism check has been implemented to review and compare all written responses for all IELTS Online tests. Suspicious responses are analysed by cross-partner investigation teams.

Continual Auditing Process

A new, continual auditing process has been implemented. This ensures necessary measures are taken to prevent security breaches as soon as we learn about new threats and challenges to the integrity of IELTS Online.

Maintaining our High Standards through Security Measures

All of the processes described above are supported by dedicated security and investigation teams who conduct additional monitoring of test-takers and are available to verify results post-test. Our stringent downstream post-test processes serve as an important safety net, as research shows that test-takers will continually try to get ahead of the system (Dawson, 2020). Combining all these measures together ensures that the test is conducted in as safe an environment as possible to maintain our high standards of test integrity in the new delivery format.

Our Continuous Programme of Research for IELTS Online

The studies outlined above have provided a solid research basis to support the development and launch of IELTS Online. We are currently conducting preliminary research into the early stages of the IELTS Online launch by gathering the perceptions of key stakeholders. We are collecting insights from test-takers on their experience of taking a remote proctored test at home, and whether they believe they were able to perform to the best of their ability. This includes test-taker perceptions of our bespoke platform and approach to remote proctoring, conducted through a post-test survey and follow-up interviews. We are collecting survey and interview data from online proctors and speaking examiners on all aspects of the test delivery, including the security, the user-friendliness of the platforms, and their perceptions of the test-taker experience. In addition, we are investigating score equivalence between in-centre and IELTS Online tests as the latter is rolled out globally. This is an ongoing process as implementation is gradual, due to the high-stakes nature of the test and the large amount of data required to perform such analyses.

References

• Association of Test Publishers, & International Test Commission (2022). Guidelines for Technology-Based Assessment. Available at: https://www.testpublishers.org/assets/TBA%20Guidelines%203-14-2022%20draft%20numbered.pdf
• Blackhurst, A. (2007). Computer-based and paper-based versions of IELTS. In O. Alexander (ed.), New Approaches to Materials Development for Language Learning (pp. 265-274). Bern: Peter Lang.
• Bruce, E. & Clark, T. (forthcoming, 2023). Hybrid Remote Proctoring and the Human Touch Model: investigating the relationship between secure online language assessment and stakeholder experiences (manuscript in preparation).
• Chan, S., Bax, S., & Weir, C. (2018). Researching participants taking IELTS Academic Writing Task 2 (AWT2) in paper mode and in computer mode in terms of score equivalence, cognitive validity and other factors. IELTS Research Reports Online, 4/2017. IELTS Partners: British Council, IDP: IELTS Australia and Cambridge Assessment English.
• Clark, T., Holland, M. & Spiby, R. (2022). Seeking Empirical Evidence to Support Online Test Validation: Building on the Indicator Assessment Model. In K. Sadeghi (ed.), Language Assessment at the Time of the COVID-19 Pandemic (in press).
• Clark, T., Spiby R., & Tasviri, R. (2021). Crisis, Collaboration, Recovery: IELTS and COVID-19. Language Assessment Quarterly, 18:1, 17-25.
• Dawson, P. (2020). Defending Assessment Security in a Digital World: Preventing E-Cheating and Supporting Academic Integrity in Higher Education. https://doi.org/10.4324/9780429324178
• Green, A. (2004). Comparisons of Computer and Paper-based Versions of IELTS Writing: A Further Investigation of Trial A Data. Cambridge: Cambridge ESOL Internal Validation Report 585.
• Isbell, D., & Kremmel, B. (2020). A literature review on perspectives from related fields with implications for IELTS@Home. Internal Research Report.
• Maycock, L. (2004). CB IELTS: Report on the Findings of Trial A (Live Trial, 2003/04). Cambridge: Cambridge ESOL Internal Validation Report 558.
• Meazure Learning. (2022). The 2021 Exam Conduct and Integrity Report: A Benchmarking Study of Integrity in Academic and Professional Assessment Settings. Available at: https://www.meazurelearning.com/resources/2021-exam-conduct-and-integrity-report
• Nakatsuhara, F., Inoue, C., Berry, V., & Galaczi, E. (2021). Video-conferencing speaking tests: do they measure the same construct as face-to-face tests? Assessment in Education: Principles, Policy & Practice, 28:4, 369-388. DOI: 10.1080/0969594X.2021.1951163
• Spiby, R. & Clark, T. (forthcoming 2023). Responding to a transformed assessment landscape: the example of IELTS. In M. Kantarcioglu et al. (eds.), Life After the Pandemic: ReVision and ReForm in EAP Contexts (in press).
• Weir, C. J., O’Sullivan, B., Jin, Y. (2007). Does the computer make a difference? The reaction of candidates to a computer-based versus a traditional hand-written form of the IELTS Writing component: Effects and impact. IELTS Research Report, 7. IELTS Partners: British Council, IDP: IELTS Australia and Cambridge Assessment English.