Mind the gap(s): exploring university teaching staff’s perceptions of IELTS writing versus university writing requirements

Abstract

This study investigated the level of awareness among academic staff with the role of admissions tutor regarding what IELTS writing scores actually represent in practice, namely what can be expected of international students (ISs) in terms of the academic writing they produce.

We investigated these academics’ knowledge of IELTS and how they had gained this knowledge, including any training they had received. We captured a baseline picture of this level of awareness through a questionnaire sent to universities throughout the UK. Subsequently, we investigated the themes from the questionnaire in more depth via interviews with a subset of the academics and then also investigated the features academics value in students’ writing, using a think-aloud procedure in which we asked the same subset to evaluate a set of IELTS scripts. A group of IELTS examiners also undertook the same think-aloud procedure in order to compare their evaluations with those of the academics to better understand the application of the IELTS descriptors in practice.

We found an overall lack of familiarity with the writing tasks and the descriptors amongst the academics, as well as differences in whether they perceived the IELTS score as representing either a) sufficient current ability or b) potential to produce the types of writing required of international students. Both groups of participants drew on a core of terms to indicate the features they valued while judging the scripts but each group also used terms not used by the other. The study also found, in addition, that some scripts were deemed by the academics to be at an acceptable level for entry to the academics’ courses whereas the IELTS score actually awarded the scripts would have precluded these candidates from entry. While it was confirmed that these academics are not generally involved in score setting, we suggest that an increased level of awareness about IELTS would still be of benefit to enhance the study experience for both students and tutors.

1 Introduction

International students (ISs) are increasingly being recruited to UK universities (Pearson, 2020) thanks to the internationalisation agenda in higher education (HE). Large numbers of ISs are entering British HE each year, with this figure totalling 605,130 in 2020-21 (HESA, 2022). Despite some recent fluctuations due to Brexit and the COVID-19 pandemic (Scott & Mhunpiew, 2021), in general, academics are more likely than ever to have ISs on their courses. IELTS often serves a key gatekeeping function so it is important that academic staff understand what IELTS scores represent in terms of ISs’ actual language abilities. A sound understanding of what these scores represent would enable academics, where appropriate, to adjust their expectations of their students accordingly (Jenkins & Wingate, 2015) and to signpost students to any relevant support, where available. This, in turn, may improve the student experience and student outcomes. The aims of this study are, thus, to explore and illuminate what academic staff at HEs value in the writing skills of international students, as evidenced in the writing they produce in an IELTS exam, and to compare this with the writing skills valued for their own subject disciplines, in order to better understand where gaps may lie.

2 Background

ISs are offered a place on a course at a higher education institution (HEI) on the basis of two components: their subject-specific ability and their English language ability. Our focus is on the latter. In the UK, the UKVI (United Kingdom Visas & Immigration) ¹ now plays a significant role in setting the threshold level of language ability requirements. Students wishing to study at degree level or above at a university in the UK are required to prove language ability at B2 level of the CEFR (Common European Framework of Reference for Languages). The threshold level of IELTS 5.5 (or equivalent), therefore, needs to be evidenced before a place on a level 4 (undergraduate first year) course can be taken up, although students may join pre-sessional ² courses with lower score. This evidence of language ability is gained from students sitting a Secure English Language Test (SELT), and IELTS is one of a short list of exams approved for such purposes. Minimum English language requirements are set by each university and differ from course to course, higher scores being required by those courses deemed more linguistically challenging in all four skills (MacDonald, 2019).

Not enough is known about how exactly the language requirements for each course are decided (Lam et al., 2021) and such processes will differ from institution to institution but typically this involves ‘limited engagement with guidance from IELTS’ (Lam et al., 2021: 5). Academic admissions tutors (defined for our purposes as staff who currently teach their subject specialism to students at universities in the UK and who also have responsibility for student admissions to a specified degree program, and are typically course leaders or equivalent), have usually been involved at some point to set the required IELTS score but, subsequently, admission on the basis of the language requirement usually becomes an administrative matter rather than an individual judgement on the part of an admissions officer, whether an academic or administrative member of staff, who will most likely be referring to a minimum required score set at some point in the course’s past. Unless a course undergoes significant alteration, there is no reason for that entry score to be regularly reviewed or amended. Lam et al. note: ‘Changes to the minimum requirements are often motivated by recruitment considerations, benchmarked against rival institutions or neighbouring departments’ (2021: 5).

1. UKVI is the government body which regulates immigration. See: Student visa: Knowledge of English - GOV.UK (www.gov.uk)

2. Courses at HEIs which ISs might attend immediately prior to the commencement of their main degree course to develop and/or consolidate English language skills (as well as study skills and maybe also subject-specific knowledge) to prepare them for study at a UK HEI.

For a candidate to gain access to, and then be successful on, their chosen course, various actors play a key role: the candidates themselves, the admission officer(s) of the target HEI (whether academic admissions tutor or administrative admissions staff), the English language skills and /or academic skills support staff at the HEI and the students’ academic tutors. From this group of actors, it is, in most cases, the academic tutors who come into most frequent contact with these international students, being the ones who deliver their lectures, and maybe tutorials and/ or seminars, and also mark their work, or have overall responsibility for such marking, and additionally may hold a pastoral care role for them. While they may not have detailed knowledge of IELTS and probably do not know the exact score each student achieved, the fact that ISs have had to pass an exam to enter university will raise an expectation of a certain level of ability in English. Research findings (Lloyd-Jones et al., 2007; Jenkins & Wingate, 2015) along with anecdotal evidence from HE tutors, suggest that there is often a gap between tutors’ expectations of ISs on entry to a university course and their achievements when writing assignments as part of their degree. This research, therefore, focuses on academic tutors and their understanding of what IELTS scores actually mean in practice; we examine to what extent the language skills which are valued for different disciplines match what the students can actually produce.

One may ask why academics should be interested in what IELTS scores mean and why they should gain a better understanding of what an IELTS performance actually represents. As well as wishing to best support ISs in their studies, the obligation on UK universities to report on a range of metrics linked to student attainment and satisfaction also likely motivates academics to attend to how well students cope with the writing demands of their degree. For example, universities must report outcomes of the National Student Survey (NSS), which aims to measure both student satisfaction and the student experience and must also provide data on student attainment, such as level of qualifications awarded, progression and retention, to the Higher Education Statistics Agency (HESA). These metrics all feed into league tables (e.g. The Times & Sunday Times Good University Guide, The Guardian University Guide and The Complete University Guide), facilitating comparison of educational providers. Increasing marketisation results in universities needing to secure student numbers in competition with comparable HEIs. This means greater effort is expended than ever before on actively ensuring student satisfaction and positive outcomes, and, above all, being able to measure this via the metrics described above.

The more academics can do, the better, to understand they should not assume that, since the ISs have gained the required IELTS score for entry, they are able and confident to tackle their written course work. Any steps towards enhancing the students’ experience of HE, such as receiving adequate support and thus improved outcomes in their written assessments, enabling them to fulfil their potential, would thus benefit the institution as a whole.

2.1 Academics’ understanding of IELTS

The current study is based on the premise that academic teaching staff are relatively unfamiliar with the requirements of IELTS in terms of the constructs assessed. Previous research has established that, in many cases, academics (and administrative admissions staff) in universities lack a comprehensive understanding of the meaning of IELTS scores (e.g. Banerjee, 2003; Coleman, Starfield & Hagan, 2003; Smith & Haslett, 2007; O’Loughlin, 2008; Hyatt & Brooks, 2009). Additionally, Hyatt (2013) found that most academic staff had limited understanding of how IELTS worked, concerning both the content and process of testing.

In an effort to guarantee transparency, the public version of the IELTS band descriptors is freely available online from the British Council’s website (which also includes information about why IELTS scores should be accepted and other information for score users) and in the IELTS Score Setting Guide (IELTS 2015: 39-40). However, it is unclear how well known these descriptors are and how they are used, if at all, to understand the demands of IELTS and consequently better understand the ISs’ abilities, beyond those in the language testing community. Valiant efforts have been made to produce materials to explain the nature and role of IELTS in the life of our international students by other bodies, such as the material produced by BALEAP, which is lodged on the BALEAP website. However, BALEAP is unlikely to be familiar to academic staff outside the sphere of EAP/TESOL. This useful material is unlikely to be well known or widely used in academia and needs wider publicity.

Hyatt (2013) suggested that one reason for academics’ lack of familiarity with IELTS may be that staff might lack time to familiarise themselves with how the assessment works and what the scores represent. Another reason for a lack of nuanced understanding of IELTS writing descriptors may be the fact that IELTS has its own symbolic value (Smith & Haslett, 2007; O’Loughlin, 2015) and high face validity due to its brand. Its functional role in indicating acceptability for admission (Coleman, Starfield & Hagan, 2003) may actually militate against investigation into what different scores mean, since admissions staff trust the IELTS brand and accept the scores at face value (see Hyatt, 2013) or as Lam et al. (2021:5) suggest, ‘IELTS scores are used and trusted as a default form of evidence to satisfy the English language requirement for admission’.

2.2 The role of writing at university

While various studies, including Hyatt (2013) and O’Loughlin (2013), have looked at HE stakeholders’ perceptions of IELTS as a whole, few have focused specifically on the writing element, despite its importance as a means of assessment in higher education. All four of the language skills assessed in IELTS contribute to the students’ communicative competence. This is reflected in the fact that minimum sub-scores for each skill are usually required by universities (Lam et al., 2021). However, of the four skills, ‘writing is the most crucial of the skills needed in an academic context’ (Alexander et al., 2008:178) and the role of writing in higher education in the UK has been the object of extensive research (among others: Lea, 1994; Lea & Street, 1998; Lea & Stierer, 2000; Nesi & Gardner, 2012). Alexander et al. (2008) even claim writing is the skill most widely used in academic communication. Although this may be less true of STEM subjects than the humanities (MacDonald, 2019), writing still constitutes the main mode of assessment through which subject knowledge is assessed overall at university in the UK (Coffin et al., 2003; Li & Hu, 2018).

Writing, it has been suggested (Li et al., 2010; Lloyd-Jones et al., 2007), poses the widest range of challenges to students because of the range of conventions and genres that have to be mastered. For instance, Nesi & Gardner (2012) identified 13 genre families commonly found at university, and Alexander et al. (2008) list 15 types of writing that students across disciplines may be required to produce. Writing effectively in these genres draws on skills and techniques which differ from non-academic writing to an extent that speaking, reading and listening skills do not. For example, all university students must learn about specialised writing conventions such as establishing a ‘disinterested’ tone of voice, structuring a written argument, and using citation. Furthermore, it must be noted all proficient users of English, whether English L1 or L2 users, must learn how to write effectively, it not being an easily acquired skill (Alexander et al., 2008). This highlights the challenges entailed.

On top of the challenges posed by writing at university, the additional challenges faced by L2 English writers must be acknowledged, as documented in the literature on the assessment of second language writing (e.g. Hamp-Lyons, 1991; Weigle, 2002; Shaw & Weir, 2007). Iannelli and Huang (2014) found that ISs performed worse in assessments, in general, than their UK counterparts, and Li, Chen & Duanmu (2010) have suggested that this is especially so in writing. The challenges may be partly explained by different approaches to assessment the ISs may have encountered prior to starting their courses. Firstly, the norms of the education system in their home country will have shaped the IS’s expectations of what is required in assessments. On reaching the UK they may then experience a degree of dissonance between teaching and assessment styles on any pre-sessional courses they attend (which focus predominantly on language skills) and faculty teaching and assessment norms (which focus on subject-specific knowledge and skills) (Johns, 1991; Erling & Richardson, 2010).

Green’s (2007) work has demonstrated that while there is a certain overlap between academic writing at university and the writing requirements of tasks in exams such as IELTS, there is not a complete match. IELTS, for example, due to its aims, scope, international nature and practical constraints which these points entail, can only assess a core of generic academic writing skills, represented in the writing band descriptors. While in many cases students will lack certain writing skills upon entry to university, despite having achieved the required IELTS score, this also means that some international students may also possess other academic skills, including subject-specific skills which are not constructs represented in the IELTS descriptors. Without awareness of these differences, academic staff are in a poor position to understand what students are able to produce on entry to their courses.

3 Research questions

Given the aims of the study, our research questions are as follows.

Research Question 1: How familiar are UK HE academic staff with the IELTS academic writing task?

1a. How familiar are academic staff with the IELTS academic writing band descriptors? 1b. How did academic staff gain this familiarity with the writing band descriptors?

Research Question 2: How do UK HE academic staff perceive the relationship between writing abilities required for the IELTS writing paper and the writing demands of degree programs?

4 Data collection

4.1 Overview

Our study addressed the questions above by drawing on three data collection techniques: a questionnaire, interviews and think-alouds (also known as verbal protocols). Our participants were academic admissions tutors at UK HEIs (henceforth referred to simply as academics). We compared the three data sources to better understand how academic staff perceive the relationship between writing required on their degree courses and what students are required to produce during the IELTS exam.

Participants in each group volunteered their time to participate (no payments were made).

The techniques were utilised in three stages. Stage 1 involved the questionnaire sent to academics, Stage 2 involved interviewing a subset of those who responded to the questionnaire, and Stage 3 involved conducting think-aloud sessions with the same sub-set. In Stage 3 we additionally undertook the think-alouds with a group of qualified IELTS examiners. The examiner data provided a baseline picture of how the descriptors are interpreted in practice and thus what the IELTS writing paper requires of students.

Table 1: Stages of data collection and participants

4.2 Participants

4.2.1 The academics

The first participant group comprised only academics. Unlike various other studies which covered admissions staff in general (e.g. Coleman, Starfield & Hagan, 2003; Hyatt, 2013), our study focused on academic staff and did not include admissions staff working in purely an administrative or managerial capacity within an academic environment (i.e. not teaching). This group was targeted as they held knowledge of both their subject areas’ academic expectations and the relevant teaching practices, but also would most likely know of, and about, IELTS, although to varying extents, which we could not assume about all admissions staff.

Academics working in 23 different universities were targeted, representing different types of university (see Table 2) from a range of disciplines, based on JACS codes from HESA (see Appendix 2) and located in different parts of the UK (covering England, Northern Ireland, Scotland and Wales). This allowed us to reach a range of informants, to compare data across different contexts, and thus potentially also reach a range of candidates for the subsequent stages of our research with academic staff. All responses to the questionnaire were entirely anonymous.

Table 2: Number and type of universities and departments contacted

Ancient refers to those universities established before the 19^th century; Red Brick include those formerly known as Civic Universities, mainly focusing on science and engineering, established in 19^th century. Newer Red Brick are as per Red Brick but established in 20^th century. Plate Glass universities were established mostly in response to a call for more universities as a result of the 1960’s socio-economic boom while the Post-92’s include any university established or granted university status after 1992 when various former UK polytechnics were granted university status.

We issued the invitation to participate on three occasions with an interval of approximately one month between each round to maximise the chance of academic staff having time to respond, aiming to accommodate the ebb and flow of the academic calendar. Those who had already responded were eliminated from the next invitation round as a courtesy to acknowledge the typically heavy flow of emails academics face.

After the third round of sending invitations, we closed access and moved to analysing the data. We received 66 responses, of which 56 responses were valid for our purposes. This was lower than we had originally envisaged. The questionnaire was open from June to September 2021, when the effects of the COVID-19 pandemic in general were increasing academics’ workloads as teaching moved online. This could account for reduced uptake of invitations to participate in research.

Academics from 20 different universities took part, covering all the university types outlined above in Table 2, with more than one academic from the same institution replying in 13 cases.

Table 3: Number and type of universities represented in the questionnaire responses

No. of different institutions represented in the responses

4.2.2 The questionnaire respondent profile

The overwhelming majority (approximately three-quarters) of our respondents had substantial experience of teaching ISs, with 68% reporting over 10 years of such experience, plus another 30% reporting 3-10 years’ experience. As shown in Figure 1, they also had considerable experience of working as admissions tutors. This meant they had sufficient experience to give a truly informed response to our queries and thus provide useful data.

Figure 1: The questionnaire respondents’ experience in the role of admissions tutor

The respondents represented 18 different academic disciplines, with the most highly represented subjects being engineering and technology, subjects allied to medicine, creative arts and design (10% each) and education (12%).

4.2.3 The interview and think-aloud participants (academics sub-group)

All questionnaire respondents were invited to participate in Stage 2 and indicated their willingness to do so by providing contact details in the final item on the questionnaire. Five academics, representing red brick and plate glass institutions, took part in Stage 2. A larger participant group was originally planned but our study was somewhat hampered by the effect of the pandemic. The extent of their experience in the role of admissions tutor ranged from one to approximately 20 years.

4.2.4 The examiners

The second group of participants involved in Stage 3 comprised IELTS examiners currently employed by the British Council to grade IELTS exam scripts. We asked the British Council to put us in contact with IELTS examiners and the only criterion we sought was that they should be currently certified examiners. Level of experience was not relevant as a range of examiners, from novice to more experienced, would reflect the nature of the current examiner pool.

The examiners’ input was sought to establish how their evaluation of the students’ work compared to that of the academics, especially in terms of what criteria their evaluations are based on. Rather than simply reviewing the descriptors ourselves to extract key terms for each band score, we needed to establish how these descriptors work in practice. Given that these participants were all trained, standardised and currently practising examiners, the expectation was that their evaluations would align closely to the IELTS marking descriptors. Seven IELTS examiners agreed to take part.

4.3 The data collection instruments

4.3.1 Stage 1: Questionnaires

The questionnaire comprised 31 items in total, addressing the following topics.

• Information sheet, consent form and participant biodata (items 1-10)

• Participant experience / training in the role of admissions (items 11-13)

• Familiarity with IELTS (items 14-15)

• Familiarity with IELTS writing tasks (items 16-21)

• Use of the IELTS marking descriptors (items 22-23)

• English writing requirements of the degree program (items 24-27)

• Perceived value of IELTS to predict writing ability (items 28-29)

• Invitation to participate in Stages 2 and 3 (items 30-31)

Item 27 presented the respondents with a list of written genres to select from in order to report which ones their students were expected to produce. This list of genres was derived from the 13 genre families identified by Nesi and Gardner (2012: 34) with the aim of representing the most widely used genres across disciplines and levels in HE. The questionnaire items can be found in Appendix 1.

The questionnaire was produced using Qualtrics software and was distributed via a link in an email sent to specific staff rather than sending out a generic message to special interest discussion lists or via social media, for example, as we believed this would result in a higher response rate. Academics were selected by trawling university websites to search for admissions tutors, course leaders or heads of department. The covering email requested that recipients forward our questionnaire to a more appropriate member of staff where necessary, as occurred in several cases.

4.3.2 Stage 2: Interview (academics only)

The second stage of the study aimed to probe key themes raised in the questionnaire. To do so, interviewees were presented with the following visual prompts (fig. 2) and asked to address them in whichever order they wished.

Figure 2: Interview prompts

The interviewer posed follow-up questions, as necessary, but this stage was designed in such a way that the majority of the output would be on the part of the academics, to be structured by them. By offering increased agency in the process, we hoped they would be more forthcoming than during a traditional semi-structured (or fully structured) interview. All prompts and instructions had been checked for clarity by a small number of colleagues not working on this project. The same interviewer undertook each interview for the sake of consistency. The first run-through of this procedure with an academic (hence referred to as Ac1) was used as a pilot but after analysis of the output, no changes were felt necessary so these data were included in the main study.

4.3.3 Stage 3: Think-alouds (academics)

Think-alouds with the academics took place immediately following the interview. Brief training on the think-aloud procedure was provided before commencement, which consisted of first viewing a video of someone undertaking a think-aloud exercise and then trying out a think-aloud by reading and commenting concurrently on a poem.

The academic was then familiarised with the nature of the two IELTS writing tasks via a brief description by the interviewer and was next given six samples of genuine IELTS Academic Paper writing scripts (provided by the British Council). The scripts consisted of three examples of Task 1 (information transfer) and three of Task 2 (essay).

The academics were asked to evaluate the scripts, based on the following prompts:

1. What are the candidates’ abilities, in your opinion?
2. What are the qualities of the script (good or bad) - what features are you looking for to make that decision?
3. Would this work meet the minimum level required for entry to your course?
4. How well (or otherwise) would this candidate cope with the future writing demands of your degree program?

Participants were asked to think-aloud concurrently while reading and evaluating the scripts to allow us insights into their perceptions of the students’ work and the criteria they were drawing on to reach their decisions.

4.3.4 The IELTS writing scripts

The British Council permitted us to use a number of genuine IELTS student scripts, produced by candidates under exam conditions, supplying 12 in all, of which we selected three of Task 1 and three of Task 2. They represented a range of scores from band 5 to band 8. This range was chosen to cover the range of ability just below and just above the most widely required entrance scores (for main degree programs, not foundation courses) (IDP Singapore, no date). To ensure balance, we selected scripts from candidates with a mixture of L1s. The British Council also supplied the scores each script had been awarded. A confidentiality and security protocol for use of the scripts was followed throughout, given that these were genuine IELTS scripts.

4.3.5 Stage 3: Think-alouds (IELTS examiners)

The IELTS examiners undertook the same familiarisation exercise as the academics and then evaluated exactly the same student scripts as those used with the academics. Prompts 3 and 4, however, differed from those used by the academics:

3. What features do you look for in making your grading decisions?

And at the end of their evaluation:

4. What score would you award and why?

The final prompt elicited a global impression of each script from each examiner, intending to prompt any observations about, firstly, the interrelation of the descriptors and overall ability and, secondly, points other than those directly linked to the descriptors. As per the academics, the first think-aloud procedure with an examiner was treated as a pilot but on analysis of the data no changes were felt necessary, so these data were included in the main data set.

4.4 Gathering qualitative data online

Initially our project was planned for face-to-face data collection but our study ‘ripened’ just as the COVID-19 virus spread so all travel was made impossible and we were forced to shift all operations online. Nevertheless, we encountered some unexpected benefits of this move as suggested by Oliffe et al. (2021). As they state: there is ‘no place like home’ (p. 1) and our online data collection may, consequently, have led to better quality data due to a reduction of any discomfort or inconvenience caused by travel.

By the point of our data collection (May to October 2021), all our participants had substantial experience of online interaction which afforded a level of confidence in the participants’ ability to effectively use the technology which aided our data collection. While various means to gather qualitative data online have existed for some time (Lobe et al., 2020), such as use of email, chat or instant messaging functions, or harvesting data from discussion lists, the advantages of video-conferencing lie, above all, in its synchronous communication, enabling immediate clarification or expansion of points raised by the participants. Being able to read body language and judge tone and stance more easily contributed to confidence that the intended message was understood. Overall, rather than viewing the shift to online data collection as purely an emergency measure or second best, we can celebrate this method of data collection for all the advantages it affords, as Teti et al. (2020) advise.

The interviews and think-alouds were conducted online using Microsoft Teams which afforded a secure, easy means to contact participants and record the interactions as most UK universities had adopted this platform by that stage. It also offered the advantage of an automatic transcription facility, which provided a starting point for creating a full verbatim transcription at the data analysis stage, as described below.

5 Ethical approval

Ethical approval for this research was granted by UCLan on 17^th December 2020, reference number: BAHSS2 0154. Participant information sheets (PIS) and consent forms constitute section 1 of the questionnaire we used with academics (see Appendix 1). Separate PIS and consent forms were sent via email to the IELTS Examiners.

Consideration of ethical issues while undertaking research is often approached solely in terms of avoiding harm to all involved. It is, however, also worth considering the benefits research can have. Comments made during the data collection did suggest that the process had been at least interesting for participants, especially the IELTS Examiners, and at best, useful as a means to reflect on and evaluate current practice. Some of the academics also commented on their increased knowledge about IELTS and assessment as a result of taking part, so this aspect of beneficial contribution appears to have been at least partly achieved.

6 Data analysis

6.1 Quantitative data analysis

For the quantitative data, simple descriptive statistics were used to provide a picture of the outcomes. More complex analysis (parametric and non-parametric analyses) was precluded by the more limited number of responses than initially hoped for. Correlational analysis would have been of interest to determine whether respondents from certain types of university, or from specific disciplines, held similar views but having fewer than 30 respondents per variable rendered this unfeasible.

6.2 Qualitative data analysis

The bulk of the qualitative data came from the interviews and the think-alouds. The questionnaire also yielded qualitative data from the open-ended questions and we applied to the latter the same qualitative data analysis approach as for the interview and think-aloud data, which is described below, but the responses to the open-ended questions were minimal.

A thematic analysis approach (as described by Braun & Clarke, 2022 and Bryman, 2012, amongst others) was employed. We applied both deductive and inductive methods of coding, given that we already had some concept-driven (deductive) concepts derived from our research aims and questions, as well as the prompts we used for collecting data, but emergent (inductive) themes were also analysed to gain a richer picture of the data set. NVivo software (Version 1.7) was used to code data and to assist in developing themes and establishing relationships between themes. See Appendix 3 for a list of the codes employed.

Agreement as to the rigour of our coding was reached via detailed discussion rather than calculating inter-rater reliability in the objectivist sense. All data were independently coded by two researchers from the project team. Regular meetings to discuss, amend and refine coding took place until we reached an understanding and approximate alignment on how we were viewing and understanding the data. Studies of a positivist inclination would usually refer to inter-reliability at this point, quoting a reliability score, such as Kappa. Our study, however, did not align with this paradigm, but rather an interpretivist approach to data analysis, where talking about the ‘reliability’ of findings makes little sense, and in its place ‘transferability’ is more apposite (Braun & Clarke 2022).

7 Findings and discussion

7.1 Overview - setting IELTS scores for course entry

The data from both the questionnaire and interviews showed that the academics were not responsible for setting the IELTS scores required to gain entry to their degree programs. Some academics interviewed applicants in order to assess their subject-specific knowledge (e.g. Academic no. 5, henceforth referred to as Ac5), while others only ever saw applications from borderline cases with unusual profiles (Ac1) or were consulted by centralised admissions processing teams only if a specific query arose (Ac2 and Ac3). The result was, according to Ac3:

‘I don’t have the full picture anymore.’ (Ac3:175) ³

The distancing of academics from the language component of the entry requirements was highlighted when Ac5 was asked what the required IELTS score was for their department and they replied:

‘Oh I can’t tell you off the top of my head because I don’t get involved with that.’ (Ac5:50)

Common across the academics’ descriptions was the routinisation of the entry process regarding the language requirement, which concurs with findings from Lam et al. (2021).

‘So what I’m sure you’ve found across a range of institutions is that once upon a time offers were - and decisions on offers were - made by academics. They’re not anymore and it’s entirely routinised’ (Ac1:23), who went on to add:

‘so there is a process by which those boundaries are set but then they’re largely administered by professional services.’ (Ac1:75)

This routinisation of the minimum score setting was received in various ways. As quoted above, Ac3 clearly felt that it had led to a loss of complete information, while some felt somewhat more positive about the centralisation of such admissions. For example, Ac1 said:

‘I remember the days when academics reviewed UCAS forms and you just ended up with unevenness and kind of lots of offers, whereas it’s much more routinised’ (Ac1:78) who went on to comment that they felt this was in the candidate’s favour and promoted fairness and consistency.

3. Direct quotes have been amended slightly to enhance readability, in that repetition and filled pauses have been omitted. Otherwise, they remain verbatim.

None of the academics reported having personal input on setting the IELTS score currently required for entry to their degree program, which is again in line with Lam et al.’s (2021) findings. This score had either been agreed at institutional level:

`the levels that are set are set centrally, they’re not, we don’t determine them’ (Ac5:122). Or, for certain courses, such as nursing, they had been set by a professional body. In addition, our interviewees reported that entry requirements had usually been ‘inherited’ i.e. decided prior to this participant coming into post (e.g. Ac3:188) or as Ac1 said:

‘IELTS boundaries were in place when I became admissions tutor.’ (Ac1:70)

The data regarding the setting of IELTS scores showed these academics were definitely at a remove from this part of the admissions process.

The academics we interviewed were fairly representative of the population of academics answering the questionnaire in terms of the scores required for entry to their courses. Of the interviewees, one was admissions tutor for a course requiring IELTS 7.5 (minimum of 6.5 in any component), three required 6.5 (6.0), and one 5.5 (5.0). This roughly mirrors the pattern of IELTS minimum overall scores and minimum writing sub-scores required for the questionnaire respondents’ courses, as shown in Figure 3 and 4.

Figure 3: Entry requirements for questionnaire respondents’ courses - IELTS overall score

Figure 4: Minimum score required on IELTS writing paper according to questionnaire respondents

Figures 3 and 4 show that band 6.0 is the most common requirement for IELTS writing, followed by 6.5 and 7.0. They also show that even if the overall IELTS score requirement is fairly low (5.0 or 5.5), the writing is usually not allowed to be lower than 5.0 and this particular minimum requirement applies in only a small minority of cases. This suggests that writing is considered to be important overall and that a relatively high level of writing ability is seen as necessary for student success on degree programs.

7.2 Research Question 1: How familiar are academic admissions staff with the IELTS academic writing task?

7.2.1 Levels of familiarity with IELTS writing tasks

There was a mixed picture regarding the academics’ familiarity with IELTS in general, tending towards familiarity rather than lack of it (as ascertained via item 14 on the questionnaire). The majority (56%) of questionnaire respondents reported being familiar or very familiar with IELTS whereas 44% said they were either unfamiliar or very unfamiliar with the exam (all figures are rounded up). However, when asked about their level of familiarity with the writing section specifically (item 15), the picture altered. This time a minority (34%) were familiar or very familiar, with 66% claiming to be unfamiliar or very unfamiliar.

In order to probe further, we investigated how the term ‘familiar’ had been interpreted by the respondents by requesting further details. Of the respondents, 77% had never seen the writing tasks (item 16) and 82% had never seen examples of the type of writing candidates produce when taking IELTS (item 17). Nevertheless, most of the respondents correctly identified (in item 19) the two genres represented in Task 1 and Task 2 (an essay and an information transfer task, i.e. a short text based on infographics, respectively). A minority believed that report, opinion piece and letter were genres required in IELTS. The latter is understandable since letters are one of the task types in the General IELTS writing exam but this was not relevant as our study related only to the Academic IELTS exam. Exactly 50% of respondents also correctly identified that citations were not required (although, of course, this means 50% thought that they were). Respondents were less clear about the expected word count for Tasks 1 and 2 combined, the majority (57%) believing that approximately 500 words were required. Only 20% correctly identified the total word count as being less than 500 words. Overall, the picture yielded by the questionnaire data was of vague familiarity at best.

The questionnaire data were supported by the interview data. In the interviews, the academics confirmed that, in general, IELTS was somewhat unfamiliar to them and the writing requirements were even less familiar. One indicator of this lack of familiarity was that one academic (Ac4) was not even sure of the possible score range (namely, 0-9). Another example was how IELTS was described by another interviewee:

‘I know … there are different parts to it. There’s a writing part and a speaking part. I think there’s a listening part as well. There’s multiple elements and there’s an overall score … that’s as much as I know.’ (Ac5:106)

and

‘I think it’s [the score required] six point nought, whatever that means …’ (Ac5131)

While this information is correct, Ac5 was clearly not confident in their description and was aware that it was rather vague. They also omitted mention of the reading paper. Likewise, Ac4 doubted their own level of familiarity with IELTS:

‘So it’s only when you come to develop programs and modules that don’t have that governance [i.e. a grade set by an external professional body] that you start to think ‘oh right well, what can, what should it be? What can it be? How can I make a decision on that?’ and I struggle with that … and that’s probably the reason why I was interested participating [in this project] ‘cause I thought well actually I know very little about this and that’s not good.’ (Ac4:51)

Ac1 commented that, in their opinion, most academics would very probably not know about IELTS unless they had been involved in the entirety of the admissions process (Ac1:267) and, as has been established, they generally are not.

Another academic realised that despite their role in admitting international students to their degree program, academic admissions tutors only had expertise in the subject-specific aspect of what the students needed to prove ability in, not the other key component i.e. language:

‘Although I’m admitting students I can’t say that I’m hugely familiar with, you know, what it takes to pass [IELTS].’ (Ac3:81)

One reason why academics were relatively unfamiliar with IELTS was suggested by Ac1, who said:

‘academics don’t really get questions on IELTS’ (Ac1:167)

meaning that their role was to answer only subject-specific queries from the administrative staff processing applications, not queries related to the candidates’ language ability. They would, thus, not be prompted to find out more.

7.3 Research Question 1a: How familiar are the academic admissions staff with the IELTS academic writing band descriptors?

7.3.1 Levels of familiarity with the descriptors

On the whole, the academics were not familiar with the IELTS writing band descriptors; the questionnaire results showed that only 39% had seen them and only 27% said they referred to the descriptors when making admissions decisions. This is somewhat in line with the responses from our interviewees, which ranged from actively referring to the descriptors while making admissions decisions (Ac3), having reviewed them carefully as a candidate when taking the IELTS test themselves about 10 years previously (Ac2), thinking they had seen the descriptors at some point in the past (Ac1, Ac5), to, finally, not being aware of having seen them at all (Ac4). Therefore, it would seem that familiarity with the descriptors varies considerably.

One interviewee explained this lack of familiarity and engagement with descriptors as being due to the grade being set by an external professional body for their course, as stated previously:

‘Well you don’t even think about it cause it just has to be and there’s no kind of exploration of why that is because that’s what it has to be in order for someone to get to get on the program.’ (Ac4:37 & 39).

These academics have no agency in the score setting, as previously discussed, and thus no reason to explore what the set score means in practice. Another reason for not engaging with descriptors was similar in that the score has been set previously and was, thus, ‘inherited’, as Ac4 said:

‘We make decisions on that [which score to be stated in course documentation] often based on lack of knowledge and probably just … I suppose just what’s gone before, rather than actually unpicking what it is and how that relates to the level of writing ability that the students need to have.’ (Ac4:19-24)

Thus, the main reason for why the academics may not be familiar with IELTS descriptors is that they do not need to engage with them to set admissions scores since these scores have already been set by someone else (as mentioned in the Background section above). While acknowledging that the interviewee group was self-selecting and therefore cannot claim to be representative of the body of academics as a whole, a receptiveness to better understanding of the descriptors and their role was evident, as expressed by Ac4:

‘I need to know more about these band descriptors so that we can make informed decisions about what it should be set at.’ (Ac4:79)

Better familiarity and understanding could help produce a more accurate picture for the academics of what the international students are actually required to produce in their language skills assessment and how far this is relevant, or not, to subject-specific writing requirements.

7.4 Research Question 1b: How did the academic staff gain this familiarity with IELTS?

7.4.1 Sources of information and training

Having considered the levels of familiarity with IELTS in relation to Research Question 1b, we needed to consider how the respondents might have become familiar with IELTS. Only 7% of questionnaire respondents reported having taken the IELTS test themselves so first-hand experience was a source of information for only very few, including one of our interviewees (Ac2). Even in the case of Ac2, who had taken IELTS themselves at least 10 years prior to the interview, as already stated, they were not confident in how much they remembered. Furthermore, simply sitting an exam does not mean the candidate necessarily understands exactly what constructs are being assessed, or how. Experience of an exam equates to only surface-level language assessment literacy and would not be sufficient knowledge to effectively inform high-stakes decision-making.

An obvious source of information about IELTS might be training relating to the role of admissions tutor, so the questionnaire asked about training for that role. Findings showed the participants fell into two groups. In answer to ‘Have you received any sort of training for this role of making admissions decisions?’ (item 9), 50% of questionnaire respondents answered in the negative and 6% could not remember. Nevertheless, 44% had received training.

While any training for the role of admissions tutor could usefully include various elements about the role, it should, presumably, also include the topic of IELTS scores (or equivalent) since visa requirements set by UKVI require international students to provide proof of sufficient language ability. This is, thus, a crucial element in the application process, so we wished to know whether admissions tutors had received any training about IELTS specifically. Of questionnaire respondents, 90% said they had received no training about IELTS (item 12) and 82% said they had had no training on any of the SELTs (item 11). This pattern was confirmed in the interviews, with some interviewees acknowledging that there had been training regarding the role of admissions officer but as Ac2 reported:

‘There was no training whatsoever about checking any particular IELTS overall or components’ (Ac2:30)

and from Ac5:

‘OK in terms of IELTS and the answer [regarding training and experience as an admission tutor] is pretty much zero.’ (Ac5:303)

Similarly, Ac 3 told us:

‘I have received some training but not specifically if you wish, in terms of language’.

Only one interviewee (Ac1) described any experience of formal training on IELTS. They reported attending:

‘sessions for academics who were involved in admissions and part of it was an instruction on OK what actually is IELTS? What are we assessing? So they had someone in from the Cambridge, somebody to explain how they were testing, what the different elements were and what they were looking for and that was quite an interesting experience. It was really quite valuable I think for the people who were there on the day.’ (Ac1:153)

This reinforced the picture of the dislocation, in general, on the part of the admissions tutors, from the area of admissions we are most interested in for this study, namely language ability.

For the 44% of questionnaire respondents who reported having undergone training, all of that training had been offered by colleagues. This included a combination of academic staff, admin staff, International Office staff or other unspecified internal staff. Ac1 confirmed in their interview:

‘There was no formalised training, so you’d be trained by the previous admissions tutor. So there was training within the academic team.’ (Ac1:17)

None of the questionnaire respondents reported having had training delivered by testing experts (this was one of the options offered in item 10), so we can assume their training related to administrative aspects of IELTS rather than the nature of, or the expectations of, the exam itself. What is more, while various of the institutions which were represented in the questionnaire responses would have had language testing experts amongst the academic staff of their university, they were not mentioned in any cases as having delivered any training.

Rather than formal training sessions, informal support from colleagues seemed to play a key role in inducting admissions tutors into the role. As an example of the types of training experienced, Ac1 reported ongoing sources of support in their admissions role being mostly other academics in their team or their centralised admissions team if on an event abroad visiting a university partner (for example, Ac1:19-24). On a similar note, Ac5 reported that their knowledge about language exams, entry scores and admissions process appeared to have come from simply picking it up as they went along having been an admissions tutor for approximately 20 years. Likewise, Ac3 claimed their knowledge of IELTS was derived:

‘mostly through experience really rather than rather than training’. (Ac3:161)

For such a high-stakes situation this would seem to be a rather haphazard approach.

7.4.2 A role for language assessment literacy

The paucity of systematised training identified here can be viewed through the framework of recent work in language assessment literacy (LAL). There has been a surge of interest in this relatively new research field (Baker, 2016) in the last decade (e.g. see Language Testing, special issue, volume 30, 2013). In tandem there has been an increased focus on the social consequences of assessment (see, for example, McNamara & Roever, 2006 and Shohamy, 2001). As Deygers et al. say: ‘Test scores convey little meaning in a contextual vacuum. A score only becomes real when it has real-life consequences, such as access to a valued position, service, or status’ (2018: 51). Both fields direct our attention to the need for wider and more nuanced understanding of assessment practice, as noted by Pill & Harding (2013).

LAL aims to shine the spotlight on how training of key stakeholders can promote quality assurance of assessments. The higher the stakes of testing, the greater the need for LAL, which is the case here where IELTS takes on a high-stakes gatekeeping role for students needing to gain entry to their desired UK university course. For this reason, it has been argued by O’Loughlin (2013), Baker et al. (2014) and Baker (2016) that both administrative and admissions staff at HEIs need better levels of LAL to inform the admission process and support international students more effectively. While these studies were from Canada and Australia the same argument also holds for UK HEIs.

7.5 Research Question 2: How do UK HE academic staff perceive the relationship between writing abilities required for the IELTS writing paper and the writing demands on students’ degree programs?

7.5.1 Perceptions of international students’ language abilities

In our interviews with academics, a positive view of international students’ language abilities emerged overall. Ac5 told us:

‘There are students who when you interview them it doesn’t make any sense [for] them to do IELTS, they just speak English really well…’ (Ac5:314)

In a similar vein, but with reference to writing in particular, Ac1 reported:

‘Sometimes their English language is actually better than mine in writing so they’ve actually got an understanding of the grammar and they’re taught the grammar.’ (Ac1:238)

Regarding writing specifically, responses to item 28 on the questionnaire (‘When ISs join your degree program, how would you rate their writing ability overall?’) indicated that academic staff were generally satisfied with international students’ written work. The writing ability of international students joining their course was rated as good or very good by $68\%$ of respondents. Only $5\%$ deemed ISs’ written work very poor. This does not mean all students’ writing is strong but would, at face value, suggest that IELTS performs a mostly satisfactory job of screening out those who lack the requisite language skills. Nevertheless, in the interviews, Ac2, in contrast to Ac5 and Ac1, voiced some concerns about students’ ability:

‘I’m not gonna lie in terms of being a concern about how well the grade reflects the ability of the students and how well they are set for achieving what we want them to do when they’re in the program.’ (Ac2:72)

These mixed results contradict those from the study by Coleman, Starfield & Hagan (2003) which found that admissions staff generally felt a higher IELTS entry score should be set as they were not satisfied with the general level of student language ability. It must be noted that this study included all admissions staff, not only academic staff, and included participants from Australia, China as well as the UK, so is not entirely comparable to our study. In addition, in the intervening time, the way UK higher education is funded has changed drastically, making universities more financially dependent on income from tuition fees. This, in turn, has increased pressure to recruit international students to UK HEIs (as they pay higher fees) which may have led to changes in staff expectations and perceptions.

7.5.2 Academic staff’s linguistic expectations of international students

In addition to asking about academics’ perceptions of ISs’ language ability, we also asked about the language demands that their degree programs placed on such students. Some interviewees felt that their programs were linguistically (and conceptually) demanding for ISs. Ac1 reported that colleagues, who (as stated before) may not necessarily know anything about IELTS, had expressed frustration when international students did not pick up class content as quickly as expected. As with Ac2, there had been discussions concerning whether to raise the IELTS entry requirement. The linguistic demands placed on international students varied according to discipline, unsurprisingly.

In the discipline of law, for example, linguistic demands were perceived to be high:

‘Actually the English language expectation is high. What we’re expecting students to do right from the beginning of the program is to be able to read complex language with specialised terminology that they won’t have come across before.’ (Ac1: 91)

‘We would normally expect students to participate in seminars and activities like that and to communicate with staff and, like, non-native speakers particularly at the beginning can find that quite challenging.’ (Ac1: 105)

We were specifically interested in the writing demands made of ISs and what academic staff expected students to be capable of, having gained entry to their degree programs. With this in mind, item 24 of the questionnaire posed the question, ‘What would you expect a student who has passed IELTS at the minimum level required for your degree program to be able to do?’ The respondents selected from a list of features typical in academic writing marking criteria ⁴. These are presented in Table 4 according to how frequently they were selected. They have been mapped onto specific features of the IELTS writing descriptors found at band 5 and above (as 5.5 is the minimum sub-score reported as an entry requirement). Where they are described as being found indirectly in the descriptors, this means that we identified overlaps between these concepts and the constructs itemised in the IELTS writing descriptors, even if the specific wording is not an exact match.

4. These features were derived from the study, ‘Writing Criteria Project - Producing Evidence-based Criteria’ undertaken at UCLan in 2016, whereby experienced EAP tutors listed the features they prioritised in the process of assessing academic essays when not marking within the parameters of any specific marking criteria/ descriptors.

Table 4: Comparison of a) features academics expected in international students’ writing and b) constructs assessed in the IELTS writing paper

* all the constructs listed under the Coherence and Cohesion category of the IELTS descriptors amount to effective structure

** constructs relating to lexical range in the Lexical Resource category of the IELTS descriptors relate to use of appropriate vocabulary

Certain features that respondents selected from the list of options in questionnaire item 24 are either not constructs reflected in IELTS marking descriptors (e.g. citation or paraphrasing) or are not genres tested in IELTS (a report). Some could not easily be addressed in a typical, centralised, high-stakes exam (e.g. ‘Work independently to produce a written assignment’) such as IELTS. These results show that academics expect international students to be able to manage some aspects of academic writing which IELTS does not require, a point which was highlighted by Ac3:

‘It’s very clear that what should be sufficient based on the descriptors that you know in the IELTS descriptors theoretically doesn’t necessarily match what we would want from them or doesn’t always or, I don’t know, match what we would expect from them once they start here.’ (Ac3:55-58)

As already mentioned, after overcoming the hurdle of gaining the required scores and sub-scores on IELTS, students then face a range of subject-specific demands and our data confirmed this. Specifically with reference to writing demands, one interviewee noted that ISs studying law need to tackle complex and varied genres such as essays, critical commentaries, case notes and letters of advice, amongst others (Ac1:114-136). They went on to say:

‘They have to give legal advice … it’s quite sophisticated and it’s got to make sense and apply to the person that they’re being asked to advise.’ (Ac1:119)

Both Ac1 and Ac2 fully acknowledged the challenges of producing a range of genres, as did Ac4 from another subject area:

‘The demands are around clinical information and clinical writing, so writing for a particular purpose and so for sharing information documentation, which is a whole other set of writing skills, it’s very different from academic writing, reflective writing. So in a relatively short three-year period they are exposed to, and are expected to perform, quite a range of different writing styles.’ (Ac4:354-359)

Similarly, Ac2 listed a range of genres the students would need to master, some beyond those we had enquired about in the questionnaire, which highlights even further the challenge students face:

‘Reports, essays, clinical reports, statistical reports, different types of reports … also we have blog assessments, so they need to be able to write more in lay terms not just academic terms - short essays that need to be succinct and it’s not the same structures with standard, longer essays.’ (Ac2:97-99)

The nature of the writing demands placed on international students varied across subjects. For example, Ac5, an academic on a STEM degree program, acknowledged the writing demands to be considerably less than for humanities subjects and explained:

‘Well, obviously it’s not as much as it would be in a humanities course’ (Ac5:178) and

‘They have to do worksheets throughout the term, and in those, in order to get very good marks, you have to kind of explain what you’re doing but … we don’t knock marks off for things like spelling mistakes or grammatical mistakes’ (Ac5:181-185) and then emphasised:

‘They do have to write things throughout the degree but they don’t do as much as they would do in some other subjects’. (Ac5:200)

That a sub-score in writing is required for all subjects, although the need for effective writing varies widely across disciplines, could constitute another topic which assessment training with academics could address.

7.5.3 Criteria drawn on by the two participant groups to evaluate the scripts

In order to better understand what the academics valued in ISs’ writing, we examined the terms they used when evaluating the IELTS scripts we presented them with. These terms highlighted both what they saw as the positive and also the negative aspects of the writing. We wanted to ascertain whether features they saw in these scripts aligned with the features they hoped to see in ISs’ writing for their coursework. The terms they used were compared with the terms (criteria) used by the IELTS examiners in order to better understand whether both groups valued the same aspects of ISs’ writing. Since the academics, understandably, did not draw on criteria derived from the descriptors, it was illuminating to examine what features they did use to evaluate the quality of the students’ work they were presented with. We hoped this would shed light on what features they saw as most important in writing in their own academic fields.

To better investigate whether any patterns and commonalities might be insightful of how the two groups perceived the writing, two word clouds were generated (using MonkeyLearn), which highlighted the range and most prominent language features they mentioned. To produce these word clouds, all the specific language features commented on by the participants when evaluating the scripts were located in the transcripts. The exact terms used (rather than glosses) were tabulated and tallied, allowing a simple quantitative comparison of the criteria drawn on. The terms can be found in Appendix 4.

Figure 5: Language features noted by the academics when evaluating scripts (n=5) Task 1 & 2 combined

Figure 6: Language features noted by the IELTS examiners when evaluating scripts (n=7) Task 1 & 2 combined

Note: LR = Lexical Resource, TA = Task Achievement, CC = Coherence & Cohesion, three of the categories in the IELTS writing descriptors.

Figures 5 and 6 illustrate that the most widely used terms by both groups of participants were grammar, vocabulary/lexis and cohesion. Some terms were used exclusively by only one group. For example, proofreading, style and use of capital letters were terms used only by academics. On the other hand, only the IELTS examiners commented on repetition, text length, format and whether there was an overview. Other features were used by both groups but more by one group than the other, such as communicative ability and argument, used more by academics and grammar, coherence and cohesion, task fulfilment and spelling mentioned markedly more by the examiners.

The terms used by the examiners reflected terminology of the descriptors, which was to be expected. When evaluating the scripts, they often used acronyms such as CC (coherence & cohesion), LR (lexical resource) and TA (task achievement, even though this category is labelled task response in the descriptors for Task 2). Due to the training and standardisation procedures IELTS examiners undergo, the use of these lexical chunks is hardly surprising. However, these constructs were not referred to by these terms exclusively; variations such a lexis or vocabulary were used when referring to LR, or flow which can be seen to relate to cohesion. Additionally, as already mentioned, overview was a feature which only the examiners noted, and noted frequently, each examiner mentioning it multiple times when evaluating Task 1 scripts. The examiners mostly shared a common procedure in evaluating the scripts. After checking for the overall length of the script in all cases (both tasks), for Task 1 the examiners checked for the presence of an overview. As Ex5 said, when asked why including an overview was important:

‘because it’s common knowledge for anyone who’s preparing the test’ (Ex5:46) and Ex6 commented:

‘an overview which clearly defines the overall process in any Task 1 is a must’. (Ex6:42)

‘Overview’ is wording found in the IELTS descriptors from band 5 upwards for task achievement in Task 1 and an examiner (Ex5: 52) suggested that this had been a feature highlighted in standardisation training, which may explain its frequent usage in this exercise.

In discussing students’ writing, a common discourse seems to have developed which inevitably relates directly back to the descriptors against which the work is being judged.

Coming from a range of disciplines, and not having been trained in using IELTS descriptors, it is not unexpected that the academics mentioned a wide variety of attributes they saw in the scripts, beyond the obvious grammar, vocabulary and organisation. For example, some mentioned style, formality, proofreading, handwriting and stance, which do not appear in any form in the descriptors. It is noteworthy though that some of the IELTS examiners also mentioned attributes which do not appear in the descriptors, such as also commenting on handwriting and stance, as well as readability and format. This indicates that despite rigorous standardisation procedures, personal opinions regarding the nature and purpose of academic writing still linger and some may influence the grading process.

These findings are similar to those of Elder (1992), who found in her study of ‘expert’ and ‘naïve’ raters of language performance, there was overall alignment of judgements from both groups but differences as regards the specific dimensions of the performance they noted. In the current study, some features were more widely used by the academics such as communicative ability and ideas suggesting that they may be using a different lens to view the work. When compared to how the IELTS examiners discussed the students’ writing, an understandable split was perceptible, akin to a top-down approach from the academics and a bottom-up approach from the examiners. For instance, as with the academics, grammar featured strongly in the examiners’ evaluations; nevertheless, while the examiners searched the scripts for features using the framework of the four main categories found in the IELTS descriptors (Task Response/ Task Achievement; Coherence & Cohesion, Lexical Resource and Grammatical Range & Accuracy), and fairly systematically commented on the level of mastery in each, the academics only mentioned specific grammatical or lexical features where they were especially notable either for excellence or the extreme opposite. For example, Ac5 particularly noticed a lack of articles (Ac5:109, 151, 165 & 311). Unless such specific features were present to the extent that they became distracting, the academics generally commented on communication of ideas, argumentation and organisation. For example:

‘You will note that I’m looking largely at the kind of structural and construction of argument points not the nitty gritty of how they’re using language.’ (Ac1: 259)

Ac4 summarised a strong script thus:

‘No concerns at all, nice flow, nice structure, good content.’ (Ac4:537)

This suggests that ideas and how they are organised took precedence for the academics over accuracy of grammar or appropriate lexical usage. The examiners, in contrast, as per their training, broke the work down into key constructs, mirroring the band descriptor categories (as listed above), using a more atomised approach, before synthesising what they saw into a score. In general, they also had far less to say than the academics about the nature of the content, or the effect of the piece of writing on the reader.

7.6 From IELTS to coping on the course

One way to interpret the academics’ understanding of the relationship between IELTS scores and the students’ writing ability is to consider whether the academics felt these candidates would cope on their degree program. For the most part, they felt they would. However, in some cases, the official IELTS scores the scripts had been awarded (which married closely with the scores suggested by the examiners) would suggest otherwise. Table 5 shows whether the academic admissions tutors felt that the writer of each script would likely cope on their degree program, whether or not the script actually achieved the minimum IELTS band score required for entry to that program and, thus, whether in a real-life scenario the student would have gained a place on their chosen course.

Table 5 aims to show whether the academics felt the writing they evaluated reached the minimum quality required for their courses. It compares firstly whether each academic felt, from looking at the student’s work, that the student would cope on their course, and secondly, whether that script had been awarded the minimum IELTS sub-score required by their university for writing, as reported by that academic (in other words, was this work good enough for their course). All the IELTS scores awarded were provided by the British Council.

Table 5: Academic admissions tutors’ assessment of whether students would cope on their degree program, compared to whether scripts actually met the minimum IELTS score for entry to that degree

*borderline/concerns about the candidate’s ability. Disparities are marked in tinted boxes.

As the table shows, most of the academics’ evaluations of acceptability were in line with entry scores required. All scripts, apart from Scripts 1B and 2A, achieved a score which would have allowed the candidates access to the academics’ degree programs. In addition, the majority of the academics evaluated all the scripts but these two to be of an acceptable level for entry. The discrepancy between the score awarded and the required entry score ranged from 1.5 bands in one case to half a band at the other extreme.

There were a few exceptions to the overall patterns of agreement. Script 1A had not been awarded the minimum score required for admission to the degree programs of Ac1 and Ac4 but both academics felt this piece of work represented a satisfactory level of ability for admission. In both cases the awarded grade was within only a half an IELTS band of the required one so this disparity is not of particular concern. Ac4 stood out as holding the most lenient view towards the quality of writing in the scripts as they also felt both Script 1B and Script 2A would be acceptable, although both were awarded a score 1.5 lower than that required for entry to Ac4’s course.

Upon reviewing Ac4’s explanation of their evaluations, it became clear that they took a slightly different approach to interpreting writing ability on entry to a degree program. Ac4 evaluated the weaker scripts in terms of potential, opining that the students would probably, with support, attain the requisite level while on the course and that they did not, at the point of application, need to evidence this level. For example, Ac4 stated:

‘I wouldn’t be sitting thinking, right this person wouldn’t be a strong candidate to come onto the program, because I think it’s a three-year program and it’s an education program and part of that education is learning to write and developing writing skills. So I wouldn’t be thinking, no, they wouldn’t be able to cope, because I do think there’s logic, there’s meaning.’ (Ac4:239-44)

This was in relation to a course for which, it was reported, students needed to master a wide range of genres and was also a course regulated by an external professional body for which specific, non-negotiable entry scores were needed.

In contrast, other academics felt the required ability should already be in place from the start:

We use specialised language and there [on their course] we expect construction of argument and use of reasoning, oral and written reasoning, right from the beginning of the program. English language expectation is high and it is also tightly enforced, so it really is quite heavily policed. We won’t accept if they are weak on IELTS’ (Ac1:33 -35)

and later added:

‘You’re just setting candidates up to fail if we allow them to come in with a weak IELTS.’ (Ac1:41)

This suggests that the academics had different understandings of what the IELTS score represented in practice. As academic admissions tutors are mostly distanced from the language entry qualifications, as already discussed, this difference in views is understandable but could have implications for what academics, not only admissions tutors, expect of their new students. It is understandable that if students’ tutors feel that having ‘passed’ IELTS, they believe students should be able to cope on entry to the course with the written assessments they face.

Such a discrepancy may explain the parts of the questionnaire data that did not align particularly closely with the data from the academic interviewees. Specifically, when asked if the required IELTS score predicts the students’ ability to cope with the writing demands of the course, 78% of questionnaire respondents replied, ‘to some extent’, which does not constitute an unqualified endorsement. The item did not specify by when students should be able to cope: immediately on entry or only later, after becoming accustomed to the course, having received formative feedback or having accessed writing support.

8 Pulling the strands together: what does this all mean?

8.1 Familiarity with IELTS and what scores represent in practice

Our study aimed to investigate the level of familiarity of academic admissions tutors with IELTS, in particular with the writing tasks and how they are marked, and thus what writing skills students can be assumed to possess as a result of achieving a certain score. The data revealed a degree of unfamiliarity overall. Training was sparse and did not cover the nature of IELTS writing requirements.

8.2 The gaps

Beyond what we looked for to answer our research questions, the data revealed there are the gaps between the examiners’ approach to evaluating the scripts versus the approach taken by the academics. The examiners represent the TESOL/EAP community, who have their own norms, habits and knowledge frameworks to describe and understand language development and use. The academics, on the other hand, not being part of that community of practice, judge students’ language use in terms of whether successful communication has been achieved, or not. The academic staff who participated in our study may be described as ‘linguistically naïve subject specialists’ (Elder, 1992:16), but as Elder reminds us, naïve raters are as able or better at judging effective communicative ability. Their potential to contribute significantly to language assessment development should perhaps be foregrounded more than it is at present.

Another gap was the difference in how IELTS is viewed by different academics. Some seemed to assume IELTS represented an aptitude test of sorts, as a ‘fitness test’, evaluating future potential to reach required standards. Alternatively, others saw it as proof of current ability to use the necessary academic skills from day one. This mismatch has consequences in terms of what would be expected of students, especially in terms of whether study skills/ language support services are seen as a vital part of students’ road to success, or not. While most universities offer EAP/study support of some kind, provision is not consistent, and many might argue insufficient, for the needs of all international students. Further provision for such support at all HEIs and consistent close co-operation between all subject staff and support staff would preclude the need for the academic staff to have a better understanding of IELTS, but unfortunately, this is not the case.

A third, and related gap, was the difference between the relatively low levels of language assessment literacy (LAL) evidenced by the academic admissions tutors in this study and the ideal level in order to best use IELTS to the advantage of both students and academics, in terms of minimum language levels required. One main aim of improved LAL is to ensure better practice at all levels. All assessment, but primarily high-stakes testing, has the potential to affect stakeholders in various ways. IELTS is usually taken in high-stakes situations and the example focused on in this study is exactly such a situation: a specific score on IELTS is the key which opens the door to a UK HE course that will shape the candidate’s future. As noted from our data, while only tentatively due to the modest size of our study, we saw that some of the candidates’ writing was deemed acceptable by the academics but may not have been awarded the IELTS score needed to enter a UK HEI. Failing to achieve the stated score closes that door.

To sum up, this study confirmed academic staff do not, or at least rarely, have to take on the high-stakes role of determining entry to university courses based on language proficiency, as per the situation of the academic staff in Baker’s (2016) study in Canada, for example. Given this lack of decision-making power in relation to IELTS scores, one could assume the academics may not be interested in what these applicants go through to gain the necessary grade, yet our data showed that our participants are indeed interested and feel a duty of care for the international students on their courses. What these academics do have control over in relation to IELTS is the quality of support these students can expect once they cross the hurdle of achieving the required entry score. Fairer and more realistic expectations of international students regarding their academic writing could be achieved by generally raising awareness amongst academics as to, firstly, the writing skills that can assumed as a result of gaining a certain IELTS score, and, secondly, those skills which IELTS does not evaluate.

9 Limitations

Undertaking this research before the global COVID-19 pandemic struck may have delivered somewhat different results, particularly in terms of being able to include a larger group of participants. Our whole research environment altered as the stress and workload in our professional lives increased (in most cases) due to the new ways of operating imposed within a very short time-frame by the pandemic. As a minimum, we would have hoped for greater participation rates but what we encountered was understandable due to such stresses.

We had hoped for a wider group of academics to take part in the interviews and think-alouds. Our participants in this part of the data collection represented only red brick and plate glass institutions. Data would have been more representative if we had secured participants from all university types. Further data could be collected in a follow-up project to plug these gaps.

10 Recommendations

10.1 Repackaging training materials

IELTS familiarisation materials already exist, such as those supplied by IELTS, produced as a result of Kerstjens and Nery’s (2000) study. Arrigoni and Clark (2015), however, claim that few stakeholders take advantage of these opportunities for familiarisation. The IELTS score setting materials (IELTS, 2015) were found by Lam et al. (2021) to be useful in familiarising staff with IELTS but it would seem from our study that many academic admissions staff would not feel the need to use this material since, in most cases, the required entry scores have already been set (as already covered). Indeed, as Lam et al. (2021) indicated, this material, unfortunately, is not widely used.

It is recommended, therefore, that this resource be ‘re-purposed’ from score-setting to ‘score understanding’ and targeted more actively to academic staff. The materials should primarily enlighten academics as to what these IELTS scores really represent in terms of the type of writing skills they could expect from students. This awareness-raising about IELTS could take various forms. For example, O’Loughlin’s (2013) suggestion for improving knowledge among admissions staff, including them taking a free IELTS test and taking part in online tutorials. IELTS workshops could also be extended to academics. Such training materials need to be concise, well-targeted and well-publicised to be effective. Therefore, awareness raising as to how such training could benefit academics should be undertaken, for example, via a sector-wide organisation such as Advance HE.

10.2 Further research

To complete the picture of academics’ understanding of IELTS, further similar studies focusing on the speaking, reading and listening skills, as assessed in IELTS, would be useful. While we assert that writing is the most important skill to investigate, some participants mentioned spoken skills as being more important than written skills in their degree programs. Research could be undertaken to examine whether there is an equal lack of understanding of what speaking sub-skills are actually represented in an IELTS exam and how far they match the speaking skills international students need to thrive at university or predict their ability to do so.

References

• Alexander, O., Argent, S., & Spence. J. (2008). EAP Essentials. A Teacher’s Guide to Principles and Practice. Reading: Garnet Publishing.
• Arrigoni, E., & Clark, V. (2015). Investigating the Appropriateness of IELTS Cut-off Scores for Admissions and Placement Decisions at an English-Medium University in Egypt, IELTS Research Reports Online Series, 2015/3. IELTS Partners: British Council, Cambridge Assessment English and IDP: IELTS Australia.
• Baker, B. (2016). Language Assessment Literacy as Professional Competence: The Case of Canadian Admissions Decision Makers, Canadian Journal of Applied Linguistics, 19 (1), pp. 63-83.
• Baker, B., Tsushima, R., & Wang, S. (2014). Investigating Language Assessment Literacy: Collaboration between Assessment Specialists and Canadian University Admissions Officers, CercleS, 4 (1), pp. 137-157.
• Banerjee, J. V. (2003). Interpreting and Using Proficiency Test Scores. Unpublished doctoral thesis, Department of Linguistics and Modern English Language, Lancaster University, Lancaster, UK.
• Braun, V., & Clarke, V. (2022). Thematic Analysis - A Practical Guide. London: SAGE.
• Bryman, A. (2012). Social Research Methods ( 4^th ed.), Oxford: Oxford University Press.
• Coffin, C., Curry, M. J., Godman, S., Hewings, A., Lillis, T., & Swann, J. (2003). Teaching Academic Writing: A Toolkit for Higher Education. London: Routledge.
• Coleman, D., Starfield, S., & Hagan, A. (2003). Stakeholder Perceptions of the IELTS Test in Three Countries, IELTS Research Reports, Volume 5, IELTS Australia, Canberra, pp. 159-235.
• Deygers, B., Van den Branden, K., & Van Gorp, K. (2018). University Entrance Language Tests: A Matter of Justice, Language Testing, 35 (4) pp. 449-476.
• Elder, K. (1992). How Do Subject Specialists Construe Second Language Proficiency, Melbourne Papers in Language Testing, 1(1):16-32.
• Erling, E. J., & Richardson J. T. E. (2010). Measuring the Academic Skills of University Students: Evaluation of a Diagnostic Procedure. Assessing Writing, 15 (3) pp. 177-193.
• Green, A. (2007). IELTS Washback in Context. Preparation for Academic Writing in Higher Education. Studies in Language Testing 25, Cambridge: Cambridge University Press.
• Hamp-Lyons, L. (1991). (Ed) Assessing Second Language Writing in Academic Contexts, Norwood: Ablex.
• HESA (Higher Education Statistics Agency). (2022). Where do HE Students Come From? Retrieved May 2022, from https://www.hesa.ac.uk/data-and-analysis/students/where-from
• Hyatt, D. (2013). Stakeholders’ Perceptions of IELTS as an Entry Requirement for Higher Education in the UK, Journal of Further and Higher Education, 37(6), pp. 844-863.
• DOI: 10.1080/0309877X.2012.684043
• Hyatt, D., & Brooks, G. (2009). Investigating Stakeholders’ Perceptions of IELTS as an Entry Requirement for Higher Education in the UK. IELTS Research Reports, Volume 10. IELTS Australia and British Council, pp. 1-50.
• Iannelli, C., & Huang, J. (2014). Trends in Participation and Attainment of Chinese Students in UK Higher Education, Studies in Higher Education, 39(5), pp. 805-822. DOI:10.1080/03075079.2012.754863
• IDP Singapore. (no date). Minimum IELTS Score Requirements for UK Universities | IDP Singapore.
• IELTS. (2015). IELTS Score Setting Guide. British Council, IDP and Cambridge English Language Assessment. Available from: www.ielts.org
• Jenkins, J., & Wingate, U. (2015). Staff and Students’ Perceptions of English Language Policies and Practices in ‘International’ Universities: A Case Study from the UK. Higher Education Review, 47(2), pp. 47-73.
• Johns, A. (1991). Faculty Assessment of ESL Student Literacy Skills: Implications for Writing Assessment, in Hamp-Lyons, L. (Ed.) Assessing Second Language Writing in Academic Contexts, Westport, CT: Ablex Publishing.
• Kerstjens, M., & Nery, C. (2000). Predictive Validity in the IELTS Test: A Study of the Relationship between IELTS Scores and Students’ Subsequent Academic Performance, IELTS Research Reports, Volume 3, IELTS Australia Pty Ltd, Canberra, pp. 85-108.
• Lam, D. M. K., Green, A., Murray, N., & Gayton, A. (2021), How are IELTS Scores Set and Used for University Admissions Selection: A Cross-institutional Case Study, IELTS Research Reports Online Series, No. 3. British Council, Cambridge Assessment English and IDP: IELTS Australia. Available at https://www.ielts.org/teaching-and-research/research-reports
• Lea, M. (1994). ‘I Thought I Could Write Until I Came Here’: Student Writing in Higher Education, in D. Graddol & S. Thomas, Language in a Changing Europe, Clevedon: Multilingual Matters, pp. 64-72.
• Lea, M. R., & Stierer, B. (Eds.) (2000). Student Writing in Higher Education: New Contexts. Buckingham, UK: SRHE & Open University Press.
• Lea, M. R., & Street, B. V. (1998). Student Writing in Higher Education: An Academic Literacies Approach, Studies in Higher Education, 23(2), pp. 157-172. DOI: 10.1080/03075079812331380364
• Li, G., Chen, W., & Duanmu, J. L. (2010). Determinants of International Students’ Academic Performance: A Comparison between Chinese and other International Students. Journal of Studies in International Education, 14(4), pp. 389-405. DOI:10.1177/1028315309331490
• Li, Y., & Hu, G. (2018). Supporting Students’ Assignment Writing: What Lecturers Do in a Master of Education Programme. Assessment & Evaluation in Higher Education, 43(1), pp. 1-13. DOI: 10.1080/02602938.2016.1274017
• Lloyd-Jones, G., Neame, C., & Medaney, S. (2007). A Multiple Case Study of the Relationship between the Indicators of Students’ English Language Competence on Entry and Students’ Academic Progress at an International Postgraduate University. IELTS Research Reports, Volume 11. IELTS Australia and British Council.
• Lobe, B., Morgan, D., & Hoffman, K.A. (2020). Qualitative Data Collection in an Era of Social Distancing, International Journal of Qualitative Methods, 19, pp. 1-8.
• MacDonald, J. (2019). Sitting at 6.5: Problematizing IELTS and Admissions to Canadian Universities, ESL Canada Journal/Revue TESL du Canada, 36(1), pp. 160-171. DOI:10.18806/tesl.v36i1.1308
• McNamara, T., & Roever, C. (2006). Language Testing: The Social Dimension. London: Blackwell.
• Nesi, H., & Gardner, S. (2012). Genres across the Disciplines: Student Writing in Higher Education. Cambridge: Cambridge University Press.
• O’Loughlin, K. (2008). The Use of IELTS for University Selection in Australia: a Case Study, IELTS Research Reports, Volume 8, ed. J Osborne, IELTS Australia Pty Ltd, Canberra, pp. 145-242.
• O’Loughlin, K. (2013). Developing the Assessment Literacy of University Proficiency Test Users. Language Testing, 30(3), pp. 363-380. DOI: 10.1177/0265532213480336
• O’Loughlin, K. (2015). ‘But isn’t IELTS the Most Trustworthy?’: English Language Assessment for Entry into Higher Education. In A. Ata & A. Kostogriz (eds.), International Education and Cultural Linguistic Experiences of IS in Australia. Samford Valley, Qld: Australian Academic Press, pp. 181-194.
• Pearson, W. (2020). Mapping English Language Proficiency Cut-off Scores and Pre-sessional EAP Programmes in UK Higher Education. Journal of English for Academic Purposes, 45, pp. 1-11. DOI: 10.1016/j.jeap. 2020.100866
• Pill, J., & Harding, L. (2023). Defining the Language Assessment Literacy Gap: Evidence from a Parliamentary Inquiry. Language Testing, 30(3), pp. 381-402. DOI:10.1177/0265532213480337
• Scott, T., & Mhunpiew, N. (2021). Impact of Government Policies and International Students on UK University Economic Stability, International Education Studies, 14 (5), pp. 1-7. DOI:10.5539/ies.v14n5p1
• Shaw, S. D. & Weir, C. (2007). Examining Writing - Research Practice in Assessing Second Language Writing. Studies in Language Testing 26, Cambridge: Cambridge University Press.
• Shohamy, E. (2001). The Power of Tests: A Critical Perspective on the Uses of Language Tests. Harlow: Pearson Education.
• Smith, H., & Haslett, S. (2007). Attitudes of Tertiary Key Decision-makers towards English Language Tests in Aotearoa New Zealand: Report on the Results of a National Provider Survey, IELTS Research Reports, Volume 7, IELTS Australia Pty Ltd, Canberra and British Council, London, pp. 137-204.
• Teti, M., Schatz, E., & Liebenberg. L. (2020). Methods in the Time of COVID-19: the Vital Role of Qualitative Inquiries, International Journal of Qualitative Methods, 19, pp. 1-5.
• Weigle, S. C. (2002). Assessing Writing. Cambridge: Cambridge University Press.

Appendix 1: Questionnaire items

Yes/No responses unless otherwise indicated.

Project Title: Mind the gap. Unpacking IELTS Academic writing band descriptors to inform university admissions decisions.

This questionnaire explores your experience of using IELTS in making university admissions decisions, your familiarity with IELTS, the English writing requirements on your degree program, and your perception of IS’ writing ability.

Large numbers of international students take the IELTS exam to gain access to higher education in the UK, where writing is generally central to assessment. This research study explores understandings of what IELTS writing scores represent in practical terms, especially for staff responsible for making admission decisions.

We are a team of researchers at the University of Central Lancashire (UCLan), in the School of Humanities, Language and Global Studies, who are interested in the use of IELTS for the purposes of admission to UK universities. The project is supported by the British Council.

This questionnaire will take no more than 10-15 minutes to complete and the data collected is completely anonymous. At the end, we will ask if you would like to participate in a follow-up interview and writing evaluation task. If you agree, identifying information such as your name and email address will be kept separately from the questionnaire data and will be used only to contact you to arrange an interview and, if you wish, to feed back on the results of the study. If you are willing to take part in this interview and task, please confirm this when prompted on the questionnaire and leave your contact details.

Any information given will be used solely for academic purposes, which may include a research report for the British Council, academic publications, academic conferences or seminar presentations. All data will be anonymised in the final report and any subsequent publications based on the data. Your name or other identifying information will not appear in any written information related to the project when it is being prepared for publication.

All data will be stored securely in digital form on password-protected cloud storage to which only the researchers will have access. The information collected will be stored for a maximum of 5 years. Further information about UCLan’s privacy policy can be found here: https://www.uclan.ac.uk/data_protection/privacy-notice-research-participants.php

If you have any questions about this study, please contact the Lead Researcher, Dr Tania Horak by email thorak@uclan.ac.uk. If you have any concerns about the way the study has been conducted, you can contact the University Officer for Ethics: (email: OfficerforEthics@uclan.ac.uk)

Thank you for your participation.

Consent

Please click on each of the text boxes below to indicate that you agree to them.

• I confirm that I have read and understand the Information Sheet for the above study. I have had an opportunity to consider the information, contact the researcher to ask questions (e.g. by email) and have had these questions answered satisfactorily.
• I understand that taking part in the study involves completing a questionnaire and then (optionally) participating in a recorded interview and writing evaluation task.
• I understand that my participation is voluntary but that questionnaire data cannot be deleted after I have completed the questionnaire, because the data are stored anonymously.
• I understand that the anonymous data aggregated from all participants will be used to understand the use of IELTS in UK university admissions decisions, and may be published in scientific journal articles and shared in permanent, publicly accessible archives accessible from any country.
• I understand that the information I provide will be held securely and in line with data protection requirements at the University of Central Lancashire.
• I understand that questionnaires will be retained in password-protected digital storage device until five years after the completion of the project.
• I understand that the research team will respect my confidentiality and I give permission for them to have access to my responses.
• I understand that confidentiality and anonymity will be maintained, and it will not be possible to identify me in any reports, presentations or publications arising from the research. No personal details will be included in the research report or any subsequent publications or presentations (i.e. name, place of work, location of place of work, length of service).
• I agree to take part in this study.

BLOCK ONE: ABOUT YOU

Q3 Are you responsible for making decisions about the admission of IS to a degree program at:

• a university in the UK?
• a UK university partner which is based abroad?
• other - please specify

Q4 Where do you work?

Q5 Please tell us which university you work at (this will be anonymised in any publications)

Q6 Which subject area do teach in? From HESA JACS categories: https://www.hesa.ac.uk/support/documentation/jacs/jacs3-principal

• Medicine and dentistry
• Subjects allied to medicine
• Biological sciences
• Veterinary sciences
• Agriculture and related subjects
• Physical sciences
• Mathematical sciences
• Computer science
• Engineering and technology
• Architecture, building and planning
• Social studies
• Law
• Education
• Business and administrative studies
• Mass communications & documentation
• Languages
• Historical and philosophical studies
• Creative arts and design

Q7 How long have you been teaching IS in your current, or any another, UK university?

• Up to 1 year

• Between 1-3 years

• Over 3 years - 10 years

• Over 10 years

BLOCK TWO: YOUR EXPERIENCE / TRAINING WITH ADMISSIONS

Q8 How long have you been involved in the admission of IS to your current, or any other, UK university?

• Up to 1 year

• Between 1-3 years

• Over 3 years - 10 years

• Over 10 years

Q9 Have you received any sort of training for this role of making admissions decisions?

• Academic colleague(s)
• Admin colleague(s)
• Language testing experts
• Other - please specify

Q10 If you did receive training, from whom?

BLOCK THREE: YOUR FAMILIARITY WITH IELTS

Q11 Have you ever attended any training / seminars / talks about the exams accepted as proof of English level for entrance to university in the UK, as decided, by UKVI (UK Visas & Immigration)?

• Yes

• No

• Can’t remember

Q12 Have you ever attended any training / seminars / talks about understanding the IELTS exam specifically?

• Yes

• No

• Can’t remember

Q13 If you did receive any training on IELTS, did you receive any explanation of what the writing section of the IELTS exam involves?

Q14 How familiar do you feel with the IELTS exam overall?

• Very unfamiliar

• Unfamiliar

• Familiar

• Very familiar

BLOCK FOUR: YOUR FAMILIARITY WITH IELTS WRITING

Q15 How familiar do you feel with the IELTS writing section in particular?

• Very unfamiliar

• Unfamiliar

• Familiar

• Very familiar

Q16 Have you ever seen the IELTS writing tasks (the questions) that candidates get in the test?

Q17 Have you ever seen examples of writing that candidates produce for IELTS?

Q18 Which of the following do you believe to be the case, based on your knowledge of the IELTS exam? Candidates have to write: (tick as many as appropriate)

• an essay
• a report
• a short text based on graphics (chart, diagram, etc.)
• a letter
• other (please specify)

Q19 Which of the following do you believe to be the case, based on your knowledge of the IELTS exam? Candidates have to write:

• about 500 words (in total for both tasks)
• more than 500 words (in total for both tasks)
• less than 500 words (in total for both tasks)

Q20 Which of the following do you believe to be the case, based on your knowledge of the IELTS exam? Candidates have to include citations:

• Yes

• No

• Don’t know

Q21 Have you ever sat the IELTS exam yourself?

• Yes

• No

• Don’t know

BLOCK FIVE: YOUR USE OF THE IELTS DESCRIPTORS

Q22 Have you ever seen the IELTS bands descriptors which the examiners use to award grades for the writing section of IELTS?

Q23 Do you refer to the IELTS writing descriptors when making admissions decisions?

BLOCK SIX: ENGLISH WRITING REQUIREMENTS ON YOUR DEGREE PROGRAM

Q24 What would you expect a student who has passed IELTS at the minimum level required for your degree program be able to do? (select all which you feel are appropriate)

• Structure an essay effectively
• Structure a report effectively
• Cite and reference accurately
• Use accurate grammar
• Use accurate spelling
• Use accurate punctuation
• Use appropriate academic vocabulary
• Paraphrase effectively
• Clearly and directly respond to the assignment task, covering all requirements
• Work independently to produce a written assignment
• Other (please specify)

Q25 In order to join your degree program, what IELTS score (or equivalent) do IS need (overall)?

• IELTS 5.0
• IELTS 5.5
• IELTS 6.0
• IELTS 6.5
• IELTS 7.0
• IELTS 7.5
• IELTS 8.0
• Other (please specify)

Q26 In order to join your degree program, do IS need a minimum score for the writing component of IELTS? If so, what?

• IELTS 5.0
• IELTS 5.5
• IELTS 6.0
• IELTS 6.5
• IELTS 7.0
• IELTS 7.5
• IELTS 8.0
• No specific minimum score for writing is needed

Q27 What types of writing are students on your degree program required to produce?

• Case study (examples: business start-up, company report, patient report, organisational analysis) (1)
• Critique (examples: academic paper review, business environment analysis, legal case report, system evaluation, teaching observation, review of a book/film/play) (2)
• Design specification (examples: database, game, product, system, website or building designs) (3)
• Essay (examples: discussion, exposition, challenge, consequential, factorial) (4)
• Exercise (examples: calculations, data analysis, statistics exercises) (5)
• Description/ Explanation (examples: concept/ job/ legislation overview, account of a natural phenomenon, instrument description, system/ process explanation) (6)
• Literature survey / review (examples: annotated bibliography, literature overview, literature review, research methods review) (7)
• Methodology report (examples: data analysis report, field report, lab report, materials selection report) (8)
• Narrative (examples: biography, creative writing, plot synopsis, reflective account) (9)
• Problem-solution scenario (examples: business scenario, law problem, logistics simulation) (10)
• Proposal (examples: research proposal, business plan, marketing plan, procedural plan) (11)
• Research report (examples: experiment write-up, research-based dissertation, research article) (12)
• Writing for a non-academic audience (examples: expert advice to industry or layperson, newspaper article, letter, information leaflet) (13)
• Other (Please specify) (14)

BLOCK SEVEN: THE VALUE OF IELTS TO PREDICT WRITING ABILITY

Q28 When ISs join your degree program, how would you rate their writing ability overall?

• Very poor
• Poor
• Good
• Very good

Q29 How well do you think a student’s IELTS score predicts their ability to successfully produce the types of writing required on your degree program?

• Very poorly

• Poorly

• To some extent

• Very well

• Little writing is required on my degree program

BLOCK EIGHT: INVITATION TO STAGE 2

Thank you for your responses.

Would you be interested in participating in a Stage 2 of the study? In Stage 2, we would like to interview you for about 30 minutes and ask you to look at samples of student writing and tell us what you think of them. We will do this online.

Survey termination message

Thank you for your time and insight. If you have shared your email address with us, we will be in touch to arrange an interview.

If you have any questions, please contact the lead researcher, Dr Tania Horak

E: Thorak@uclan.ac.uk

Appendix 2: Subject areas targeted

Based on JACS 3.0 Principal Subject Codes (introduced 2012/13).

JACS (the Joint Academic Coding System) is a way of classifying academic subjects and modules, used by HESA (Higher Education Statistics Academy)

https://www.hesa.ac.uk/support/documentation/jacs/jacs3-principal

Appendix 3: Coding scheme

Appendix 4: List of criteria used by participants (during think-aloud stage) to judge the scripts and number of mentions

Author biodata

Dr Tania Horák is a senior lecturer at the University of Central Lancashire (UCLan) in the northwest of England. She is currently course leader of MA in TESOL and Applied Linguistics and also teaches and supervises students on the Professional Doctorate in Education program. She has previously worked in TESOL in the Czech Republic, Bangladesh and Lithuania. Her research interests lie in foreign language testing and assessment, above all washback, impact and social consequences of assessment.

Dr Sharon McCulloch is a senior lecturer in language and education at the University of Central Lancashire. She is currently course leader of the Professional Doctorate in Education program. She has been teaching English language and EAP since 1994 in the UK, the Czech Republic and Japan. Her research interests lie in academic literacies, including both student writing and the writing and publishing done by professional academics.

Elena Gandini is an EFL/ESOL lecturer at the University of Central Lancashire, teaching on EFL and EAP courses, contributing to test development as a member of the language testing team and working on pre-service teacher training intensive courses. She has previously worked as a language teacher and teacher trainer in Italy, Austria and Germany. Her research interests are in the area of language testing and assessment, multilingualism and material development.