The attitudes of IELTS stakeholders: student and staff perceptions of IELTS in Australian, UK and Chinese tertiary institutions

Abstract

This report presents the findings of a three-nation study which examined stakeholder attitudes to the International English Language Testing System (IELTS). With research undertaken in Australia, the People’s Republic of China and the United Kingdom, the perceptions and perspectives of university staff and students were measured via quantitative and qualitative methodologies. Staff and student surveys were distributed, culminating in a total respondent group of 624. To triangulate the data, 37 qualitative interviews were conducted. The student group consisted of currently enrolled tertiary degree students who had provided an IELTS score in partial fulfilment of university admissions. The staff group consisted of persons in academic and administrative positions at the focal institutions.

As an inductive study, various themes were highlighted in the research instruments. These included: knowledge areas; perceptual/attitudinal areas; beliefs regarding the predictive nature of IELTS vis-à-vis university demands; the appropriateness of entry levels; and perceptions of unprincipled activity. In general, the greatest areas of difference existed between the staff and student groups, irrespective of institutional location. Students were, on the whole, more knowledgeable than staff on a wide range of themes related to the IELTS test. Both staff and students indicated that the purpose of the IELTS test is primarily functional (ie. university entry), with a secondary learning/skill improvement role.

Overall, respondents perceived the IELTS test to have high validity, but staff and student respondents differed over the predictive nature of the IELTS test score in relation to English levels required for university study. Students were generally satisfied with the IELTS entry scores used by their institutions and were, on the whole, positive regarding the relationship between English language skills demonstrated by an IELTS score and language abilities necessary to succeed in a university environment. In contrast, staff respondents generally wished for an increase in their institution’s minimum IELTS entry score, and were on the whole less satisfied with English language abilities.

There were some perceptions of unprincipled uses of the IELTS test. This related particularly to the admission of students with IELTS scores below the institutional minimum entry cut-off value.

Among the general conclusions, it is suggested that past IELTS participants should be included in policy reviews of institutional English language standards. Greater effort should be made to educate tertiary staff about the form and function of the IELTS test, and about the meaning of IELTS scores. Research into the evolving nature of authentic classroom genres is suggested and it is proposed that the washback effects of IELTS might be used to better prepare students for the complex ‘real-world’ academic language tasks they will encounter.

2.0 Background

Since the inception of the International English Language Testing System (IELTS), much research has been conducted on its operations. Many studies have focused on the reliability and the validity of IELTS in assessing the readiness of candidates to train or study in the medium of English (Ingram 1991; Buell 1992; Charge and Taylor 1997; Kerstjens and Nery 2000). Other research has examined the position the IELTS test occupies in the learning process of NESB students, reporting on approaches to the test itself (Mickan 1997), and situating the IELTS within a broader learning context (Barrett-Lennard 1997).

The current study is akin to the latter category of IELTS related research. In recent years, there has been growing emphasis in language testing on the importance of consulting with stakeholders and taking their views seriously (Rea-Dickins 1997). This is part of a larger concern with the societal impact of language tests, particularly the interplay of high-stakes language tests and market forces, and with the ethical

implications of testing (Davies 1997; Chaloub-Deville and Turner 2000; Hamp-Lyons 1997, 2000; McNamara 2001; Shohamy 2001; Spolsky 1997). These themes reflect a change in focus in language testing, which is mirrored more broadly in the field of applied linguistics through the so-called ‘social turn’ in recent research. The social turn is a recognition of language learning and teaching (and now testing) as social activities, involving the participation of socially-located individuals in widely-differing, and often unequal, social contexts (McNamara 2001). Rea-Dickins identifies a broad range of potential stakeholders in any testing situation, including: language testers; applied linguists; teachers; parents; administrators; teacher educators; funding bodies; sponsors; government bodies; the public; national and international examination authorities; test-takers and test users. This last category (test users) is particularly relevant to the current study, for it includes “university admission officers who need to interpret scores on national and international proficiency tests (such as TOEFL and IELTS)” (Rea-Dickins 1997, p. 305).

Also relevant to this study is Hamp-Lyons’ (2000) observation that the voices of test-takers are the least heard of all stakeholders, yet they have the largest stake in testing. Furthermore, Wall notes that it is important to distinguish students’ perceptions of tests “as opposed to their teachers’ impressions of these perceptions” (Wall 2000, p. 506). While acknowledging that the surveying of stakeholders’ perceptions might be interpreted as mere ‘lip service’ to consultation, Rea-Dickins (1997) argues that stakeholder involvement can lead to greater democratisation and fairness - provided that it is not merely a forum for complaints - if the process encourages more active stakeholder participation and feedback.

Research into the socio-educational impact of language tests distinguishes ‘washback’ - the impact that tests have on curriculum, learning and teaching - from ‘impact’ (see Hamp-Lyons 1997; Wall 1998, 2000), which refers to the broader, sometimes unintended, societal effects of tests. Messick redefines construct validity to include what he calls its ‘consequential aspect’. That is, “evidence and rationales for evaluating the intended and unintended consequences of score interpretation and use in both the short- and long-term, especially those associated with bias in scoring and interpretations, with unfairness in test use, and with positive and negative washback effects on teaching and learning” (Messick 1996, p. 251). Hamp-Lyons argues that while washback is a significant component of language testing and can have both negative and positive effects on learning and teaching, language testers, in particular, need to consider the impact of testing: “many more studies are needed of students’ views and their accounts of the effects on their lives of test preparation, test-taking and the scores they have received on tests” (Hamp-Lyons 1997, p. 299).

This study sought to access the viewpoints of two key IELTS stakeholder groups (IELTS participants and academic/administrative staff), critically evaluating perceptions and interpretations of IELTS. As the project title suggests, the research sought to identify how IELTS is understood in university settings, from both an administrative and an educational perspective. As will be discussed, the intended and unintended uses of the IELTS test, washback effects, attitudes towards test preparation/test-taking, and interpretations of the meaning of IELTS scores are all relevant to the current research. IELTS is the ‘preferred’ English language test of many UK universities, and is the sole test acceptable to all Australian universities (Coley 1999). Its very prominence means that perceptions of and attitudes towards IELTS are highly relevant to understanding the socio-cultural place that IELTS - and English language testing in general - has come to occupy in the modern higher education sector.

3.0 Methodology

The study made use of both survey and interview methodologies. This had the advantage, firstly, of collecting quantitative and qualitative data, and secondly, of triangulating data in a multi-modal methodology.

One student group (‘successful’ IELTS participants) and two tertiary staff groups (academic and administrative) were contacted. In this research context, a ‘successful’ IELTS participant is defined as an NESB student who provided an IELTS score in partial fulfilment of student admissions at one of the focal institutions. A random selection of all IELTS student participants at UNSW, the University of Ulster and the Guangzhou branch campus were surveyed. A lesser number of IELTS participants were interviewed.

The academic/administrative staff groups represented a cross-section of tertiary employees who, on a regular or intermittent basis, may sight IELTS scores and make decisions based on these scores in relation to university admission, subject selection, degree progress, et cetera. However, no effort was made to exclude from the study staff with little or no formal knowledge of the IELTS system. We felt it was important to capture a realistic cross-section of staff, and hence, of staff perceptions and knowledge of the IELTS test. In other words, we were as interested in recording what staff do not know about IELTS as we were in documenting what they do know. Consequently, the academic/administrative staff groups were selected in a manner that would maximise an appropriate cross-section of staff experience with the IELTS test itself and with IELTS participants. As with the student group, a lesser proportion of academic/administrative staff were interviewed.

3.1 Research sites

The research was conducted on-site at the University of New South Wales, Australia, the University of Ulster, Northern Ireland, and a Guangzhou-based branch campus. The following is a brief summary of the three institutions.

3.1.1 The University of New South Wales

The University of New South Wales (UNSW) is in Sydney, Australia. UNSW offers undergraduate and postgraduate courses through nine faculties. The university operates three campuses.

UNSW is a highly suitable locus for research into the attitudes of IELTS stakeholders. Of the 32,000 students on campus, over 6,000 are international students, making the student body one of the most international in the world (Uniken 1999; UNSW 2000). Just under half of the student body (14,793) was born outside of Australia, the majority from Asia and the Middle East (UNSW 1998). Research conducted in the Faculty of Commerce and Economics, the university’s largest faculty, found that 59 different languages were spoken by their students, and that 48% of students spoke a language other than English at home (UNSW 1997).

3.1.2 The University of Ulster

The University of Ulster is in Northern Ireland. With campuses in Belfast, Jordanstown, Coleraine and Magee, Ulster’s six faculties cater to 20,000 undergraduate and postgraduate students. At present, there are approximately 800 international students studying at the University of Ulster, including exchange students from the European Union. The majority of exchange students (ie. ERASMUS/SOCRATES participants), are from European language backgrounds, while a large proportion of full degree international students are from the Asia-Pacific region.

3.1.3 The University of Ulster/Hong Kong College of Technology, Guangzhou

Established in the late 1990s, the Guangzhou campus operates as a franchise of the University of Ulster, administered in joint venture with the Hong Kong College of Technology. Students who successfully complete the Higher National Diploma in Computing Sciences at South China Agricultural University may apply for admission to the two-year franchised course, leading to a Bachelor of Science in Computing Science. This is offered jointly by the School of Informatics, University of Ulster, and Hong Kong College of Technology. This program has been validated by the University of Ulster and audited by the UK Quality Assurance Agency. It is delivered at the College of Sciences, South China Agricultural University. The Centre for English Language Teaching (CELT) of the University of Ulster is responsible for the delivery of IELTS preparation programs by university appointed lecturers.

To enter the first year of the franchised course, students must have achieved an IELTS score of 5.0; to proceed to the second (final year), they must achieve an IELTS score of 6.0. Participating students complete at least part of their degree at the Guangzhou campus. Students have the option, but are not obliged, to complete the final year of undergraduate courses in Northern Ireland.

3.2 Advantages of the focal institutions

The smaller proportion of international and/or NESB students at the University of Ulster provides an excellent counter-point to UNSW’s much larger NESB population. For instance, it may be hypothesised that there is a differential knowledge base and understanding of IELTS in direct relation to the size and relative primacy of the NESB student body at the two universities.

As noted, a substantial proportion of the NESB students at the University of Ulster are from European language backgrounds. This is in contrast to the largely Asian-Pacific linguistic heritage found at the University of New South Wales. The Guangzhou campus consists, at present, entirely of local Chinese students. These varied institutional demographics, distributed across three countries, has the potential to provide a rich and representative insight into stakeholder perceptions of IELTS.

Furthermore, with the rapid expansion in offshore educational provision, branch campuses and joint venture arrangements in higher education, it is meaningful to include an off-shore campus in the overall research design. Figures indicate that just over 50 off-shore campuses and twinning arrangements were operated by Australian universities in 1996 (Back et al. 1996). By 1999, Australian universities operated nearly 600 off-shore programs, thus representing a ten-fold increase in off-shore operations in just three years. The enrolment at off-shore campuses accounted for one-quarter of Australia’s international student enrolments in that year, or in real terms, 25,000 individuals (IDP 1999). The university sectors of many OECD countries have undergone similar accelerations in their transnational growth. Despite these significant developments, few independent assessments or evaluations of external programs have been undertaken (for a detailed discussion, see Coleman 2000). Thus, the inclusion of respondents from the Guangzhou campus provides data on the substantial, but often overlooked, transnational nature of international student enrolment.

3.3 Survey

The purpose of the survey was to gather staff and student comments on a range of perceptual, factual and attitudinal themes in relation to the IELTS system and test. Included in Appendix 1.0, the staff and student surveys had 42 questions in common. The remaining questions (10 additional for staff; 19 additional for students) related to issues specific to that group, that of the assessor/interpreter group (staff) and that of the assessed/participant group (student). The survey was developed in light of existing University of Cambridge Local Examinations Syndicate (UCLES) work in this area, and with close coordination between the UK, Chinese and Australian research sites.

While presented to respondents as single section questionnaires, the survey may be sub-divided into a number of thematic sections, as a range of attitudinal, perceptual and belief-system questions were posed - as well as questions relating to the respondents’ knowledge of the testing system.

The following general categories were included in the survey design.

3.3.1 Preparatory materials

To gain a comprehensive understanding of stakeholder perceptions of IELTS, it was useful to query IELTS participants on their attitudes to test preparatory materials (as applicable). For instance, it was considered informative to establish whether students tend to utilise preparatory materials as a means of performing well in the IELTS test itself (outcome orientation) or as a means of improving overall English language skills (process orientation).

Student respondents were specifically queried on: the sort of IELTS preparation they undertook, as applicable (Q. 5, see Appendix 1.1); the rationale for undertaking IELTS preparation, and; the relative importance of a range of decision-making rationale (Q. 6, see Appendix 1.1).

Additional survey questions aimed to establish whether IELTS participants believed that preparing for/taking the IELTS test helped to improve their English language skills. These questions sought greater refinement to the outcome/process orientation questioning. For example, students may have initially approached the IELTS from an outcome perspective (university entry), but participation may have nonetheless resulted in a process benefit (language skill improvement) (See Q. 7-11, Appendix 1.1).

3.3.2 Perceptions of IELTS

We queried staff and students on their perceptions of IELTS, in particular its purpose and how effective participants perceived the test to be in assessing preparedness for various language demands.

Topics included:

• speaking skills
• listening skills
• writing skills
• reading skills
• mixed skill environments (ie. group work scenarios that require rapid alternations between reading, speaking, listening and writing)
• university study overall
• language skills as a whole.

Other perceptual/attitudinal queries asked of both respondent groups included questions relating to the validity, reliability and predictiveness of the IELTS test, as well as questions about the respondents’ perceptions of unprincipled use of test results. These are discussed in the Results section.

3.3.3 Self-assessment of language levels

IELTS participant (student) and interpreter (staff) perspectives on the English language level necessary to succeed in a university environment are instructive. Respondents were asked to indicate what they construe to be the overall IELTS score necessary for academic success at differing levels of university study (ie. undergraduate, postgraduate). In this questioning, IELTS participants and interpreters were encouraged to reflect upon their actual experience of English language in a tertiary context, and to comment upon the appropriateness of their university’s IELTS admission levels in the identification of linguistic ability.

3.3.4 Knowledge areas

Various fact- and experience-based questions were asked. Questions related to respondent knowledge of IELTS cut-off entry scores overall, and whether differential IELTS entry scores applied in relation to student country of origin, faculty or by level of study. Respondents were asked to identify the highest possible IELTS band score.

3.3.5 Congruent measures

Identical staff and student surveys were administered at the three research sites. A common survey was important for the research outcomes and the data analysis, for it allowed the researchers to conduct quantitative analyses across the entirety of the intra-group respondents (ie. all students, across the three research sites). As noted, the staff and student questionnaires shared 42 common questions. This allowed statistical analyses between the staff and student respondent grouping (inter-group analysis).

3.3.6 Demographic information

To gain a refinement in the quantitative analysis of survey results, various demographic details were sought (gender, country of origin, primary language). In addition, an IELTS performance history was sought for both staff and students.

For students, some further demographic information was collected, including level of study, faculty, and year of study. Also asked was the number of IELTS test attempts made, their IELTS scores on the most recent attempt, and whether the student had participated in non-IELTS English language testing. Student demographic information is summarised in Table 1.

For staff, additional demographic queries included their primary role at the university, level of seniority, and relevant representative positions held. Staff demographic information is summarised in Table 2.

3.3.7 Survey sampling

All groups were sampled using the method of systematic sampling with a random start. In this sampling method, the first sample is selected by means of a random numbers table, and each subsequent sample is taken at a sampling interval. The interval is calculated from the total size of the sub-group divided by the number of students to be sampled. For example, suppose the total number of Ulster students who had provided an IELTS score for admission purposes was 200, and we wished to sample 50 of those students. As 200 divided by 50 is four, the sample group would include every fourth student - beginning at the number provided by the random start. So, if the random numbers table gave the number 191 to start, the group would include student number 191, 195, 199. One then recommences at the top of the list, to student 3, 7, 11 etc. For a further description of this sampling methodology, see Babbie (1989).

At the University of New South Wales, staff and student contact information was accessed via the Office of Human Resources and the Student Administration Department, respectively. A staff list - inclusive of academic and administrative staff - was provided by Human Resources. UNSW has a total staff population of 5,000, consisting of 2,075 academic (full-time equivalent) and 2,917 general staff (full-time equivalent) (UNSW 2000). Following systematic sampling, 1,000 surveys were sent via the university’s internal mail system. A self-addressed return envelope was included.

For the student survey, only those students who had provided an IELTS score as part of their admissions process were selected. As only international students are required to provide a standardised test score as evidence of English language proficiency, the UNSW student group thus represented a sampling of the university’s international student population. In 2000 (the most recent statistics available) the international student population at UNSW numbered 6,860 (UNSW 2000).

The UNSW Student Administration Department provided a list of 1,019 students, consisting primarily of postgraduate students (n=874), with a lesser number of undergraduate (n=131) and non-award (n=14) students. Given the large number of international students at UNSW, and the prominence of IELTS amongst the range of English language tests available (Coley 1999), it is likely that the actual number of international students (particularly undergraduates) who provided an IELTS score as part of their admissions was in fact much higher. However, it was determined that a sample set of 1,019 students was sufficient. Surveys were mailed to the entire sample. A self-addressed postage-paid return envelope was included.

A similar method was followed at the University of Ulster. With total employees, including academic and general, numbering around 3,500 (Ulster 2001) and approximately 800 on-campus international students (Ulster 2000), a total of 300 staff surveys were sent via the university’s internal mail system, and 300 student surveys were sent via the regular post. As with UNSW, return envelopes (and postage where applicable) were provided.

The much smaller campus at Guangzhou called for a different method. Surveys were circulated within lecture and tutorial sessions, but survey completion remained voluntary. Approximately 100 student surveys were distributed. As noted in the description of the Guangzhou campus, participating students undertake repeat IELTS testing, rather than the single pre-admission English proficiency testing familiar to most universities. To enter the first year of the program, students must achieve an IELTS score of 5.0; to proceed to the second (final year), they must achieve an IELTS score of 6.0.

No staff surveys were distributed in Guangzhou, primarily because of the complex institutional relationships involved. Described earlier, the Guangzhou franchise is operated as a joint venture between the University of Ulster and the Hong Kong College of Technology. The course itself is delivered at the College of Sciences, South China Agricultural University, taught in the main, however, by Hong Kong College of Technology staff. During the data collection phase, one visiting Ulster staff member was wholly responsible for IELTS preparation (in the period since, two EFL lecturers have been appointed by Ulster). Four local teachers, employed by the College of Humanities at South China Agricultural University, assisted the visiting Ulster staff member at various points during the course. Given such fractional involvements of staff members, distributed across multiple institutions, a decision was made to forego staff surveys in Guangzhou.

In the research planning stage, the researchers aimed for a minimum of 500 survey respondents overall. For the student survey, a reasonable cross-section of all disciplines and level of study was a central aim. In the staff survey, a reasonable balance of academic and general staff was sought.

Tables 1 and 2 provide a breakdown of student and staff respondent figures. While sample totals are certainly of sufficient size to undertake robust statistical analysis, one is reminded of the low response rates typical to postal surveys. Postal survey response rates are commonly in the range of 10-30% (Chiu and Brennan 1990; DSS Research 2001). The staff response rates at the Australian and UK institutions (14%; 20% respectively) and the 30% response rate among the student samples at the Australian and UK institutions are all within the expected range.

A total of 429 student surveys and 195 staff surveys were returned, giving a total respondent group of 624. Respondent information is broken down by institutional origin, gender, origin by region, and several other variables.

Table 1: Survey respondents, student demographics

*Gender values do not equal the sample total as nine students did not give a response. **Level of study values do not equal the sample total as 11 students did not give a response

Table 2: Survey respondents, staff demographics

*Gender values do not equal the sample total as five staff did not give a response.

3.4 Interview methodology

Given the formal nature of the postal survey, an informal interview schedule was designed. The benefit of an informal interview design is that it allows the use of probe and follow-up questions, and may facilitate more subtle expressions of viewpoints in respondents than is possible with formal interviews. In research studies such as this, it is advisable to utilise two methodologies to triangulate the data and to increase the validity and reliability of the findings.

Firstly, to gain more detailed explorations of participant perspectives, interviews were conducted with a cross-section of IELTS participants. Secondly, interviews were conducted with a cross-section of academic/administrative staff at the UK and Australian research sites. For logistical reasons, it was not possible to conduct interviews in Guangzhou.

Common staff and student interview guides were used in the interviews, allowing for consistency in the topics covered (interview guides are at Appendices 2.1 and 2.2). The interview guide also served to complement the questionnaire topics. Interview facilitation was shared between the authors. To limit interviewer bias, five of the Australian staff interviews were conducted with two facilitators. Five of the UK staff interviews were conducted by one of the Australian researchers.

3.4.1 Staff interviews

The purpose of the academic/administrative staff interviews was to gain insights into how decision-makers interpret and utilise IELTS scores in administrative and educational contexts. A range of academic/administrative staff was interviewed. Given the time and logistical constraints involved, it was considered important to interview across as many institutional strata as practicable. For inductive research into perceptions and attitudes, interview responses are most valid and reliable when they represent the greatest range of respondents and responses.

A great diversity of university personnel are involved in the interpretation of IELTS scores and/or rely upon interpretations of IELTS scores in their decision-making. While it may be argued that only a narrow band of university staff require a comprehensive understanding of IELTS (university admissions for example), staff throughout universities do make admissions decisions and formulate judgements regarding a student’s ability to succeed in courses subsequent to the student’s initial entry. Examples may include admissions to specific intra-university programs and decisions regarding pre-requisites and advanced standing. At the postgraduate level, it is not uncommon for entry assessments to be determined at the school/department level, with the Head of School holding the right to waive English language entry pre-requisites.

Similarly, staff who may not be directly involved with admissions decision-making may nonetheless have substantial interaction with NESB and/or international students. Their interpretations of English language levels, and the perceived relationship between the IELTS test and university admission standards is equally informative. As key stakeholders in tertiary education, the views of a broad range of academic and general staff offer an inclusive and subtle portrayal of perceptual/attitudinal positions regarding the IELTS test and how it is used. Thus, it is useful to learn how a diversity of university staff may interpret the use and meaning of IELTS in language assessments. This premise of a diversified staff interaction with IELTS and IELTS scores, also informed the survey methodology.

A number of staff survey respondents had indicated their willingness to participate in a follow-up interview (see staff questionnaire, Appendix 2.2). These staff members were contacted. Additional staff were approached individually to participate. These were selected to provide representative cross-section of views by faculty, level of responsibility/seniority, and category of employment (academic or non-academic).

3.4.2 Student interviews

The main goal of the student interviews, as with the staff interviews, was to add greater depth and nuance to the thematic areas highlighted in the survey. In the student groups, we similarly hoped to gain a

representative distribution of perspectives. As mentioned earlier, all students interviewed were ‘successful’ IELTS participants, that is, international students who had attained sufficiently high IELTS scores to gain entry to tertiary study.

Among those interviewed were several students who approached the researchers, indicating their willingness to participate in a follow-up interview (see student questionnaire, Appendix 2.1). Additional interviewing followed a method of fortuitous sampling. Students who took part in programs administered by the Centre for English Language and Teaching (CELT; University of Ulster) and The Learning Centre (TLC; University of New South Wales) were asked to participate in an interview.

Staff and student interview respondent demographics are shown in Tables 3 and 4. A total of 37 interviews were conducted. In the student group, the preponderance of postgraduate interviewees perhaps reflects a bias in the CELT and TLC user groups. The demographics by gender, faculty and continental origin is relatively balanced, however. While student origin is shown by continental grouping, student interviewees were drawn from 12 countries. In the staff group, prospective respondents were contacted on the basis of the position they held within the university. Gender was a secondary variable. This perhaps accounts for the gender skew within and between the Australian and UK institutions.

Table 3: Interview respondents, students

Table 4: Interview respondents, staff

4.0 Results and discussion

Results will be discussed by thematic area:

• knowledge areas
• student-specific queries
• perceptual/attitudinal areas
• beliefs regarding the predictive nature of IELTS vis-à-vis university demands
• the appropriateness of entry levels, and
• perceptions of unprincipled activity.

The thematic areas are discussed intra- and inter-institutionally. Where no differences exist within either the staff or student cohort across the three research sites, staff and student responses are collapsed into two natural clusters. This is a useful methodology, for it maximises sample sizes and thus enhances the robustness of the data. Overall, the greatest areas of difference existed between the student and staff responses. Where differences exist within either the student or staff groups by institutional origin, the data is presented and discussed in light of these inter-institutional differences.

In the reporting of interview data, staff and student respondents are identified by the national location of their institution, and their stakeholder grouping. For example, a staff member from the UK institution will be identified as ‘staff, UK’. A student respondent from the Australian institution, will be identified as ‘student, AUS’. The interview data has been thematically integrated with the quantitative findings.

4.1 Knowledge areas

Staff and student respondents were asked a variety of knowledge-based questions or, in other words, questions related to the factual elements of the IELTS test and its administration. A number of questions were asked in relation to their institution’s minimum IELTS cut-off scores.

Both staff and students were asked to respond ‘yes’/‘no’/‘don’t know’ to the question: “there are different IELTS entry scores at the undergraduate and postgraduate levels” (staff Q. 1, Appendix 1.2; student Q. 15, Appendix 1.1). At the Australian university, 60% of staff responded ‘don’t know’, with the remaining responses evenly distributed between ‘yes’ (18%) and ‘no’ (22%). In comparison, 51% of students responded ‘yes’ (11% ‘no’; 37% ‘don’t know’). In the 2001 academic year, the correct answer for the Australian institution was ‘yes’, as each faculty had the opportunity to set its own IELTS cut-off scores, resulting in varying IELTS minimums between faculties and across undergraduate and postgraduate

programs. This changed in the 2002 academic year, with the IELTS minimum score for entry to the Australian university standardised at IELTS 6.0 across all faculties and for undergraduate and postgraduate courses ¹. Given this period of changeover, some staff may have been unsure of current policy. Alternatively, many staff respondents may simply not have known.

At the UK university, 59% of staff responded ‘yes’ to this question, 23% ‘no’, 18% ‘don’t know’. Students recorded similar, if more affirmative results: 74% ‘yes’; 2% ‘no’; 22% ‘don’t know’. Interestingly, though, the UK university had (in 2001) a more standardised IELTS cut-off system than the Australian university: the minimum acceptable IELTS score is 6.0 overall, but individual subjects may impose higher score requirements. So, although there is apparently a more standardised institutional IELTS admissions score at the UK university, staff and students perceived more variable undergraduate/postgraduate entry standards than staff and students at the Australian university.

On a related factual question - “Faculties can have different IELTS entry scores ² ” - policy knowledge was clearly differentiated between the staff and student groups. 74% of all students were aware that faculties may set different entry scores. In comparison, only 43% of all staff answered in the affirmative. A Chi-square test of staff and student responses to this question revealed a highly significant difference (Pearson=59.6; p<0.0005). Therefore, students were significantly more knowledgeable about faculty entry standards than staff members.

It is worth noting staff responses by institution. Staff responses at the Australian institution were almost equally divided between ‘yes’ and ‘don’t know’ responses (45% and 46%, respectively), while only 9% believed that faculties may not vary their IELTS entry scores. As outlined above, during the research period, faculties at the Australian university had the right to set their own IELTS entry scores, and there was great variability across the institution. At the UK institution, responses were almost equally divided between ‘yes’/‘no’/‘don’t know’ (38%, 31%, 31%). Again, as noted, while individual subjects at the UK institution may implement a higher IELTS entry score, there exists an institution-wide minimum standard of 6.0. It is not, of course, imperative that all staff know the administrative standards across their own institution. This knowledge does, however, become relevant in double degree programs, and to students enrolled in degrees across two or more faculties.

Staff and students were asked to respond ‘yes’/‘no’/‘don’t know’ to the question: “there are different IELTS entry scores depending on the student’s country of origin” (staff Q. 2, Appendix 1.2; student Q. 16, Appendix 1.1). The official policy of all relevant institutions is meritocratic: the IELTS entry score is standard and is not varied by country of origin. Staff responses reflected this, with $57\%$ of all staff responding in the negative. However, $40\%$ of staff responded ‘don’t know’. While student responses had similarities - $54\%$ answered ‘no’ - there were important differences. $15\%$ of all students, compared to $3\%$ of all staff, responded ‘yes’, indicating a belief that IELTS cut-off scores are varied by a student’s country of origin. A Chi-square test demonstrated a significant difference $(p < 0.0005)$ . The Guangzhou student respondent group heavily influenced this result, with $50\%$ of respondents believing a country bias exists.

A related question - “there are different IELTS entry scores depending on the student’s language background” (staff Q. 3, Appendix 1.2; student Q. 17, Appendix 1.1) - revealed similar results. While the majority of staff and students responded in the negative (56% and 58%, respectively), there was great variance in the positive responses. While just 3% of all staff felt that language background influenced the calculation of IELTS entry scores, 12% of all students felt this was the case. This was sufficiently different to achieve significance (Chi-square, p<0.001). Again, the Guangzhou student cohort had a major influence on the result, with 38% answering in the affirmative, compared to 6% at the Australian institution and 11% at the UK institution.

Staff and students were asked to indicate the highest possible score on the IELTS test. Respondents were asked to tick the appropriate box, ascending from 5.5 (by 0.5 intervals) to 11.0, plus 15, 20 and 100 (staff Q. 36; student Q. 54). One may suggest that the multiple choice format provided a degree of guidance, in that the response was not left open-ended. More than half of all staff (56%) responded ‘don’t know’. This compares with 9% of all students. The most common response to a value was ‘9.0’ (staff 24%; students 63%). This is the correct response. As may be expected, a number of respondents gave the value ‘10.0’ (15% of staff; 8% of students). The results also suggest that a proportion of students in fact guessed on this question. 16% of student responses (n=66) were dispersed between ‘5.5’ and ‘8.5’. This finding was shared between student respondents in the three research sites.

As two-thirds of students, compared with one-quarter of staff, knew the IELTS maximum score, one can assert with some confidence that students are far more knowledgeable of the IELTS band score structure than staff. This finding is supported by extreme statistical significance (Chi-square, p<0.0005).

4.1.1 Student-specific queries

Students were asked to indicate their reason for taking the IELTS test (Q.7, Appendix 1.1). A 12-item list was provided, and students were asked to rank all items that they deemed important to their decision-making. Respondents were given the option to rank as many or as few as they deemed appropriate. A thirteenth item - ‘other - please specify’ - gave students the option to detail further rationales not included in the 12-item list.

Responses are shown in Table 5. Far and away the most important rationale for all students was the item “I had to take the IELTS test to get into my faculty/university of choice”. The second ranked item, as a measure of items ranked first overall by respondents and overall ranking (ranked within the top ten), was closely related (“to help my chances of getting into my faculty/university of choice”). Third overall, selected by one-quarter of respondents as the first (most important) item, was “to improve my English language skills overall”. The fourth most important item for respondents returned to an instrumental rationale (“to gain a standardised English language qualification”). The remaining items declined rapidly in first rank motivation, but the development of specific English language skills remained important to roughly a fifth of respondents. The most frequently reported ‘Other’ reasons were to qualify for a scholarship and for immigration purposes.

The overall rank ordering of responses reflects the high stakes nature or the test, for without a sufficiently high score on the IELTS test (or its equivalent), NESB international students will not be admitted to their tertiary course, or possibly institution, of choice.

Student respondents (n=429)

Student interview respondents made comments congruent with the survey data. Below are typical responses.

It tells you your standard and gets you into university. It is only for this purpose. (student, UK)

I think the main purpose is actually for us to learn the skills - reading, writing, listening and speaking. And also for this university to assess our levels of skills in English before we are taken into the course. (student, UK)

I just did it [IELTS] because I needed a score to get admission. It was just about starting my course, and getting my offer letter. (student, AUS)

As far as I know, usually Australian universities need the students to meet some English language requirement. So they have to take the IELTS or TOEFL test. (student, AUS)

It is to test students’ English ability for listening, speaking, writing. Some people do it for this, but it is expensive to do more than one time. People do it to get into college. (student, UK)

It is of interest to note that staff responses to a related question (“what is the purpose of the IELTS test?”, Q. 7, Appendix 1.2) exhibited similar results. As a measure of items ranked first overall and by overall ranking (ranked within the top ten), staff responses indicated a belief that “for a student to gain a standardised English language qualification” was the most important purpose of the test. This item was ranked number one by 40% of staff respondents, and 62% of respondents ranked it within their top seven. Second in weight was “to improve English language skills overall” (37.9% ranked this first; 50.8% selected this item). Third, was “students have to take the IELTS to get into a faculty/university of choice” (14.4%; 37.4%) and fourth “to help a student’s chances of getting into a faculty/university of choice” (11.3%; 41%). In other words, there was a consensus between the student and staff sample sets (n=624) vis à vis the purpose of the IELTS test, albeit ranking these top four items in differing orders.

Below are some typical staff responses on this theme.

Some students are forced to do this [IELTS] because if they want to come to university they have to meet the English language requirement. For some students there is no choice. IELTS is the most convenient or recognised test from their country of origin. (staff, AUS)

It seems to me students do it [IELTS] because they want to get into university. I can’t really see students doing it just to know how they stand with English. But it’s a good preparation for study in English on a degree course. (staff, UK)

[IELTS] can be a learning tool. It is an assessment tool. But probably just as significantly it is a learning tool…in preparation for the examinations themselves. There is always a certain amount of learning that takes place in an examination situation but critically it is the preparation for those examinations, in terms of skills, and also students’ confidence. (staff, UK)

I suppose it’s a screening tool for language skills in English. From what I understand it has four different measures. It’s actually looking at four aspects of language, ie. listening, speaking, reading and writing. I guess it seems to be used here as a way of screening people for entry into university. (staff, AUS)

As determined by the staff and student survey respondents, of the four most important rationales for IELTS participation, three related to technical/functional motivations, and only one related to skill development. Skill development did feature as a secondary or tertiary motivation, however, with a relatively stable proportion of respondents indicating that improvements in writing, speaking, listening and reading (in that order) are of overall importance. It is worth noting that external factors (part of schooling; parent/teacher decision) had apparently little impact. This suggests that students make an active decision in their selection of the IELTS test - the decision is not made for them. Yet, in a way, participation is ‘enforced’: NESB international students generally must participate in a narrow band of accepted English language proficiency tests or officially recognised training courses if they seek admission to an English language medium university.

Respondents were asked if they had undertaken any IELTS training or preparation, and if so, in what form (Q. 5). Multiple responses were possible. The vast majority of respondents had done some form of IELTS preparation, with only 6% (n=24) indicating they had not. The most commonly used preparation was IELTS practice books (57%), followed by IELTS classes (51%), audio tapes and videos (37%) and IELTS course books (32%). IELTS course books are likely linked to IELTS classes, whereas IELTS practice books, audio tapes, and videos may be used in individual study.

The rationale for undertaking IELTS training/preparation reflects the rationale given for test participation, discussed above. Asked to rank their responses, the most important reason for preparing for the IELTS test was “to improve my English language skills overall”, ranked number one by 35% of respondents. The second most important rationale was “to help me get a higher IELTS score” (27%) and third was “to help my chances of getting into my faculty/university of choice” (26%). The fourth most important rationale related to desired improvements in a specific English language skill (writing), but represented less than 12% of number-one rankings. External factors (ie. part of schooling; told by parent or teacher to do so) were the lowest ranked variables, suggesting active decision-making on the part of IELTS candidates.

As with the comments related to the rationale for test participation, the student interviewees gave responses which suggest a strategic purpose to their preparatory activities: that is, to achieve higher scores. The following are representative comments.

They [the training courses] just train you on how to sit the exam. It doesn’t improve your English at all, I think. It just gives you the tricks, how you can pass the exam, you know, that’s all. Just tricks for how to score 6.0 or 5.5, but not for a 7. (student, AUS)

I’ve used tapes and I’ve borrowed some books. And some sample IELTS tests. I just want to improve my listening ability, so every day I just listen to the test about one hour and now I think I’m familiar with this type of examination…I think it’s not good for improving my English level, it’s only good for improving my ability in taking part in the examination. (student, AUS ³ )

I used this training material where they have these tools like the wordbook and also cassettes. The cassettes actually help us to improve our skills in listening, so I was trying to prepare myself for the test…I used these tools to get myself adapted to the format of the questions. (student, UK)

Some schools teach you how to prepare for the IELTS test, and the teachers give you more models of the IELTS test and after you have done all of the test models, you can know how to take part in the IELTS test. Because in the IELTS test, if you are familiar with the examination type it is easier to get a good mark. (student, AUS)

I had some classes for one year. We had two or three American teachers to teach us the speaking, and our teacher from Greece to teach us the reading, writing and listening. We had tapes and books for the examination. I think a lot of materials are important because you can get used to the exam. Before the real exam you have the simulated tests. You know what kind of questions and how long you have to do the test. (student, UK)

Other student interviewees felt that preparation for the IELTS test did improve their English language skills.

I studied English for five months before I took the IELTS test. We did examples of the test, we practiced speaking and writing. This helped a lot. For example, in listening skills, the teacher used chores, like watching the news. We listened and the teacher distributed a sheet and we tried to answer questions that were quite similar to the real test. We practiced reading and writing too. In writing skills,…we had to explain a chart or a graph. If we didn’t get used to this type of expression, we can’t write well. (student, AUS)

There is a great similarity, then, between the rationale for taking the IELTS test and the motivation for engaging in some sort of IELTS preparation. As with IELTS test-taking, participation in IELTS preparation represents a combination of formative motivation (overall English skill development) and summative motivation (higher scores and university entry).

This conclusion is substantiated by the findings on further motivation-related questions. 50% of respondents had taken some other standardised English language test, 36% of whom had taken the TOEFL. Asked why they had taken another or an additional test (Q.9), the now familiar grouping of summative/formative motivations dominated: “to satisfy the entry requirements…” (35%), “to improve my English language skills overall” (27%), and “to get a better English language test score” (17%). IELTS-related motivations had little impact. The item “the IELTS is too hard - you can’t get a good score” only gained 6% of overall responses and was ranked number one by just four respondents. “Taking the IELTS test is too expensive” gained a similarly low volume of responses. From this we may infer that student motivation for participation in other English language tests is not as a negative response to, or dismissal of, IELTS (ie. too hard; too expensive). Rather, participation in the TOEFL or other tests reflected the high stakes of English language testing in general - that is, students must often explore every means to gain the required entry score.

The interview data raised two further themes, that of availability and familiarity, and may reflect market penetration. Examples of such reasoning by students are given below.

I just found the IELTS test recently, because before I didn’t know about it. I just knew the TOEFL. (student, AUS)

I think in China most students are familiar with the TOEFL examination, because the TOEFL has a long history. Most people are familiar with this test, so it is popular. The IELTS, people are still not confident with this test. They don’t know what skills you should have to take part in the examination. (student, AUS)

I took the TOEFL because IELTS was rarely held then. (student, AUS)

Of the respondents who had taken the IELTS test more than once (Q.11; n=120; 28%), the most important reason for doing so was to satisfy entry requirements (22% of total responses), to get a better IELTS score (16%) and to improve English language skills overall (10%). Percentage values for this question are of course lower, as the majority of respondents had not taken repeated IELTS tests. For this group, the IELTS test itself was not considered to be a good learning tool: only 4% of respondents indicated that this was a rationale for repeat participation.

Further formative/summative questions were posed to determine the relative weighting of these motivational constructs. Three inter-related questions were asked. On a five-point Likert Scale, students were asked to respond to the statement: “my IELTS score was an important factor in determining which faculty/university I could get into” (Q.29). Nearly two-thirds of respondents (63%) ‘strongly agreed’ or ‘agreed’ that this was the case. However, only 22% of respondents selected ‘strongly agree’/‘agree’ for the statement “if I had got a higher IELTS score, I would have enrolled in a different degree” (Q. 32), with 55% ticking ‘disagree’/‘strongly disagree’. One further set of responses helps to shed light on these results. Students were asked: “the main purpose of the IELTS is technical: if you want to go to university, it is just one of the things you have to do” (Q. 44). 60% of students ticked ‘strongly agree’/‘agree’.

The results to these three questions suggest that students have a sophisticated understanding of the administrative gate-keeping function of the IELTS test. Students recognise that an IELTS test score can play a major part in their application (Q. 29 and Q. 44).

As represented by the responses to Q. 32, however, this technical role of the IELTS score is separated from their own experience. That is, nearly 80% of students did not feel that they were blocked from pursuing a chosen degree stream as a consequence of their IELTS score. Put another way, the majority of student respondents did not perceive that they had been ‘punished’ by this high stakes test. While an English language test score is seen by respondents as simply part of admissions, the majority of respondents did not believe that their score acted as an entry barrier to their preferred field. Recall that the respondents do represent ‘successful’ IELTS post-participants: these are the responses of individuals who successfully used an IELTS score in partial fulfilment of a university’s admission requirements.

Finally, two further student-specific questions were asked. On a five-point Likert Scale, students were asked: “preparing for the IELTS test helped me improve my English language skills” (Q. 43). Nearly three-quarters of respondents (74%) answered in the affirmative. As respondents were enrolled university students, a question was asked in relation to ongoing language skill improvements: “my English language ability has improved in the time since I took the IELTS test” (Q. 27). As may be expected (or hoped), given the English medium of instruction (each of the research sites) and external language environment (for the Australian and UK sites), 59% responded in the ‘strongly agree’/‘agree’ range.

Overall, the responses to the student-specific questions indicate a deep understanding of the IELTS test and how scores are used by administrative bodies. The pattern of responses between survey questions indicates good internal consistency in the test instrument. Of particular interest is the combination of

formative and summative motivations in relation to test participatory and test preparatory motivations. These findings suggest that students view the IELTS test as both a means to an end, and an end in itself.

4.2 Perceptual/attitudinal areas

Questions grouped under this category relate to perceptions of the IELTS test, its purpose, rationale for participation, its reputation, and the validity and the reliability of the test. It must be emphasised that these are the perceptual and attitudinal positions of the respondents or, in other words, their ‘informed’ opinions based on experience. This section includes both staff and student data.

On a five-point Likert Scale, staff and students were asked to rate their reaction to the statement “in my experience, the overall IELTS score accurately measures English language ability” (staff Q. 9, Appendix 1.2; student Q. 18, Appendix 1.1). Less than half (43.1%) of staff ‘strongly agreed’/‘agreed’ with this statement as compared with three-fifths (58.9%) of students. This difference is sufficiently large to achieve extreme significance (Chi-square, p<0.0005). Roughly equal numbers of staff and student respondents were ‘not sure’ (33.3% and 26.7%, respectively). Nearly a quarter (24%) of staff and 14% of students disagreed with the statement.

Respondents were asked to rate their reaction to the statement “in my experience, all students who get the same overall IELTS score have the same English language ability” (staff Q. 10; student Q. 19). This question is concerned with perceptions of the reliability of the overall IELTS test score, or the consistency of English language ability as reflected by an IELTS score. Responses to this question were remarkably similar between the two groups: 13.7% of staff and 16.5% of students selected ‘strongly agree’/‘agree’; 39.6% and 31.2% selected ‘not sure’, and; 46.7% and 48.7% selected ‘disagree’/‘strongly disagree’. The degree of similarity between the staff and student groups meant that no significant difference was found between the groups. With nearly half of all respondents disagreeing with the statement (n=286), one may suggest that both staff and students are apparently unconvinced of the reliability of overall IELTS scores as standard measures of generalised English language ability.

It should be noted that respondents were only asked about candidates with the same overall IELTS score. However, as an overall score is an average of the four sub-scores, two test-takers who gain the same overall score may have achieved very different sub-scores. For instance, a candidate who achieved a speaking band 4.0 and an overall band 6.0 may appear to have quite different abilities to a candidate who achieved a speaking band 7.0 and an overall band 6.0. To account for potential variability in the composition of the overall score, many IELTS test users stipulate a minimum overall band score as well as minimum requirements in each of the four sub-skills.

Student interviewees gave detailed appraisals of the testing system, identifying specific factors that they felt affected their test performance. The speaking sub-test, perhaps in part due to the interpersonal nature of this testing, was separated out for special attention by a number of interviewees. The following are indicative comments.

I think the IELTS sometimes measures our ability in English, but sometimes it isn’t so right. I took the test twice, and the first time on the speaking test I think I had a good examiner. He tried to make me feel very comfortable. But the second time I didn’t like the examiner. She acted like she was going to scare me. And it wasn’t only me, I think almost all the candidates said she was not good. (student, AUS)

I hated my examiner last time [laughs]. Actually I met him when I took the test the second time, and at that time he marked my speaking score and then my score dropped from 6 to 5. And on the third time, he was an examiner again, and this time he marked my writing score, and my writing dropped from 6 to 5. And it didn’t just happen to me, it also happened to my friends. For some people, they got 7 for speaking, but when they took the test with this examiner their score dropped from 7 to 6, or from 6 to 5. He didn’t give more than 6 to anybody. (student, AUS)

The reliability and the validity of the IELTS oral interview has been the subject of recent research, including examinations of the impact of gender (O’Loughlin 2000) and of rater orientation in the awarding of scores (Brown 2000). The interview data collected in this study suggests that students perceive the oral interview as the area of greatest subjectivity in the assignation of scores.

Student interviewees commented more generally on the perceived validity of their scores. The following comments reflect a range of experiences and observations:

I think there are differences in the different sessions of the exam, and I think it might affect some students’ scores. For example, some exam sessions may be more difficult than other ones. For me the reading part [of the IELTS exam] was more difficult than the listening part, and the listening part was simpler than the listening parts that I saw in the handouts I got from the Australian Embassy. I think this may lead to a lot of differences between the scores in different sessions. (student, AUS)

I think that if the score is more than 6.0, it is easier to understand. If it is lower than 6, I don’t know, it depends on the centre where they took the test, the person - it’s difficult to be sure of their ability or knowledge. But if it’s more than 6, I can understand. I can recognise the people who get 6, 6.5, or 7. (student, AUS)

I know that my English is good, and that is what IELTS confirmed for the university. Therefore it is reliable. But some other students on my program have 6.5, but I think maybe my English is better. But I was so nervous at the examination, because I thought I would not do it properly. (student, UK)

It is very hard to say that a person who gets 7 and a person who gets 6 are different. I have a friend who got 7.5 but actually I don’t think he’s better than me, and I got 6.5. I mean, some people who get 6.5 are better than people who get 7 or 7.5. I think it’s because they are good in a test but not good in English generally. (student, AUS)

I got 7 for listening, but when I came here everyone [sounded like] ‘do-do-do-do’ - and I couldn’t understand. I don’t try to tell everyone ‘oh, I got 7 in listening’ because my listening is so bad. (student, AUS)

I know my level of English, and I think the IELTS mark I have is a true measure of it. Therefore, I believe it is reliable. (student, UK)

If I score a 7, that doesn’t mean my English is at that level. Myself, I’m 7, and another student he’s 7, but maybe our levels are different. I can see a difference between the speaking levels of someone who has 6 and someone who has 7. But there can be differences between students with the same score. (student, AUS)

Of interest is the following comment, for while the student perceives the experience to indicate unreliability, it may in fact reflect the test’s reliability.

I got 7 on reading the first time, and I really didn’t understand anything about the test. I just scanned to the right passages, saw the same word and then I ticked the answer. And I got 7. The second time I took the test, I understood the text quite well, and I got 7. The same [laughs]. (student, AUS)

On the basis of content knowledge, the student apparently believes that she/he should have had different scores on the reading sub-test across the two attempts. However, this experience may in fact reflect good validity and reliability in IELTS testing. As a test of language ability, rather than content knowledge, a like score over two attempts suggests that the IELTS did effectively measure the candidate’s actual reading language proficiency.

Staff are often in the position of interpreting IELTS scores for the purpose of university entry, and to determine eligibility for advanced standing, inter alia. Indeed, 41% of staff respondents stated that they make admissions decisions (Q. 5), and 27% stated that in making these admissions decisions “…an English language test score - such as the IELTS - forms some part of the decision-making process” (Q. 6). With over a quarter of university staff, as represented in the survey respondent figures, making decisions in which an English language test score features, it was useful that the staff questionnaire included two further questions regarding reliability and validity, and the function of the IELTS test score. Staff were asked to rate their responses to the statement “when I see a student’s IELTS score, I have a very clear idea of their English language proficiency” (Q. 22). Responses were somewhat more affirmative than responses to Q. 10 outlined above: 42% of staff selected ‘strongly agree’/‘agree’; 24% were ‘not sure’, and; 34% selected ‘disagree’/‘strongly disagree’. This set of responses suggests that although staff may be reluctant to generalise scores across test-takers, there is greater confidence (42% positive responses to Q. 22 versus 13% positive to Q. 10) in their own ability to predict a student’s English language ability on the basis of an IELTS score. In other words, the results to Q. 10 and Q. 22 indicate that staff have greater confidence in the validity of the IELTS test, than they do in its reliability.

Below are some illustrative comments.

I think that in any test of this type, given the variety of individual learning rates, maturity and motivation, the difference has to be looked at very carefully…Probably some at 5.0 coming here would probably do as well as somebody at 6 or 7, or maybe even better. We don’t make that judgement, because that is an administrative judgement. (staff, UK)

I think [IELTS] is reliable in that it is run internationally and they have quite high standards of checking and comparing and so on. I think on the whole it is quite reliable. Although then on the other hand, you do hear anecdotes. There is some room for variation. (staff, AUS)

I suppose if you look at ten people with an IELTS of 6.0, they have all got pretty much the same level. I mean, their skills may be different, they may be better at speaking or writing, so they do level out. I think in that respect it is a reliable test. (staff, UK)

I especially notice the differences in students who come in on 6.5. You get somebody coming in with 6.5 and 6 in each band who is actually very proficient in English and does quite well. And then you would get another student who would come in with exactly the same and I would barely understand what they were trying to say to me. I don’t know how this happened. (staff, AUS)

A number of staff interviewees were well aware of the distributed nature of international student admissions. This administrative staff member (below) described the admissions procedure.

With the undergraduate program, the [admissions] team has the authority to make decisions on whether or not somebody is eligible for entry. All paperwork stays within the office. The postgraduate program for academic entry requirements is different. The file then goes to the faculties. They make the decisions on whether someone is eligible or not…[Faculties] advise on whether they are willing to waive the English requirement. (staff, AUS)

Several gave advice on possible quality improvements.

Most course directors haven’t a clue [about IELTS], and that’s who should be looking at the scores, not the admissions officers, who don’t really have any idea of the subject area. I think it would be really useful if course directors could have presentations on what IELTS really means. For example, what 6.0 in writing and what 4.0 in speaking means. (staff, UK)

As a lecturer, I guess I would like to know, as the students come through the door, what their IELTS score is. As early as possible, as it might acknowledge possible problems. (staff, AUS)

To gain a further sense of staff understandings of the IELTS system, staff respondents were presented with a factual/perceptual statement: “a student’s IELTS score tells you a lot about their overall intelligence”

(Q. 23). Happily, only 8% of respondents answered in the affirmative. Nearly two-thirds (63.6%) ‘disagreed’/‘strongly disagreed’ to the statement and 28% were ‘not sure’. Therefore, the majority of staff respondents were aware that the IELTS does not function as an intelligence aptitude test.

Staff and student respondents were asked several further questions relating to perceptions of IELTS in an organisational or reputational sense. The statement “the IELTS test has a good reputation” was posed (staff Q. 18; student Q. 28 ⁴ ). The same number of staff and students ‘strongly agreed’ (n=33, both groups), while a total of 52% of staff and 50% of students responding in the affirmative. Close to 40% of both groups were ‘not sure’, with just under 10% giving negative responses.

Less similarity was found in staff and student responses to “the IELTS test gives you/a student the best chance to get into university in an English-speaking country” (staff Q. 19; student Q. 30). While 35% of staff gave affirmative responses, 63% of students indicated ‘strongly agree’/‘agree’. And although 26% of students were ‘not sure’ how to respond, 55% of staff were ‘not sure’. This inter-group difference was highly significant (Chi-square, p<0.0005). Significant inter-group differences were again found for the interpretive statement “the IELTS is the best English testing system available” (staff Q. 20; student Q. 31). The main points of difference relate to affirmative responses (27% of staff ‘strongly agree’/‘agree’; 35% of students), and non-committal responses (65% of staff; 53% of students).

Finally, respondents were asked to rate their attitudes to the statement “an IELTS score is an internationally-recognised qualification” (staff Q. 21; student Q. 33). Nearly a quarter (22%) of staff ‘strongly agreed’ compared with 11% of students. Students on the whole gave more affirmative responses, however, with the sum of ‘strongly agree’/‘agree’ replies equalling 71%, as compared to the staff’s 64%. 22% of students and 31% of staff were ‘not sure’. Extreme inter-group significance was again reached (Chi-square, p<0.0005).

Several staff interview respondents specifically noted the international recognisability of IELTS ⁵. For instance:

[IELTS] provides some sort of standard that is nationally and internationally recognised. It provides a general standard for assessing people to some extent of what background they bring that may or may not make them capable of coping with university demands. (staff, AUS)

We see it as an internationally recognised test for university entrance. Not necessarily that they are ready to run just because they have got IELTS, but it is a minimum starting point depending on the score that has been set. (staff, UK)

I believe the IELTS is possibly the best at the moment. That’s my perception, compared to the others like TOEFL and so on. (staff, AUS)

Students, too, had an awareness of national and/or international recognition:

I know that IELTS has an authority that comes from the Government. (student, UK)

I knew that the TOEFL was accepted worldwide. But when I had some contact with some Canadian universities, for example, I asked them if they accepted the IELTS test - is it a valid test or not? And they told me yes, we accept it. (student, AUS)

Most European countries use this system, so I think it is reliable. (student, UK)

I feel it has been used for many years and also it is a measurement of student skills and it is internationally recognised, except in the USA where most of the universities want to use TOEFL. (student, UK)

I trust Cambridge. It is a very famous university name. (student, UK)

In summary, the distribution of staff and student responses to questions of reputation indicate large points of difference, mostly in the respondents’ degree of certainty. As shown in Table 6, this may be largely attributed to ‘not sure’ responses, with the percentage of staff non-committal replies larger than student percentage in three of the four reputation questions asked.

Table 6: The reputation of IELTS: ‘Not sure’ responses by staff and students

The majority of staff perceive that IELTS is internationally recognised and that IELTS has a good reputation. The majority of students, too, affirmed the international recognition of IELTS and attributed to it a good reputation. However, the majority of staff were uncertain (ie. ‘not sure’ responses) regarding the ‘functional’ reputation of IELTS - that is, whether an IELTS test score afforded students the best chance of gaining entry to an English-medium university or whether IELTS represents the best English testing system available. Students were far more willing to comment on the functional reputation, with the majority nominating an IELTS test score as the best means through which to fulfil the English language component of their admissions. Students, though, were less sure (yet more sure than staff) about asserting that IELTS is the best English testing system available.

4.3 Beliefs regarding the predictive nature of IELTS vis-à-vis university demands

As the IELTS test consists of four sub-sets, questions were posed regarding the IELTS test score as a whole, and in relation to reading, writing, speaking and listening. In the current section, questions related to student preparedness for the university environment are discussed. In the section following, questions related to the appropriateness of the university’s IELTS cut-off score are discussed.

While these themes are clearly related, there is an important distinction. On the latter, staff and students were asked to comment on administrative decision-making - that is, how well the IELTS cut-off scores for university admission, determined by the university’s themselves, reflect the English language demands in university study. On the former (discussed in this section), respondents were asked to comment on how the IELTS system itself prepares participants for university study. While we are aware that IELTS is not specifically designed to prepare participants for the English language demands of tertiary study, it is nonetheless an indicator of preparedness, and is most certainly interpreted as such. And although not all

IELTS test-takers choose to use IELTS training materials, many do. Again, we are aware that these training materials are designed to aid prospective test-takers for the test itself, and not necessarily for future study. However, as the questionnaire and interview data suggest, stakeholders do perceive a link between IELTS training materials and English language environments beyond the test itself.

Staff and students were asked a variety of questions, each commencing with the statement “the IELTS system prepares [you/students] well for…”. For instance, staff were asked “the IELTS system prepares students well for academic uses of English, such as reading for university subjects” (Q. 12, Appendix 1.2). The equivalent question for students was “the IELTS system prepares you well for academic uses of English, such as reading for university subjects” (Q. 21, Appendix 1.1). Shown graphically in Table 7, staff and students were asked to rate their responses on a five-point Likert Scale in relation to reading, writing, listening and speaking (staff Q. 12-17; students Q. 21-26). It was the relationship of the IELTS to applied usages of discrete language skills that was being queried, and one will note that respondents were asked to reflect upon their own experience.

In addition, student respondents were asked to comment on informal English skill development (“…such as having conversations in English outside the classroom environment”) and abilities in mixed skill environments (“…in which you have to shift between reading, writing, listening and speaking skills (such as group work and laboratory practicals)”).

It is possible to discuss these six questions simultaneously, for in each case the difference between the staff and student groups evinced extreme difference (Chi-square, p<0.0005). The nature of this difference between staff and student responses is also somewhat consistent across the six variables.

Table 7: “The IELTS system prepares participants well.” Various English language skill sets

On the whole, students gave positive responses regarding the preparatory nature of the IELTS system, its applicability to university study, and for life in an English language environment. On each of these variables, more than 50% of students selected ‘strongly agree’/‘agree’. In comparison, 35% or fewer of staff respondents ticked ‘agree’, with a negligible number (n=2) selecting ‘strongly agree’. Furthermore, staff were more negative on the whole, with ‘disagree’/‘strongly disagree’ responses ranging from 16% (informal English) to in excess of 34% for the writing and speaking sub-sets. Writing and speaking were similarly the sub-sets which drew the greatest proportion of negative responses from students, accounting for 19% and 25% of responses, respectively. Staff were, on the whole, less certain about the preparatory nature of

IELTS, with 35-53% of staff respondents ticking ‘not sure’ on the various English skill sub-sets. Students were relatively more certain, with between 20-34% of students selecting ‘not sure’ on this set of variables.

This difference in the degree of certainty is not surprising: as students have taken the test and staff have not (only 4% of staff respondents indicated that they had taken an IELTS test themselves; Q. 48), one would expect former test participants to be more willing to offer opinions regarding the test’s preparatory nature or its suitability for English language demands in a university setting. Staff, too, may be entirely unaware of the actual English language test students take. Not only does this lack of sure knowledge account for the high rate of uncertainty, it suggests caution in the interpretation of positive or negative staff responses. Staff may, for instance, be reflecting on standardised English language preparation and/or testing as a whole, rather than thinking specifically of the IELTS. Much greater certainty is possible with the student group, as respondents had at least participated in the IELTS test, if not necessarily informal/formal IELTS preparation.

Overall, these findings give some insight into whether the IELTS test and system is interpreted as having a formative or summative function. The more positive student responses on the preparatory nature of IELTS suggest that students do perceive a formative function. Student responses indicate that participation in the IELTS system, in whatever way, does have direct preparatory benefit for life in an English medium university.

As shown in the examples below, interview respondents noted the preparatory benefits of IELTS participation, across a range of English language skills.

When I studied English at first, I hadn’t studied writing. But when I had to take the IELTS test I had to study writing. I think it was good for me - so with the writing background from IELTS I can write critical reviews now. So it’s very useful. (student, AUS)

Yes, I learned. Particularly for me in the speaking. At the start I was shy and did not like to speak, but our teachers made us confident. They told us we can make mistakes, and they did not stop us when we talked. Only afterwards they told us about them…I could read better too, but not quickly enough. (student, UK)

Especially for writing, if you do well it means you will do well in your studies. (student, AUS)

The preparation classes show you how to read large difficult texts without a dictionary. This was a great problem for me, as our [home country] university teachers taught us to use a dictionary and translate words … IELTS teaches you how to read for the essential information you need. You don’t have to spend a lot of time trying to understand the parts you don’t need. This is a valuable skill, and it has helped me to do a lot of reading on this course…I have no problem taking notes as I listen. I learned this skill in IELTS training. (student, UK)

The following student comment, while not necessarily related to the preparatory nature of IELTS, gives some insight into the nature of the test itself, and its applicability to authentic language settings. Referring specifically to the Speaking sub-test, this comment is representative of comments made by several student interviewees regarding the speaking test:

In speaking, the style of the speaking test has changed … The examiner will ask you a question but he will ask just only once. But in my opinion, I think if the purpose of the IELTS test is to show the use of English in daily life, then I don’t know why in the test we can’t ask the examiner to repeat the question. I think in real life you can ask people to elaborate, to explain more. But in the new speaking test, we can’t. You are just given a [prompt] card, and you just have to talk, just talk talk talk like we are talking to ourselves. It’s not like we participate with each other, something like this. It is quite foolish. But the old style was ok. We were given a card, and we could ask questions regarding the content of the card. I think that’s better. (student, AUS)

Note: The student is referring to Part 2 of the Speaking test, which is an unassisted short talk. Parts 1 and 3 involve question-and-answer interaction between examiner and candidate.

The greater proportion of negative responses amongst staff survey respondents suggests that staff are less willing to associate a formative function to IELTS. While these staff responses do not necessarily indicate a belief in a summative function in and of themselves, a relationship may be drawn to other test items. Recall that in Q. 7 (“What is the purpose of the IELTS test?”), three of the top four staff responses related to summative rationale: to gain a standardised English language qualification; to assist entry to a faculty/university of choice; to achieve entry to a faculty/university of choice. These findings suggest that students perceive a mixed formative/summative function for the IELTS, while staff perceive more of a summative function. These findings are corroborated by the staff interview data (examples below).

There is a difference between the [IELTS] levels and what they [students] can do. The nature of essay writing, or listening to a long lecture or discussing a tutorial presentation at university are really very different tasks in their complexity and so on from what the [IELTS] exam demands. So they are not prepared for that, I don’t think. (staff, AUS)

The value of IELTS as a learning tool dissipates where it is situated in a market assessment context. As it does with A-levels or whatever you have in any country, SAT tests in America, you know, the content isn’t any longer the issue. It is the benchmark that is the issue. (staff, UK)

4.4 Beliefs regarding the appropriateness of entry levels

The set of questions discussed in this section are thematically related to those examined above. In this case, staff and student respondents were asked to reflect upon IELTS entry scores at their institutions, and to comment upon the accuracy of the cut-off scores as a measure of the English abilities necessary to succeed in a university environment. A Likert Scale was used, with respondents asked to rate the relationship, in their view, of IELTS cut-off scores and reading, writing, speaking and listening abilities necessary to succeed, as well as an overall assessment. For example, the question text for reading proficiency was “in my experience, the IELTS entry score required by the university is an accurate measure of the English reading abilities necessary to succeed at university” (staff Q. 24-28, Appendix 1.2; student Q. 34-38, Appendix 1.1).

It is again possible to discuss the various English language sub-sets simultaneously, as there was a certain consistency within and between the staff and student respondents groups across the five variables. Extreme statistical significance between the two groups was found on each sub-set and in relation to overall English proficiency (Chi-square, p<0.0005). The findings are summarised in Table 8 (next page).

Table 8: “The IELTS entry score required by the university is an accurate measure of English language abilities necessary to succeed at university.” Various English language skill sets

In common with the previous section, students gave, on the whole, more positive responses than staff. On each of the sub-sets (except speaking), more than 50% of student respondents ticked ‘strongly agree’/‘agree’. That finding indicates that the majority of students interpret the institutional IELTS cut-off score is an appropriate indicator of the level of English language they face at university. In other words, the majority of students believe that the English language ability encapsulated in their institution’s minimum IELTS score is an accurate appraisal of the English language proficiency necessary to succeed in university study. Other results suggest that these responses were based on knowledge of the actual cut-off scores used by their institutions. Asked to indicate faculty-based IELTS cut-off scores, 74.7% of student respondents put this correctly in the IELTS 5.5-6.5 range (student Q. 4).

There were institutional differences. In relation to each of the IELTS sub-sets and overall English language proficiency, a rank-order comparison between the three student groups found that the Guangzhou student respondents were most positive (in excess of 63% positive responses on Q. 34-38), the Australian university students were less positive (between 42% positive responses on speaking and 57% on listening) and the UK university students were least positive (49% on reading and writing, and 57% on speaking and listening) ⁶. The greatest difference existed in the speaking sub-set. Guangzhou respondents had 63% positive responses, the UK respondents gave 57% and the Australian respondents gave 42% positive responses on the accuracy of their institution’s IELTS entry score as a reflection of the speaking ability necessary to succeed in the university environment. On each of the sub-sets and for the overall query, the Kruskal-Wallis $H$ -test (rank sums of k independent samples) achieved significance to at least the 0.05 level, indicating that for this set of questions, the student respondents should be treated as three separate groups, rather than as a single cohort. It is worth reflecting that the Guangzhou campus has the lowest IELTS entry score, requiring 5.0 for first year entry, and 6.0 to advance to second year. The UK institution requires a minimum IELTS of 6.0. During the research period, entry to the Australian institution varied from IELTS 6.0 to 7.0, dependent on faculty or degree stream.

Some interviewees gave considered appraisals of the relationship between an IELTS score and the abilities necessary in tertiary study. This student broke it down into constituent skills:

The [IELTS] score may be important but, for example, in reading, when we take the test we have to use our skimming and scanning skills, but in real life, in my studies I can’t use these skills. Because I have to understand exactly what the content is, and if I use skimming and scanning, I can’t understand. But for listening, it’s ok. Writing and speaking too, yeah, apart from reading I can apply it to my studies. (student, AUS)

Staff responses were, on the whole, less positive. Summing ‘disagree’/‘strongly disagree’ responses, between 27% (reading ability) and 38% (writing ability; and 37% overall English) of staff indicated that they did not perceive the IELTS cut-off score used by their institution was an accurate measure of the English language ability to succeed in a tertiary environment. As seen in Table 8, staff were most critical of student writing as assessed by the IELTS cut-off score, with only 21% responding in the affirmative.

Staff respondents at the Australian institution were on the whole less positive than staff respondents at the UK institution. A Chi-square analysis of staff responses per institution found significant differences (in excess of p<0.005, each variable), indicating that staff should not be treated as a single group for these five questions, as the distribution of responses is sufficiently different to warrant separate analysis. This is an important finding for two reasons. Firstly, during the research period (2001) IELTS cut-off scores at the Australian institution varied by faculty, with several faculties requiring minimum sub-set and overall scores of 6.5 and 7.0. At the UK institution, the university’s official minimum was 6.0 overall. Thus, with higher average minimums, staff at the Australian institution were nonetheless less affirmative and more negative about the sufficiency of the IELTS entry score.

Staff survey results run counter to the student findings: student respondents at the Australian institution were more positive about the accuracy of IELTS cut-off scores and the demands of university study than their UK-based counterparts. One way of interpreting this finding is that students at the Australian institution perceive the higher entry score - sometimes to an IELTS 7.0 - as a more realistic reflection of the English language ability necessary to succeed. The UK students, on the other hand, perceive a gap between the IELTS minimum (6.0) and the somewhat higher demands in university study. For their part, Australian staff - with a much higher proportion of NESB students in their classes than UK staff - still perceive an English language proficiency shortfall, even with the higher IELTS entry cut-off score. It should be remembered that staff may not be responding to the IELTS per se, but rather the English language levels that they perceive, either in their academic or administrative relations with NESB students. The finding, however, remains unchanged. Whether or not staff responses related to the IELTS entry score specifically or institutional English language benchmarks in general, the point is unequivocal: one-third or fewer of staff at the Australian and UK university believed that the English language cut-off value required by their institutions was sufficiently high enough to assure success in university study.

It is interesting, however, that the Guangzhou-based students - requiring the lowest IELTS entry score of the three student groups - were most positive about the appropriateness of the IELTS entry score as a reflection of the English language ability necessary for successful study. This may be due, in part, to the nature of the off-shore campus. A number of the lecturers at the Guangzhou campus are locally engaged, seconded from positions at Chinese institutions, and as such may share a common primary language with the students. As other research has shown, local staff at off-shore campuses may use a shared language other than English to explain complex topics or concepts if they perceive students are having difficulty in the medium of English (Coleman 2000). These students may, therefore, be less reliant on their English language proficiency in their university studies than their counterparts in UK- or Australia-based institutions.

These staff members, who work in a language support capacity, believe that staff may mis-attribute student difficulties to the IELTS, rather than the institution’s own entry standards:

They [staff] only know the scores from the fact that it is in the prospectus, and they know that on quality assurance procedures, they can’t accept students under those certain scores. They, like any person

with common sense, will assume that the score therefore sets the standard at which you can write essays, do seminars, do debates in a university and that … is not necessarily the case. (staff, UK)

[Staff] expectations are in excess of the level set for entrance to the university, so their expectation is basically normalised English language in an academic environment. So the fact that students do have difficulty with essays, seminars, debating, et cetera is an assumption that the IELTS is faulty and that the training is faulty. In fact, probably neither of those are true. It has to do with matching expectations to the level at which [students] come in. (staff, UK)

Rather than focusing on the minimum score, this staff interview respondent saw university minimums as simply a starting point:

I would like to think that a student at 6.5 would not sit back and stick at that. I would rather see 6.5 as a base from which to work. Again 6.5 should be seen as a starting point by teachers and students. (staff, UK)

These staff members compared the skills measured by the IELTS test, and the skills necessary to succeed in the university environment:

Writing is one of their biggest problems. Even for many students with 6.5, they are still not up to scratch…They have to write assignments of 3000 words - extended writing is not catered for in the test…Speaking skills are still commonly weak, even if they have a good IELTS score. They’re not really prepared for all the different areas, like lengthy presentations and seminars. They need a lot more practice. IELTS preparation seems to be skimpy, and focuses on areas they need to pass the test. (staff, UK)

I would place [an overall IELTS score of 6.0] at upper intermediate level, but I certainly wouldn’t think that they would be able to just fly on their own at university. I think they would probably need some help as well…The problem is that when you see the global score you don’t know what the individual scores are inside that, so perhaps somebody is high on speaking but maybe less on writing. I think really I would be happier to look at individual scores rather than an overall 6.0. (staff, UK)

Amongst student interviewees, too, there was a perception that at the higher IELTS score levels, there was greater reliability than at lower levels:

I can see that if a person gets 6 and over, they must try hard, so I think it’s quite trustworthy. But below 6, I’m not sure if it’s reliable or not. (student, AUS)

I don’t know why, but it seems very difficult to jump from 6 to 6.5. I studied very hard for a year and I hoped to get this mark, but I couldn’t. (student, UK)

If you are clever enough or do enough practice, you can get 5.5. It’s not very hard to get 5.5. But more than 5.5 - for example, getting a score of 4 is very easy, you don’t have to do anything, just go and write the test. But 5.5 is a good level. After that, if you want to reach 6 you have to do a lot of work, much more than reaching 4.5 or something. Getting from 6.5 to 7 is really more difficult. (student, AUS)

As indicated, the IELTS entry score at the Australian institution will be standardised in the 2002 academic year to a minimum of 6.0 on each of the sub-sets. For a number of faculties, this represents a lowering of the overall IELTS cut-off score. These findings from 2001 data suggest that staff interpreted cut-off levels as already too low, while Australia-based students were relatively more positive than their lower cut-off score UK-based counterparts. Further unsupported supposition on this topic is unnecessary, as staff and students were specifically asked whether their institution’s IELTS entry score should be varied, and if so, in which direction.

Staff and students were asked if the minimum IELTS entry score should be changed - no direction was stated, with only normative ‘yes’/‘no’/‘don’t know’ options available (staff Q. 32; student Q. 39). Staff responses exhibited no clear preference, with responses almost evenly split three ways. Student responses were similarly divided. Only 18% of students indicated that IELTS cut-off scores should be changed, versus 34% of staff. ‘Yes’ respondents were asked to indicate their preferred cut-off value. Nearly 40% of the staff who answered this question indicated that the minimum IELTS score necessary for university entry should be 6.5 or above. In comparison, 30% of students indicated that the minimum IELTS should be 6.5 or above. However, 40% of students believe that the university’s IELTS minimum should be between 5.0-6.0. In comparison, only 9% of staff indicated that the IELTS minimum should be in the 5.0-6.0 range. 50% of staff ticked ‘don’t know’ for this question of preferred entry standards.

On a five-point Likert Scale, staff and students were asked whether postgraduate and undergraduate IELTS entry scores should be higher (staff Q. 34-35; student Q. 41-42). Amongst staff respondents, 44% believe postgraduate and 36% believe undergraduate cut-off scores should be higher (sum of ‘strongly agree’/‘agree’ responses). A lesser proportion of students believe that postgraduate (30%) and undergraduate (20%) IELTS minimum scores should be raised. This finding is consistent with those discussed above. For these questions, staff uncertainty is lower than student uncertainty (approximately 40% ‘not sure’ staff responses versus roughly 72% ‘not sure’ student responses). Across the entire data set, this result of greater staff certainty is a rarity. The point here is that although staff respondents showed high levels of uncertainty regarding many aspects of the IELTS system, the topic of greatest surety was entry standards.

The comments of student interviewees provide further insight into the question of entry score levels for undergraduate and postgraduates. Several respondents believed that English language entry scores should in fact be lower for postgraduate research students than for undergraduate coursework students:

I think 6.0 should be reasonable enough [as an entry score]. For research I think it needs to be lower…because I think for research you don’t do much communicating, speaking. I think the most part is reading and writing, especially for the thesis report, and you do lesser on speaking. Except when you finish your thesis report, and you have to do a presentation. (student, UK)

I think in Bachelor of Science students, 18 to 20 year olds, it is necessary to have a good ability [in English], because they have a lot of subjects, everyday they have to go around. But research students…maybe this is a very strong student, and they [the university] need this student. Everywhere in the street you can find many people in this country who can speak English fluently, but they don’t have a knowledge of, for example, physics or petroleum … They need me for my knowledge. I was speaking with my supervisor, and they are experts because they have a lot of overseas students. They prefer to just check my listening and speaking, and then bring me over. (student, AUS)

I don’t understand why the levels are the same for all students. When a student is a professional, a postgraduate, and has a lot of professional knowledge, I think maybe they don’t need such high English. It is different for undergraduates, because they are so young and don’t have world knowledge. Then they need high English to manage, because they cannot apply experience to their study … I am familiar with databases and I do not need to learn how to use a library and things like that. That is what I mean by experience, to apply it to your study. (student, UK)

But this viewpoint on undergraduate and postgraduate scores was not the consensus. This interviewee, for example, believed the reverse should be the case:

Maybe for an undergraduate program 5.0 is ok because it is taught. But the postgraduate students have to be independent and do research, so they need to have good English. I don’t mean just language, but be able to research and work with a supervisor, and make reports, write papers and present papers to conferences. They also have to publish papers. This is impossible for a person with 5.0. I know, because I got 5.0 the first time. (student, UK)

Staff also made comments regarding their preferred entry minimums:

Certainly in the past, we have had discussions where we would like to increase it to 7.0, purely because students arrive and they have a very good educational experience…I am very concerned now that these [admission] scores have dropped to 6.0. (staff, AUS)

If I was totally honest, I would love to see it higher than 5.5 or 6. I think that students have a difficult enough time coping with English as a second language. I think if the scores were higher I know the students would have a happier time. Some of them would have a life here, not just a life in the library … I think it should be 7 for postgraduates. (staff, UK)

Finally, respondents were asked to reflect upon the means through which they believe institutional IELTS entry scores are established. Given the various questions in relation to the appropriateness of entry scores as an indicator of a student’s linguistic readiness for tertiary study, it was useful to ascertain perceptions of the methodology through which universities set their lower boundaries.

Participants were asked, firstly, to rate their responses to the statement “in my opinion, the university determines its IELTS entry levels based on careful consideration of the appropriate English language ability necessary for university study” (staff Q. 31a; students Q. 45a). Shown in Table 9, two-thirds of students ‘strongly agree’/‘agree’ that a careful consideration of the appropriate English language ability necessary for university study explained the setting of IELTS entry scores. Two-fifths of staff believed this to be the case. Participants were asked, secondly, to rate their responses to the statement “in my opinion, the university determines its IELTS entry levels based on a wish to match the IELTS entry levels of other universities” (staff Q. 31b; student Q. 45b). Positive responses were cut nearly in half for both groups, with 37% of students and 22% of staff believing that inter-university competition was a prime motivator in the setting of IELTS cut-off minimums. Uncertainty levels (‘not sure’ responses) were relatively high for both groups on both queries.

Table 9: The rationale for institutional IELTS entry levels: Language requirements versus inter-university competition

Interview respondents noted an interplay between the linguistic needs and the competitive nature of standards-setting. Staff members observed:

It mightn’t be a bad idea if the scores were raised, but there again the university is thinking commercially as well as academically. (staff, UK)

I think it [the standardisation of IELTS entry scores across the university] is because it’s been different strokes for different folks all over the university…and they want to have a uniform computer system. I just wonder whether the university also just wants to present a front of English requirements for the university, and not have the different scales for different faculties. Therefore anybody from Admissions can give a quick response but not an honest answer on what is the criteria … That would be my perception of why it’s been done. (staff, AUS)

The decision [of IELTS entry levels] is made on the basis that someone else in the international framework has decided what would be appropriate, and since there is a kind of international agreement on it, the assumption is that it has been tested for reliability and validity and factors like that. That has all been done otherwise it wouldn’t be internationally recognised. All the other universities set similar standards along that scale, that way, in quality assurance terms, you can be seen to be doing the appropriate thing with the students. It is a procedural agreement rather than a content consideration or a contextual assessment. (staff, UK)

Certainly on the business side of the university, we are a selecting university, not a recruiting university. I don’t think it would harm the standing of the university or our ability to attract international students if our scores went up. (staff, AUS)

Students, too, were aware of two-fold institutional decision-making processes. Some queried the rationale:

When I entered the university, the IELTS entry score for my faculty was 6.5. But now it has decreased to 6.0. I think for me, personally, I got 6.5 but I still have had difficulties in studying. How about the people who only get 6? Or 5.5? Sometimes I think that maybe the university is wrong to decrease the IELTS score, just to get more students, especially international students. But why aren’t they concerned with our studies, whether we can cope with our studies or not. And if we fail, we have to pay the tuition fee again. Maybe it’s not fair to students. (student, AUS)

Many student interviewees gave carefully considered responses to the setting of institutional and/or discipline-based standard setting. For example:

My entry requirement of IELTS was 6.5. And I saw that the requirement for the Bachelor of Science was 6.5, but for Engineering it’s 6.0. So, I don’t know what’s the difference between Science and Engineering, but I think that a 6 for both is ok at the Bachelor level. But for students now studying Commerce, they need more. I live with a friend and he studies Commerce, and he has to write lots of reports and assignments. I don’t have to write reports, so I think the requirement must be different between the students in different faculties. Yeah, I think for Commerce students, it should be 6.5. (student, AUS)

I think that the Foundation Studies English requirement is a bit low. Maybe it is better to adjust to 5.5, because to get an average score of 5.0 is quite easy. For the university, it depends on which faculty. Maybe the Engineering score of 6.0 is fair. Commerce students need a higher score, because Commerce students do lots of reading and writing, they have to write long essays or do more oral presentations. But Engineering students do not need to do lots of writing, they only have to write in computer programming languages. However I think the Engineering students also need to improve their English. I think the level of 6.5 for Commerce students and 7 for Medicine is the right level. (student, AUS).

I think it should be a 6.5 minimum in my course, because if you want to be a language teacher, you cannot make mistakes. If someone is doing computing science you just have to sit in front of the computer, but for my course there is a lot of reading and writing. I think the scores should depend on the course. (student, UK)

In my opinion, for Engineering they should have a lower point…I think 5.0 is enough. They are just here for the technology. In the future they will use their mind, not often use their mouth. For the culture students, for the Arts students, it should be higher, because in the future they have to use the language to demonstrate their abilities, or use language to work in their field. For example, lawyers and in education, they have to use language…I think they should have 7.0. (student, AUS)

These subtle appraisals of the English language skills needed to succeed at university give credence to Shohamy’s observations (2001) regarding the democratisation of standard-setting. The inclusion of student stakeholders in processes of admission standard reviews could be of great use, reflecting as it does the

‘lived’ experience of the minimum English language abilities necessary to succeed, and the views of participants of the English language tests themselves.

Staff interviewees also discussed the benefit, if not the necessity, of differential entry scores by field of study. Appraisals of the language intensiveness of specific fields underlay the staff comments:

Cut-off scores are satisfactory for certain programs, not all…For example, a Japanese student doing Anglo-Irish literature will find it incredibly difficult to write extended essays and do critical analysis. The same student, doing Informatics, would probably not have the same problems…I think the university needs to take account of different study areas. Courses that are linguistically demanding, like Philosophy, Literature, whatever, are asking more of the students than Informatics or Art and Design. Entry requirements should not be just 6.0 - they should specify 6.0 for Course A, and 7.0 for Course B, for example. (staff, UK)

I think Law and Medicine are extremely demanding courses so I would feel that they would be fine if they came in with at least 6.5 or 7. (staff, AUS)

Whether staff perceive that linguistic factors play a greater or lesser role in disciplinary success is of potential concern. It is particularly so for the Australian institution, which from 2002 has removed the English language entry level distinction between linguistically demanding and linguistically less demanding fields of study ⁷.

Staff, particularly from the UK institution, referred to a further issue - that of trust-building related to the British Council. Amongst the interview sample, it is of interest that no staff at the Australian institution attributed the setting of entry score levels to a supra-institutional entity.

I would have to rely on the British Council. And whoever is involved in standardising and moderating their work on a year to year basis. (staff, UK)

My own feeling is that they [universities] probably followed others, they probably followed a pattern that had been established in higher education…one might even be guided by whatever the experience of the British Council has been. I mean, have they said in any documentation that 6 is an acceptable level? I don’t know, but presumably universities have made judgements on an overall basis. But equally I think there may well be issues that require faculties and other schools to make different determinations depending on the subject content. (staff, UK)

It is administered by very well known and reputable organisations like the British Council and the University of Cambridge Local Examinations Syndicate. (staff, UK)

4.5 Perceptions of unprincipled activity

The final survey section to be discussed relates to perceptions of unprincipled usages of or activities surrounding the IELTS test. Noted earlier, one goal of this study was to provide data on the degree to which staff and students truly believe that unprincipled activities actually occur in respect to English language entry standards. By providing questionnaire and interview data on this theme, it is hoped that some evidence, as opposed to allegation, may be brought to bear on the perceptions and/or the reality of unprincipled practice.

With the intention of separating hearsay from direct experience, a series of paired questions were asked. In the first pair of questions, staff and students were asked if they had heard of bribes being paid to gain higher IELTS scores, and if so, the frequency with which they had heard of this. The second pair asked if respondents were positive that bribes had been paid, and to nominate the number of times they were certain this had occurred (staff Q. 37-40; student Q. 46-49).

The findings are summarised in Table 10. A small proportion of staff and student respondents agreed that they had “heard that people can pay a bribe to get a higher IELTS score” and a similarly small proportion were “sure people can pay a bribe to get a higher IELTS score”. Frequencies are included in Table 10, for although proportions are low, event numbers are nonetheless high. A total of 82 respondents felt “sure people have paid a bribe to get a higher IELTS score” at least once. The majority of event responses came from students.

In the interviewee groups, most of the respondents stated that they had no awareness of briberies paid. Of those who did, this was generally in the form of hearsay, or observation at a distance (examples below).

Some people in my country say this is possible but I don’t know. Who could I bribe? Even if I did bribe a person, I would be afraid that something would be wrong with the certificate and the university would know and reject me. My parents would feel such shame. (student, UK)

At the Embassy, in the queue, somebody asks you what you are doing, they give you this number…they say maybe the IELTS is a big trap. It’s easy to pay $2000, maybe $10,000. At the Embassy there are many notices saying don’t talk to and give money to other people - we do not know them and they are not related to us. These people are in the queue and in the street. They find you and you are a student or a teenager, and you want to go overseas. They want to talk to you about your problems. Why have you come for six months to the Embassy? They find out if you have money, and if you can pay $300 or $400 just to get this IELTS mark. (student, AUS)

Staff interviewees, too, only reported hearsay. This staff member placed the issue in terms of cultural relativism:

I have heard of it in foreign states where IELTS might be taught, but the term ‘bribe’ which you have used is not necessarily understood the same way in other cultures. Therefore, I am saying that the intentionality in certain cases is not to bribe. In certain cultures it would be quite normal to offer gifts in return for support and so forth, and that gives a further expectation. That is what we would sometimes see in our culture as a bribe. (staff, UK)

This staff member’s comment (below) perhaps relates to reliability, but the experience of variable ability at fixed score levels raised the question of security breaches in the interviewee’s mind:

My experience with students with the IELTS test has been that where they get a consistently high score in each band - and when I say a high score, it used to be 6.5 or more, with a minimum of 6.0 in each band - I found that some students coming in with that score were very good. Other students clearly struggled, and it made me wonder whether there was a security in place that made sure the students actually did sit the test and how in fact the security was done. You do hear sometimes of stories where people have paid people to come in and sit a test for them. (staff, AUS)

The falsification of certificates is apparently a larger issue. This first comment reflects on how this may be done:

You can take the number 6 and cut and just use glue and take a print and you can go just give the copy. The university doesn’t need the certificate - you just get one certificate. The universities, other places, they just need a copy certified by you. And they can’t check all of them, because they have a lot of documents, and maybe it’s easier to pay $2,000 for. Many people will decide to use a different way to reach a destination. (student, AUS)

This comment explains how it has been done:

I heard that there is some, what’s that called - bribing? There is definitely, you know. I explored and found a person who does all these things…I found a person and he took somewhere around $3,000 and he gave me a very identical certificate and he put all 7 … I don’t know, maybe he had a connection with the British Council because I compared everything [on the certificate] and everything was so correct - they just made it a little different in that code number. It was so perfectly made, you know. The signature was similar, everything was similar. So no one could see it was false. (student, AUS)

And this staff comment details how it has been detected:

One of my colleagues spotted a certificate, or rather a certified copy of a certificate, that he thought might be questionable … On further investigation with IELTS in Canberra, it turned out that it was fraudulent, and it’s actually opened up a whole new can of worms in that we discovered recently a fair proportion of photocopied certificates that seemed to be fraudulent … Now that’s the negative thing in that IELTS, for all its administrative thoroughness and efficiency, the system appears to have been broken and therefore that then calls to question how good the testing administration is. On the other side, to put a positive spin on it, IELTS are obviously taking it very seriously and they are in the process of putting in a number of measures to try and trace the forgeries to see if there is an operation or scam going on, in order to clamp down on what has happened. (staff, AUS)

Of the IELTS testing environment itself, student interviewees universally noted the strict security in force. These are typical descriptions:

In the examination setting, they are very careful about it, checking names and photos and all that … And also the speaking part is completely separated from the other three parts, so if you send another person to your exam session, what do you do for the speaking part? So the separation of the two parts makes it difficult to send another person other than yourself. (student, AUS)

Staff at the exam need the students to show their ID card or to identify themselves. So it is quite secure. (student, AUS)

The security to take part in the exam is very high. It’s difficult because they check your picture many times - your passport or your identity card. It is difficult to ask your friend to go and take part. (student, AUS)

Table 10: Perceptions of unprincipled activity

Staff and students were asked similar paired questions regarding the admission of students with IELTS scores below the minimum entry cut-off value. As may be seen in Table 10, reported occurrences were far higher. More than 40% of staff and students had “heard of students getting into the university below the minimum IELTS entry score”. Percentages remained high when asked if they were “sure students get into the university with scores below the minimum IELTS entry score”, as did reported frequencies. More than 200 respondents - that is, one-third of the entire sample - were “sure that students have gained admission to the university with scores below the minimum IELTS entry score”. Frequency values indicate that this may occur with some regularity. For instance, 47 respondents indicated that they were personally certain of six or more occurrences of sub-minimum entry. This, of course, has a multiplier effect: 47 respondents each certain of six or more occurrences equals a minimum of 280 separate instances. The true total, if one were to calculate each of the column values, is in fact greater than 600.

This staff member explained the nature of sub-minimum entry, in their experience:

It’s the course director’s decision whether or not to admit students. They do take students below the [university’s IELTS entry] level, because they want them on their course, or they need the numbers. They are very keen to get international students because they pay more. Maybe they think that the students are going to cope. Maybe they know they can get help with English for the student, or maybe they know in advance that the student will fail, but they’re not really bothered. (staff, UK)

I’ve had some research and PhD students who have got in. I have no idea how they came in and I don’t know how they got past the interview. I think some people just have friends in very high places. These students can’t communicate to anyone and their time here is going to prove very difficult. (staff, AUS)

While the rumour and/or certainty of bribery and of sub-minimum entry have been grouped under a single heading, the source of and responsibility for these two themes are distinct. The issue of bribes paid for higher IELTS scores ⁸, whether as hearsay or certainty, is likely of concern to IELTS administration. This data, gained independently from staff and students across three countries, shows that a minority of stakeholders are certain bribes occur, and that a fallibility exists within the administration of IELTS. The site of fallibility may exist outside the administrative reach of IELTS, but may be seen to rest with IELTS itself. ⁹

The responsibility for students gaining admission to university with IELTS scores below the institutional minimum rests elsewhere. IELTS administration deliberately remains entirely independent of the utilisation of IELTS scores for admissions purposes. Responsibility for sub-minimum admissions, then, rests with the universities themselves. Staff responses indicate that this does not involve a simple indictment of central admissions, as a large proportion of staff, at various levels of seniority, make admissions decisions that involve, in part, a review of a standardised English language test such as IELTS. The interview data further corroborates this decentralisation of international student admissions, particularly at the postgraduate level. That minimum English language standards are regularly flouted - particularly in the context of staff concerns regarding the ability of NESB students to handle the English language demands of their degree programs - is certainly a practice that may have serious long-term consequences. The interview data provides some illustration of the possible impacts the admission of ill-prepared students may have upon university standards and upon university financial resources.

For example, staff interviewees made the following observations.

I believe [IELTS] is reliable because of the number of students that I see with 6.5 who succeed, and I can understand them. Students who come here with 7.0 or 7.5 I have absolutely no difficulty with ever. So the interaction I have with students, the scores that I see convince me that it’s a reliable test. That’s why when I see somebody with what I consider to be a reliable test score who doesn’t meet the

requirement verbally and in written work and we have terrible difficulty with them, it doesn’t make me wonder about the reliability of the test. It makes me wonder about the security of knowing who they are, and if they did the test. (staff, AUS)

Course directors who take students with poor English are only storing up trouble, as the faculty finishes up paying for extra support for the student. (staff, UK)

I am very concerned now that these scores have dropped to 6 - still with a minimum of 6 in each band, fair enough - but people who were rejected and indeed required to do an extra session of English before they came in, they manage much better than people who come in with a straight 6. We are disadvantaging the students. (staff, AUS)

I think there are market and different pressures undoubtedly influencing the provision of education in the UK. Increasingly I think there will be pressures to seek overseas markets. But I think if we are interested in, firstly, the student and, secondly, the quality of the certification that we give them, I think we will either need to provide quality English improvement classes, or we must lay down standards that people have to meet before they arrive here. (staff; UK)

5.0 Further discussion and conclusion

In this section, some of the more salient findings will be discussed. While kept brief, this section identifies some of the more trenchant issues that emerged from the current research and flags areas of future research.

It should be pointed out that there is, particularly from the student perspective, a generally positive perception of IELTS combined with a clear recognition of its gate-keeping function within institutions. There would seem to be a reasonably clear perception of the positive washback of IELTS via preparation programs and materials, but this is, however, less evident in the sub-skill and the mixed skill areas.

In terms of staff perceptions, the findings are less positive. As there was uncertainty amongst staff interviewees, and a high proportion of ‘not sure’ responses to the survey, the less positive staff findings must be tempered by a generalised ‘contamination’ effect: a perception of poor language skills amongst the NESB student population in general.

Perceptions of both the appropriate level of English language ability for university study, as represented in an IELTS score, and perceptions of how English language minimum IELTS scores are determined, can provide important insights into the management of higher education. It should be stated that the focal institutions, and the majority of English medium universities, set their minimum IELTS score below British Council guidelines.

The IELTS management consortium (the British Council, IELTS Australia and UCLES) is reluctant to formally state what may be the ‘correct’ minimum score for a given field of study, preferring instead to leave standard setting to individual institutions. As stated:

The appropriate level required for a given course of study or training is ultimately something which institutions/departments/colleges must decide in the light of knowledge of their own courses and their experience of overseas students taking them. (IELTS 2001, p. 22)

Nonetheless, the IELTS management consortium has “used its experience of placing overseas students to establish certain guidelines relating to acceptance on courses” (IELTS 2001, p. 22). Their suggested band scores for academic courses are reproduced in Table 11. The key point to note is that the overall IELTS score of 6.0 - the minimum score necessary (from 2002) for entry to all degree programs at the focal institutions - still requires, in the opinion of the IELTS management consortium, further English study for both linguistically less and linguistically more demanding courses. For linguistically less demanding courses, such as mathematics and computer-based work, a minimum IELTS of 7.0 is suggested. For

linguistically more demanding courses, such as medicine and law, an IELTS score in excess of 7.5 is advised. IELTS is also clearly aware of the multiple variables which intervene in student success at university and is appropriately cautious about claims it makes regarding the test’s predictive validity (IELTS 2001).

One staff member, who had been directly involved with the setting of institutional English entry score levels, stated an awareness of the consortium’s suggested minimums. Interestingly, the institution’s minimum entry score was set below that advised by the consortium:

The only real interest we had in the IELTS score was in setting the University policy, which was five or six years ago at the least…We got a fair amount of information from the British Council about the testing standards and what they represent so we had a lot of background information about it. The British Council set out a grid of what competence a particular score would represent and I think we would have thought 6.5 was a safer score for overall competence to cope with academic degree courses … We reviewed policy and set the standard requirement of 6.0. (staff, UK)

Table 11: Suggested minimum IELTS score by linguistic demand. IELTS consortium guidelines

Source: IELTS (2001, p. 22)

The questionnaire results suggest that university staff would prefer an increase in the minimum IELTS entry score at their institutions. While students were more equivocal regarding the appropriate level of English language cut-off scores for university entry, supporting data suggests that, post-entry, students with higher IELTS scores felt more prepared for university study than students with lower scores. That is, students at the Australian institution, with a higher minimum IELTS entry score (at the time of analysis), were more positive regarding their preparedness of study in the medium of English than the UK institution students, with a lower IELTS minimum.

A simplistic reading of these results, however, is not possible. IELTS represents a high-stakes test, the results of which can determine the education or immigration prospects of NESB individuals. Approximately 40% of student respondents, for instance, believed that the appropriate range for the IELTS university cut-off should be 5.0-6.0. The Chinese institution students, with first year IELTS entry scores of 5.0, were most positive regarding their preparedness for university study in an English language medium.

Rather than reflecting simple opportunism, the student interview data provides sophisticated moral rationale for a lowered - or at least attainable - IELTS entry standard. These students sought the opportunity to enter university, and remarked upon the great difficulties inherent in language skill development in environments that do not allow active use of the English language:

There are some countries where few people can speak English - Japan, Iran, maybe, I don’t know, everywhere. This is not my fault. This is my government, my culture. They are not English. Sometimes people just don’t think about this. (student, AUS)

I don’t think they should have it like you must score a certain amount in each [IELTS] item. That is very difficult for overseas students. We’re not native speakers, we can’t score, for example, 7.0 in each item. That’s very difficult. (student, AUS)

When people are in their home countries, there are few opportunities to improve their English, especially in listening and speaking. I think 5.5, as a basic knowledge, is enough to come here and start to improve their English. For example, if they want to improve their English from 5.5 to 7.0, in their home country I think it would take maybe one year. But if they came here, I think three months is really enough to improve their English to 7 or 6.5. (student, AUS)

On the other hand, many staff interviewees were concerned about the hardships borne by students with weak English language skills, and the consequent moral responsibility that must be assumed by universities that admit seemingly under-prepared students. In addition to comments on this theme already cited, staff interviewees made these observations:

I have a few students at the moment who really are a bit stressed because they are struggling. There are tremendous cultural demands on them to ensure that they succeed…I think if they are struggling with English as well as the other academic components of the course, and the cultural pressures, it seems to me, God help them, they have got a lot on their plate. While we try very hard to accommodate all those wider pastoral needs, if there is a deficit in English, well I am the last person who can help them. (staff, UK)

I guess sometimes the students don’t realise just how hard it’s going to be in the first few months to make that adjustment. I don’t think that all staff are aware of just how difficult it is for the students to make the adjustment as well. They probably have different expectations of students than is realistic. (staff, AUS)

I would like to ask lecturers to come into my English language class and try to teach the student who has just arrived to do a PhD. In some cases that student may never have written an essay or have any idea of the structure and some day they are expected to go off and do research and a PhD. I have a PhD myself and I understand the system. It’s all so extremely difficult. I really feel for students with English as a second language. It’s difficult for me - I can’t imagine how stressful it is for them. (staff, UK)

Clearly, the issue of English language entry standards - that is, the use of IELTS and other standardised tests as an indicator of linguistic preparedness for university-level study - is extremely complex. What needs to be remembered, however, are the moral and ethical dimensions, on top of the administrative, bureaucratic and resource-related components. The admission of linguistically under-prepared NESB students may not only involve a large amount of ongoing language support, but may also require pastoral care to a large degree. Institutions must consider carefully their rationale for the setting of minimum language requirements. While it could obviously be argued that the institutional uses to which the tests are put are well beyond the control of the IELTS management consortium, these uses may nevertheless impact on IELTS and its reputation. Ironically, IELTS strong reputation in the market-place, in conjunction with a lacking awareness in the finer points of the test, perhaps ‘allows’ institutions to place too great a reliance on the test’s predictive power.

Universities also need to be conscious of the impact on their own reputations when accepting students with low test scores, especially when admissions decisions are not made in light of the relative availability of English language support at their institution. As these student and staff interviewees observed:

If a student comes here and they cannot speak [English] or listen, when they go back to Taiwan, if they do not improve, what will the people back home think of the university. They cannot understand why they can come if their English is not good. (student, UK)

The reason for lowering the standard is that we might get more students. It would mean benefits for the university, they would get more money, but the problem then is the standard has dropped. Then a

student may not have a positive experience or maybe does not get a good degree at the end of it, which is a problem. (staff, UK)

A strong finding was the lack of awareness among staff of the meanings of IELTS scores (see 4.2). IELTS could consider ways of facilitating communication of its criteria, its methods of ensuring reliability and validity, and other themes to staff on a more regular basis. The comment below typifies the responses of staff:

To be blunt, I think that maybe somebody in my position and indeed those lecturing overseas students might do well to have greater insights into what the test is designed for, how it is constructed, what it measures and what is the significance of the various score levels. (staff, UK)

The low level of knowledge on the part of staff contrasted with the students’ often very clear and detailed knowledge of the test and its constituent parts. IELTS may wish to consider consulting on an on-going basis with student stakeholder groups with a view to improving perceptions of both reliability and validity. Institutions, too, could benefit from the inclusion of IELTS participants when considering a modification in entry levels. The student survey and interview data detailed subtle understanding of IELTS, and its relationship with tertiary study in an English language environment. Such consultation would be in line with current concerns in the language testing community, with greater stakeholder involvement, greater democratisation of testing procedures and greater accountability of language testers urged (see Hamp-Lyons 2000; Shohamy 2000).

There is evidence of positive washback in the student survey responses. This reflects the power of IELTS to promote teaching and learning contexts, which encourage the development of the kinds of skills and knowledge needed for success in the real worlds of the university. This goes to broader issues of the consequential effects of validity and test impacts. The data consistently support the view that IELTS does have the power to affect student language learning outcomes, in both positive and negative ways (see 4.1.1 for a detailed discussion). The IELTS training industry that has emerged generally serves to prepare candidates for success on the IELTS test. Washback effects have the potential to impact upon the form in which academic English is taught. This in turn influences what students learn in these programs and what students internalise through independent use of IELTS materials.

Staff perceptions of washback were complex, and on the whole, less positive than students. Some staff perceptions, particularly the belief that IELTS ‘certifies’ student readiness for tertiary study, may be due to the general ‘contamination’ effect noted above. An important point, made most clear in the interview data, was the relationship between the constructs of academic language proficiency assessed by the IELTS test, and the curriculum genres that students face in undergraduate and postgraduate study. For instance, this staff interviewee compared what IELTS participants must produce for the written test with the demands of university study:

The second part [of the writing test], the essay, it is only 250 words. The students here in the first semester are asked to write maybe three or four assignments of 2,000 words. I really don’t think they are well enough prepared for that. (staff, UK)

Taking these complex intervening variables into account, IELTS could fruitfully investigate the real-world contexts at both undergraduate and postgraduate levels, and the authentic classroom and assessment genres students need to master. IELTS may wish to consider funding ethnographic or qualitative studies that investigate real-life contexts and evolving classroom environments. Some work has been done in this area, such as Buell’s study of the correlation between teacher observation of linguistic performance and test scores (Buell 1992). Much recent research into academic literacy in higher education points to the complex and dynamic disciplinary contexts (Johns 1997; Zamel and Spack 1988) in which students negotiate academic literacy. Further identification of the sociolinguistic and discourse competencies that would enhance students’ likelihood of successful study could usefully be carried out, with consideration given to the operationalisation of these constructs. In a recent workshop presentation, Appleby and Karacsony (2001) argued that IELTS does not prepare students for the academic English requirements of tertiary study. In their view, IELTS preparation programs focus on a “specific set of testable language skills and test techniques rather than the richness of language, attitudes towards learning and knowledge valued in tertiary education” (p. 1).

The IELTS ‘brand’ is very positively perceived at the UK institution. With favourable survey response rates, the interview respondents noted the link between IELTS, the British Council and the University of Cambridge, and the relationship with UK quality assurance processes, as rationale for their trust of the IELTS system. While also favourably rated at the Chinese and Australian institutions, supra-institutional links were less well known. These could be further developed, particularly amongst the academic and administrative staff. IELTS may also wish to respond to perceptions that student English language proficiency is attributable to IELTS rather than to an institution’s choice of minimum entry-levels (see 4.4).

It is hoped that applied benefits have emerged from this research project. Some possible applications include:

• institutional refinement in the use of IELTS for admission purposes
• a contribution to an understanding of how IELTS is perceived by key student and staff stakeholder groups (this may serve as market research for IELTS Australia and the British Council)
• a contribution to an understanding of how IELTS is used, and the manner in which formative and summative orientations to the test are prioritised (this could contribute to greater market refinement, and some perspective on the production of resources and support materials)
• an identification of the breadth of tertiary staff members which require further information regarding the IELTS test and its function.

Acknowledgments

The researchers would like to thank Valerie Caulcutt for her administrative support and to acknowledge all the students and staff of the three institutions who willingly gave of their time to participate in the study.

Bibliography

• Appleby, R. and Karacsony, A. (2001). ‘Is McEnglish enough? In search of substance for English language in the 21^st Century’. Unpublished workshop presentation. University of Technology, Sydney.
• Babbie, E. (1989). The Practice of Social Research. Fifth Edition. Belmont: Wadsworth
• Back, K., Davis, D. and Olsen, A. (1996). Internationalisation and Higher Education: Goals and Strategies. IDP Education Australia. Canberra: Australian Government Publishing Service.
• Barrett-Lennard, S. (1997). ‘Encouraging Autonomy and Preparing for IELTS: Mutually Exclusive Goals?’ Prospect. Vol. 12, No. 3. p. 29-40.
• Brown, A. (2000). ‘An Investigation of Rater’s Orientation in Awarding Scores in the IELTS Oral Interview’. IELTS Research Report. Volume 3. Canberra: IDP Education Australia.
• Buell, J. G. (1992). ‘TOEFL and IELTS as Measures of Academic Reading Ability: An Exploratory Study’. Presented at the 26th Annual Meeting of the Teachers of English to Speakers of Other Languages. Vancouver, 4-7 March.
• Chaloub-Deville, M. and Turner, C. E. (2000). ‘What to look for in ESL admission tests: Cambridge certificate exams, IELTS and TOEFL’. System, 28, p. 523-539.
• Charge, N. and Taylor, L. B. (1997). ‘Recent Developments in IELTS’. ELT Journal. Vol. 51, No. 4. p. 374-380.
• Chiu, I. and Brennan, M. (1990). ‘The Effectiveness of Some Techniques for Improving Mail Survey Response Rates: A Meta-analysis’. Marketing Bulletin, 1, p. 13-18.
• Coleman, D. (2000). The Core and the Periphery: The Case of Foundation Studies Programs in the Organisational and Educational Transformations of Higher Education. Doctoral dissertation. University of New South Wales.
• Coley, M. (1999). ‘The English language entry requirements of Australian universities for students of non-English speaking background’. Higher Education Research and Development. Vol. 18, No. 1. p. 7-17.
• Davies, A. (1997). ‘Demands of being professional in language testing’. Language Testing, 14 (3), p. 328-339.
• DSS Research (2001) Mail Survey Response. http://www.dssresearch.com/library/general/mailresp.asp
• EET (2001) Universities in Crisis. Senate Employment, Workplace Relations, Small Business and Education Committee. Commonwealth of Australia. http://www.aph.gov.au/senate/committee/EET_CTTE/public%20uni/report/index.htm
• Hamp-Lyons, L. (1997). ‘Washback, impact and validity: ethical concerns’. Language Testing, 14 (3), p. 295-303.
• Hamp-Lyons, L. (2000). ‘Social, professional and individual responsibility in language testing’. System. 28, p. 579-591.
• IDP Education Australia (1999). International Students in Australian Universities, First Semester 1999. Sydney: IDP Education Australia.
• IELTS (2001). International English Language Testing System: Handbook. Co-produced by: University of Cambridge Local Examinations Syndicate; the British Council; IDP Education Australia.
• Ingram, D. E. (1991). ‘The International English Language Testing System (IELTS): The Speaking Test’. Working Papers of the National Languages Institute of Australia. Vol. 1, No. 1. p. 101-114.
• Johns, A. (1997). Text, role and context: Developing academic literacies. Cambridge: Cambridge University Press.
• Kerstjens, M. and Nery, C. (2000). ‘Predictive Validity in the IELTS Test’. IELTS Research Report. Volume 3. Canberra: IDP Education Australia.
• McNamara, T. (2001). ‘Language assessment as social practice: Challenges for research’. Language Testing. 18 (4), p. 333-349.
• Messick, S. (1996). ‘Validity and washback in language testing’. Language Testing, 13 (3), p. 241-256.
• Mickan, P. (1997). ‘What do Writing Tests Test for Non-Native Speakers?’ Presented at the HERDSA Annual International Conference. Adelaide, 8-11 July, 1997.
• Noonan, G. and Contractor, A. (2001). ‘The new import-export trade’. The Age, 20 January.
• O’Loughlin, K. (2000). ‘The Impact of Gender in the IELTS Oral Interview’. IELTS Research Report. Volume 3. Canberra: IDP Education Australia.
• Rea-Dickins, P. (1997). ‘So, why do we need relationships with stakeholders in language testing?’ A view from the UK. Language Testing, 14 (3), p. 304-314.
• Reilly, G. (1998). ‘Fee education’. The Age. 18 April. Russell, A. (2000). ‘Overseas push gets lost in translation’. The Age. 3 November.
• Shohamy, E. (2001). The Power of Tests. Sydney: Pearson Education.
• Spolsky, B. (1997). ‘The ethics of gatekeeping tests: What have we learnt in a hundred years?’ Language Testing, 14 (3), p. 242-247.
• Ulster (2000). University of Ulster Annual Report, 1999-2000.
• Ulster (2001). University of Ulster - Profile.
• Uniken (1999). ‘Building an international reputation’. December, Issue 10. p. 7.
• UNSW (1997). Minutes of the Teaching and Learning Committee of the Academic Board. University of New South Wales. 7 April.
• UNSW (1998). 1998: Pocket Statistics. Sydney: UNSW Publication.
• UNSW (2000). Pocket Statistics 2000. Sydney: UNSW Publication.
• UNSW (2000a). Minutes of meeting 00/10 of the UNSW Academic Board, 7 November. Resolution AB00/97.
• Wall, D. (1998). ‘Impact and washback in language testing’. In Clapham, C. M. and Corson, D. (eds.). Language Testing and Assessment. Encyclopedia of Language and Education. Vol 7. Dordrecht: Kluwer. p. 291-302.
• Wall, D. (2000). ‘The impact of high-stakes testing on teaching and learning: Can this be predicted or controlled?’. System, 28, p. 499-509.
• Way, N: (2000). ‘Degrees for sale’. Personnel Risk Management. 28 July.
• Zamel, V. and Spack, R. (eds) (1998). Negotiating academic literacies. New Jersey: Lawrence Erlbaum.

Appendix 1.1

IELTS Survey: Student perceptions of the International English Language Testing System

The Learning Centre, University of New South Wales and the Centre for English Language Teaching, University of Ulster

We are a team of independent researchers, and we are really interested to learn what you think of the International English Language Testing System, sometimes known as the IELTS test. We would really appreciate it if you could take a few moments to fill out this survey. Your participation is voluntary, and your decision whether or not to participate will not affect your relations with the University in any way. If you choose to complete the survey, all of your answers are of course confidential, and you will remain anonymous. If you have any questions, please feel free to contact Dr. Sue Starfield, Director of the Learning Centre, University of New South Wales (s.starfield@unsw.edu.au). If you have any further questions or concerns, please contact the Ethics Secretariat, University of New South Wales, UNSW Sydney 2052, Australia (phone +61-2-9385-4234, fax +61-2-9385-6648, email ethics.sec@unsw.edu.au).

We thank you for taking the time to fill out this survey. Once you are finished, please post the completed survey in the enclosed reply-paid envelope. Thank you very much for your participation.

Dr. Sue Starfield, University of New South Wales; Dr. David Coleman, University of Sydney; Ms. Anne Hagan, University of Ulster

1. Have you ever taken the IELTS test?

[ ] yes [ ] no

2. If YES, which IELTS test did you take?

[ ] general IELTS test [ ] academic IELTS test [ ] don’t remember

3. Did you submit your IELTS score as part of your admissions application to the university?

[ ] yes [ ] no

4. Do you know what your faculty requires as a minimum overall IELTS score to gain entry? Please tick the appropriate box.

5. What IELTS training/preparation, if any, did you do? Please tick all appropriate boxes

6. If you did any IELTS training/preparation, why did you do so? Please number the items that were important to you. Put a 1 beside the most important reason, a 2 beside the second most important and so on, for ONLY those items that were important to you.

_ to improve my English language skills overall _ to improve my English reading skills _ to improve my English writing skills _ to improve my English listening skills _ to improve my English speaking skills ____ to improve another specific English language skill:

_ to help me get a higher IELTS score _ to help my chances of getting into my faculty/university of choice

_ it wasn’t my choice - the IELTS training was part of my schooling

_ it wasn’t my choice - someone else said I had to do it (ie. parent or teacher)

_ to get a job of choice

_ other - please specify…

7. Why did you take the IELTS test? Please number the items that were important to you. Put a 1 beside the most important reason, a 2 beside the second most important and so on, for ONLY those items that were important to you.

_ to improve my English language skills overall _ to improve my English reading skills _ to improve my English writing skills _ to improve my English listening skills _ to improve my English speaking skills _ to improve another specific English language skill:

_ to gain a standardised English language qualification

_ to help my chances of getting into my faculty/university of choice

_ I had to take the IELTS to get into my faculty/university of choice

_ it wasn’t my choice - the IELTS test was part of my schooling

_ it wasn’t my choice - someone else said I had to do it (ie. parent or teacher)

_ to get a job of choice

_ other - please specify…

8. Did you take other English language tests before entering your university course? If YES, please tick all appropriate boxes.

[ ] TOEFL

[ ] CULT

[ ] UCLES (Cambridge) First Certificate

[ ] Foundation Studies Program EAP

[ ] other - please specify…

9. If you did take other English language tests, what was your reason? Please number the items that were important to you. Put a 1 beside the most important reason, a 2 beside the second most important and so on, for ONLY those items that were important to you.

_ to improve my English language skills overall

_ to improve a specific English language skill:

_ to satisfy the entry requirements of my faculty/university of choice

_ to get a better English language test score

__ the IELTS is too hard - you can’t get a good score

_ it wasn’t my choice - the other test was part of my schooling

_ it wasn’t my choice - someone else said I had to do it (ie. parent or teacher)

_ to get a job of choice

____ taking the IELTS test is too expensive

_ other - please specify…

10. How many times have you taken the IELTS test? Please tick only one box.

[ ] never (zero times)

[ ] 1 time

[ ] 2 times

[ ] 3 times

[ ] 4 times

[ ] 5 or more times

11. Only answer this question if you took the IELTS test more than once:

If you took the IELTS test more than once, what was the reason? Please number the items that were important to you. Put a 1 beside the most important reason, a 2 beside the second most important and so on, for ONLY those items that were important to you.

_ to improve my English language skills overall

_ to improve a specific English language skill:

____ to satisfy the entry requirements of my faculty/university of choice

_ to get a better IELTS test score

__ the IELTS test is a good learning tool

__ the IELTS is too hard - you have to take it more than once to get a good score

_ it wasn’t my choice - taking the IELTS test more than once was part of my schooling

_ it wasn’t my choice - someone else said I had to do it again (ie. parent or teacher)

_ out of interest

_ I had taken the IELTS more than two years before applying to university, so my previous IELTS score was out of date

_ other - please specify…

12. Do you remember your overall score on your most recent IELTS test? Please tick the appropriate box.

[ ] below 4.0

[ ] 4.0

[ ] 4.5

[ ] 5.0

[ ] 5.5

[ ] 6.0

[ ] 6.5

[ ] 7.0

[ ] above 7.5

[ ] don’t remember

13. In what YEAR did you take your most recent IELTS test?

Please answer the following questions by ticking the most appropriate item.

14. Each faculty has a different IELTS entry score

[ ] yes

[ ] no

[ ] don’t know

15. There are different IELTS entry scores at the undergraduate and postgraduate levels

[ ] yes

[ ] no

[ ] don’t know

16. There are different IELTS entry scores depending on the student’s country of origin

[ ] yes

[ ] no

[ ] don’t know

17. There are different IELTS entry scores depending on the student’s language background [ ] yes [ ] no [ ] don’t know

18. In my experience, the overall IELTS score accurately measures English language ability

Options: [ ] strongly agree [ ] agree [ ] not sure (neither agree nor disagree) [ ] disagree [ ] strongly disagree

19. In my experience, all students who get the same overall IELTS score have the same English language ability

Options: [ ] strongly agree [ ] agree [ ] not sure (neither agree nor disagree) [ ] disagree [ ] strongly disagree

20. I think a student’s English language ability should be fully formed before they begin a university degree

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

For the following questions, please think about the IELTS test as well as any IELTS training/preparation you may have done.

21. The IELTS system prepares you well for academic uses of English, such as reading for university subjects

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

22. The IELTS system prepares you well for writing assignments, reports and essays at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

23. The IELTS system prepares you well for listening in lectures and tutorials at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

24. The IELTS system prepares you well for academic uses of English, such as speaking in seminars and tutorials at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

25. The IELTS system prepares you well for informal uses of English, such as having conversations in English outside the classroom environment

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

26. The IELTS system prepares you well for environments in which you have to shift between reading, writing, listening and speaking skills (such as group work and laboratory practicals)

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

27. My English language ability has improved in the time since I took the IELTS test

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

28. The IELTS test has a good reputation in my country

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

29. My IELTS score was an important factor in determining which faculty/university I could get into

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

30. The IELTS test gives you the best chance to get into university in an English-speaking country

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

31. The IELTS is the best English testing system available

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

32. If I had got a higher IELTS score, I would have enrolled in a different degree

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

33. An IELTS score is an internationally-recognised qualification

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

For the following questions, please think about the parts of the IELTS test itself.

34. In my experience, the IELTS entry score required by the university is an accurate measure of the English reading abilities necessary to succeed at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

35. In my experience, the IELTS entry score required by the university is an accurate measure of the English writing abilities necessary to succeed at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

36. In my experience, the IELTS entry score required by the university is an accurate measure of the English speaking abilities necessary to succeed at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

37. In my experience, the IELTS entry score required by the university is an accurate measure of the English listening comprehension abilities necessary to succeed at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

38. Overall, the university’s IELTS entry score accurately reflects the English language level required to succeed at the university level

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

39. In my opinion, the IELTS entry score for my faculty/university should be changed

[ ] yes [ ] no [ ] don’t know

40. If YES, the university should change the minimum IELTS entry score to:

41. In my opinion, the minimum IELTS entry score should be higher than it is now for postgraduate students

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

42. In my opinion, the minimum IELTS entry score should be higher than it is now for undergraduate students

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

43. Preparing for the IELTS test helped me improve my English language skills

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

44. The main purpose of the IELTS is technical: if you want to go to university, it is just one of the things you have to do

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

45. In my opinion, the university determines its IELTS entry levels based on:

a) careful consideration of the appropriate English language ability necessary for university study

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

b) a wish to match the IELTS entry levels of other universities

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

46. I have heard that people can pay a bribe to get a higher IELTS score

[ ] yes [ ] no

47. If YES, how frequently have you heard that people can pay a bribe to get a higher IELTS score?

[ ] once [ ] two times [ ] three to five times [ ] six or more times

48. I am sure people can pay a bribe to get a higher IELTS score

[ ] yes [ ] no

49. If YES, how frequently are you sure people have paid a bribe to get a higher IELTS score?

[ ] once [ ] two times [ ] three to five times [ ] six or more times

50. I have heard of students getting into the university below the minimum IELTS entry score

[ ] yes [ ] no

51. If YES, how frequently have you heard of students getting into the university below the minimum IELTS entry score?

[ ] once [ ] two times [ ] three to five times [ ] six or more times

52. I am sure students get into the university with scores below the minimum IELTS entry score

[ ] yes [ ] no

53. If YES, how frequently are you sure that students have gained admission to the university with scores below the minimum IELTS entry score?

[ ] once [ ] two times [ ] three to five times [ ] six or more times

54. What is the highest possible score on the IELTS?

Some questions about you:

55. What is your gender?

[ ] female [ ] male

56. What is your country of origin? Please specify … .

57. What is your primary language? Please specify … .

58. What is your level of study?

[ ] undergraduate [ ] postgraduate

59. What is your year of study?

[ ] first year [ ] second year

[ ] third year [ ] fourth year

[ ] fifth year or more

61. Any additional comments?

Thank you!

Please place your completed survey in the reply-paid envelope and mail at your first convenience.

If you are interested in discussing the IELTS further, we would very much like to hear from you. Please send us an email (d.coleman@edfac.usyd.edu.au or a.hagan@ulst.ac.uk) or phone (David Coleman in Sydney: 9351-2625 or Anne Hagan in Belfast: 7032-4198) to arrange a time when we can meet. These discussions will be confidential and will not take more than thirty minutes of your time. We look forward to hearing from you!

Appendix 1.2

IELTS Survey: Staff perceptions of the International English Language Testing System

The Learning Centre, University of New South Wales and the Centre for English Language Teaching, University of Ulster

We are a team of independent researchers, and we are really interested to learn what you think of the International English Language Testing System, sometimes known as the IELTS test. Many prospective students take the IELTS test in partial fulfillment of university admission.

We would really appreciate it if you could take a few moments to fill out this survey. Your participation is voluntary, and your decision whether or not to participate will not affect your relations with the University in any way. If you choose to complete the survey, all of your answers are of course confidential, and you will remain anonymous. If you have any questions, please feel free to contact Dr. Sue Starfield, Director of the Learning Centre, University of New South Wales (s.starfield@unsw.edu.au). If you have any further questions or concerns, please contact the Ethics Secretariat, University of New South Wales, UNSW Sydney 2052, Australia (phone +61-2-9385-4234, fax +61-2-9385-6648, email ethics.sec@unsw.edu.au).

We thank you for taking the time to fill out this survey. Once you are finished, please put the completed survey in the enclosed envelope and send via the internal mail. Thank you very much for your participation.

Dr. Sue Starfield, University of New South Wales; Dr. David Coleman, University of Sydney; Ms. Anne Hagan, University of Ulster

1. There are different IELTS entry scores at the undergraduate and postgraduate levels

[ ] yes [ ] no [ ] don’t know

2. There are different IELTS entry scores depending on the student’s country of origin

[ ] yes [ ] no [ ] don’t know

3. There are different IELTS entry scores depending on the student’s language background

[ ] yes [ ] no [ ] don’t know

4. Faculties can have different IELTS entry scores

[ ] yes [ ] no [ ] don’t know

5. In my job, I sometimes have to decide whether or not to admit students to specific subjects or degree streams

[ ] yes [ ] no

6. In my job, I sometimes have to decide whether or not to admit students to specific subjects or degree streams, in which an English language test score - such as the IELTS - forms some part of the decision-making process

[ ] yes [ ] no

7. What is the purpose of the IELTS test? Please number the items that you feel are important. Put a 1 beside the most important reason, a 2 beside the second most important and so on, for ONLY those items that you feel are important.

_ to improve English language skills overall

_ to improve English reading skills

_ to improve English writing skills

_ to improve English listening skills

_ to improve English speaking skills

_ to improve another specific English language skill:

____ for a student to gain a standardised English language qualification

____ to help a student’s chances of getting into a faculty/university of choice

_ students have to take the IELTS to get into a faculty/university of choice

_ to get a job of choice

_ other - please specify…

8. To your knowledge, which of the following other English language tests does the university accept? Please tick all appropriate

[ ] TOEFL [ ] CULT [ ] UCLES (Cambridge) First Certificate

[ ] Foundation Studies Program EAP [ ] other - please specify…

Please answer the following questions by ticking the most appropriate item.

9. In my experience, the overall IELTS score accurately measures English language ability

Options: [ ] strongly agree [ ] agree [ ] not sure (neither agree nor disagree) [ ] disagree [ ] strongly disagree

10. In my experience, all students who get the same overall IELTS score have the same English language ability

Options: [ ] strongly agree [ ] agree [ ] not sure (neither agree nor disagree) [ ] disagree [ ] strongly disagree

11. I think a student’s English language ability should be fully formed before they begin a university degree

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

12. The IELTS system prepares students well for academic uses of English, such as reading for university subjects

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

13. The IELTS system prepares students well for writing assignments, reports and essays at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

14. The IELTS system prepares students well for listening in lectures and tutorials at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

15. The IELTS system prepares students well for academic uses of English, such as speaking in seminars and tutorials at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

16. The IELTS system prepares students well for informal uses of English, such as having conversations in English outside the classroom environment

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

17. The IELTS system prepares students well for environments in which they have to shift between reading, writing, listening and speaking skills (such as group work and laboratory practicals)

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

18. The IELTS test has a good reputation

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

19. The IELTS test gives a student the best chance to get into university in an English-speaking country

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

20. The IELTS is the best English testing system available

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

21. An IELTS score is an internationally-recognised qualification

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

22. When I see a student’s IELTS score, I have a very clear idea of their English language proficiency

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

23. A student’s IELTS score tells you a lot about their overall intelligence

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

24. In my experience, the IELTS entry score required by the university is an accurate measure of the English reading abilities necessary to succeed at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

25. In my experience, the IELTS entry score required by the university is an accurate measure of the English writing abilities necessary to succeed at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

26. In my experience, the IELTS entry score required by the university is an accurate measure of the English speaking abilities necessary to succeed at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

27. In my experience, the IELTS entry score required by the university is an accurate measure of the English listening comprehension abilities necessary to succeed at university

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

28. Overall, the university’s IELTS entry score accurately reflects the English language level required to succeed at the university level

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

29. In my opinion, non-English speaking background students should commence university study with a minimum English language competency, but their language proficiency could be expected to improve over time

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

30. The university has a standard IELTS entry score for all students

[ ] yes

[ ] no

[ ] don’t know

31. In my opinion, the university determines its IELTS entry levels based on:

a) careful consideration of the appropriate English language ability necessary for university study

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

b) a wish to match the IELTS entry levels of other universities

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

c) Other reasons. Please specify…

32. In my opinion, the university should change the minimum overall IELTS score necessary to gain entry

[ ] yes

[ ] no

[ ] don’t know

33. If YES, the university should change the minimum IELTS entry score to:

[ ] below 4.0

[ ] 4.0

[ ] 4.5

[ ] 5.0

[ ] 5.5

[ ] 6.0

[ ] 6.5

[ ] 7.0

[ ] above 7.5

[ ] don’t know

34. In my opinion, the minimum IELTS entry score should be higher than it is now for postgraduate students

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

35. In my opinion, the minimum IELTS entry score should be higher than it is now for undergraduate students

Options: [ ] strongly agree [ ] agree [ ] not sure [ ] disagree [ ] strongly disagree

36. What is the highest possible score on the IELTS?

37. I have heard that people can pay a bribe to get a higher IELTS score

[ ] yes [ ] no

38. If YES, how frequently have you heard that people can pay a bribe to get a higher IELTS score?

[ ] once [ ] two times [ ] three to five times [ ] six or more times

39. I am sure people can pay a bribe to get a higher IELTS score

[ ] yes [ ] no

40. If YES, how frequently are you sure people have paid a bribe to get a higher IELTS score?

[ ] once [ ] two times [ ] three to five times [ ] six or more times

41. I have heard of students getting into the university below the minimum IELTS entry score

[ ] yes [ ] no

42. If YES, how frequently have you heard of students getting into the university below the minimum IELTS entry score?

[ ] once [ ] two times [ ] three to five times [ ] six or more times

43. I am sure students get into the university with scores below the minimum IELTS entry score

[ ] yes [ ] no

44. If YES, how frequently are you sure that students have gained admission to the university with scores below the minimum IELTS entry score?

[ ] once [ ] two times [ ] three to five times [ ] six or more times

Some questions about you:

45. What is your gender?

[ ] female [ ] male

48. Have you ever taken the IELTS test?

[ ] yes [ ] no

49. What is your primary role at the university

[ ] central administration [ ] faculty/school-based administration

[ ] academic - research only [ ] academic - teaching and research

50. At roughly what level of seniority would your position be classified:

academic staff:

[ ] associate lecturer/tutor [ ] lecturer/senior lecturer [ ] associate professor/professor

non-academic staff:

[ ] non-managerial [ ] middle-level managerial [ ] high-level managerial

51. Do you hold other representative positions within the university? Please tick all appropriate

[ ] Head of School [ ] school/department committee [ ] faculty committee

52. Any additional comments?

(Please add an extra sheet if you require more space)

Thank you!

Please place your completed survey in the envelope supplied and put in the internal mail at your first convenience.

If you are interested in discussing the IELTS further, we would very much like to hear from you. Please send us an email (d.coleman@edfac.usyd.edu.au or a.hagan@ulst.ac.uk) or phone (David Coleman in Sydney: 9351-2625 or Anne Hagan in Belfast: 7032-4198) to arrange a time when we can meet. These discussions will be confidential and will not take more than thirty minutes of your time. We look forward to hearing from you!

Appendix 2.1

Student interview guide

Was an IELTS score part of your admissions application?

What is the purpose of IELTS?

Why did you take the test?

How many times have you taken the IELTS test?

if multiple times, why?

Did you take any other English language tests (TOEFL, etc)?

if YES, why?

if NO, why not?

Did you use any IELTS training materials?

if YES, why?

Is IELTS a learning tool?

Does it serve a gate-keeping function?

Are you aware of the IELTS scoring system?

if YES, please describe

if NO, what if anything do you know about IELTS scores?

If you saw someone had an IELTS score of 6.0, what would that mean to you?

In your view, are your English language skills high enough to handle the university workload?

probe to specific skills

-reading, writing, speaking, listening

-formal/informal

In your view, are the English language skills of other students high enough to handle the university workload?

probe to specific skills

-reading, writing, speaking, listening

-formal/informal

**if feel English language levels are too low

What explains the low level of English language proficiency?

if don’t mention IELTS

does the IELTS system have anything to do with it?

Should the university change its English language entry cut-off levels?

if YES, to what?

IELTS score and/or other measure

How would/do you know that IELTS is a reliable test?

How would/do you know that IELTS is a trustworthy test?

How would/do you know that IELTS is a comprehensible test (ie. you know what the scores mean)?

Should students commence study at the university with fully formed English language skills, or can they arrive with a minimum proficiency and be expected to improve over time?

Have you heard of students entering the university below the minimum IELTS cut-off score?

if YES, please explain

Have you heard of students paying a bribe to get a higher IELTS score?

if YES, please explain

Demographics

-ug/pg

-faculty/course

-year

-country

-maternal/first language

-age

What did you get on the IELTS, overall?

Appendix 2.2

Staff interview guide

What is the purpose of IELTS?

why do students take the test?

Is IELTS a learning tool?

Does it serve a gate-keeping function?

Have you ever had to make a decision (admission, standards, advanced standing…) where an IELTS score was a contributing factor?

if YES, please explain

Are you aware of the IELTS scoring system?

if YES, please describe

if NO, what if anything do you know about IELTS scores?

If you saw a student had an IELTS score of 6.0, what would that mean to you?

In your experience, are the English language skills of students high enough to handle the university workload?

probe to specific skills

-reading, writing, speaking, listening

-formal/informal

**if feel English language levels are too low

What explains the low level of English language proficiency?

if don’t mention IELTS

does the IELTS system have anything to do with it?

Should the university change its English language entry cut-off levels?

if YES, to what?

IELTS score and/or other measure

How would/do you know that IELTS is a reliable test?

How would/do you know that IELTS is a trustworthy test?

How would/do you know that IELTS is a comprehensible test (ie. you know what the scores mean)?

Should students commence study at the university with fully formed English language skills, or can they arrive with a minimum proficiency and be expected to improve over time?

Have you heard of students entering the university below the minimum IELTS cut-off score?

if YES, please explain

Have you heard of students paying a bribe to get a higher IELTS score?

if YES, please explain

Position/category within the university

-administration

-academic

-student support services

-OTHER DETAILS?