The use of IELTS for university selection in Australia: A case study

Abstract

Grant awarded Round 10, 2004

This study examines the use of IELTS in selecting international, full fee-paying students to a large faculty at a major Australian university.

In recent years there has been growing interest in how high-stakes language test scores are interpreted and used in specific contexts. This report describes a study which examined the use of IELTS in selecting international, full fee-paying students for a large faculty at a major Australian university. Using institutional documents, questionnaires and interviews the study attempted to, firstly, map how IELTS was used in the selection process and, secondly, explore the knowledge and beliefs which staff (both administrative and academic) and students had about the Test. Key findings included:

1. a well-articulated, if somewhat inflexible, selection policy and clear guidelines about minimum English language requirements including IELTS
2. a generally high level of procedural compliance with university policy and procedures on the part of staff directly involved in selection
3. variable levels of knowledge about the IELTS (both of the Test and the scores it produces) among staff and students including a lack of understanding among both groups as to what different IELTS scores imply about a student’s language ability, their readiness for university study and their need for further English development
4. the prevalence of “folkloric” beliefs particularly amongst staff about English language proficiency and the IELTS Test, some with a firmer basis in reality than others. Such beliefs include a scepticism about the validity, reliability and “trustworthiness” of IELTS scores and an unrealistic expectation about their power to predict academic success
5. a lack of clearly established equivalence between the IELTS Test and other acceptable evidence of English proficiency in university selection policy.

These findings and their implications are discussed in detail in this report.

1 BACKGROUND AND RATIONALE

In recent years the use of language tests for gate-keeping purposes has come under close scrutiny by researchers. More generally, the political dimension of language testing has begun to receive the attention it has long deserved with various researchers and writers examining and critiquing current practices from a policy and social perspective (see, for example, Hamp-Lyons 1997; McNamara 1997; Pennycook 2001; Shohamy 2001).

The interpretation and use of language tests, in particular the evidence they provide about individuals’ language ability for a specified purpose, is of fundamental concern in any evaluation of test validity. As Messick (1996, pp 245) has pointed out:

Messick’s (1989) model of validity as a unified concept is conceptualised in terms of four facets: a) the evidential basis of test interpretation (construct validity), b) the evidential basis of test use (construct validity and relevance/utility), c) the consequential basis of test interpretation (value implications) and d) the consequential basis of test use (social consequences).

This study relates particularly to the second, third and fourth of these facets: the need for test scores to be relevant and useful in the testing context, the need to recognise that all interpretations of test scores involve questions of value and the need to investigate what specifically happens when a test is implemented in terms of both its washback on teaching and learning and its impact more generally (McNamara 2001, pp 335-336).

The IELTS test has established itself as the most powerful high-stakes English language testing instrument in Australia. It is the only test used by the Australian Federal Government to screen the English language proficiency of applicants whose first language is not English for both permanent residency and (where appropriate) for study visas. Given this gate-keeping function and, hence the influence it has on the lives of individuals, it is extremely important to evaluate the impact of the use of IELTS for these purposes. One such use is university selection. IELTS has firmly entrenched itself in policy and practice as the most widely recognised language test for determining whether prospective applicants, whose first language is not English, have met the minimum English language entry requirements in Australian universities.

IELTS consists of four sub-tests in Listening, Reading, Writing and Speaking. Candidates are given a separate Individual Band Score for each of the sub-tests from 0 to 9. Listening and Reading scores increase by increments of 0.5 while Speaking and Writing increase in whole numbers (until 1 July 2007 when they were also changed to 0.5 increments). An Overall Band Score, an unweighted average of the four results, is then produced. The Overall Band Score and Individual Band Scores are then recorded on the Test Report Form given to candidates.

IELTS is not a certificated pass/fail examination. Instead, individual institutions must determine the minimum scores for entry to particular courses at undergraduate and postgraduate levels. In general, an Overall Band Score of between 6.0 and 7.0 in the Academic module is accepted as satisfactory evidence of English proficiency by institutions of higher education worldwide. As suggested in the IELTS Annual Review (2002, pp14-15) IELTS Overall Band Scores reflect English language proficiency and cannot be read as predictors of academic success or failure independently of other relevant personal and social factors. Nevertheless, the IELTS partners (British Council, IELTS

Australia and Cambridge ESOL) have provided guidelines relating to the minimum Overall Band Scores which might be acceptable for particular courses (IELTS Handbook, 2005, p 5). In these guidelines a distinction is made between linguistically demanding courses such as law, medicine and journalism and less linguistically demanding ones such as pure mathematics, agriculture and information technology. It is recommended that linguistically demanding courses require a higher minimum entry level than less linguistically demanding courses. Furthermore, while levels 7.0 and above are indicated to be ‘acceptable’ or ‘probably acceptable’ for both kinds of courses, levels 6.5 and below are accompanied by the recommendation, ‘English study needed’. The IELTS Handbook (2005, p 5) adds the important caveat that:

This recommendation suggests that the numeric scores should not be used exclusively when considering the suitability of potential students for particular courses. In their review of the major university English admissions’ tests used around the world, Chalhoub-Deville and Turner (2000, pp 537-538) suggest that past academic performance and performance at a selection interview should also be taken into account. Rees (1999) notes that, while proficiency tests can provide information about the approximate level of English language competence a student may have at any one time, they cannot predict the rate of likely improvement in proficiency within the academic context or, indeed, academic success. Rees suggests that a student’s language learning potential might also be usefully gauged though language aptitude and even IQ testing.

In setting entry levels on IELTS many universities not only stipulate an Overall Band Score but also minimum Individual Band Score levels on one or more of the sub-tests. As Chalhoub-Deville and Turner (2000, p 537) suggest, good selection practice will take into account the scores in the different skills areas, as well as the overall score on a test like IELTS, as different academic programs may require different profiles of language ability.

While a baseline Overall Band Score and in some cases, Individual Band Scores, are specified within any given university’s general admissions policy, individual faculties - and sometimes departments - are often able to decide on their own entry scores for different courses at both undergraduate and postgraduate levels. This means that an individual faculty’s or course’s entry score may be higher - and sometimes even lower than the level stipulated in the university’s general admissions policy. In some universities selection officers, working alongside academics, are able to exercise discretion in relation to individual cases, especially at the postgraduate level. While this process can allow factors other than students’ English language proficiency to be considered in the selection process, there is also a danger that they may be admitted before their English proficiency is adequate to commence their course of study.

More generally, Chalhoub-Deville and Turner (2000, p 537) argue that test users such as selection officers and academics need to be knowledgeable about the test instruments they employ and ensure appropriate and ethical interpretation and use of the results they yield. They also highlight the importance of monitoring and evaluating the impact of cut-off scores to ensure valid and ethical use of the test in the university context.

With the reduction in Federal Government funding to Australian universities over the last decade, tertiary institutions attempt to recruit as many full fee-paying students as possible each year (most of these are international students whose first language is not English). In this climate the competition for full fee-paying students is intense. This development has had a direct effect on the use of IELTS (and the Test of English as a Foreign Language, TOEFL, where it is also accepted) with universities being under pressure to lower their entry scores to attract more students. An alternative strategy has been to build English language pathways which do not culminate in IELTS or TOEFL including stand-alone intensive English for Academic Purposes programs or programs where English is one part of the curriculum such as Foundation Studies or Year 12. Both ‘solutions’, however, have been highly controversial and continue to be widely debated within universities. In relation to alternative measures of English proficiency, Coley (1999) argues that IELTS is the most stringent measure of students’ readiness for academic study and that many of the other alternatives used by Australian universities are inadequate.

Recently there have attempts to lower IELTS entry scores for particular courses and require students admitted to them to undertake further English study, particularly in their first year (see Elleringtion and Bayliss 2004 for example). More radically, some universities are now considering changing their entry requirements so that students will only have to demonstrate what is now the entry level by the end of their course. It could be argued, however, that this strategy threatens to erode standards of English language proficiency in Australian universities.

Factors which influence improvement in English language proficiency appear to be wide ranging and complex. Elder and O’Loughlin (2003) examined the progress made by 112 students in Australia and New Zealand studying pre-university intensive English over a 10-12 week period. Students sat an IELTS Test at the beginning and end of this period. They found that the average amount of improvement over this period was 0.5 of a band but that there was strong variability among the students with some making no progress at all. In addition, it was found that the more proficient students improved less on the IELTS Test than the less proficient students. Using data collected from questionnaires and interviews with students and teachers, they found that the reasons for improvement (or the lack of it) included motivation, accommodation, self-confidence and the extent to which students used or accessed English outside the classroom.

As suggested previously, IELTS results reflect English proficiency alone and are unlikely, in themselves, to be accurate predictors of academic success. Nevertheless, there have been a number of studies in the last decade which have examined the Test’s predictive power (for example, Broadstock 1994; Cotton and Conrow 1998; Hill, Storch and Lynch 1999; Kerstjens and Nery 2000; Dooey and Oliver 2002). The findings in these studies generally show that IELTS has only weak to moderate predictive power: the common conclusion is that language is only one of many important factors contributing to academic success or failure. Such factors include, among many others, student motivation, financial support, adequate study skills and the use of English outside the classroom. In the most recent of these studies, Dooey and Oliver (2002) found that achieving the institution’s minimum Overall Band Score (in this case 6.0) was only a partial predictor of academic success. They also found that some students who did not achieve this minimum level but were still admitted for other reasons were successful in the first year of their studies. This finding underscores the importance of examining factors other than an applicant’s IELTS result when assessing their suitability for tertiary study.

In recent years there has been a growing interest in studying the use of IELTS in tertiary institutions. Several important studies funded by the IELTS partners are described briefly below.

Deakin (1997) surveyed the attitudes of English teaching professionals at the tertiary/adult level in Australia. He found that while they viewed IELTS as a reasonably good proficiency test, they also thought that it was important to recognise its limitations in predicting the kinds of difficulties that international students faced as they learned to operate with the academic culture of Australian universities.

McDowell and Merrylees (1998) surveyed academics and administrative staff in a wide range of Australian tertiary institutions to explore, firstly, which were using IELTS, secondly, whether it was serving their needs and, thirdly, to establish what other measures of English proficiency were being used. They found that IELTS was the most commonly used and most preferred English proficiency test.

A more recent study into the use of IELTS by Coleman, Starfield and Hagan (2003) examined the attitudes of both students and staff (administrative and academic) towards IELTS in three institutions in Australia, China and the UK. Coleman, Starfield and Hagan found that, while all participants in the study were generally positive towards the Test, overall, students were more knowledgeable about the Test and more convinced that the institution’s IELTS entry level was appropriate for the course they were undertaking. Staff generally felt that the IELTS scores should be higher and that many students’ English language ability was not adequate for their chosen course. Perhaps the most disconcerting finding was that the university staff (administrative and academic) in the three participating institutions demonstrated low understanding of the meaning of IELTS scores. The issue of how staff in other institutions interpret and use the test results clearly warrants further research.

While these studies have provided extremely useful information about the use of IELTS in tertiary selection at a broad, macro level, there is an equally strong need for more micro level case studies to explore its use in tertiary selection in detail. Banerjee (2003) has undertaken an important study into the use of proficiency test scores, including IELTS, in the selection of postgraduate degree courses at a UK university. She found that the selection process at the University of Lancaster was a complex, holistic decision-making process based on the recommendation of an academic staff member taking into account a wide range of criteria. Yet, like Starfield et al (2003), Banerjee found that academic admissions officers were not very knowledgeable about the meaning of proficiency test scores.

In a recent study conducted at the University of Bristol, Rea-Dickins, Kiely and Yu (2007) also found that university admissions staff were not always sufficiently knowledgeable about the meaning of IELTS test scores. They argue for stronger training of admissions tutors so that they become better informed about the meanings of IELTS score profiles. This includes awareness of, and access to, the IELTS website.

Given that Banerjee (2003) and Rea-Dickins et al (2007) focused exclusively on UK selection practices there is a clear need for other locally based studies examining the use of IELTS in the selection of international students at universities in other countries. This was the purpose of this study which examines the use of the Test within a large faculty at a major Australian university.

2 RESEARCH QUESTIONS

The central research question investigated in this study was:

To what extent are IELTS test scores used in valid and ethical ways for the purposes of university selection?

3 CONTEXT OF THE STUDY

3.1 Research site

The Faculty of Economics and Commerce, University of Melbourne, Australia.

The Faculty of Economics and Commerce is one of the University of Melbourne’s largest faculties with the highest number of international student enrolments. In 2005, a total of 5463 students were enrolled in undergraduate and postgraduate programs; of these 2585 were international students.

3.2 Participants

1. Senior academic and administrative (‘professional’) staff within the Faculty of Economics and Commerce and across the wider university.
2. Admissions officers within the university’s international admissions office and selection officers within the faculty itself (professional staff in both instances).
3. Selected students enrolled in undergraduate and postgraduate courses in the Faculty of Economics and Commerce.

4 METHODOLOGY

4.1 General approach

The study employs a case study approach to investigate the research question. Merriam (1988, pp 9-10) provides the following definition of a case study:

In a sense, of course, this could be a description of any form of empirical research and yet what is distinctive about the case study is its holistic focus on the ‘bounded system’ in context. Thus, for Yin (1989, p 23) “a case study is an empirical inquiry that investigates a contemporary phenomenon within its real-life context; when the boundaries between phenomenon and context are not clearly evident; and in which multiple sources of evidence are used”. As Yin suggests, the other main distinguishing feature of case study research is its use of a variety of evidence (both qualitative and quantitative) to explore the issue(s) under investigation.

Stake (1994, p 237) distinguishes between intrinsic and instrumental case studies. In the intrinsic case study the researcher is focused entirely on the particular case. In the instrumental case study, on the other hand, a particular case is investigated to throw light on an issue or theory. In this instance, Stake suggests:

The research reported here is an example of an instrumental case study. In focusing on how IELTS is used for selection purposes in the Faculty of Economics and Commerce at the University of Melbourne, the study aims to shed light on the use and impact of IELTS in university selection as an educational and administrative practice.

4.2 Data collection

There were three main forms of data collection: search and analysis of relevant university policy and procedures documents; separate questionnaires administered to staff and students; and interviews conducted with selected staff and students.

4.2.1 Selection policy and procedure documents

Initially, a range of university selection policy and procedure documents were collected to ascertain a) the relevant English language requirements for prospective undergraduate and postgraduate international students in the Faculty of Economics and Commerce, particularly the IELTS entry scores and b) the place of English language requirements in the selection process.

The documents relating to a) the university’s (including the faculty specific) English language entry requirements were all publicly available and accessible online. The starting point for this first stage of the data collection was the information available to all international students applying for a course at the University of Melbourne in 2005. The links to the Faculty of Economics and Commerce’s specific undergraduate and postgraduate English language entry requirements were then followed. Access to more restricted documents relating to b) selection procedures and specifically how evidence of English language proficiency is used in the selection process were obtained from the Manager, International Admissions at the university. This work was completed between May and July 2005.

4.2.2 Questionnaires

Different versions of the questionnaires were then developed for staff and student participants, as described below.

Staff Questionnaires contained a combination of forced-choice and open-ended items. These items were designed to elicit information about:

• participants’ roles in international student selection
• their self-rating of their knowledge of English language proficiency requirements for admission of international students to the university and faculty
• their understanding and opinions regarding the use of IELTS in selecting prospective international students into programs in the Faculty of Economics and Commerce
• their opinions about the use of IELTS in selection.

The final item provided an opportunity for participants to comment on any other aspect of their experience with, or opinion about, the use of the IELTS Test and other measures of English language proficiency (see Appendix 1).

Student Questionnaires also contained a combination of forced-choice and open-ended items. The items were designed to elicit information about:

• the participants’ personal and academic identity
• their English language abilities at the time they were accepted into their courses at the university
• their experience as international students, in terms of the English language demands of their courses.

The final item in the Student Questionnaire provided an opportunity for participants to make further comments about the matters addressed in previous items (see Appendix 2).

4.2.3 Interviews

Semi-structured staff interviews were conducted by the researcher and research assistant with a sample of the participants. The interviews were based on participants’ questionnaire responses, using a staff interview template in which the sections closely followed those in the questionnaire (see Appendix 3).

Semi-structured student interviews were also conducted with a sample of the participants. As was the case with staff interviews, the student interviews were based on their questionnaire responses using a template in which the sections closely mirrored the Student Questionnaire (see Appendix 4).

Less structured interviews were also conducted with two ‘expert’ academic staff from outside the Faculty of Economics and Commerce who possessed a great deal of knowledge and experience of issues related to international student admissions at the university.

4.3 Procedures

4.3.1 Pilot study

Before the main phase of the study, a small-scale pilot study was conducted with small groups of staff and students in August 2005. The principal aim of the pilot study was to develop, trial and refine the instruments to be used in the main study. These instruments included the Staff Questionnaire, Student Questionnaire, Staff Interview and Student Interview. The pilot study was conducted in the Faculty of Education, the home faculty of the researcher and research assistant involved in the project. During the pilot, Staff and Student Questionnaires were drafted, revised and then administered to small groups of faculty colleagues and students. In both cases, the participants in the pilot study were asked to respond to the items on the questionnaires, as a simulation of the main study. In addition, participants were asked to provide constructive feedback on the format of the questionnaires and the nature and phrasing of the individual items in the questionnaires. Participants’ responses were analysed; the constructive feedback was evaluated; and then further revisions were made to the questionnaires.

These revisions were most significant in the Staff Questionnaire. They included both minor and major rephrasing of specific items, in the interests of clarity and precision of focus. This was particularly the case in items where the intention was to ask participants to comment on faculty policies and procedures. The overall format of the questionnaire was also redesigned, to create obligatory sections that could be responded to by both professional and academic staff participants and optional sections that could be responded to by either professional or academic staff, as relevant. This was based on feedback from a range of staff participants in the pilot study, but also directly addressed concerns expressed by a senior professional staff member that items requiring an opinion or value statement regarding the use of IELTS in selection would be “inappropriate” for most professional staff participants given their role was to implement university selection policy and procedures rather than to develop them.

Short interviews were then conducted with staff and students, to refine the process of probing aspects of the data that was collected through means of the questionnaires. Here again, the significant revisions were made to the interview structure, based on feedback about process issues from the participants, the researcher and research assistant. Specifically, both staff and student interview sequences were streamlined, to focus on selected sections in the questionnaires, and on selected items within each section. This format gave a stronger focus to the interviews, without compromising opportunities for participants to provide extended responses and commentaries in a conversational format. At the same time, it also provided a mechanism for managing time effectively within the interview context, particularly for staff participants. The pilot interviews conducted with staff participants also revealed the benefits to be gained from placing relatively junior and/or professional staff participants in paired and/or small-group interview contexts.

The process described above involved considerable time and effort, but yielded participant-friendly and comprehensive instruments for use in the main study. Feedback on the phrasing of items in the Staff Questionnaire was especially valuable, leading to increased clarity and precision in the items eventually included in the main study instruments. Streamlining of the interview process also created a clear and sustainable focus of issues related to the use of IELTS in selection.

4.3.2 Main study

The main study began in September 2005 and continued through until early December 2005. Recruitment of participants took place in two broad phases. First, a list of the administrative staff (known as ‘professional staff’ at this university) most directly involved in the selection of international students at the university in general and in the Faculty of Economics and Commerce was compiled. Associate deans, heads of departments and program directors were added to this list. An invitation to participate in the research project was then sent to each person on this list, via email. Those individuals who responded in the affirmative were sent a staff participant package which included a

cover letter, plain language statement, consent form, Staff Questionnaire and a return envelope. Eleven professional and nine academic staff completed the questionnaire.

Staff Interviews were conducted following initial analysis of the Staff Questionnaires. Interviews were conducted by the principal researcher and the research assistant with a sample of the staff participants. Six professional and four academic staff took part in the interviews. These participants were selected following initial analysis of the data obtained via the questionnaires in order to reflect a representative range of roles and responsibilities within the selection process. The sample included academic and professional staff of the university and faculty, at different levels of responsibility and seniority. Some interviews were conducted on an individual basis and others in the context of a small group, according to the nature and level of the participants’ roles in the selection and/or teaching and supervision of international students. Academic and senior professional staff were interviewed individually while more junior professional staff were interviewed in groups. Both the researcher and the research assistant were present at all interviews.

As previously outlined, the interviews were planned using the staff interview template and the particular questionnaire responses of selected participants. Initially, participants were shown their completed questionnaires to refresh their memories and to provide a starting point for the interview. The interviews themselves allowed opportunity for both clarification and extension of these responses. The researcher played the role of interviewer allowing the research assistant to take detailed field notes while the interviews were being conducted. Audio-tape recordings were made of the interviews for subsequent analysis.

Two senior academic members of the university, who were experts in the area of student selection, were also interviewed. These interviews were conducted to obtain a broad historical background on the use of IELTS in selection at the University of Melbourne, and to seek information on the status of IELTS as a measure of English language proficiency within the university. The experts were interviewed individually, a tape-recording of each interview was made and detailed field notes were taken by the research assistant.

Student participants were recruited by a number of means. In the first instance, several academic staff participants allowed the research assistant to attend scheduled undergraduate and postgraduate lectures and to present a brief outline of the project to the students in attendance. Student participant packages were then left in the lecture theatre/classroom for interested volunteers to collect. These packages contained a cover letter, plain language statement, consent form, Student Questionnaire and a return envelope. The research assistant also attended two economics and commerce tutorials held at a residential college of the university with the permission of senior staff. A similar procedure to the on-campus context was followed. In addition, the research assistant met with the co-ordinator of the university’s overseas students’ association to explain the project. An item regarding the project was published in their newsletter. A small number of student participant packages were held at the reception desk of the association for collection by interested volunteers. Ten undergraduate and 10 postgraduate international students completed the questionnaire.

Student interviews were conducted in a similar vein to the staff interviews. The participants were selected to include both undergraduate and postgraduate students enrolled in the faculty. A total of five postgraduate and five undergraduate students participated in this stage of the study. They were selected on the basis of their questionnaire responses to ensure a good representation of the 20 respondents. The interviews themselves were built around participants’ questionnaire responses, using a student interview schedule template. As in the staff interviews, the interviews allowed opportunity for both clarification and extension of participants’ questionnaire responses, particularly in relation to their English language capacity on entry to courses, the level of language demand in those courses, and the language support provided for students during their courses. Most interviews were conducted on an individual basis, according to participant availability and level of study. As in the staff

interviews, participants were shown their completed questionnaire responses to refresh their memories and to provide a starting point for the interviews. Field notes were taken during the interviews which were also audio-taped for subsequent analysis.

4.4 Participant information

Eleven professional and nine academic staff completed the Staff Questionnaire. The first section of the questionnaire elicited relevant bio-data about individual respondents and their role in international student selection. The analysis revealed that all of the participating professional staff were directly involved in the selection of international students and most had more than one year of experience in this work. Several, particularly those at senior administrative levels, had five or more years’ experience. Only four of the academic staff were directly involved in selection as part of their role as director of academic programs within the faculty. The other five academic staff included staff who lectured in specific undergraduate subjects or who worked in the faculty’s Teaching and Learning Unit.

Relevant information about the 20 students from the Faculty of Economics and Commerce is summarised in Table 1 below. All but one of the 20 student participants indicated that they had used their most recent IELTS test scores in their application for their current course within the Faculty of Economics and Commerce. The single exception indicated that she had used her most recent IELTS scores to gain admission into the undergraduate program but was now studying at postgraduate level in the faculty (cf Q11, Student Questionnaire).

Table 1: Student participants - background data

4.5 Methods of analysis

Firstly, the university selection policy and procedure documents were examined to build up a clear understanding of the policy and procedures related to the English language requirements for undergraduate and postgraduate study in the Faculty of Economics and Commerce, particularly the IELTS entry scores and the place of English language requirements in the selection process. The policy documents relating to the use of IELTS in selecting international students for both undergraduate and postgraduate courses at the University of Melbourne in general and the Faculty of Economics and Commerce in particular can be accessed by following the relevant links on the university website at http://www.unimelb.edu.au. These documents provided authoritative and current information about the university’s policy and procedures on English language requirements.

The Staff and Student Questionnaire data were coded and entered into separate databases. Some of the questionnaire data were categorical (eg sex and country of birth) and others were continuous or ordinal (eg years of involvement in selection or level of proficiency on IELTS). Some of the data was also narrative in character (eg optional further comments on selected questionnaire items). Data pertinent to each item on both the Staff and Student Questionnaires were entered as discrete elements in the relevant databases. The complete breakdown of results for each question is presented in Appendices 5 and 6.

Initially, running summaries of themes, issues and ideas raised in the staff and student interviews were made, and then checked and cross-referenced with field notes taken during the actual interviews. Overall themes and issues raised in each interview were noted for comparison across interviews. The principal researcher and research assistant independently checked the interview summaries against the field notes to maximise the reliability of this process. They then met to discuss and resolve any inconsistencies noted by either of them. The summaries are included as Appendices 7 and 8 respectively. Where appropriate, selected direct quotes by staff and students were transcribed for the purpose of incorporating them in this report.

5 RESULTS

Following analysis of the different data collected, the results were grouped into three broad themes:

1. how IELTS is used to select international students and to plan for their future language learning
2. what knowledge professional and academic staff as well as students have about IELTS, English proficiency and the selection process
3. what beliefs staff and students have about IELTS, English proficiency and the selection process.

5.1 How IELTS is used to select international students and to plan for their future language learning

This section will examine firstly, the English language requirements of both the University of Melbourne in general and the Faculty of Economics and Commerce in particular; secondly, how the use of the IELTS is monitored and evaluated for this purpose; and thirdly, the extent to which IELTS scores are used to plan for future English language learning.

5.1.1 English language requirements

The various kinds of English language evidence accepted by the university for undergraduate and postgraduate study are included at Appendix 9. For local students these include English studies in the Australian and New Zealand Year 12 English subjects and International Baccalaureate. For international students these include IELTS, TOEFL, secondary school English studies in countries where English is an official language and satisfactory completion of either one year within the last two years or two years within the last five years of university study in an institution where English is the language of instruction and assessment.

Given that IELTS is only one of a number of possible ways of satisfying the university’s English language requirements, it was important to investigate firstly, what percentage of students used IELTS results to enter the Faculty of Economics and Commerce compared to the most other popular forms of evidence recognised by the university. Data was provided by the Faculty of Economics and Commerce for this purpose. There were four types of evidence used by international students for undergraduate or postgraduate entry in 2005: IELTS, TOEFL (computer and paper-based), secondary school level English or English as a Second Language and previous study in an English-medium tertiary institution (including Foundation Studies for the undergraduate cohort). The results are summarised in Table 2.

These figures show that IELTS was the most widely used form of evidence of English proficiency, although only marginally so at the undergraduate level. It is noteworthy that a significant number of students satisfy the English language requirements by virtue of having studied in an English-medium institution in Australia or elsewhere of either one year within the last two years or two years within the last five years.

Table 2: Type of English language evidence used to enter the Faculty of Economics and Commerce in 2005

Table 3 shows the minimum IELTS entry scores for the university in general and the Faculty of Economics and Commerce in particular. These are given separately because individual faculties are able to interpret the university guidelines for their own particular needs. Some faculties (but not the one used in this study) even distinguish different IELTS minimum scores for individual courses, particularly at the postgraduate level. Note that applicants must have obtained these scores listed on the Academic module of the IELTS Test within the previous 24 months.

There are a number of important features of the faculty policy to be noted here. Firstly, it appears that a shared sense of the required English standards as well as market forces had a strong influence in the original setting of IELTS entry levels. One of the ‘expert’ academics interviewed underscored this point in relation to the Overall Band Score of 6.5 as the undergraduate entry requirement:

[this] seems to be standard across most Australian universities, so I suspect that…it was the zeitgeist at this particular time…and also of course, market forces…if other universities are saying 6.5…then, I just think everyone considers that it’s reasonable for what you need to do in an undergraduate degree. (Expert Interview 2)

Secondly, while there is a provision for a lower entry at the undergraduate level (an Overall Band Score of 6.0), it had not been used in recent years because of the very high demand for places according to the faculty’s General Manager. Thirdly, when this provision is applied it carries with it a condition that students must undertake an ESL credit subject in their first year of study. This is the only instance in the university’s policy where an IELTS score is used to guide future learning. Lastly, the same IELTS minimum scores are required for all postgraduate courses of study including PhD.

Table 3: Minimum IELTS scores used by the university and the Faculty of Economics and Commerce in 2005

5.1.2 The selection process

As shown in Appendix 10, undergraduate selection is carried out by a team of admissions officers (who are professional staff) in the university’s International Admissions office. Where a student’s secondary level qualifications are well-known to the university, these admissions staff have “delegated authority” to assess the application. Where an undergraduate applicant’s qualifications are not well-known to the university, the application is sent to the relevant faculty, in this instance the Faculty of Economics and Commerce, for assessment by the Undergraduate Selection Committee. As summarised in Appendix 11, the selection process for international students seeking entry into postgraduate courses is the same as the undergraduate one except that all assessment of applications is carried out at the faculty level by the Postgraduate Selection Committee.

In the assessment process for both undergraduate and postgraduate courses, the primary emphasis is on the applicant’s academic qualifications and only secondarily on whether they have adequate evidence of meeting the relevant English language requirements (ie using IELTS scores or other evidence). This stage of the process does not normally require interpretation or judgement on the part of the admissions/ applications officer - applicants are deemed to have either met or not met the academic and English requirements and a recommendation is made. However, it is possible for the officer to be unsure about the status of the applicant’s academic record in which case it would be referred to a more senior member of staff.

The three possible recommendations made at this stage of the selection process are “accept”, “make a conditional offer” or “reject”. The recommendations made by the admissions officers in the International Admissions office or the application officers in the faculty are then checked and “signed off” by a more senior administrative staff in the admissions office or a senior academic staff member representing the relevant program on either the undergraduate or postgraduate faculty selection committee. Applicants are accepted if they meet both the academic and English requirements, made a conditional offer if they do not meet all of the academic prerequisites and/or the English requirements, or rejected outright if they clearly do not meet the academic requirements. Applicants cannot accept a conditional offer until all requirements stipulated in the letter of offer are met. These conditions can include meeting certain academic and/or English language requirements. In the letter of conditional offer applicants are referred to relevant websites for information on how to meet these requirements.

It is worth underscoring at this point that an application cannot be rejected purely on the basis of insufficient evidence of English language proficiency - conditional offers are routinely made to applicants who meet the academic but not the English requirements. Applications are only rejected outright if the applicant’s academic qualifications are clearly inadequate.

While academic staff members may be involved in making judgements about the quality of an applicant’s academic record, particularly at the postgraduate level, they rarely examine their evidence of English language proficiency - it is the university admissions or faculty selection officers who do this checking at the initial stage of the selection process. Moreover, neither professional nor academic staff are normally allowed to vary the English language requirements for any given application. This ensures a consistency of approach. The only exception, according to the faculty’s professional manager, might be for a research higher degree applicant with an outstanding academic record who just fails to meet the English requirements.

One anomaly noted during this study was that the assessment checklist used by International Admissions for undergraduate applications only specified “an overall band score of 6.5 or more” without any reference to the minimum requirement for Writing which is 6.0. When queried, one of the admissions managers suggested that this additional requirement was common knowledge among the admissions officers although at the same time conceded this was an omission that needed to be rectified. The problem with this omission is that the Individual Band requirements are likely to be checked less systematically than the Overall Band Score requirement.

The interviews conducted with the two senior academic staff who were experts in selection underscored the fact that selection of international students has always been a problematic issue at the University of Melbourne. Both of these staff indicated that rising numbers of international applicants have put considerable pressure on the university and its faculties to establish appropriate English language proficiency entry requirements. Indeed, Expert#1 stated that the issue of the selection of international students has been:

the most serious, significant and controversial issue that has ever come to Selection Committee, Student Pathways, and will forever remain the…most controversial issue. (Expert Interview 1)

Both experts claimed that matters of most pressing concern are located around the university’s expressed mission of selecting students who are “most likely to succeed” and the readiness of international students to commence studies in English in an Australian university. For example, Expert#1 explained that:

the [Selection] Committee is concerned about English language preparation, both from the point of view of readiness to commence and from the point of view of successful completion. Because we are supposed to be an English language university that teaches and assesses in English and produces graduates who can function well in English. (Expert Interview 1)

Expert#1 also raised the dilemmas created by the multiple pathways that the university allows applicants to use to establish their English language proficiency:

Now, our problem is that we recruit from a wide variety of student pathways, we lack a common indicator…we have standard tests like the IELTS, TOEFL and so on, there is already a question - are all versions of those tests…equivalent or can you benchmark correctly? How do two different tests…benchmark…and how are these related to other major feeder streams such as the IB [International Baccalaureate] and the four different English language subjects offered under the VCE [Victorian Certificate of Education]? (Expert Interview 1)

The issue of comparability of measures used to gauge English language proficiency is fundamentally related to the principle of equal access for applicants to the university:

…given the plethora of pathways…it will always be difficult to ensure that we treat applicants from possible entry streams equitably. (Expert Interview 1)

Expert #1 positioned this concern for equal access against the expressed commitment of the university to select the most able applicants:

The dominant principle of selection here at Melbourne [University] is that we must select from those students who are most likely to succeed in the course. (Expert Interview 1)

In relation to this imperative, Expert#1 explained that the issue of minimum IELTS scores for entry to the university is increasingly overshadowed by concerns about the equivalence (or otherwise) of alternative pathways for establishing English language proficiency. Specifically, he expressed concerns about the particular ‘suitability’ of the ESL subject which forms part of the local secondary school qualification, the Victorian Certificate of Education (VCE), as evidence of English proficiency:

The weakest of all students we have here [at the University] are fee-paying, overseas students [who entered] via VCE ESL, who took… Chinese [language] and are Chinese native speakers. (Expert Interview 1)

Both experts noted that IELTS has become an increasingly popular means of establishing English language proficiency during their tenure within the university.

They both expressed confidence that the policies and procedures regarding the use of IELTS scores for selection were consistently and fairly applied. In relation to the established minimum IELTS scores required for entry to the university, Expert#1 noted that:

I don’t think that there’s a professional belief that 6.5 is a real comfort zone for entry. It’s on the high side by national and international standards, which is I guess, one of the reasons why we keep it. But, I think it is professionally understood that the IELTS scores are imperfect indicators…and 6.5…6.5 is being cautious and ensures that we don’t get too many people who are [going to struggle]. (Expert Interview 1)

Both experts also confirmed that procedures exist for faculties to vary minimum IELTS scores for entry into the university’s programs, and for individual applicants to be granted an English language proficiency waiver. For example, Expert#2 cited the potential for the Faculty of Music to accept applicants who don’t meet the English language proficiency requirements but have outstanding talent in their respective performance fields (Expert Interview 2). Expert#2 also acknowledged that different areas of study should be able to have different entry requirements. He explained that:

…it seems to me that…in Computer Science, or course work anyway…there probably isn’t any need for an IELTS of 7 for postgraduate work. But I think that in the sort of…discipline areas and professional areas…that are linguistically demanding, so I can see why Arts has a 7. (Expert Interview 2)

In relation to this matter, Expert#2 did note that, in his opinion, there appeared to be a move within senior executive levels of the university to ‘free up’ the requirements and/or pathways for establishing English language proficiency entry levels (Expert Interview 2).

In summary, the university has clear policy and procedures around the selection of international students including the use of IELTS scores as the most popular form of evidence of English language proficiency. There is a high level of procedural compliance on the part of staff, particularly the professional staff who carry out the verification of students’ academic qualifications and evidence of English language proficiency. However, concern was expressed by the two academic experts regarding the comparability of the various types of evidence of English language proficiency accepted by the university and the extent to which individuals and faculties might hold unrealistic expectations regarding the IELTS Test and test scores, in terms of predicting or guaranteeing applicants’ success in their studies.

5.1.3 How the use of IELTS is monitored and evaluated

This section looks at how the use of IELTS is monitored and evaluated by the university and/or the Faculty of Economics and Commerce. As Chalhoub-Deville and Turner (2000, p 237) suggest, it is important that test-users carry out local investigations to ensure that the minimum scores on English language tests are appropriate for entry to their academic programs. This relates to the valid and ethical use of test scores. If the entry requirements are set too low then students will be accepted into courses before they are ready. If the entry scores are set too high students may be unreasonably excluded from the courses.

Relevant professional staff indicated by email communication at the conclusion of the study that the faculty had not undertaken formal tracking of international students’ academic performance in relation to either IELTS entry scores or the other main types of English evidence shown in Table 2 in recent years. However, the use of an English Screening Test developed by the Language Testing Research Centre at the University of Melbourne in 2002 revealed that international students who had completed secondary school in Victoria, including the VCE ESL subject, performed significantly lower in the first semester of their undergraduate studies than other international students in the Faculty of Economics and Commerce (Hughes, 2002). The report concludes that:

It is by no means obvious that all students who barely meet the criterion relevant to their application have equivalent competency [in English]. (Hughes 2002, p 65)

The report recommends that the minimum score for VCE ESL be increased until “suitable changes …are made to improve their relevance to selection into university and performance into university” (Hughes 2002, p 10).

The important point for this study is that there have been no other studies conducted in recent years which have monitored or evaluated the various forms of English evidence accepted for entry to the faculty including IELTS.

5.1.4 How IELTS scores are used to guide future learning

A further dimension of the use of IELTS relates to whether and how it is used to guide students’ future learning. In response to both questions 16 and 19 of the student questionnaire (Were you required by the University to successfully complete additional English studies before you started your course? and Has the University required you to complete English language subjects since commencing your course?), none of the participants reported that they had been required to complete additional language studies before or during their course as a result of their IELTS entry scores.

The interview with staff from the faculty’s Teaching and Learning Unit (TLU) indicated that neither IELTS scores nor other evidence of English language proficiency provided by students for entry to the university is used to guide future English language learning. Students are either referred to the unit by their lecturers and tutors or else self-select for the support provided there. They can also attend the university’s Language and Learning Skills Unit (LLSU) which offers diagnostic testing and additional language support to both individuals and groups of students.

The IELTS Handbook recommends that students who have obtained an Overall Band Score of 6.5 (or less) and who are admitted to “linguistically demanding” courses should undertake further English study. Most of the faculty’s courses would be in this category. Currently, university policy only requires faculties to follow this guideline for students who are admitted with an Overall Band Score of 6.0 or less, not 6.5 (see Table 2 above). As noted above, this is the only instance in the university’s policy where an IELTS score is used to guide future English language learning.

Academic Expert#1 explained that he would prefer to see the university move to a “two strand entry arrangement” where students with lower than minimum English language proficiency standards were allowed to enrol in programs, particularly at postgraduate level, provided they agreed to undertake further diagnostic testing and “ESL remediation (sic)” (Expert Interview 1). He noted that this would only be feasible in programs where there was the opportunity for elective studies, and thus where there was the capacity to “remediate for credit” (Expert Interview 1). He emphasised that:

taking in students who were fragile and then expecting them, either by compulsion or election, to undertake additional work, while they remain struggling…is untenable.

(Expert Interview 1)

5.2 Knowledge about IELTS, English proficiency and the selection process

This section reports on how well staff and students understand the nature of English proficiency, IELTS and its use in selection. Because of the potential for loss of face for participants, this understanding was investigated through a mixture of self-assessment and direct questions about the use of the Test.

5.2.1 Staff knowledge

Section B of the Staff Questionnaire examined how staff rated their own knowledge of particular aspects of IELTS, English proficiency and the selection process as follows:

Respondents were required to rate their knowledge about each topic as either Extensive, Good, Limited or None. Table 4 below shows the results for this section for the 20 respondents in terms of raw frequencies.

Table 4: Frequency distribution of responses, Staff Questionnaire, Items 1-11

In relation to Question 6 the University’s English language entry requirements, 17 out of the 20 participants indicated that they thought they had a ‘good’ or ‘extensive’ level of knowledge of the university’s English language proficiency requirements. Overall, professional staff participants rated their knowledge as ‘good’ or ‘extensive’ more consistently than academic staff participants. Only one professional staff participant compared to two academic staff participants rated their knowledge of the university’s English language proficiency requirements as ‘low’ or ‘none’.

The pattern observed in relation to Question 6 is also evident in the participants’ rating of their knowledge of the use of IELTS scores in the selection of international students (Question 7), the specific IELTS scores that are set for entry into the University (Question 9) and IELTS scores that are set for entry into the Faculty (Question 10). Here the ratings of ‘good’ or ‘extensive’ were recorded by 16, 15 and 17 of the participants respectively. Again, professional staff participants tended to rate their own knowledge as ‘good’ or ‘extensive’ more often than did the academic staff participants.

On the other hand, only nine respondents rated their knowledge as ‘good’ or ‘extensive’ for how the IELTS Overall Band Score is calculated (Question 8) and only 12 rated their knowledge as ‘good’ or ‘extensive’ for other evidence that can be used to satisfy English language proficiency requirements (Question 11).

Overall, the results for Section B of the questionnaire suggested that staff rated their knowledge of the university’s English language proficiency requirements quite strongly. In general, professional staff rated their knowledge more highly than their academic colleagues.

In Section C of the Staff Questionnaire respondents were asked a series of questions which more directly tested their knowledge of how the IELTS was used in selection. These questions were as follows:

As indicated above, respondents were required to answer Yes, No, or Unsure for each question and were also given the opportunity to explain their answers. Despite the relatively strong self-assessments about selection matters indicated by the participants’ responses to Section B, the data in Section C revealed some considerable levels of uncertainty and/or inaccuracy with respect to the specific aspects of the use of IELTS scores in selection, particularly on the part of academic staff. The summary of results, in terms of frequencies, is shown in Table 5 below.

Table 5: Frequency distribution of responses, Staff Questionnaire, Items 12-20

For Question 12 (Are the minimum level IELTS Band Scores and/or Individual Band Scores required for entry into the Faculty different for undergraduate and postgraduate students?), six participants recorded a ‘Yes’ response (the correct response), seven recorded a ‘No’, six recorded ‘Unsure’ and one did not respond (see Table 5). In the main, professional staff participants recorded a ‘Yes’ response; academic staff participants recorded a ‘No’ or ‘Unsure’ response.

Similarly, in response to Q13 (Are applicants with lower than the minimum required IELTS Overall Band Score sometimes accepted into undergraduate programs in the Faculty?), eight participants recorded ‘Yes’ (the correct response), four recorded ‘No’, seven recorded ‘Unsure’ and one did not respond. More professional staff participants recorded an ‘Unsure’ response to this question; academic staff participants recorded a very mixed range of ‘Yes’, ‘No’ and ‘Unsure’ responses.

In response to Q14 (Are applicants with lower than the minimum required IELTS Overall Band Score sometimes accepted into postgraduate programs in the Faculty?), the majority of participants (11 out of 20) recorded ‘No’ (the correct response). Only two participants (one professional staff and one academic staff) recorded ‘Yes’, but seven participants (four professional staff and three academic staff) indicated that they were unsure.

The pattern of mixed responses (see Table 5 above) is also evident in relation to:

• Question 15 (Are applicants with lower than minimum required IELTS Individual Band Scores sometimes accepted into undergraduate programs in the faculty?
• Question 17 (Are applicants admitted to the faculty with scores lower than the minimum required IELTS usually required to enrol in additional English language credit subjects?)
• Question 20 (Is the IELTS test more commonly used than other recognised measures of English language proficiency by applicants for entry into Faculty courses?).

In all three cases the correct response is ‘Yes’.

For Question 16 (Are applicants with lower than the minimum required IELTS Individual Band Scores sometimes accepted into postgraduate programs in the Faculty?), more than half of the participants recorded ‘No’ (the correct response). However, seven participants indicated that they were unsure. Similarly, more than half of the participants recorded ‘Yes’ (the correct response) to Question 18

(Is the date of the applicant’s last IELTS test taken into account in the selection process?) but five indicated that they were unsure. All of these were academic staff participants.

In response to Question 19 (Is the particular module of the IELTS test - academic or professional training - which an applicant has taken, checked in the selection process?), nine participants recorded ‘Yes’ (the correct response), but eight participants recorded ‘Unsure’. These participants included both professional and academic staff.

Analysis of the Staff Interview data revealed that both professional and academic staff interviewed held a sound understanding of the ‘rules’ and ‘processes’ of selection and, in particular, the need to confirm applicants’ English language proficiency. Professional staff participants were particularly explicit and precise in their discussion of these matters, often making clear and direct references to university policy documents and guidelines. They attributed their knowledge of IELTS (including the minimum requirements for admission to the university or faculty) to first-hand experiences as a result of working as admissions or selection officers, information provided in university policy documents and guidelines (web and paper-based) related to the selection of international students and information and advice provided by other colleagues within the university. For example, professional staff participant P10 explained:

Well, we have a Policies and Procedures Guide on Selection…and our IELTS scores and other English language scores are approved by our academics and go through our Faculty Board for final approval, and that’s all initially written down…it’s all written down, and as part of the Selection process, you need to go through the Policies and Procedures [documents] and learn as much as you can on what you have to look for, where you have to look for it, etc…It’s all there. You don’t make up anything. (Staff Interview 4)

Similarly, professional staff participant P4 explained:

My knowledge started just with…just through…being told. And since then, I have been able to access the web and the Selection Procedures and like in the Regulations and stuff like that. I’ve learnt where they are, whatever. We have it documented in our office, it’s on the internet, it’s on our webpage… (Staff Interview 4)

All professional staff and academic staff participants interviewed argued that there is general consistency in the application of university and faculty rules for selection of international students, with regards to the meeting of English language entry requirements in general and the minimum IELTS scores in particular. In regard to these matters, professional staff participants commonly described themselves and colleagues as operating on a ‘need to know’ principle. As professional staff participant P3 explained:

We’re not required to know specifically how the test is conducted, we just need to know the score. (Staff Interview 4)

Despite this, many participants also indicated that they were interested in learning more about IELTS and the consequences of setting different entry levels for different courses.

Related to the ‘need to know’ principle, most of the professional and academic staff participants indicated that they were unsure about how the IELTS Overall Band Score was calculated (as revealed in the Staff Questionnaires), and what Individual Band Scores actually meant in terms of an applicant’s English language proficiency. As professional staff participant P1 explained when asked to elaborate on why she felt unsure about the meaning of particular IELTS scores:

I think the information [about IELTS scores] is there, and I have actually looked at the [IELTS] website and the reports they send out, but it’s not something I have needed to know, so I am not filling up my brain with it. (Staff Interview 2)

Professional staff participant P5 also noted that:

I think people understand what the requirements are, but I don’t think they understand what the score actually means, in real terms. (Staff Interview 2)

Overall, there was considerable variability in the responses given by the staff about the use of IELTS scores in selecting international students in Section C of the Staff Questionnaire with academic staff expressing greater uncertainty than professional staff about detailed aspects of the faculty’s practices. This is probably no surprise as the professional staff deal more directly with these issues than academic staff in selection. The issue of staff knowledge about IELTS is taken up further in Section 6: Discussion.

5.2.2 Student knowledge

Analysis of the responses to Question 12 of the Student Questionnaire (Please list your most recent IELTS Overall and Individual Band Scores) reveals a range of Overall and Individual Band Scores amongst the participants. These results are summarised in Table 6 below. These results indicate a fairly wide range of proficiency levels when students sat their IELTS Tests. Note that there was one student who incorrectly reported gaining 6.5 on the Speaking sub-test (0.5 increments were not assigned on the Speaking or Writing sub-tests before 1 July 2007).

Table 6: Student participant responses to Question 12, Student Questionnaire, (Please list your most recent IELTS Overall and Individual Band Scores)

Data supplied by students in relation to Question 14 of the Student Questionnaire (What are the minimum IELTS scores required for entry into your course?) indicated variable knowledge about the test. Only nine of the 20 participants gave the correct Overall and Writing Band requirements for entry to their relevant level of study (undergraduate or postgraduate), eight indicated that they were unsure about this issue, two gave incomplete responses and one participant recorded higher than required minimum scores. Two participants gave incomplete responses, recording either just an Overall Band Score or the Writing Band Score.

5.3 Beliefs about IELTS, English proficiency and the selection process

The data on staff and students’ participants’ beliefs was gathered by seeking their opinions on various issues related to IELTS, English proficiency and the selection process. It is important to stress at this point that their responses and comments need to be considered cautiously as they tended to be based on anecdotal information and did not always reflect strong understandings of the Test, in particular its purpose, strengths and limitations. As academic staff participant, P19, suggested:

I think there are a lot of myths out there about…language testing.

One important myth was the mistaken assumption, apparent among staff particularly, that all international students had undertaken IELTS to enter the university. While this was true for a slim majority of international students entering the Faculty of Economics and Commerce in 2005, many students also entered using TOEFL (computer and paper-based) scores, secondary level English and previous study in an English-medium tertiary institution (see Table 2 above). This assumption seems to have led some academic and professional staff into ‘scapegoating’ the IELTS Test when they perceive students’ English proficiency to be inadequate.

5.3.1 Staff beliefs

In Section D (Questions 21-30) of the Staff Questionnaire respondents were asked to give their opinions on a range of issues about the use of IELTS as follows:

This section was designated optional on the questionnaire. Some professional staff had indicated they would be unwilling to complete it when shown an earlier draft as their opinions about the Test were not relevant to their work.

As shown in Table 7 below the majority of the participants were willing to complete the section. Only four participants (two professional staff and two academic staff) declined. The academic staff indicated on their questionnaires that since they had little if any direct responsibilities in selection, they did not feel able to express opinions. One other academic staff recorded an ‘unsure’ response for every question which may have indicated a similar reluctance.

Table 7: Frequency distribution, Staff Questionnaire, Items 21-30

In terms of the responses to specific statements in Section D, the data revealed an equally divided response pattern on certain questions. For example, in response to Question 21 (I believe that the current IELTS entry levels are adequate for students entering the Faculty’s program), six participants recorded ‘Yes’, six recorded ‘no’ and four recorded ‘Unsure’. This trend is also evident in the participants’ responses to Question 23 (I believe that some undergraduate courses of study should require higher IELTS scores than others) with five ‘Yes’, six ‘No’ and four ‘Unsure’ responses, Question 24 (I believe that some postgraduate courses of study should require higher IELTS scores than others) with six ‘Yes’, five ‘No’ and five ‘Unsure’ responses and Question 25 (I believe that postgraduate courses of study should require higher IELTS scores than undergraduate courses) with seven ‘Yes’, five ‘No’ and three ‘Unsure’ responses.

However, Questions 22, 26, 27, 28, 29 and 30 yielded much stronger consensus among the participants. With respect to Question 22 (I believe that the IELTS scores provide accurate evidence about an applicant’s English language proficiency), 10 of the 16 respondents indicated that they disagreed with the statement. In response to Question 26 (I believe that an applicant’s IELTS scores should be considered in relation to factors such as age, motivation and language learning history), nine of the 15 respondents disagreed with the statement. This is interesting because the IELTS Handbook (2005) recommends that these factors should be taken into account in the selection process. In the case of Question 27 (I believe that a selection interview would be a useful addition to IELTS scores, if resourced properly), 12 of the 16 respondents agreed with the statement. There were 11 ‘No’ responses to Question 28 (I believe that IELTS scores are good predictors of academic success). As discussed earlier in this report the ‘No’ response is strongly supported by research conducted into the predictive validity of the Test. However, it is unclear whether this response suggests that these respondents assume that IELTS should have high predictive power and are critical of the Test because it doesn’t. In the case of Question 29 (I believe that all staff involved in selection have a good understanding of IELTS test scores), 10 of the 16 respondents agreed with the statement.

Finally, in response to Question 30 (I believe that an applicant’s English language proficiency is as important as their academic record in making selection decisions), 11 of the 16 respondents recorded a ‘Yes’ response.

Analysis of the additional comments made by a small number of participants in the other optional section of the Staff Questionnaire (Section E) revealed that they held considerable reservations about the relationship between what applicants need to do and/or demonstrate in the IELTS testing situation, what the applicants can actually do and/or demonstrate in terms of English language proficiency, and the capacity of applicants who achieve minimum IELTS levels to meet the English language demands of studying at an English-medium university. In total, eight participants - five professional staff and three academic staff - chose to make additional comments in this section. A sample of these comments, two from professional staff participants and one from an academic staff participant, are reproduced below:

P5 (Professional Staff): We have concerns that while some applicants meet our IELTS requirements this does not prepare them for the difficulties of comprehending the language in the classroom and they are going to struggle.

P10 (Professional Staff): In the [postgraduate] programs we have had students enter the program with IELTS of 7, but their English understanding is very poor. I believe that English proficiency is extremely important in obtaining good results. I also believe that prospective students can take advantage of schemes that gain them good results in the tests, but they have not actually obtained these results honestly.

P16 (Academic Staff): I am very confident that our Faculty adheres to the protocols (for English proficiency assessment). However as a teacher of postgraduate students I observe some international students with what I would regard as ‘poor’ oral and written skills. This observation then begs the question: Are the IELTS scores used for selection poor indicators of English proficiency or is the written evidence tendered by applicants for their English proficiency ‘misleading’?

In the staff interviews a number of participants raised questions about the accuracy of IELTS, and about the minimum IELTS levels currently set for entry into the university at particular levels of study as well as for different programs or courses. In particular, concerns were expressed about the way in which IELTS might be administered and/or scored in different contexts or countries. Participants also indicated that there are widely held concerns about the ‘actual’ and ‘theoretical’ level of some international students’ English language proficiency, at least at the beginning of their courses. For example, in elaborating on her opinions about the accuracy of students’ IELTS scores (cf Q21 Staff Questionnaire), one professional staff participant (P7) commented:

Well, if they [the students] score 6 on Speaking and in a conversation with you they cannot…really speak, then the score must be inaccurate. (Staff Interview 2)

Professional staff participant P5 expressed further concerns about the accuracy of IELTS scores, and the possible variations in how IELTS Tests were conducted:

From experience, you’ll sit down with someone whose scores are exactly the same [as another student’s] and he’ll talk to you and he’ll have no idea what I’m saying and that’s why you have this issue that it [the score] doesn’t provide much evidence about how the test is run, who ran it, who marked it, a whole range of questions we have. (Staff Interview 2)

Later in the same interview, this participant claimed that “without any doubt” students with the same IELTS results were presenting with very different levels of English language proficiency in their interactions with faculty staff (Staff Interview 2). He emphasised that this was “not a trivial matter”, but a very important issue for the Faculty of Economics and Commerce, raising questions about the reliability of students’ IELTS results. These concerns were echoed by academic staff participants. For example, academic staff participant P15 commented that:

I note very little variance in the [IELTS] scores…very few applicants do better than four 6s and a 6.5. So, I mean, that…and that surprises me…and would make an anecdotal observation that people with the same scores…display huge differences in language once they get here. (Staff Interview 3)

Another academic staff participant, P20, noted that she was:

…not sure what [the IELTS scores] tell us about their ability to cope with academic type language requirements. (Staff Interview 5)

One staff participant (P19) raised the possibility that students’ results on their IELTS Test might be attributable in some way to coaching:

There’s anecdotal [evidence] - I don’t know whether it’s true or not - that there’s a lot of support or coaching for IELTS. (Staff Interview 5)

Another participant reported that:

[Students] come and say to me “Oh yeah, the IELTS is easy, it’s easy to get a good [score] because I knew what I was doing but now I’m here, ahhh, I can’t cope”. (Staff Interview 5)

Despite these reservations, both professional and academic staff participants generally expressed more faith in IELTS as a measure of English language proficiency, than other currently acceptable pathways. For example, one academic staff participant, P15, explained:

I am the selection officer for the Master of International Business, one of the faculty’s oldest postgraduate professional programs, and one element of selection is that students should provide evidence of competency in English language. So…once upon a time there used to be TOEFL…we’ve given up on that…and use IELTS. (Staff Interview 3)

However, this apparent lack of confidence in TOEFL may be at least partly due to its declining popularity and even availability in recent years as the same participant later pointed out:

I have in the back of my head that this faculty insists on IELTS, and I would see hundreds of applications…in the years…I have seen thousands over time, you know…it’s been an awful long time since I’ve seen a TOEFL one. (Staff Interview 3)

Responses to Questions 23 and 24 of the Staff Questionnaire revealed mixed views among respondents about whether the IELTS entry levels are adequate for the various courses at both undergraduate and postgraduate level offered by the Faculty of Economics and Commerce. Many of the participants in the staff interviews commented that tensions existed between university and/or faculty marketing agendas and desired and/or expected standards of English language proficiency. In speaking of the levels of support needed by “admittedly a handful” of international students, professional staff participant P3 explained that:

You don’t want them to fail…especially…they’re paying so much. (Staff Interview 4)

Professional staff participants who identified themselves as having active roles in the marketing initiatives of their faculty and/or the university as a whole were particularly explicit and expansive on this matter.

Both professional and academic staff participants consistently referred to perceived variations in the language demands associated with different courses. For example, staff participant P15 explained that his particular postgraduate program, the Master of International Business, was “language intensive”, and he observed that international students tended to “stay away from” such programs (Staff Interview 3). He argued that international students with lower levels of English language proficiency are drawn to the “less language intensive” programs such as “the Master of Applied Finance and the Master of Accounting and Economics [as] the universal language of economics is maths”. (Staff Interview 3)

This participant argued that for these and other programs, the current IELTS levels were sufficient, but that for language intensive programs, “where there are complex sets of ideas and complex sets of languages” the IELTS entry requirements need to be higher. (Staff Interview 3)

Other participants suggested that clearer distinctions could be made between courses where students can be expected to improve their English language proficiency over the duration of their study and those where little if any significant language improvement could be expected. One academic staff participant, P20, also explained that students themselves are sometimes surprised to find that they need to keep developing their English language skills during their studies. She explained:

The students themselves, they come in thinking, “Well I got in, so why are you pushing me to do more of anything?” (Staff Interview 5)

A number of the participants who commented on this matter drew particular attention to the significance of written language skills in success at university, and to the level of the challenges faced by international students in this area, particularly at postgraduate levels of study. Professional staff participant P4, for example, remarked that higher IELTS minimum scores for postgraduate study should be required, particularly where students will be involved in:

…more research-based and thesis writing and all that sort of stuff…I just think that if you don’t have a particularly strong grasp of English writing then it’s automatically going to disadvantage you…even if your content is good. (Staff Interview 4)

Professional staff participant P10 reiterated this point, noting that in her experience:

The English language students who have the most difficulty are the ones that only have the 6 and the 6.5. (Staff Interview 4)

Some academic staff participants echoed these concerns. Participant P15, for example, commented that he “admired these students” for their bravery in attempting a postgraduate higher degree program in a second language. (Staff Interview 3)

With regard to the selection process itself, most professional staff expressed discomfort with the idea that they might be responsible for taking other factors, such as an applicant’s motivation to succeed and/or their language learning ability, into account when determining an applicant’s English language proficiency specifically on the possible consideration of issues (cf Q26 Staff Questionnaire).

Professional staff participant P5 suggested that this:

…would open the floodgates for arguments about [student entry]…and who are the selection officers to make those sort of judgements on somebody’s motivation or their language learning ability? (Staff Interview 2)

Professional staff participants were less supportive than academic staff participants regarding the possible inclusion of the use of additional selection interviews (cf Q 27 Staff Questionnaire) not only because of the problem of resourcing them but also of assessing them. Professional staff participant P1 was most articulate about this matter, but her comment reflects the reactions of other professional staff participants:

I think that’s the nice thing about the IELTS. Whether it’s wrong or right, you’ve got to be…black and white, you’re in or you’re out. (Staff Interview 2)

A number of academic staff participants observed that the IELTS Test might be viewed as being able to do more that it actually can (cf Q 28 Staff Questionnaire). They reported anecdotal evidence that an IELTS score is often regarded by teaching staff as the single most reliable indicator of the likelihood of an international student’s capacity to succeed in his/her studies. Both academic selection ‘experts’ endorsed this claim. Expert #2 was particularly explicit about this, criticising:

…the unthinking reverence with which [IELTS scores] are treated…it just continues to amaze me when…everyone…on [the university’s] Selection Procedures Committee keep on being told…that it [IELTS] doesn’t have this kind of predictive significance that they think it has. (Expert Interview 2)

Some participants revealed that they held different views of language, and that this ‘coloured’ their opinions in relation to IELTS and the overall selection processes. One common view expressed was that language is ‘just a tool’, and that international students, who often bring novel and exciting ideas and perspectives to their studies can excel, irrespective of their language limitations. One academic staff participant, P17, noted that:

We hear a lot of noise about all the overseas students, their English, blah blah blah… Certainly I think the mind, the thought, the idea, the logical thinking, are more important than language itself. I think we judge students, especially overseas students, on the expression of language too much… rather than following their logical thinking, the ideas… Scholarly development is not just involve (sic) language. Language is a tool. (Staff Interview 1)

This participant went on to comment that:

Overseas students are fantastic! Their ideas are so wonderful… Our local students is (sic) boring! Very narrowly focused. (Staff Interview 1)

Another academic staff participant, P19, explained that:

I totally agree that postgraduate study requires higher order thinking, I am not sure that it requires higher order language skills. (Staff Interview 5)

The two academic staff both thought that the current IELTS entry levels could be too high.

In contrast, a very small number of participants expressed commitment to the idea of an intimate link between language (discourse) and cognition (conceptualisation), arguing that international students often struggled with the intellectual or substantive demands of their studies as a direct result of language limitations. One academic staff participant, P15, noted that:

[Many international students, commonly those from mainland China,] can write a ‘nice essay’, say at school level, but [lack] the ability to think and reason at higher levels… if your language doesn’t have those words and there isn’t a book to find substitutes, you can’t go to the dictionary…it’s at a more profound level when you come to advanced reasoning. (Staff Interview 3)

This participant also raised the significance of social knowledge and differing world views in meeting the demands of studying in language intensive postgraduate programs such as his own, and questioned the capacity of the IELTS Test to provide evidence of students’ proficiency in these areas (Staff Interview 3).

A very small number of participants interviewed also commented on the English language difficulties experienced by local students. These participants commented that, while international students who meet minimum IELTS requirements for entry into the university might experience subsequent language difficulties in the course of their studies, they are not the only students who ‘struggle’.

Staff participants from the Faculty of Economics and Commerce Teaching and Learning Unit (P19 and P20) indicated that quite large numbers of local students contact the unit for help with English language issues (Staff Interview 5). Further, these and other participants commented that in their experience, international students’ levels of language proficiency were not the only factor implicated in their success as students of the university. Staff participant P19 also noted that while quite a number of international students appear before progress committees, they are not over-represented in university or faculty (Staff Interview 5). She noted that:

My experience of this, the student progress [committee], is that students we are seeing there, [the problem] is not language, it’s not working…it’s a lot more issues…and different styles of teaching and learning. (Staff Interview 5)

To sum up, staff agreed on some issues and not on others in the sections which cued their beliefs in the Staff Questionnaire and Interviews. It was interesting to note that staff were evenly divided in their questionnaire responses on whether the IELTS scores should generally be higher for university entry but that a clear majority of them did not believe the IELTS scores provided accurate (in the senses of valid and/or reliable) evidence about students’ language proficiency or that they were good predictors of academic success. There also seemed to be particular concern about the “trustworthiness” of the IELTS results in the sense of whether students’ had obtained their results honestly or not. However, the interview data indicated that staff believed IELTS to be more accurate than the other measures of English proficiency. The issue of staff beliefs will be taken up further in the Section 6: Discussion.

5.3.2 Student beliefs

Student participants generally had a positive view of how accurately their IELTS scores captured their English proficiency.

In response to Question 13 (Do you think all of your IELTS results were accurate?), 16 of the 20 student participants answered in the affirmative, with four participants indicating that they were unsure about this matter (see Table 8 below).

Table 8: Frequencies of responses to Question 13, Student Questionnaire (Do you think all your IELTS scores were accurate?)

Five student participants took the opportunity to provide a descriptive comment in response to this question. In all five cases, the comments appeared to elaborate on or justify the participant’s response. For example, student participant SP1 recorded ‘Yes’ and then commented:

I think they were accurate in the sense that they reflected my performance on that day. My performance was affected not only by my English skill but also other factors such as anxiety on the day.

Similarly, student participant SP 18 recorded ‘Yes’ and then explained:

I guess they were relatively accurate. My grasp of the English language is not fantastic but not too bad either.

By comparison, student participant SP9 recorded an ‘Unsure’ response and then commented:

I am not sure cause (sic) I thought my writing and reading was quite good. The score didn’t come out as I expected.

In response to Question 15 (Do you think the IELTS scores required for entry into your course are appropriate?) most participants (16) indicated that they thought the IELTS scores required for entry into their course were appropriate (see Table 9 below). Three participants reported that they thought they were not appropriate and one reported being unsure about this matter.

Table 9: Frequencies of responses to Question 15, Student Questionnaire (Do you think the IELTS scores required for entry into your course are appropriate?)

In response to Question 17 (At the start of your course, did you think your English was good enough to succeed in your studies?), 15 of the student participants answered ‘Yes’ (see Table 10 below). Two participants indicated that they did not, and three indicated that they were unsure about this.

Table 10. Frequencies of responses to Question 17, Student Questionnaire (At the start of your course, did you think your English was good enough to succeed in your studies?)

These responses suggest that some students (particularly those who had only just satisfied the IELTS entry requirements) may have had unrealistic expectations about the adequacy of their English skills for successful completion of their courses. However, in response to Question 18 (Do you expect your English to improve while you are completing your course?), the overwhelming majority of students believed that their English proficiency would improve during their course (see Table 11 below).

Table 11: Frequencies of responses to Question 18 (Do you expect your English to improve while you are completing your course?)

None of the respondents had been required to complete additional ESL subjects since starting their course (Question 19. Has the University required you to complete English language subjects since commencing your course?) Yet, as shown in Table 12 below, most participants indicated feeling that they needed other English language support during their studies in response to Question 20 (Have you felt that you needed other English language support this year?).

Table 12: Frequencies of responses to Question 20, Student Questionnaire (Have you felt that you needed other English language support this year?)

In response to Question 21 (Have you been given any additional English language support this year?), only 10 participants (five undergraduate and five postgraduate) reported that they had sought support from the university’s Language and Learning Skills Unit (LLSU) or the faculty’s Teaching and Learning Unit (TLU). Specifically, they reported seeking help with writing assignments, grammar, listening, speaking, presentation and writing. For example, student participant SP4 commented that she had sought help with:

writing my assignments: the LLSU and TLU have helped me a lot guiding me how to write good academic assignments.

Similarly, student participant SP1 explained that

it helps me with my writing assignments, just to check the grammar, not to change the idea.

Student participant SP7 explained that he had sought help with:

listening, public speaking, presentation and writing.

Taken together, the responses to Questions 17, 18, 20 and 21 suggest that as their course continued most students had gained a more realistic view of their English proficiency and the need for support to improve it and to successfully complete their studies. However, only half of the respondents had actually accessed the available services for these purposes.

In the final section of the Student Questionnaire (Section E), eight participants took the opportunity to make additional, mostly critical, comments on matters to do with the use of IELTS in selection. Many of these participants reported concerns regarding the “trustworthiness” of IELTS results. For example, student participant SP3 commented:

IELTS is a good tool in helping the unis to make decisions on the basis of students English capacity. However, there is a problem. I believe the test is manipulated. Some candidates prepared very well by memorising some part exam answers without really acquiring the knowledge. Especially spoken test. I still remember that I took one four years ago when I was still in China and didn’t speak much English. My English tutor gave me some spoken text topics and gave me some very well written answers to let me recite. It turned out to be effective and I got 6. After three years undergraduate study in Australia, I found my English was improved and I am quite confident with talking to locals and giving presentations in front of class. I took another IELTS for the purpose of migration application. Of course I was too confident to prepare any spoken topics and it turned out that I got the same score (6). I had a really good conversation with the examiner, but I believe the problem was I didn’t follow the structure of the test which was because I didn’t recite the fixed answers.

In a similar vein, student participant SP4 explained that:

In my opinion the IELTS test is a good tool to know your level of English. However, I would like to know why in Colombia the people says that if you take the exam here in Australia is better because you can have higher scores than in Colombia. It shouldn’t be the same criteria to mark the test everywhere around the world?

Student participant SP9 expressed his concern as follows:

I think IELTS is misleading sometimes. I can’t say why. Cause people with decent scores sometimes don’t show as much aptitude in the application and understanding of the language. Is there something wrong with the test or something wrong with the students? Maybe exam cram helps them pass IELTS, but leaves no lasting impression on their collective understanding of the language.

Several student participants spoke of being ‘drilled’ in practice IELTS test items. As postgraduate student participant SP2 explained:

…sometimes I think the test is not conducted so well, mmm…some place in some countries, sometimes…like for me, you are just shown and how to pass the test. (Student Interview 6)

These participants suggested that this might be a widespread practice in particular areas or countries, and felt that it accounted for some of the language difficulties that they and others experienced in their first weeks and months of study. One postgraduate student participant, SP4, also raised questions about “getting different results if you take the test in different places” (Student Interview 7). A few student participants also suggested that their results (and those of people they knew) depended on how they felt on the day of their test, especially with respect to the Speaking and Listening components. For example, student participant SP19 “was quite nervous” about taking the Speaking test (Student Interview 1) and found the Listening test quite hard to do:

The place where I sat for our test was quite noisy, like there were cars outside, airplane, and then there was one part where I didn’t listen, and…I couldn’t answer. (Student Interview 1)

Some participants expressed their concern that the university and faculty applied IELTS scores too inflexibly in selection. For example, student participant, SP20, commented that:

IELTS as a requirement for course entry should be considered leniently. For example: students with Band 5.5 or 6 should also be considered to enter their course provided they can prove later that they fulfil the English language requirement by taking equivalent tests.

Some participants also recorded their concerns about the relationship between the difficulty of tasks they had completed successfully on the IELTS Test and the more complex language demands of studying at university. For example, one participant who had achieved high IELTS scores, SP18, commented that:

I feel frustrated sometimes because I have many ideas but I do not know how to express myself. Sometimes I feel that what I have written in my essays could have been expressed better or more eloquently. I used to be a little self-conscious when speaking in class or in any other situation because I was afraid that people would not understand my accent.

The student interviews indicated that most participants believed their IELTS test results to be accurate (cf Q13 of the Student Questionnaire). Student participant S18, an undergraduate participant, spoke about her confidence in the IELTS Tests and results:

Well, now I have a good idea of what I am writing in English, what I can say and do. (Student Interview 4)

Other student participants found that their scores reflected their own self-assessment and knowledge of their individual English language proficiency. For example, one of the postgraduate student participants, SP1, explained that:

…for me, I think, I know my English skills, not accurately, but I know because, can I communicate well, can people understand me, can I write something and can people read and understand what I write…so…and I think, it’s not bad, it’s quite good. So when my…the result…gave me sort of good results, I thought well… “Yes!” (Student Interview 3)

Similarly, postgraduate student participant SP4 commented:

Well, you know what you can do [after taking the test]. I speak English in and use English in many places, so I can…use…make my own personal judgement. (Student Interview 7)

One undergraduate student participant, SP16, explained that she “was just pleased to get the minimum requirements” to get into the university (Student Interview 5).

A few student participants indicated that they were surprised by their IELTS scores. As one postgraduate student participant SP19 explained:

I didn’t think my English was that good [laughter] ‘cos I sort of speak in Malaysian more… I went to New Zealand for exchange last year. Maybe I learned a lot last year. (Student Interview 1)

Some student participants raised questions about whether the minimum scores set for university entry are too high (cf Q 15 Student Questionnaire). For example, undergraduate student participant SP20 indicated that:

I passed the tests and got the scores I needed, but I didn’t feel that I had done a great effort or well. Then when I started uni, it didn’t seem to matter ‘cos I could have done the work without my scores anyway. I could have got lower scores and still did the work. (Student Interview 2)

One postgraduate student participant, SP3, also argued that:

…the test doesn’t show all the aspects…other things can be important in the student’s success in their studies as well as their English. (Student Interview 6)

In contrast, another postgraduate student participant, SP5, commented that:

IELTS is quite good in illustrating what you will [be] facing in the real [university] situation. (Student Interview 3)

Some student participants noted that they didn’t expect their English to improve significantly and/or at all during their studies (cf Q 18 Student Questionnaire). These were most often participants who also noted that they did not mix with English-speaking peers or participate in tutorials and lectures.

As student participant SP19 explained:

I still mix with my friends from Asia. I think that’s the main problem. You speak our own language. (Student Interview 1)

By comparison, a number of student participants did indicate that they fully expected their English to improve as a result of mixing with English-speaking peers and/or as a function of the amount of reading and writing they would do or had already done in their courses. Postgraduate student participant SP5 explained that:

[I] expected that, because, during my course I am reading quite a lot, so, I notice that…when I am doing postgraduate and reading a lot more…information…it will help me to you know get more knowledge, and also living in Melbourne and having to communicate with people, it will improve my communication skills in English. (Student Interview 3)

In the main, ideas similar to this were expressed by other postgraduate participants, but some undergraduate participants also noted that this would be a “natural thing” (SP18 in Student Interview 4).

Many student participants indicated that they would have liked more help with English, especially assignment-related written English, during the course (cf Qs 20 and 21 Student Questionnaire). For example, undergraduate student participant SP19 indicated that she would have liked more assistance with both language and content matters:

Maybe before we hand in the assignment, proofread for us…overall presentation…not [just] the language problem but the material… (Student Interview 1)

Another postgraduate student participant, SP6, explained that:

…I have got help with spelling and grammar…from the Teaching and Learning Unit. I also needed help with overall structure and academic expression. I would like more help in more assignments. (Student Interview 8)

In almost all cases where student participants indicated that they had sought help with their English, they had received assistance from either the Language and Learning Skills Unit (LLSU) of the university or the Teaching and Learning Unit (TLU) of the Faculty of Economics and Commerce or both. One participant, SP1, spoke at length about the need to “find the right person” in these places (Student Interview 3):

…not everybody in TLU can help you…I tried three [people]…lucky I found the third was good, so I thought, ‘OK, I’ll just stick with you’. (Student Interview 3)

However, some student participants explained that they didn’t really need the support of the LLSU or the TLU. As postgraduate student participant SP2 explained, she “felt confident to work alone” (Student Interview 6). Undergraduate student participant SP20 also explained that:

…I could have got more help doing assignments, you know, spelling and grammar, writing reports and essays the way you have to, organising what you say…Still, I am doing ok, so maybe I didn’t need it. (Student Interview 2)

While acknowledging the support provided for them, a number of undergraduate and postgraduate student participants also noted that they thought it was more important to have strong content background and understanding than to be “perfect” in English spelling, grammar and punctuation (Student Interview 1). Some participants indicated that their lecturers and tutors also gave them this message. Other student participants considered that initial and ongoing success in their studies had more to do with social and cultural issues than language proficiency issues.

Overall, the questionnaire and interview data gathered from students indicated they had a more positive view of the IELTS than did the staff in terms of the Test’s capacity to provide accurate information about their English proficiency and the appropriateness of the scores set for entry by the university. However, like the staff, there were concerns about the reliability and “trustworthiness” of test results in some instances. There were also a range of views among these students about how important English language competence was to academic success and the best way to improve their social and academic proficiency.

6 DISCUSSION

This section relates the results to the central research question posed at the outset of this report:

To what extent are IELTS test scores used in valid and ethical ways for the purposes of university selection?

6.1 Messick’s (1989) unified model of validity

Most discussions of “validity” and “ethicality” in recent times are grounded in Messick’s unified model of test validity (1989) summarised in Table 13 below. Of particular relevance to this study are the second, third and fourth cells, ie the evidential basis of test use, the consequential basis of test interpretation and the consequential basis of test use.

Table 13: Facets of validity (Messick, 1989)

The second cell stresses the need for test scores to be relevant and useful in the particular testing context. The third cell, the value implications of test interpretation, refers to the fact that the interpretation of test scores is not value-free and that test scores are “read” and acted upon by various test stakeholders in different ways. As Messick (1995, p 748) suggests:

The value implications of score interpretation are not only part of score meaning, but a socially relevant part that often triggers score-based actions and serves to link the construct measured to questions of applied practice and social policy…value implications are not ancillary but rather integral to score meaning.

As McNamara (2006) suggests, this cell has received comparatively little attention compared to the fourth cell of Messick’s (1989) progressive matrix, the social consequences of test use. This focus on the fourth cell has led to two different responses from the language testing community. The first view maintains that language testing practice can be made ethical and stresses the individual responsibility of test developers to ensure that it is. The other view argues that tests are basically sociopolitical constructs which “are designed as instruments of power and control” and “must therefore be subjected to the same kind of critique as are all other political structures in society” (McNamara 2006, p 43). In both cases the foci of analysis are the test and, more particularly, the test developer or test agency which provides it.

6.2 Ethical language testing, accountability and the test user

A very neglected stakeholder in discussions around responsibility in language testing, especially in “ethical language testing”, is the local test user who is strongly implicated in ensuring that test scores are relevant and useful in the given context, determining what actions flow from the interpretation of scores and even influencing test impact. This leads to the notion of accountability which, to date, has been mainly restricted to the unidirectional sense of responsibility that the test developer and/or test agency has to the test-taker and the test user. However, as suggested above, the test user is also responsible to both the test-taker and test developer/agency at various levels. Indeed, the American Educational Research Association’s “Standards for educational and psychological testing” suggest that “the ultimate responsibility for appropriate test use and interpretation rest lies predominantly with the test user” (AERA et al, 1999, p 112). Referring to this excerpt Chalhoub-Deville and Turner (2000, p 537) argue that:

Test-users need to be cognisant of the properties of the instruments they employ and ensure appropriate interpretation and use of test scores provided. Test-users need to carry out local investigations to make sure that their admission requirements are based on an informed analysis of their academic programs and the language ability score profiles necessary to succeed in these programs.

The International Test Commission (ITC 2000) has also published guidelines on test use.

The guidelines suggest that “competent test users will…interpret results appropriately, communicate the results clearly and accurately to relevant others and review the appropriateness of the test and its use”. (TTC 2000, p 1)

Finally, the language testing profession is in the process of developing a code of practice following the establishment of its own code of ethics. Section E “Responsibilities of users of test results” of the Draft Code of Practice (Version 3) produced by the International Language Testing Association (ILTA) stipulates that:

Persons who utilise test results for decision making must:

1. Use results from a test that is sufficiently reliable and valid to allow fair decisions to be made.
2. Make certain that the test construct is relevant to the decision to be made.
3. Clearly understand the limitations of the test results on which they will base their decision.
4. Take into consideration the standard error of measurement (SEM) of the device that provides the data for their decision.
5. Be prepared to explain and provide evidence of the fairness and accuracy of their decision-making process. (ILTA 2005, p 4)

The main focus on the ensuing discussion is on the University of Melbourne in general and the Faculty of Economics and Commerce in particular as users of the IELTS.

6.3 The selection of international students

The findings in this study indicate that two principal sources of evidence are examined in the selection of international students at the University of Melbourne: the applicant’s academic record and evidence of English language proficiency. English language proficiency is positioned as less important than the applicant’s academic record. If the applicant meets the relevant academic but not the English requirements then they are routinely made a “conditional offer”. This means that their acceptance into the course they have applied for is contingent upon them meeting the English language requirements. If they do not meet the faculty’s academic requirements then their application is normally rejected irrespective of whether the English language requirements have been met. However, IELTS scores in this study were not generally used to guide future English language learning, the underlying assumption being that if applicants meet the faculty’s English language requirements then they should be able to succeed in their studies without being required to undertake additional English studies. Students do have access to the faculty’s Teaching and Learning Unit and the university’s Language and Learning Unit but attendance is voluntary and the assistance students receive may be of a one-off nature targeted to a particular assessment requirement, for example.

Contrary to the recommendations made by the various professional bodies above, the Faculty of Economics and Commerce has not monitored or evaluated its IELTS requirements at all in recent years. The lack of such empirical evidence is likely to hamper any increase in understanding of the Test in the future among key policy makers and jeopardises the integrity of the university’s use of the Test.

The selection process, including the verification of English language proficiency, is generally managed and administered entirely by professional staff. There is a high level of adherence to the minimum English language requirements stipulated for undergraduate and postgraduate study among these staff although there seems to be less careful checking of the required Individual as opposed to Overall Band Scores. More importantly, there is no flexibility in how these requirements are applied to each individual applicant. This approach certainly ensures consistency but is arbitrary in its approach. English language proficiency scores including IELTS are assumed to be definitive sources of evidence about a student’s language ability and readiness to undertake study in an English-medium institution. There is no account taken of the standard error or measurement associated with these scores or of factors which might guide their interpretation. The IELTS Handbook (2005, p 5), for example, suggests that factors such as age, motivation and language learning history be taken into account when interpreting IELTS scores. Such an approach is probably unworkable in the current system which simply calls for clear and unambiguous “objective” decisions requiring low levels of knowledge about English language proficiency scores and academic qualifications. This view was endorsed by the majority of staff participants in the study who were opposed to the notion of considering such factors in interpreting test scores. The danger of more complex procedures requiring “subjective” judgement is that they could be, or be perceived to be, unfair, especially if admissions and selection officers are not sufficiently trained in making decisions based on multiple factors.

The approach to the selection of international students at the University of Melbourne described here is not universal. For example, the selection of postgraduate international students at Lancaster university, meticulously documented in Banerjee’s (2003) study, is radically different to the one described here. There the selection of international students is a complex, holistic decision-making process primarily based on the “subjective” recommendation of an informed academic staff to the university’s senior postgraduate admissions officer. The range of criteria taken into account is extremely rich including the applicant’s academic background, intellectual capacity, evidence of English language proficiency (IELTS or other recognised measure), work experience, the applicant’s own argued case for selection, reports from academic and work referees, personal characteristics (such as motivation, age and adaptability) and, in some instances, a follow-up telephone interview. Such factors can be either “indicators of success” or “sources of struggle” (Banjeree 2003, pp 236-237). Other factors influencing their decisions include the offer-acceptance ratio, recommendations from other academic colleagues, the reports of agents and scholarship agencies. Successful applicants are described as either a “clear accept”, “safe bet” or “risk” and appropriate recommendations are made to students in the second and third categories about undertaking additional English study or seeking additional academic assistance. Banjeree’s findings (2003, p 235) indicated that if a risk is taken, it is more likely to be taken with language proficiency scores than with an applicant’s academic or professional background.

Although complex and time-consuming, the selection process Banjeree describes ensures that no one single factor such as academic record or English language proficiency scores dominates the selection process. In the sense that they are interpreted flexibly and balanced against other “indicators of success” it could be argued that English proficiency test scores are used more validly and ethically in her context. However, Banerjee (2003, p 235) warns that:

…making admissions decisions is far from straightforward. It involves balancing sometimes conflicting evidence and making a judgement. Each feature of the application is seen to interact differently with others. Consequently, it is not simply a case of applying a ‘tick-list’ to an application and only admitting students whose applications meet the stipulated criteria. Instead, the criteria represent fixed parameters and the applicants offer varying evidence of their suitability for their chosen course, often not quite meeting one criterion while exceeding another.

Moreover, the degree to which it could be reliably and practically implemented with large numbers of applicants is questionable. In theory, it should have a highly positive washback effect in so far as academic staff would need to be well informed about a wide range of criteria relevant to selection, including English proficiency test scores. Yet Banerjee (2003, p 245) suggests that:

Neither of the approaches to selection, including the use of English proficiency test scores at the two different universities seems to be ideal, one being overly arbitrary, “objective” and reductive and the other unwieldy, excessively “subjective” and over-inclusive. There is clearly a need to strike a balance between these two extremes.

6.4 Knowledge and beliefs about English language proficiency and IELTS

The findings in this study show that IELTS is the most common form of English language evidence used by students to apply for places in the Faculty of Economics and Commerce at both undergraduate and postgraduate levels. It is also the most common point of reference for academic and administrative staff when discussing the English proficiency of international students. However, as indicated in Sections 5.3 and 5.4 above, this is not always underpinned by a sound knowledge of the nature, purpose and predictive power of proficiency tests in general, and the IELTS Test in particular, on the part of either staff or students.

While staff rated their knowledge about IELTS quite highly, their responses to detailed questions about the use of the Test in selection indicated a high level of variability in their actual understanding, with academic staff expressing more uncertainty than professional staff. This is hardly surprising given that it is the professional staff who check the evidence of English language proficiency in course applications. The important question here is “how much knowledge is sufficient?” Within the current system, appeals to the “need to know” principle, whereby staff only acquire the knowledge they need to perform their work, seem reasonable. This is particularly true for professional staff who check English language evidence provided they are not asked to make more “flexible” decisions about applicants. Yet, these same staff may also be required to advise students about how to meet the requirements in terms of courses (including test preparation) and the Test. In this instance, they do require additional knowledge about the Test. Moreover, for academic staff engaged in formulating selection policy, the level of knowledge about IELTS and other measures of English proficiency used by the university definitely needs to be higher. Even the two senior academic staff who were “experts” in selection did not display a particularly strong level of understanding of English language proficiency, IELTS scores or their implications for future English learning.

In the absence of a sound knowledge base it appears that a set of “folkloric” beliefs has grown up among professional and academic staff around English language proficiency and the IELTS Test, some with a firmer basis in reality than others. These beliefs include scepticism about the validity, reliability and “trustworthiness” of IELTS scores and an unrealistic expectation about their power to predict academic success. There seems to be a widespread misconception that achieving the specified minimum IELTS score implies that students’ English proficiency is sufficient to successfully complete rather than commence their courses. For some staff the Test appears to serve as a scapegoat for their dissatisfaction with the English proficiency of international students more generally, based on the assumption that all of these students have satisfied the university’s English requirements with IELTS scores. However, the fact that staff were equally divided on whether the IELTS entry scores should be higher or not suggests that this dissatisfaction is not shared universally. One of the academic “experts” on international student selection construed English language development in purely “deficit” terms in his consistent reference to the need for English language “remediation” among students with lower than expected proficiency. The implication seems to be that proficiency is something to be corrected rather than developed.

Notwithstanding their greater firsthand knowledge of the Test as test-takers, students displayed variable levels of knowledge about IELTS requirements for entry to courses in the questionnaires and interviews. They did indicate a more positive view of IELTS than the staff in terms of the Test’s capacity to provide accurate information about their English proficiency and the appropriateness of the scores set for entry by the university. However, like the staff, there were concerns about the reliability and “trustworthiness” of test results in some instances. There was also a range of views among these students about whether there should be greater flexibility in the interpretation of IELTS test scores, how important English language competence was to academic success and the best way to improve their social and academic proficiency.

6.5 The relationship between IELTS and other recognised measures of English language proficiency

IELTS scores feature in university policy as the most prominent and widely referenced measure of the English proficiency required for entry to faculties and their courses. Because of this, and the fact that most students applying to the university use IELTS scores to satisfy its English language requirements, the Test is the most widely discussed measure of English proficiency eliciting both praise and, more often, criticism. Certainly its popularity seems to have contributed to a lack of knowledge about other accepted measures of English proficiency among both staff and students. This, however, does not appear to have adversely affected the recognition of these other measures, at least in the university’s selection policy and procedures.

The most important issue related to the use of different measures of English language proficiency in the selection process is the complete lack of empirically established equivalence between IELTS and other accepted measures of English proficiency in the university. This problem has been exacerbated in recent years as some faculties have varied their IELTS entry requirements for different courses, particularly at postgraduate level. Apart from the TOEFL for which the university has developed “equivalencies” to IELTS scores, the minimum requirements of the other forms of evidence of English language proficiency (such as performance in VCE ESL, Foundation Studies or academic programs in other English-medium institutions) have not been adjusted in line with these changes. This issue will certainly need to be addressed in future.

7 CONCLUSION

This study has examined the use of IELTS in the selection of international students in one faculty of a large Australian university. Clearly, there is a need for other such studies to be conducted so that this issue can be critically examined in the various institutional contexts in which IELTS is employed around the world. Such research is warranted in the interests of identifying valid and ethical selection procedures to inform policy and practice in this area. In methodological terms, this study has demonstrated the importance of carefully piloting data collection instruments, such as questionnaire and interview schedules, which have been developed to investigate complex and sensitive issues relating to the use of the IELTS for high-stakes purposes. It is recommended that future studies incorporate careful trialling of such instruments.

Individual institutions are then responsible for ensuring that students are selected on a rational, transparent and equitable basis. The IELTS test partners need to ensure that universities are adequately informed about the meaning of the test scores and for advising them about their potential use in the selection process. To this end, it is suggested that they play a more active role in designing face-to-face or multi-media based training programs for key academic and administrative personnel who are engaged in the interpretation and use of the Test. Perhaps test users such as universities and governments, also need to be held more accountable to the IELTS test partners, and in so doing, to test-takers, for the ways in which test scores are interpreted, used and monitored. How such a system might work could be the subject of a new research study.

ACKNOWLEDGEMENTS

I am grateful to IELTS Australia for funding this study and to the staff and students of the University of Melbourne who participated in the project. I also gratefully acknowledge Dr Merle Iles for her assistance with the design of the questionnaires and interview schedules, the data collection and data analyses.

References

• AERA, APA and NCME, 1999, Standards for educational and psychological testing, American Educational Research Association, Washington, DC
• Banerjee, JV, 2003, Interpreting and using proficiency test scores, unpublished PhD thesis, Department of Linguistics and Modern English Language, Lancaster University
• Broadstock, H, 1994, The predictive validity of the IELTS and TOEFL: a comparison, unpublished Masters thesis, Department of Applied Linguistics and Language Studies, University of Melbourne
• Chalhoub-Deville, M and Turner, C, 2000, ‘What to look for in ESL admission tests: Cambridge certificate exams, IELTS and TOEFL’ in System, vol 28, pp 523-539
• Coleman, D, Starfield S and Hagan, A, 2003, ‘The attitudes of IELTS stakeholders: student and staff perceptions of IELTS in Australia, UK and Chinese tertiary institutions’ in IELTS Research Reports, Volume 5, IELTS Australia Pty Ltd, Canberra, pp 160-207
• Coley, M, 1999, ‘The English language entry requirements of Australian universities for students of non-English speaking background’ in Higher Education Research & Development vol 18, no 1, pp 7-17
• Cotton, F and Conrow F, 1998, ‘An investigation into the predictive validity of IELTS amongst a group of international students studying at the University of Tasmania’ in IELTS Research Reports, Volume 1, IELTS Australia Pty Ltd, Canberra, pp 72-115
• Deakin, G, 1997, ‘IELTS in context: issues in EAP for overseas students’ in EA Journal, vol 15, no 2, pp 7-15
• Dooey, P and Oliver R, 2002, ‘An investigation into the predictive validity of the IELTS test as an indicator of future academic success’ in Prospect, vol 17, no 1, pp 36-53
• Elder, C and O’Loughlin, K, 2003, ‘Investigating the relationship between intensive English language instruction and band score gain on IELTS’ in IELTS Research Reports, Volume 4, IELTS Australia Pty Ltd, Canberra, pp 153-206
• Ellerington, K and Bayliss, A, 2004, ‘Keeping the customer satisfied: maintaining a balance between expectations and experience for postgraduate NESB students’ in TESOL in Context, vol 13, no 2, pp 9-15
• Hamp-Lyons, L, 1997, ‘Washback, impact and validity: ethical concerns’ in Language Testing, vol 14, no 3, pp 295-303
• Hill, K, Storch, N and Lynch, B, 1999, ‘A comparison of IELTS and TOEFL as predictors of academic success’ in IELTS Research Reports, Volume 2, IELTS Australia Pty Ltd, Canberra, pp 52-63
• Hughes, BD, 2002, The English screening test for economics and commerce, report presented to the University of Melbourne Selection Procedures Committee, October
• IELTS, 2005, IELTS Handbook, co-produced by the University of Cambridge ESOL Examinations, the British Council and IELTS Australia Pty Ltd
• IELTS, 2002, IELTS Annual Review, co-produced by the University of Cambridge ESOL Examinations, the British Council and IELTS Australia Pty Ltd
• International Language Testing Association, 2005, Draft code of practice: version 3, accessed 18 June 2006 from http://www.ilttaonline.com/code.htm
• International Test Commission, 2000, International guidelines for test use, accessed 18 June 2006 from http://www.intestcom.org/
• Kerstjen, M and Nery, C, 2000, ‘Predictive validity in the IELTS test’ in IELTS Research Reports, Volume 3, IELTS Australia Pty Ltd, Canberra, pp 85-108
• McDowell, C and MerryLees, B, 1998, ‘Survey of receiving institutions’ use and attitude to IELTS’ in IELTS Research Reports, Volume 1, IELTS Australia Pty Ltd, Canberra, pp 116-139
• McNamara, T, 1997, ‘Policy and social considerations in language assessment’ in Annual Review of Applied Linguistics volume 18, ed B Grabe, pp 304-319
• McNamara, T, 2001, ‘Language assessment as social practice: challenges for research’ in Language Testing, vol 18, no 4, pp 333-349
• McNamara, T, 2006, ‘Validity in language testing: the challenge of Sam Messick’s legacy’ in Language Assessment Quarterly, vol 3, no 1, pp 31-51
• Merriam, SB, 1988, Case study research in education: a qualitative approach, Joosey-Bass, San Francisco
• Messick, S., 1989, ‘Validity’ in Educational measurement, 3rd edition, ed RL Linn, American Council on Education/McMillan, New York, pp 13-103
• Messick, S., 1994, ‘The interplay of evidence and consequences in the validation of performance assessments’ in Educational Researcher, vol 23, pp 13-23
• Messick, S, 1995, ‘Validity of psychological assessment’ in American Psychologist, vol 50, no 8, pp 741-749
• Messick, S, 1996, ‘Validity and washback in language testing’ in Language Testing, vol 13, pp 241-56
• Pennycook, A, 2001, Critical applied linguistics, Lawrence Erlbaum, Mahwah, NJ
• Rea-Dickins, P, Kiely R and Guoxing Yu, 2007, ‘Student Identity, Learning and Progression (SILP): The affective and academic impact of IELTS on ‘successful’ candidates’ in IELTS Research Reports, Volume 7, IELTS Australia Pty and British Council, Canberra, 2007, pp 59-136
• Rees, J, 1999, ‘Counting the cost of international assessment: Why universities may need to get a second opinion’ in Assessment and Evaluation in Higher Education, vol 24, no 4, pp 427-438
• Taylor, L, 2004, ‘Testing times: research directions and issues for Cambridge ESOL examinations’ in TESOL Quarterly, vol 38, no 1, pp 141-145
• Shohamy, E, 2001, The power of tests, Longman, London
• Stake, RE, 1994, ‘Case studies’ in Handbook of qualitative research, eds NK Denzin and YS Lincoln, Sage, London
• Yin, R, 1989, Case study research, 2nd edition, Sage Publications, Beverly Hills, CA

APPENDIX 1: STAFF QUESTIONNAIRE

IELTS RESEARCH PROJECT

Questionnaire: Staff

Office Use Only

Date administered: ____

SECTION A: In this section, we would like some information about you and your role in international student selection, so that we can better understand the other information you provide. Please note that we require your name so that we can contact you for further information if necessary.

Please write your responses in the spaces provided.

1. Your title (Miss Mrs Mr Dr etc) and full name.
2. Your position in the University or Faculty.
3. Your role(s) in the University and/or Faculty international student selection process.
4. If you are a Professional Staff member, how many years have you been involved in international student selection?
5. If you are an Academic Staff member, what responsibilities do you have in relation to international students?

SECTION B: In this section, we are interested in how you rate your knowledge of English language proficiency requirements for admission of international students to the University and the Faculty.

Please indicate the extent of your knowledge about the following topics by circling one of the response options.

SECTION C: In this section, we are seeking information about the use of IELTS scores in selecting prospective international students into programs in the Faculty of Economics and Commerce.

Please respond to the following questions by circling one of the response options, and then providing an explanatory comment.

12. Are the minimum level IELTS Overall Band Scores and/or Individual Band Scores required for entry into the Faculty different for undergraduate and postgraduate students? Yes No Unsure

Please explain:

13. Are applicants with lower than the minimum required IELTS Overall Band Score sometimes accepted into undergraduate programs in the Faculty? Yes No Unsure

Please explain:

SECTION D (OPTIONAL): In this section we are seeking your opinions about the use of the IELTS. If you believe your opinions are not relevant to your role in the University, please go straight to SECTION E.

Please circle the option that best matches your response to the statement.

SECTION E: In conclusion, we would welcome any other comments about the use of the IELTS test and other measures of English language proficiency.

THANK YOU FOR YOUR ASSISTANCE

APPENDIX 2: STUDENT QUESTIONNAIRE

IELTS RESEARCH PROJECT

Questionnaire: Students

Office Use Only

Date administered: ____

SECTION A: In this section, we are interested in finding out about you.

Please write your responses in the spaces provided.

1. Family name: ____
2. Given name: ____
3. Sex:
4. Date of birth: ____
5. Country of birth: ____
6. Nationality: ____
7. First language: ____
8. Other languages spoken: ____
9. Current program and year level: ____
10. Date of most recent IELTS test: ____

SECTION B: In this section we are interested in finding out about your English language abilities at the time you were accepted into your course at the University.

Please circle the appropriate response option or write your response in the spaces provided.

11. Did you use your most recent IELTS results in your application for admission to the Faculty? Yes No

12. Please list your most recent IELTS Overall and Individual Band scores.

Overall score:

Listening:

Reading:

Writing:

Speaking:

13. Do you think all of your IELTS results were accurate? Yes No Unsure Please explain:

14. What are the minimum IELTS scores required for entry to your course? Unsure Overall score: Writing:

15. Do you think the IELTS scores required for entry into your course are appropriate? Yes No Unsure

16. Were you required by the University to successfully complete additional English language studies before you started your course? If ‘yes’, list the English language studies you completed. Yes No Unsure

SECTION C: In this section we are interested in finding out about how you are coping with your current course.

Please circle the appropriate response option or write your response in the spaces provided.

17. At the start of the year did you think your English was good enough to succeed in your studies? Yes No Unsure
18. Do you expect your English to improve while you are completing your course? Yes No Unsure
19. Has the University required you to complete English language subjects this year? Yes No Unsure

If ‘yes’,

a) what English language subject or subjects?

b) how have these subjects helped you with your English (e.g., listening and taking notes in lectures, reading, writing your assignments)?

20. Have you felt that you needed other English language support to cope with your course this year (e.g. help with your assignments)? Yes No Unsure

21. Have you been given any additional English language support this year? Yes No Unsure

If ‘yes’,

a) who thought this would be a good idea? (for example, a lecturer, a tutor, yourself)

b) who gave you the support?

c) how has this support helped you with your English (e.g., listening and taking notes in lectures, reading, writing your assignments)?

SECTION D: In this section we would like you to add any additional comments you have about the issues covered in this questionnaire.

Please make further comments on matters relating to your English language abilities, the IELTS test, your IELTS results, and/or selection into your course and success in your studies.

Thank you for your assistance.

APPENDIX 3: STAFF INTERVIEW SCHEDULE

Interview Schedule: Staff

Participants given own completed questionnaires to peruse.

1. INTRODUCTORY COMMENTS:

Thanks

Purpose of interview

Focus:

(i) your individual questionnaire responses (ii) key issues in the use of IELTS in Selection

Length of interview

Taping

Access to transcripts of interview?

2. CORRECTION/ELABORATION OF INDIVIDUAL RESPONSES

Is there anything in your responses to the items on the questionnaire that you would like to change/correct or elaborate on?

3. POINTS FOR FURTHER DISCUSSION:

(see over)

A. Roles/responsibilities of the range of staff involved in Selection

a. policy making; b. assessing and determining conditional and/or ‘unusual’ entry; monitoring/checking applications and/or documentation; c. ‘front counter’ administration

B. Selection staff’s (self-rated) knowledge of English language proficiency requirements for admission

• levels of knowledge of general University requirements, Faculty requirements,
• any discrepancies between these,
• differences between Overall and Individual Band Scores,
• differences between General Training and Academic Modules,
• other means by which students might meet English language proficiency requirements
• reasons for level of knowledge
• operation of ‘need to know’ principle

C. Opinions about the use of IELTS

• Value of other measures of an applicant’s English language proficiency
• Impact of use of IELTS on use of other measures of English language proficiency
• Impact of use of IELTS on course/subject design, teaching and assessment
• Reliability/validity of IELTS scores
• Suitability of current IELTS requirements

D. Opinions about the use of IELTS

• Value of other measures of an applicant’s English language proficiency
• Impact of use of IELTS on use of other measures of English language proficiency
• Impact of use of IELTS on course/subject design, teaching and assessment
• Reliability/validity of IELTS scores
• Suitability of current IELTS requirements

Other

APPENDIX 4: STUDENT INTERVIEW SCHEDULE

Interview Schedule: Students

Participants given own completed questionnaires to peruse.

1. INTRODUCTORY COMMENTS:

Thanks

Purpose of interview

Focus:

(iii) your individual questionnaire responses (iv) key issues in the use of IELTS in Selection

Length of interview

Taping

Access to transcripts of interview?

2. CORRECTION/ELABORATION OF INDIVIDUAL RESPONSES

Is there anything in your responses to the items on the questionnaire that you would like to change/correct or elaborate on?

3. POINTS FOR FURTHER DISCUSSION:

(see over)

• A. Course/level of study; use of IELTS in application Confirmation of course/year levelConfirmation of use of IELTS in application
• B. Perceptions of your English language abilities at the time you were accepted into your course at the University. Accuracy of your IELTS results Probe participant’s responses to Q’s 13, 14, 15 and 16
• C. English language demands of current course Select from Q’s 17, 18, 19And/orSelect from Q’s 20 and 21
• Other comments

APPENDIX 5: STAFF QUESTIONNAIRE RESULTS

SECTION A: QUESTIONS 1-5

STAFF - SECTION B: QUESTION 6. Knowledge of the University’s English language requirements

SECTION B: Q 7. Knowledge of the use of IELTS test scores in the selection of international students)

STAFF - SECTION B: QUESTION 8. Knowledge of how the IELTS Overall Band Score is calculated.

SECTION B: QUESTION 9. Knowledge of the IELTS scores that are set for entry into the University.

STAFF - SECTION B: QUESTION 10. Knowledge of the IELTS scores that are set for entry into the programs of the Faculty of Economics and Commerce.

SECTION B: QUESTION 11. Knowledge of evidence other than IELTS scores that can be used to satisfy the Faculty’s English language proficiency entry requirements eg TOEFL.

STAFF - SECTION C: Question 12. Are the minimum level IELTS Overall Band Scores and/or Individual Band Scores required for entry into the Faculty different for ugrad and pgrad students?

STAFF - SECTION C: Question 13. Are applicants with lower than the minimum required IELTS Overall Band Score sometimes accepted into undergraduate programs in the faculty?

SECTION C: Question 14. Are applicants with lower than the minimum required IELTS Overall Band Score sometimes accepted into postgraduate programs in the faculty?

STAFF - SECTION C: Question 15. Are applicants with lower than the minimum required IELTS Individual Band Score sometimes accepted into undergraduate programs in the faculty?

SECTION C: Question 16. Are applicants with lower than the minimum required IELTS Individual Band Scores sometimes accepted into postgraduate programs in the faculty?

STAFF - SECTION C: Question 17. Are applicants admitted to the faculty with scores lower than the minimum required IELTS scores sometimes usually required to enrol in additional English language credit subjects?

STAFF - SECTION C: Question 18. Is the date of the applicant’s last IELTS test taken into account in the selection process?

SECTION C: Question 19. Is the particular module of the IELTS test (ie Academic or General Training), which an applicant has taken, checked in the selection process?

STAFF - SECTION C: Question 20. Is the IELTS test more commonly used than other recognised measures of English language proficiency (eg TOEFL) by applicants for entry to Faculty courses?

STAFF - SECTION D (OPTIONAL): Question 21. I believe that the current IELTS entry levels are adequate for students entering the Faculty’s programs.

SECTION D (OPTIONAL): Question 22. I believe that the IELTS scores provide accurate evidence about an applicant’s English language proficiency.

STAFF - SECTION D (OPTIONAL): Question 23. I believe that some undergraduate courses of study should require higher IELTS scores than others.

SECTION D (OPTIONAL): Question 24. I believe that some postgraduate courses of study should require higher IELTS scores than others.

STAFF - SECTION D (OPTIONAL): Question 25. I believe that postgraduate courses of study should require higher IELTS scores than undergraduate courses.

SECTION D (OPTIONAL): Question 26. I believe that an applicant’s IELTS scores should be considered in relation to factors such as age, motivation and language learning history.

STAFF - SECTION D (OPTIONAL): Question 27. I believe that a selection interview (face-to-face or by phone) would be a useful addition to IELTS scores, if resourced adequately.

SECTION D (OPTIONAL): Question 28. I believe that IELTS scores are good predictors of academic success.

STAFF - SECTION D (OPTIONAL): Question 29. I believe that all staff involved in selection have a good understanding of IELTS test scores.

SECTION D (OPTIONAL): Question 30. I believe that an applicant’s English language proficiency is as important as their academic record in making selection decisions.

STAFF - SECTION E: In conclusion, we would welcome any other comments about the use of the IELTS test and other measures of English language proficiency.

APPENDIX 6: STUDENT QUESTIONNAIRE RESULTS

SECTION A: Questions 1-10

STUDENTS - SECTION B: QUESTIONS 11 and 12

11. Did you use your most recent IELTS results in your application for admission to the Faculty?
12. Please list your most recent IELTS Overall and Individual Band Scores.

*Has taken an IELTS recently; completed an IELTS to gain ugrad entry; repeated IELTS for immigration purposes. # Student incorrectly recorded this score.

STUDENTS - SECTION B: QUESTION 13. Do you think all of your IELTS results were accurate?

STUDENTS - SECTION B: QUESTIONS 14-16

14. What are the minimum IELTS scores required for entry to your course?
15. Do you think the IELTS scores required for entry into your course are appropriate?
16. Were you required to successfully complete additional English language studies before you started your course?)

SECTION C: QUESTIONS 17 -20

17. At the start of the year, did you think your English was good enough to succeed in your studies?
18. Do you expect your English to improve while you are completing your course?
19. Has the university required you to complete English language subjects this year?
20. Have you felt that you need other English language support this year?

STUDENTS - SECTION C: QUESTION 21. Have you been given any additional English language support this year?

STUDENTS - SECTION D: Please make any further comments on matters relating to your English language abilities, the IELTS test, your IELTS results, and/or selection into your course and success in your studies.

APPENDIX 7: STAFF INTERVIEW SUMMARIES

Interview 1: Staff Participant P17

Points of note/interest:

• Knowledge/opinions well developed and well defended, with reference to particular examples
• Raised the colonial/hegemonic status of English vis a vis Asian languages
• Commented on the interplay between cognition and language
• Argued that university/faculty needs to use a test that is discipline specific

Interview 2: Staff Participants P1, P5 and P7

Points of note/interest

• Clear distinction between the roles of General Staff Selection Officers and Academic Staff, although some senior/experienced General Staff do hold and are willing to express views about the use of IELTS in selection.
• Significant concerns about the ‘accuracy’ and ‘reliability’ of the test/levels, given the difficulties some international students present with in the early stages of their course.
• Sense of precision and solidarity in the application of the IELTS requirements in the Selection process.

Interview 3: Staff Participant P15

Points of note/interest:

• Clear use of IELTS as a mechanism for sorting ‘able’ from ‘unable’ applicants, BUT concerns about whether IELTS is a reliable instrument: significant tension!
• View of language intensive subjects being equivalent with higher-order thinking/complex reasoning.
• Seems to operating on a lang as a social semiotic platform, and to see the demands of areas of study for international students as a function of their experience in the field of study/language of that field. Not a common view.

Interview 4: Staff Participants P3, P4 and P10

Points of note/interest:

• Concerns about the ‘true’ status of IELTS scores, for at least a small number of students.
• Active acceptance of the distinction between academic and general staff responsibilities in the processing of applications. Clear understanding of ‘what is needed to know to do the job’ of selection.
• Awareness of tensions between marketing and academic quality/standards.
• Identification of the significance of writing in success at university, and of the challenges faced by international students in this area.

Interview 5: Staff Participants P19 and P20

Points of note/interest:

• Interesting view of language.
• Sense that students are quite low in English language proficiency at entry/in first year, but that these are not necessarily international students.

Interview 6: Academic Selection Expert #1

Points of note/interest:

• General comfort with IELTS and IELTS levels.
• Strong commitment to ‘remediation’ of language difficulties, without creating ‘additional study’ load.
• Concern about the validity and reliability of alternative pathways for establishing English language proficiency.

Interview 6: Academic Selection Expert #2

Points of note/interest:

• Confirmation of university and Faculty processes.
• Suggestion of ‘guarded use’ of IELTS as a measure of English language proficiency.

APPENDIX 8: STUDENT INTERVIEW SUMMARIES

Interview 1: Student Participant SP19

Points of note/interest:

• General confidence in the IELTS test as an indicator of English language proficiency. Clear evidence of ‘settling in’ difficulties in university classes and assignments, but nothing major.
• Evidence of the enclave model of international student experiences at ‘foreign’ universities.

Interview 2: Student Participant SP20

Points of note/interest:

• Limited faith in IELTS.
• Not convinced that scores are appropriate
• Sense that the content not the language is the issue in academic success.

Interview 3: Student Participants SP1 and SP5

Points of note/interest:

• Faith in the accuracy of IELTS.
• Level of expectations for language improvements as a result of their study.
• Clear distinction between the relative importance of ‘grammar’ and ‘content’.
• Possible tensions between generic language support and more faculty-focused support.

Interview 4: Student Participant SP18

Points of note/interest:

• Faith in IELTS as a measure of English language proficiency.
• Concern about expression of ideas rather than grammar (related to high IELTS OB and IB scores?)
• Expectation that language would improve during her studies.

Interview 5: Student Participant SP16

Points of note/interest:

• Low level of confidence and proficiency in English.
• Low contact with English-speaking peers.
• Limited involvement in language intensive subjects or components of subjects.
• Enclave model of international student experience.

Interview 6: Student Participants SP2 and SP3

Points of note/interest:

• Doubts about the reliability of the IELTS scores (not the test itself).
• Concern that IELTS be seen as entry level proficiency, not as the sum total of a student’s experiences in learning English.

Interview 7: Student Participant SP4

Points of note/interest:

• Self-awareness of language proficiency.
• Expectations of improvement through course experiences.
• Proactive in seeking support for language learning.

Interview 8: Student Participant SP6

Points of note/interest:

• Faith in IELTS.
• Awareness of variations in subject demands.

APPENDIX 9: ENGLISH LANGUAGE ENTRY REQUIREMENTS

English language entry requirements, University of Melbourne (as at August 2005)

UNDERGRADUATE ENTRY

• Satisfactorily completing secondary studies in a country where English is the official language and gaining a satisfactory pass in an approved final-year English subject. (Applicants from countries with more than one official language may be required to meet TOEFL or IELTS requirements.)
• Obtain a grade in one of the following General Certificate of Education subjects: General Paper (AO Level), or AS Level General Studies, English Language, English Literature, English Language and Literature; which is deemed by the Academic Board to be at least the equivalent of applicants for the course who have completed the VCE.
• Satisfactorily completing the final two years of secondary studies in an approved secondary school with English as the language of instruction and gaining a satisfactory pass in English in the final year.
• In an institution where English is the language of instruction and assessment for the entire institution, satisfactorily completing either:
  • at least the first year of a tertiary course within the last two years;
  • OR
  • at least a two year tertiary degree within the last five years.

Formal advice from the institution to support claim to have satisfied this requirement.

• Obtain the following grade in English:

• Meeting the University’s TOEFL or IELTS (academic test) requirements within the 24 months preceding application:

TOEFL:

Paper based test - a score of 577 or more including a score of 4.5 in the Test of Written English.

Computer based test - a score of 233 or more including an Essay Rating score of 4.5. Internet based test - a score of 90 or more including a Written score of 21.

IELTS:

An overall band score of 6.5 or more in the International English Language Testing System (IELTS), including the completion of the Academic Reading and Academic Writing modules. A minimum score of 6.0 must be achieved in the Academic Writing module.

POSTGRADUATE ENTRY

• Satisfactorily complete secondary studies in a country where English is the official language and gain a pass in final year English (applicants from countries with more than one official language may be required to meet TOEFL or IELTS requirements).

• Satisfactorily complete the final two years of secondary studies in an approved secondary school with English as the medium of instruction and gaining a pass in English in their final year.

• In an institution where English is the language of instruction and assessment for the entire institution, satisfactorily complete either:

  • at least the first year of a tertiary course within the last two years;
  • OR
  • at least a two year tertiary degree within the last five years.

• Obtain a grade in the General Paper, General Studies, English Language, English Literature, English Language and Literature at the General Certificate of Education AO/AS level which is deemed by the Academic Board to be at least the equivalent of applicants for the course who have completed the VCE.

• Obtain a pass of VG (grade 4) in English in the Norwegian Vitnemal or the Swedish Avgangsbetyg.

• Satisfactorily complete additional English Language Studies which, in the opinion of the Selection Committee, is the equivalent of the other tests specified above.

• Meet the University’s TOEFL or IELTS requirements in a test taken no more than 24 months prior to application. Your original TOEFL or IELTS test report form must be included with your application. Certified copies and TOEFL examiner’s score records are not accepted. (The accepted levels for IELTS and TOEFL vary from one faculty to another and are listed in an appended table).

APPENDIX 10: INTERNATIONAL UNDERGRADUATE ADMISSIONS PROCESS FLOWCHART

APPENDIX 11: INTERNATIONAL POSTGRADUATE ADMISSIONS PROCESS FLOWCHART

Author Biodata

Kieran O’Loughlin is Senior Lecturer in TESOL and Assistant Dean (International Programs) in the Faculty of Education at the University of Melbourne. He has many years experience in ELT as a teacher, manager, teacher educator and researcher. He currently teaches a range of subjects on the postgraduate TESOL program and his main research interests include second language assessment, English for Academic Purposes and second language teacher education.