Investigating IELTS exit score gains in higher education

Abstract

A much under-researched issue in English-medium higher education is the extent to which international students for whom English is a second/additional language improve their proficiency in the language during their studies. This report describes a study which examined the improvement made by full-fee paying international students (N = 63) from a large faculty at a major Australian university on the Academic version of IELTS over the duration of their studies. Using official pre- and post-course IELTS results, student questionnaires, and student and staff interviews, the study investigated the rate and nature of the improvement, as well as the educational, personal and social factors influencing this improvement.

Key findings included the following points.

1. The greatest average improvement was in Listening and Reading and the least average improvement was in Writing.
2. The average improvement on Listening, Reading and Writing (but not Speaking) was significantly correlated.
3. Students with lower initial results in Listening, Reading and Writing tended to improve significantly more than students with higher results.
4. Undergraduate students improved more than postgraduate students.
5. The degree of English language support students sought within the university and the degree of contact they had with English outside the university strongly influenced their English language improvement.

These findings and their implications are discussed in detail in the report.

1. BACKGROUND AND RATIONALE

There are several crucial English language proficiency issues for international higher education students who have a first language other than English and who are studying in English-medium contexts. Firstly, there is the question of whether students’ proficiency is adequate to commence university study. Secondly, the issue of whether students require additional English language instruction as part of their course of study. Thirdly, the vexed question of whether students should be required to demonstrate a particular level of proficiency at the end of the course as part of the certification process.

The issue of whether there should be an assessment of English proficiency when students exit a higher education course where the medium of instruction is English has been the subject of some debate in recent years. There are good reasons why such assessments might be useful. Exit tests could be viewed as one measure of accountability to stakeholders in English-medium higher education to demonstrate the level of proficiency attained in English by students in all disciplines. The use of exit tests may lead to positive washback on the academic curriculum in so far as ongoing English language development could assume a higher priority for universities as a result. Exit tests can also provide information to prospective employers about a student’s competence in English. In addition, they provide students with an indication of their overall progress in the language during the time they have spent studying in an English-medium context.

Why, then, has there been an international reluctance to use exit tests to date? The answer is complex. Berry and Lewkowicz (2000), for example, suggest a number of reasons why exit tests have been resisted in Hong Kong, which are mainly related to their possibly negative impact and washback effects. They suggest that exit tests may:

• provide unfair comparative measures of inter-institutional performance as they would not necessarily take into account the differing English entry levels and the kinds of English language support provided both across and within institutions
• have a negative washback effect on the existing curriculum in so far as teachers and students may be overly focused on the exit assessment
• represent a retrograde tendency in the assessment of university study and may set up too strong a barrier to graduation for some students
• not necessarily lead to an upgrading of language competence.

Berry and Lewkowicz believe that language tests may not have been readily accepted by teachers and students for these reasons. While these are all legitimate reasons, they are largely speculative. However, in the same study, Berry and Lewkowicz (2000) surveyed 1418 undergraduate students at the University of Hong Kong to gather students’ views about the desirability of introducing compulsory language assessment prior to graduation, and found that 30% of respondents believed that an exit test should be a requirement for all graduating students. Interestingly, the majority of these students (50%) favoured an international test such as IELTS or TOEFL. The main reasons for this choice were that: firstly, these tests are already internationally recognised which means that employers know their value and can relate to them; secondly, they are fair to all as they are standardised and reliable measures of language proficiency; and thirdly, they can be taken on more than one occasion and students are therefore not bound by a single test date. However, some concern was voiced about ensuring that any assessment used was formally recognised, especially by potential employers, and that students should be offered more optional language courses at university to prepare for the test.

Those against the use of these tests felt that such tests may not be relevant to Hong Kong students and that they may place too much pressure on them to reach the required standard to graduate.

In comparing the various options for an exit assessment of students’ English language competence Berry and Lewkowicz (2000, p 37) note that tests like IELTS were designed for a different context and purpose, ie to assess students’ ability for future academic study in English. In this sense, IELTS fulfils the traditional requirements of a proficiency test. An exit test, on the other hand, is an achievement test - it looks back at what has been learnt. However, as Davies (1992, p 40) points out, this is not a straightforward distinction as these two kinds of tests have overlapping functions. Achievement assessments, for instance, do not always relate to a fixed or defined syllabus. From this perspective, it is quite feasible that a test like IELTS could serve the dual purposes of assessing students’ readiness to undertake higher education as well as their exiting competence.

To date, research on score gains based on the IELTS Test has been limited mostly to studies examining the effect of short, intensive, pre-sessional English programs taken by students preparing to enter university (Brown 1998; Archibald 2001; Read and Hayes 2003; Elder and O’Loughlin 2003; Rao et al 2003). Elder and O’Loughlin (2003), for example, examined the progress made by 112 students in Australia and New Zealand studying pre-university intensive English over 200 to 240 hours of instruction. Students sat an IELTS Test at the beginning and end of this period. They found that the average amount of improvement over this period was 0.5 of an overall band but that there was strong variability among the students, with some making no progress at all. In addition, it was found that the more proficient students had a lower level of improvement on the IELTS Test than the less proficient students. In Green’s study (2005), while a student with a Writing band score of 4 on entry to a two-month pre-sessional course was likely to obtain 5 at the end of the course, a student who obtained a Writing band score of 6 was more likely to obtain a score of 6 again than advance to a 7. This pattern is consistent with the well-recognised effect of regression to the mean in test scores. Green (2005) also notes considerable individual variation in rate of gain.

While these studies have added to our understanding of score gains on IELTS, the important issue of improvement on the Test over longer periods of time, such as from the beginning to the end of higher education courses conducted in English, has not been seriously examined. Therefore, a study such as this is needed, not only to investigate the potential suitability of IELTS as an exit test in higher education, but also because it may shed more light on the amount and type of improvement required to move up from an overall band score of 6.5 to a score of 7 or higher. This is an important question if the Test is to be used to discriminate at these levels in the future. Such discrimination has already started to occur in some courses in higher education in Australia. For instance, international students applying to specialise in Teaching English to Speakers of Other Languages (TESOL) within the Graduate Diploma of Education at The University of Melbourne now require an overall band score of 8 on IELTS. This decision was made in a bid to ensure that all students have adequate English language proficiency to undertake their teaching practicum in local secondary schools. However, more empirical data is needed about the nature of English language proficiency at the more advanced levels (7 and above), how difficult it is to achieve them and what enabling conditions (educational, personal and social) might assist students in reaching them. Elder and O’Loughlin (2003) examined a wide range of factors influencing score gains amongst international students. Using data collected from questionnaires and interviews with students and teachers, they found that the reasons for improvement (or the lack of it) included motivation, type of housing accommodation, self-confidence and the extent to which students used or accessed English outside the classroom. However, their study focused on students with an initial score of less than 6 enrolled in intensive pre-sessional English courses. It is worth investigating whether the same factors are salient for students starting with higher levels of proficiency undertaking higher education courses.

2. RESEARCH QUESTIONS

This paper seeks to answer the following questions.

1. How much improvement on the IELTS Test can be expected of undergraduates and postgraduates who are completing higher education courses in an English-medium context?
2. What educational, personal and social factors influence this improvement?

3. THE IELTS CONSTRUCT

To interpret score gain on IELTS, it is necessary to define the construct being assessed in the Test. The IELTS Handbook (2007, p 2) suggests that the Test “is designed to assess the language ability of candidates who need to study or work where English is the language of communication”.

The test is available in two formats - Academic and General Training. The Academic Reading and Writing tests assess whether a candidate is ready to study or train in the medium of English at an undergraduate or postgraduate level. The Listening and Speaking tests are common to both the Academic and General Training formats. Each candidate takes four tests, one in each of the four skills - Listening, Reading, Writing and Speaking. The IELTS Handbook (2007, p 3) states that the “tests are designed to cover the full range of ability from non-user to expert use”. Candidates receive a band score for each of the four modules, as well as an overall band score which is a rounded average of these four results.

In this study, candidates sat for an official Academic IELTS Test on two occasions - once before they applied to enter their undergraduate or postgraduate course at The University of Melbourne (referred to as Test 1) and again in the final semester of their course (referred to as Test 2). As all test administrations took place before July 2007, the scores for Listening and Reading were reported in whole and half bands but for Writing and Speaking in whole bands only. (After this time, all four skills were reported in whole and half bands.)

What ability or abilities are assessed in the IELTS Academic Test? It is fundamentally an academic language proficiency test, ie, it predicts the extent to which a candidate will be able to begin studying through the medium of English.

In terms of the model of proficiency underlying the Test, Davies (2008, p 106), in his comprehensive history of IELTS, suggests that the Test samples what has been called “communicative language ability”, taking a more abstract view of communicative competence than had been the case up until the latest revision of the Test in 1995. He suggests that the current IELTS Test is supported by Bachman’s (1990) Interactional Ability (IA) model as opposed to a Real Life (RL) one, the latter including more discipline-specific Academic Reading tests in the early 1990s. Davies (2008, pp 109-111) argues that “tests cannot be authentically real-life: the best they can do is simulate reality” and that the current IELTS Test “represents a kind of regression to the mean, a (good) compromise between the extremes of the structural and the communicative”.

Davies suggests that IELTS is a test of generic academic language ability, basing this claim on his definitions of ‘academic language’ and ‘language proficiency’. Academic language according to Davies (2008, p 113) is:

…the language of coherent argument where implications are understood and inferences made. It is, above all, the discourse in which, as reader, as listener, as speaker and as writer, the candidate makes sense of what has gone before and responds, and continues to respond appropriately.

And academic language proficiency is (Davies 2008, p 113):

…skilled literacy and the ability to move easily across the skills. In other words, it is the literacy of the educated, based on the construct of there being a general language factor relevant to all those entering higher education, whatever specialist subject(s) they will study.

The current IELTS Test therefore “quite deliberately eschews any claim to specificity because what it wishes to claim is that the test is generic, potentially generalisable to any type of academic language use” (Davies 2008, p 108).

The Academic Test could be said to include a test of general academic literacy through the Academic Reading and Writing tests. The fact that the Listening and Speaking tests are common to both the Academic and General Training formats does not necessarily imply that the skills being assessed are not academic. However, they are certainly less explicitly academic in character than the other two tests.

4. CONTEXT OF THE STUDY

4.1 Research site

The study was conducted in the Faculty of Economics and Commerce, The University of Melbourne, Australia from June 2006 until December 2007. This research site was also used in another funded project undertaken during 2005 by O’Loughlin (2008) to examine the use of IELTS in university selection.

The Faculty of Economics and Commerce is one of The University of Melbourne’s largest faculties and has its highest number of international student enrolments. In 2005, a total of 4585 students were enrolled in its undergraduate and postgraduate programs; of these, 46% were international students. In 2006, the total number of students was 4900 and within that cohort, 48% were international students.

The faculty offers a range of coursework and research programs in four departments: Accounting and Business Information Systems; Economics; Finance; and Management. In addition to its award programs, the faculty provides a range of non-award support programs and services through its Teaching and Learning Unit (TLU) to enhance the quality of teaching and learning for both local and international students. The development of students’ academic English skills is one major focus of the unit’s activities. The extent to which participating students benefit from its programs and services is examined in this study.

4.2 English language entry requirements

In 2006-7, the minimum IELTS requirements for entry to courses in the Faculty of Economics and Commerce at the university were:

• undergraduate courses - an overall band score of 6.5 including at least 6.0 in Writing
• postgraduate courses (excluding PhD) - an overall band score of 6.5 including at least 6.0 in all four individual bands.

These results must have been obtained within the 24 months preceding their course application.

While the official IELTS Handbook advises that an overall band score of 7.0 is ‘probably acceptable’ for ‘linguistically demanding’ academic courses, overall band scores of 5.5 to 6.5 are accompanied by the recommendation ‘English study needed’ (2005, p 5). This advice implies that it is the responsibility of universities to provide additional English language support to students when admitted to ‘linguistically demanding’ courses at these lower levels. However, some subjects offered by the faculty are less ‘linguistically demanding’ than others. Therefore, this advice would need to be more strongly heeded in some subjects than others.

5. METHODOLOGY

5.1 General approach

The study employed a case study approach to investigate the research questions.

Stake (1994, p 237) distinguishes between intrinsic and instrumental case studies. In the intrinsic case study, the researcher is focused entirely on the particular case. In the instrumental case study, on the other hand, a particular case is investigated to throw light on an issue or theory. In this instance, Stake (1994, p 237) suggests:

The case is of secondary interest; it plays a supportive role, facilitating our understanding of something else. The case is looked at in depth, its contexts scrutinised, its ordinary activities detailed, but because this helps us pursue the external interest.

The research reported here is an example of an instrumental case study. In focusing on exit score gains in the faculty at the university, the study aims to shed light on the rate of improvement which can be expected of undergraduate and postgraduate students over the duration of their higher education course as well as the educational, personal and social factors impacting on this improvement. This does not mean that all faculties across a single university or all universities are identical in terms of the international student experience but, rather, the study of one case has the potential to illuminate important aspects of this experience in other contexts.

5.2 Data collection

There were three main forms of data collection:

1. Test 1 and Test 2 IELTS scores
2. student questionnaires
3. interviews conducted with students and academic staff.

5.2.1 IELTS Test 1 and Test 2 scores

Students presented an original copy of the IELTS results used for entry into their current course in the faculty. These results are referred to as IELTS Test 1 scores.

The students undertook a second IELTS Test for the study in the last semester of their courses at the university. The results of this test are referred to as Test 2.

5.2.2 Student questionnaires

Students were asked to complete a questionnaire as a component of their participation in the study (see Appendix 1). The questionnaires contained a combination of forced-choice and open-ended items and were designed to elicit information about:

• the participants’ personal and academic background
• their perceptions of their English language abilities before and during their course
• the factors that assisted or hindered their English language development while studying in their course.

The final item in the student questionnaire provided opportunities for the participants to comment on any other aspect of their experience.

5.2.3 Interviews with students and academic staff

Semi-structured focus group interviews were conducted with a sample of student participants following the release of their Test 2 results. The purpose of the interviews was to explore the factors that assisted or hindered their improvement over the period of their study. Interviews were conducted by one or both of the researchers. The interview schedule is provided in Appendix 2.

Interviews with academic staff were conducted to elicit information about the learning environment and the factors that they saw as important in influencing the English language development of international students for whom English is a second language. The interviews were conducted with selected pairs of staff participants. Both researchers were involved in conducting staff interviews. The interview schedule is provided in Appendix 3.

5.3 Procedures

The original plan was to recruit 100 students (50 undergraduate and 50 postgraduate students). However, initial difficulties in recruiting students led to a re-negotiation of this number and the timeline of the project with the funding body, IELTS Australia. A second round of recruitment followed, resulting in a total of 63 students participating in the study.

The study began in May 2006 and continued through to October 2007. Ethics approval for the conduct of the project was gained from The University of Melbourne Ethics Committee before the recruitment of participants. Advice for recruiting students was sought from the Director of the Teaching and Learning Unit within the faculty and staff in the faculty office. Initially, contact with students was made through email, which resulted in some students volunteering to participate. However the numbers were low. Advertisements were then posted in and near the faculty and academic staff were asked to advertise the project in lectures. The email notices and the advertisements clearly indicated the date of the IELTS Test and that it was offered free of charge. These recruitment strategies resulted in 25 students volunteering to participate in the study by June 2006.

After re-negotiations with IELTS Australia, the recruitment period was extended so that more students could be sought. The second phase of recruitment then began in February 2007. The research assistant sent emails to students, attended lectures to brief students about the project and posted advertisements in and near the faculty. This activity resulted in the recruitment of 38 students.

In each of the two recruitment phases, the same procedures were adopted. The data collection was planned so that students could complete a questionnaire, sit for their second IELTS Test and participate in group interviews arranged according to their IELTS results during their final semester of study while they were still accessible in Australia. This meant that they needed to sit for the IELTS mid-way through their final semester rather than at the end. This is a constraint for studies which seek to test the exit level of international students, particularly if interviews are used for data collection.

Students were first invited to attend a project briefing session. They received information about the project including a plain language statement, consent form (including permission for the researchers to be given access to their Test 2 results) and student questionnaire. The researchers described the project, then the students were given time to read the information and sign the consent form, after which they were asked to complete the student questionnaire. They were asked to produce their original IELTS Test 1 results and their record of enrolment at the university to ensure that they were studying in the final year in the faculty. Once this information had been collected, students were given details for enrolling in IELTS Test 2.

Students sat for their official IELTS Test 2 on either 2 September 2006 or 31 March 2007 at RMIT English Worldwide in Melbourne. On the first date, there were 16 undergraduate and nine postgraduate students. On the second date, there were 14 undergraduate and 24 postgraduate students.

Student interviews were organised after the results of Test 2 had been sent to the researchers. Emails were sent to students with information about the scheduled interviews and requesting student volunteers. A total of 22 students participated in the interviews. The interviews themselves were built around participants’ questionnaire responses. The interviews allowed for clarification and extension of the student questionnaire responses, particularly in relation to the factors that influenced their English language development within and outside the university during their course. All interviews were conducted as focus groups. Participants were shown their completed questionnaires to refresh their memories. The interviews were audio-taped for subsequent analysis.

Staff participants were recruited for the study after all of the student data had been collected.

An invitation to participate was emailed to all academic staff in the faculty. The respondents were sent the plain language statement and the consent form. Interviews were organised to suit the availability of staff involved. Three staff interviews were organised, each involving two participants and the two researchers. The interviews began with discussion of the preliminary analysis of the score gains between the students’ Test 1 and Test 2 results, as well as some of the general comments made by students in the interviews. The preliminary findings were used as prompts to elicit discussion about the learning environment and the factors that they saw as important in influencing the English language development of international students for whom English is a second language. Audio-tape recordings were made of the interviews for subsequent analysis.

5.4 Study participants

A total of 63 volunteer international students for whom English was a second/additional language were recruited for the study. The cohort consisted of 30 undergraduate and 33 postgraduate (but not PhD) students, who were invited to participate in the study in their final semester.

All undergraduate students were in the final semester of the Bachelor of Commerce degree of three years’ full-time duration. The postgraduate students were also in their final semester of study and were completing a wide range of courses of between 12 to 18 months full-time at the Graduate Diploma and Masters levels.

One uncontrollable source of variability in the results reported here relates to the time when Test 1 was undertaken by the students. Given that IELTS Test results are considered valid for up to two years before they are used for an entry application, it is possible that students’ proficiency had either improved or declined between Test 1 and starting their course. Whether or not they improved depended on factors such as their exposure to the use of English and additional courses that they may have undertaken after Test 1. The other variable factor was the amount of time students had been studying their higher education course when they took Test 2. Undergraduate students took Test 2 two and a half years after commencing their studies and postgraduate students six to 12 months.

Due to the problems of recruiting adequate numbers of students in the study, these variables could not be held constant. Obviously these sources of variability reduce the ability to generalise the findings in relation to score gains. However, we consider that this is the price of conducting a ‘real life’ study where such variables cannot be tightly controlled. As a result, the study is high in ecological validity. Nevertheless, the statistical findings presented later in this report should be viewed cautiously in light of these two different sources of variability and the small sample size (N=63). The findings should be read as suggestive of trends rather than as authoritative results.

The statistical results formed the basis on which to select the case studies of students. To participate in the study, each student needed to have satisfied the faculty’s English language entry requirements with results from an IELTS Test and be able to produce an official record of their results. This enabled the researchers to conduct a direct comparison of their entry and exit proficiency as evidenced on the IELTS Test. They also needed to have already successfully completed two and a half years’ equivalent full-time undergraduate study or at least six months’ equivalent full-time postgraduate study. In return for their participation, all students were invited to take an official IELTS Test free of charge.

Relevant information about the 63 students from the Faculty of Economics and Commerce is summarised in Table 1 below. All had indicated that they had used their IELTS Test 1 in their application for their current course within the faculty. Even though the formal entry requirement is an overall score of 6.5, one student had an IELTS Test 1 overall score of 5.5, and four had an overall score of 6.0.

Study level

Sex

Table 1: Student participants - background data

Six academic staff participated in the study. Of the three disciplinary staff, two were professors and one was a senior lecturer. All coordinated subjects in undergraduate and postgraduate programs and were involved in teaching international students. They ranged in experience from six years to 30 years in university teaching. The Teaching and Learning Unit staff were all experienced in supporting international students’ academic learning. One was a senior lecturer involved with postgraduate students and the other two were lecturers who worked with undergraduate students in the faculty.

5.5 Methods of analysis

5.5.1 Test scores

The test scores obtained by student participants for IELTS Test 1 and Test 2 were analysed quantitatively to answer Research Question 1 regarding how much improvement can be expected by students. This question was answered by analysing what improvement the students made, as well as the nature of that improvement across the four skills.

To answer the question of the improvement made by the participants on the IELTS Test, data from Test 1 and Test 2 were collected and cross-tabulated. These included individual scores for Listening, Reading, Writing, Speaking and overall. Descriptive statistics for Test 1, Test 2 and score gains from Test 1 to Test 2 were calculated. These included the mean, standard deviation and standard error of the mean figures.

To investigate the nature of the improvement, Principal Components Analysis (PCA) was used to explore whether the pattern of improvement was the same or different across Listening, Speaking, Reading and Writing. Next, regression analyses were undertaken to examine whether students with lower or higher initial scores were improving most across the same four skills.

5.5.2 Student questionnaires

The student questionnaire data was analysed both quantitatively and qualitatively to explore Research Question 2. The responses were coded and entered into a database. Some of the questionnaire data was categorical (eg sex and country of birth), some ordinal (eg years of studying English or years in Australia) and others experiential (eg language support offered at university). Some of the data was also narrative in character (eg optional further comments). The complete breakdown of the results for each of the questions is presented in Appendix 6.

Regression analyses were conducted to investigate the relationship between score improvement and students’ responses to the questionnaire they completed when they agreed to participated, ie prior to Test 2.

5.5.2 Interviews with students and academic staff

The staff and student interviews were examined qualitatively in relation to Research Question 2. Initially, they were transcribed by the research assistant. Running summaries of the themes, issues and ideas raised in the interviews were made by the two researchers. Positioning theory (Harré and van Langenhove 1999) informed the analyses of the interviews in relation to the agency of the students and the teachers within the institutional practices of the university. Implicit within this model is the agency that people demonstrate in achieving, accepting or rejecting positioning acts within focus group interviews. Overall themes and issues raised in each interview were noted for comparison across interviews. The summaries of the student and staff interviews are Appendix 7 and 8 respectively. The interview data and the questionnaires were used in the analysis to answer the second research question concerning the educational, personal and social factors that influence student improvement.

Following the initial analyses of the student interviews, six were chosen by the researchers as case studies illustrating firstly, the most common trends in the test score data and secondly, the major themes in the interview data. The test scores and interview contributions of these six students then formed the basis of the case studies.

6. RESULTS

The results of the study are organised under the two main Research Questions.

1. How much improvement on the IELTS Test can be expected of undergraduates and postgraduates who are completing higher education courses in an English-medium context?
2. What educational, personal and social factors influence this improvement?

6.1 How much improvement on the IELTS Test can be expected of undergraduates and postgraduates who are completing higher education courses in an English-medium context?

This question was addressed by examining the rate and nature of students’ improvement from Test 1 to Test 2.

6.1.1 How much improvement did the participants make from Test 1 to Test 2?

The amount of improvement made by the 63 student participants is reported below in terms of their Listening, Reading, Writing, Speaking and overall scores. Obviously, the small sample size means generalising to the wider population is problematic. Between 2006 and 2007 when the study was conducted, the Listening, Reading and overall scores were expressed from 1.0 - 9.0 in increments of 0.5 while Writing and Speaking were expressed from 1.0 - 9.0 in whole numbers only.

6.1.1a Cross-tabulations

Initially, the data from Test 1 and Test 2 was cross-tabulated to analyse the amount of improvement made by students in Listening, Reading, Writing, Speaking and overall. There were no missing observations in the data.

These cross-tabulations show that many of the student participants exceeded the minimum entry scores for both undergraduate and postgraduate study in the faculty at the university in Test 1. Recall that these levels were overall 6.5 with Writing no less than 6.0 for undergraduate study and an overall score of 6.5 with no band less than 6.0 for postgraduate study (excluding PhD). On average, student scores showed improvement both overall and across each of the four skill areas although there was more improvement in some skills than others. The amount of improvement is described in detail below.

Listening scores

Table 2 shows that, for Listening, the scores ranged from 5.5 to 8.5 on Test 1 and from 6.5 to 8.5 on Test 2. The mode was 7.0 for Test 1 (17 students) and 8.0 for Test 2 (14 students).

Test 2 results

Table 2: Listening scores. Number of students at each IELTS band at Test 1 and Test 2

Reading scores

Table 3 below shows that the scores ranged from 5.0 to 9.0 on Test 1 and from 5.5 to 9.0 on Test 2. The mode was 6.5 for Test 1 (20 students) and 7.5 for Test 2 (21 students).

Test 2 results

Table 3: Reading scores. Number of students at each IELTS band at Test 1 and Test 2

Writing scores

Table 4 below shows that the scores ranged from 5.0 to 8.0 on Test 1 and from 5.0 to 9.0 on Test 2. The mode was 6.0 for Test 1 (26 students) and 6.0 for Test 2 (28 students).

Test 2 results

Table 4: Writing scores. Number of students at each IELTS band at Test 1 and Test 2

Speaking scores

Table 5 below shows that the scores ranged from 5.0 to 9.0 on Test 1 and from 5.0 to 9.0 on Test 2. The mode was 6.0 for Test 1 (33 students) and 7.0 for Test 2 (27 students).

Test 2 results

Table 5: Speaking scores. Number of students at each IELTS band at Test 1 and Test 2

Overall scores

Table 6 below shows that the scores ranged from 5.5 to 8.5 on Test 1 and from 6.0 to 8.5 on Test 2. The mode was 6.5 for Test 1 (26 students) and 7.0 for Test 2 (18 students).

Test 2 results

Table 6: Overall scores. Number of students at each IELTS band at Test 1 and Test 2

6.1.1b Descriptive statistics

Tables 7, 8 and 9 below show the means, standards deviations and standard error of the mean for the Listening, Reading, Writing, Speaking and overall scores for a) Test 1, b) Test 2 and c) improvement from Test 1 to Test 2.

Table 7: Descriptive statistics, Test 1 (N=63)

Table 8: Descriptive statistics, Test 2 (N=63)

Table 9: Descriptive statistics for improvement from Test 1 to Test 2 (N=63) Table 7 indicates that for this cohort of students, on average, the strongest skill was Listening followed by Reading, Speaking and Writing respectively on Test 1. Table 8 shows that this order was the same for Test 2. The spread of scores (as measured by the standard deviation) was fairly similar. Table 9 shows that the greatest average improvement in Test 2 was in Listening and Reading (0.500 and 0.532 respectively), and that the Writing test showed the least improvement (0.206). It is worth remembering here that, unlike the Listening and Reading tests, the Writing and Speaking tests results were still reported in whole bands during the period in which this study was conducted and therefore, unlike Listening and Reading, did not register smaller amounts of progress. Nevertheless, the finding that the students showed the least improvement in Writing suggests that students may not always be required to develop their writing skills in their higher education courses.

6.1.2 What was the nature of the improvement from Test 1 to Test 2?

In this section, we examine the patterns of improvement in the data set. Initially, the issue of whether improvement in all four Bands (Listening, Reading, Writing and Speaking) followed the same pattern was investigated. This was done using a factor analytic approach, specifically Principal Components Analysis (PCA).

6.1.2a Principal Components Analysis (PCA)

Table 10 reports the unrotated factor matrix for the four aspects of language learning, namely improvement in Listening, Writing, Reading, and Speaking. Here we see that the while Listening, Writing and Reading group together on the first principal component (ie are strongly correlated), improvement in Speaking is separate from these, loading on the second principal component.

Table 10: Unrotated factor matrix

Normally, it is prudent to rotate the factors, to ‘even out’ the loading of variance on the factors. In the unrotated analysis, the first factor accounts for as much as possible. In the rotated analysis, the variance is spread more equitably among the main factors. However, as is commonly the case when the number of variables is small - as in this case (just four variables) - the rotated and unrotated analyses generally give very similar results. In Table 11, the rotated factor matrix is displayed, and again, it is clear that improvement in Listening, Writing and Reading group together on the first principal component, while improvement in Speaking stands apart. In other words, there is no correlation between the improvement in Speaking and the average of the improvement in Listening, Reading and Writing.

Table 11: Rotated factor matrix (varimax rotation)

The factor analyses imply that there is some correlation between those three skills which load on the first factor, but this needed to be checked using a measure of the inter-correlation between the three items such as Cronbach alpha. The result for this test was 0.3598. While reasonably strong, this figure may have been weakened by the relatively small sample size (N=63). Nevertheless, it is clear that the pattern of improvement in Speaking is significantly different from the Listening, Reading and Writing.

Figure 1 plots the improvement in Speaking (X-axis) against the average improvement in Listening, Reading and Writing (Y-axis) with each dot representing one student. For example, the figure shows that there is one student who improved by 1.5 bands in Speaking and an average of 2.0 on the other three skills. The graph shows clearly that there is a very weak correlation between the average Listening, Reading and Writing scores and the Speaking scores.

Thus, it makes little sense to use the overall score, the adjusted average of the four skills, as an accurate measure of individual improvement for the data obtained in this study. The results suggest it is more meaningful to average the results on the Listening, Reading and Writing tests only and to treat Speaking separately.

The findings indicate that the pattern of scores for Speaking is different from the other three skills. The reasons as why this occurs are difficult to interpret. From the perspective of construct validity, this finding suggests that Speaking constitutes a different dimension of ‘communicative language ability’ than the other three skills. However, it should be recognised that there is a potentially weaker reliability associated with scores in Speaking than in the other three subtests. This unreliability derives mainly from the fact that the student performances are only rated by one IELTS Examiner (as is the case for Writing) but also because Examiners may positively or negatively impact on these performances in their role as interlocutors.

Figure 1: Improvement in Speaking versus improvement in Listening, Reading and Writing

6.1.2b Regression analyses

In the next stage of the analysis, we wished to test the hypotheses that students with lower initial scores improved more than students with higher initial scores for each of the language learning components. We tested this improvement firstly, in each of the four skills, secondly, for the overall scores and lastly, on the average of Listening, Reading and Writing using Ordinary Least Squares regression (OLS). Note that in these and subsequent regression analyses reported in this study, p<0.05 is taken as the threshold for statistical significance and is denoted by a single asterisk (*). Statistical significance at the 0.01 level is denoted by two asterisks (**).

Tables 12 to 16 provide the results for each of the four skills. In each case, the explanatory variable (the initial scores), predicts the dependent variable (the respective improvement scores). We notice that in each case, not only is the explanatory variable statistically significant in each regression, but the T coefficient is negative. This confirms our hypothesis that students with lower initial scores improved more than those with higher initial scores in each of the four skills tests.

Dependent Variable: Improvement in Listening score

Notes: Overall regression F-stat=17.658 with 1 degree of freedom (regression) and 61 degrees of freedom (residual) was significant on the F test at p=0.0001. The r-squared figure is 0.2245.

Table 12: Regression results for improvement in Listening scores

Dependent Variable: Improvement in Reading score

Notes: Overall regression F-stat=19.984 with 1 degree of freedom (regression) and 61 degrees of freedom (residual) was significant on the F test at p=0.0000. The r-squared figure is 0.2468.

Table 13: Regression results for improvement in Reading scores

Dependent Variable: Improvement in Writing score

Notes: Overall regression F-stat=7.802 with 1 degree of freedom (regression) and 61 degrees of freedom (residual) was significant on the F test at p=0.0070. The r-squared figure is 0.1134.

Table 14: Regression results for improvement in Writing scores

Dependent Variable: Improvement in Speaking score

Notes: Overall regression F-stat=6.426 with 1 degree of freedom (regression) and 61 degrees of freedom (residual) was significant on the F test at p=0.0138. The r-squared figure is 0.0953.

Table 15: Regression results for improvement in Speaking scores However, when we tested the overall scores (the adjusted average of all four skills tests), the result was not statistically significant as shown in Table 16 below.

Dependent Variable: Improvement in overall scores

Notes: Overall regression F-stat=2.219 with 1 degree of freedom (regression) and 61 degrees of freedom (residual) was not significant on the F test at p=0.1415. The r-squared figure is 0.0351.

Table 16: Regression results for improvement in overall scores

The non-statistically significant result is probably because - as we discovered from the Principal Components analysis (PCA) - Listening, Reading and Writing do not load on the same factor as Speaking. To put it another way, the individuals with high improvement scores in the first three skill areas are not necessarily the same individuals with high improvement scores in the Speaking area. This also implies that a certain proportion of those students with low initial scores in Listening, Reading and Writing (and who may be showing great improvement in these areas) are also those with high initial scores in Speaking (with consequent lower improvement scores in Speaking).

When we average the improvement in Listening, Reading and Writing (those which loaded on the same factor in the PCA), we get the result in the final regression output in Table 17. As should be expected, the average of the initial scores predicts the amount of improvement in the average of Listening, Reading and Writing in Test 2. Again, the regression coefficient is negative (and statistically significant), with an r-squared value of 0.968 indicating that the initial scores account for just under 10% of the explanation in variance of the improvement scores.

Dependent Variable: Improvement in Listening, Reading and Writing scale

Notes: Overall regression F-stat=6.535 with 1 degree of freedom (regression) and 61 degrees of freedom (residual) was significant on the F test at p=0.0131. The r-squared figure is 0.9680.

Table 17: Regression results for average improvement in Listening, Reading and Writing scores

When this is plotted (see Figure 2) with initial scores on the horizontal axis and improvement scores on the vertical, we obtain a negative regression line confirming that students with higher initial test results in Listening, Reading and Writing improve much less than those with lower initial test scores. Since some data points may be directly on top of one another, a small random variable has been added to show the scatter. An OLS regression line is also shown - the slope (negative) is statistically significant (T-stat = 2.5560; significance=0.0131).

These cross-tabulations show that many of the student participants exceeded the minimum entry scores for both undergraduate and postgraduate study in the Faculty of Economics and Commerce at The University of Melbourne in Test 1. Recall that these levels were overall 6.5 with Writing no less than 6.0 for undergraduate study and an overall score of 6.5 with no band less than 6.0 for postgraduate study (excluding PhD).

Figure 2: Average improvement in Listening, Reading and Writing from Test 1 to Test 2 with regression line

6.2 What educational, personal and social factors influenced students’ improvement from Test 1 to Test 2?

Research Question 2 was examined by looking firstly at the student questionnaire responses, secondly, at individual case studies incorporating test profiles and data from the student questionnaires and interviews, and thirdly, at the staff interviews.

6.2.1 Student questionnaire responses

The first stage in addressing this research question was to examine student questionnaire responses and their average improvement in Listening, Reading and Writing only (since the pattern of test score improvement for Speaking was different). Again, a series of OLS regressions was employed to examine the relationship between student responses to the questionnaire and average score improvement in Listening, Reading and Writing from Test 1 to Test 2.

The questionnaire responses analysed were:

• age
• country of birth
• gender
• level of study (undergraduate/postgraduate)
• years since IELTS Test 1
• years of studying English in school
• years studying English at university
• years studying English at a private language school
• years of being taught English by a private tutor
• length of time in Australia
• type of accommodation in Australia
• whether they had previously lived in an English-speaking country (other than Australia)
• length of time in an English-speaking country (other than Australia)
• belief about adequacy of English at the start of their university course
• belief about whether their English had improved while completing their course
• whether the university has required them to complete additional English studies
• whether they felt the need for English support to cope with their course
• whether they had been given additional English support during their course.

The only statistically significant results in these analyses were for four variables as outlined below.

1. Level of study

The results in Table 18 below show that undergraduate students improved much more than postgraduate students. This was not surprising given that the minimum English entry requirements for admission to undergraduate courses were not as rigorous as for postgraduates and also, because undergraduate students had studied for longer than the postgraduate students in this study (2.5 years versus 0.5-1.5 years respectively).

Dependent Variable: Relationship between level of study and test improvement

Notes: Overall regression F-stat=4.9092 with 1 degree of freedom (regression) and 61 degrees of freedom (residual) was significant on the F test at p=0.0305. The r-squared figure is 0.0745.

Table 18: Regression results for relationship between level of study and Test improvement

2. Years since last IELTS Test (Test 1)

The results in Table 19 indicate that the longer the time period since students took Test 1, the more likely they were to improve on Test 2.

Dependent Variable: Relationship between years since Test 1 and test improvement

Notes: Overall regression F-stat=6.1502 with 1 degree of freedom (regression) and 61 degrees of freedom (residual) was significant on the F test at p=0.0160. The r-squared figure is 0.0930.

Table 19: Regression results for relationship between years since Test 1 and Test improvement

These findings reinforce those based on the previous variable - level of study. As IELTS Test results are considered valid for up to two years before they are used for an entry application, undergraduate students may have sat Test 1 up to four and a half years before Test 2, whereas the maximum time lapse for postgraduate students would have been three years. A limitation of this study was its exclusive focus on the period when students were undertaking their degrees. In retrospect, it would have been strengthened by also examining what students may have done to improve their English between taking Test 1 and commencing their university studies.

3. Years studying English at university

The results in Table 20 below show that the longer students had previously spent studying English at university, the lower their improvement was from Test 1 to Test 2. The probable reason for this is that the students who had spent a longer time studying English at university were more likely to be postgraduates, who, as shown in Table 18 above, were less likely to improve their proficiency than undergraduates. However, it is not clear whether the years spent studying English at university were prior to Test 1, between Test 1 and Test 2 or during both periods of time.

Dependent Variable: Relationship between previous English study at university and Test improvement

Notes: Overall regression F-stat= 5.2531 with 1 degree of freedom (regression) and 61 degrees of freedom (residual) was significant on the F test at p=0.0258. The r-squared figure is 0.0887.

Table 20: Regression results for relationship between English study at university and Test improvement

4. Whether they have previously lived in an English-speaking country

The results in Table 21 suggested that students were more likely to improve on Test 2 if they had previously lived in an English-speaking country. Again, this finding is difficult to interpret meaningfully without knowing if the students had lived in an English-speaking country before Test 1, between Test 1 and Test 2 or during both periods of time.

Dependent Variable: Relationship between previous stay in an English-speaking country and Test improvement

Notes: Overall regression F-stat= 4.7437 with 1 degree of freedom (regression) and 61 degrees of freedom (residual) was significant on the F test at p=0.0333. The r-squared figure is 0.0722.

Table 21: Regression results for relationship between previous stay in an English-speaking country and test improvement

The questionnaire data was useful for beginning to examine the educational, personal and social factors influencing score improvement in this study. However, these results suggest that it was of limited use in exploring the complex relationship between test improvement and these factors. In order to do this, we undertook a series of individual case studies which provided much more illuminating data about the reasons for student improvement or the lack of it from Test 1 to Test 2. These are presented in the next sub-section of the report.

6.2.2 Student case studies

The six case studies below were selected as representative of the following categories of students:

1. Undergraduate student with large overall score gain

2. Postgraduate student with large overall score gain

3. Undergraduate student with no overall score gain

4. Postgraduate student with no overall score gain

5. Student with greater improvement in Speaking than in Listening, Reading and Writing

6. Student with lower improvement in Speaking than in Listening, Reading and Writing

6.2.2a Case study 1 - Charmaine

Undergraduate student with large score gain Charmaine was a 20-year-old undergraduate student from Hong Kong. She had been in Australia for three years and one month, and was in the final semester of her Bachelor of Commerce degree. Charmaine’s Test 2 results in Table 22 below show that she improved by 1.5 bands in her overall IELTS score, with improvements in all the skill areas. Her scores increased most noticeably in Speaking and Writing. The duration between Test 1 and Test 2 was two years and eight months.

Table 22: Test 1 and Test 2 scores for Charmaine

Although Charmaine had studied English for five years at secondary school in Hong Kong and had a private tutor for two years, she indicated in her interview that she experienced difficulties in communicating when she first arrived in Australia. She attributed her improvement to the fact that she had lived with an Australian host family for the three years that she had been studying for her degree.

Charmaine not only improved in her Speaking skills, but also developed a distinct Australian accent. She was amazed by her progress: “Guess it’s amazing what an Australian family can do for you” (Student Interview 4, 2:40). According to Charmaine, her host family was one of the main factors in developing her Speaking and Listening skills.

Charmaine said that other factors also contributed to her English language development. As a student in Hong Kong, Charmaine’s experience of classes taught solely in English increased her confidence in her ability to cope with the English teaching environment. She was generally confident about her proficiency in terms of her academic studies. In addition, she studied an elective in communication skills, which was designed for ESL learners and aimed at developing academic English language skills. According to Charmaine, this subject helped her with her commerce studies. Another contributing factor was the decision by Charmaine to always speak in English:

If they [students from Hong Kong] live with Hong Kong people, generally their English does not improve a whole lot. That’s my main observation…My friends and I speak English even if Australian friends are hard to find. (Student Interview 4, 23:00)

For Charmaine, it was important to find opportunities to speak English. For example, she and her group of friends would communicate in English, even though they shared common languages of Cantonese and Mandarin.

Charmaine indicated that there were limited opportunities to develop her spoken language skills at university. Her commerce course was mainly assessed through writing: “there is no subject that I have done, apart from my communication skills subject, where I had to do an oral presentation” (Student Interview 4, 32:00). Charmaine knew a few international students who had limited prospects for developing their speaking skills while studying in Australia. She offered the following example:

He [Charmaine’s friend] lives with five other Hong Kong people. He travels to university by driving. No communication. He lives in Box Hill, a very Chinese community. Over four years he got around not using much English. He did well at uni because his course is maths based. In commerce and accounting you have to read and write a bit but there’s maths involved with every subject. (Student Interview 4, 29:33)

According to Charmaine, international students may successfully complete their studies without having to engage with the English-speaking community either inside or outside the university.

Course choice may be a factor hindering students’ English language development. In recounting her experience, Charmaine said there were limited opportunities within her course to develop her written English skills:

[Now] I’m stronger in non-academic English development. I think I’ve written less than five essays with commerce/economics…so I would have written around 10,000 words and dot points are allowed… When I first arrived my academic skills were stronger than social skills. Now definitely the other way around. The academic bit has improved but not as much as the social. (Student Interview 4: 35:00 and 36:20)

Her reversal of strengths in English language skills at the completion of her course is attributed in part to the homestay environment. Charmaine was also active in seeking opportunities to develop her English skills. Her actions and her homestay environment created situations that supported her development of oral English. She was less concerned with her writing and reading as her entry scores were strong for an undergraduate course and she was not required to complete extended written assignments for her course. Her advice for future international students neatly summarises her viewpoint:

Get out of your comfort zone and don’t stay with someone from your area.

(Student Interview 4, 38:40)

6.2.2b Case study 2 - Carlos

Postgraduate student with large score gain

Carlos was a 28-year-old postgraduate student from Peru. He had been in Australia for one year and five months. For most of that time he had been studying for his Master of Applied Commerce. His test results in Table 23 show that he had made an overall improvement of one score band, with improvements in all the skill areas apart from Reading, which remained the same.

Table 23: Test 1 and Test 2 scores for Carlos

The duration between the two tests was two years and one month.

Carlos had studied English for five years in secondary school. He also studied English at a private language school in Peru for 18 months after completing secondary school. Carlos indicated that he wanted to learn English because he planned to study abroad and believed that it was important for his future employment. He had completed an English preparatory course for university study before commencing his Masters course, as he was concerned about starting the course without a “high level of English” (Student Interview 2, 8:00).

In Australia, Carlos lived with his wife, who was also from Peru. When he began his studies he felt that:

It was going to take me a long time to adapt to the new learning environment.

The Australian accent was a key constraint. (Questionnaire - q.18)

For Carlos, one of the main factors in choosing to study at the university was to improve his language skills (Interview 2, 23:30). He found that working at a hotel helped develop his speaking skills:

I applied for a casual job and my university English language adviser told me that it wasn’t a good strategy as it might obstruct my studies. But I wanted to be able to speak all the time…you practically don’t speak at all at university.

(Student Interview 2, 12.30 and 16.00)

Carlos also credited activities such as listening to the radio and watching television as reasons for his improvement:

You have to make an effort to listen and understand Australian English. You don’t have subtitles to help you. (Questionnaire - q 23)

He placed himself in situations that he believed would help further develop his English skills. He believed that there were very few opportunities to develop spoken English within the university.

While his listening and speaking skills appeared to improve through actively seeking situations where he could be exposed to Australian English, Carlos also sought the services available within the university to develop his academic writing skills.

I have used services to help me improve some of my assignments, as I have grammar problems. Lecturers are hard on grammar so I don’t want to lose points.

(Student Interview 2, 19:15)

Carlos sought opportunities both inside and outside the university to develop his English language skills. He believed that that his actions had attributed to his significant English language improvement over 17 months. He suggested that his speaking and listening skills had developed through his life outside the university while his reading and writing skills had improved through his study at the university.

6.2.2c Case study 3 - Daisy

Undergraduate student with no overall score gain

Daisy was a 22-year-old undergraduate student from China. She had been in Australia for two years and eight months, and was in the final semester of her Bachelor of Commerce degree.

Daisy’s Test 2 results in Table 24 show that there was no overall score gain between Test 1 and Test 2. Nevertheless, she had improved by half a band in Listening and Reading. However, she remained the same in Speaking and decreased by one band in Writing. The duration between the two tests was three years and three months.

Table 24: Test 1 and Test 2 scores for Daisy

Daisy had studied English in China, including eight years of learning English at secondary school and two years at university level. She felt that English was important for studying and travelling abroad, as well as future employment. Daisy had decided to find employment in Australia and apply for permanent residency (Student Interview 8, 33:20).

When she began her course, Daisy did not think that her English skills were adequate to cope with her studies. Daisy indicated that spoken English was important for her academic work and engagement with other students in her class:

Sometimes we have to do group assignments and know different people…so it is

important to develop the social English for that.

(Student Interview 8, 33:20)

Daisy emphasised the importance of developing her spoken English skills in the interview. She believed that her English skills had improved by her general exposure to English through attending lectures and tutorials, making friends who spoke English and watching television. She indicated that she spoke English with other international students and this assisted her spoken English development. However, her opportunities were limited. One of the main obstacles for Daisy was finding people with whom she could speak English:

I actually got some problems. My friends always speak Chinese and I don’t have the real environment to practice English. I can only read the newspaper and watch TV and I work in a hotel, but I’m doing accounting assistance, and working there helps.

(Student Interview 8, 12:50)

Her limited opportunities for developing her speaking skills were further constrained by Daisy living with Chinese students and spending most of her time socialising with them, and speaking Mandarin. Yet she was aware that there were different strategies that Chinese students use to develop their spoken English skills.

Actually the majority of Chinese students have speaking problems, but different students have different ways. Some live in homestay, if they have good financial support, or join society to make more local friends. (Student Interview 8, 22:00)

This emphasis on spoken language development is further reflected in her advice for international students studying in the faculty:

I think the most important thing is to build up confidence. If they speak in front of Asians, they feel more confident but in front of local students they don’t want to speak to them. So speak to other international students from different background. You cannot put this person in a local environment directly. It will destroy their confidence, so they have to improve step by step. Watch more TV and read newspapers so you know what happens in Australia. So when other people talk about it, you know what they are talking about.

(Student Interview 8, 44:30)

Daisy’s strategy of slow immersion into the English environment was different from Carlos’ and Charmaine’s approach where they elected to place themselves in situations where they were forced to communicate in English, ie they were more willing than Daisy to take risks with their English language communication.

Daisy’s Writing score decreased by one band and the reasons can in part be attributed to the nature of her accounting degree, where there is little emphasis on writing large amounts of text. Her only comment concerning the development of her writing skills took the form of a complaint - as she completed her assignments on the day they were due she was not able to access the services of the faculty English language support unit as tutors needed to have the assignment in advance to provide individual feedback on written assignments (Student Interview 8, 19:35). Developing her writing skills may not have been important for Daisy to successfully complete her course. However, her lack of progress was reflected in her Test 2 Writing result.

6.2.2d Case Study 4-Truc

Postgraduate student with no overall score gain

Truc was a 27-year-old postgraduate student from Vietnam. She had been in Australia for nine months, and was completing a Master of Applied Commerce.

Her Test 2 results in Table 25 show that while there was no overall score gain, she had improved by one band in Listening. However, her scores had decreased by two bands for Writing, and had remained the same in Speaking and Reading. The duration between the two tests was one year and four months.

Table 25: Test 1 and Test 2 scores for Truc

English study was compulsory at both secondary school and university, which she studied for seven years and four years respectively. She also studied at a private English language school in Vietnam for three months before undertaking Test 1. She indicated that English was important for her to study abroad and for employment purposes.

When Truc began her course, she was confident that her English language skills were adequate because her friends had told her that reading and writing were important for study. Her Test 1 results indicated that these were her stronger skill areas. While she was studying, opportunities to talk with other international students in English and living in an English-speaking country further improved her listening and speaking skills.

Truc was happy with her result, even though her Writing score was lower in the exit test. When questioned about why she received a lower Writing score, she said that while she had many ideas to write down, she felt that the word limit constrained her, and that she was unable to decide on what to write and ran out of time. She also felt that she was stronger in Reading than her Test 2 score indicated because she had developed the skills of skimming and scanning.

Truc lived with other Vietnamese students. When she first arrived, she made many international friends and socialised with them in an attempt to develop her spoken English skills. She looked for opportunities to engage with the university community:

I apply for several different voluntary jobs and I meet new friends…I mentor in the International Centre and my faculty. I communicate with staff and new students. Good university activity for me to improve my English. (Student Interview 4, 20:30)

She indicated that it was important to speak English with other international students as a way of developing her spoken English. However, as she progressed with her studies, she spent more time at home with her Vietnamese flatmates:

Last few months I always stay at home and speak with my Vietnamese friends, so my English is very poor. (Student Interview 4, 23:55)

Opportunities for Truc to speak English were limited both within and outside the university. One of her biggest challenges was that the Vietnamese language is spoken so widely within the community: “There are so many Vietnamese in Melbourne. It is used in all places, the market, university and at home” (Questionnaire: q 19). A further obstacle was that there were few opportunities to use English in tutorials.

Even some subjects require presentation but worth 10-12% of overall mark assessment so doesn’t really make much difference if you say anything or not. (Student Interview 8, 31:00)

Truc observed that international students were able to be successful in their studies without needing to engage with the English-speaking community (Student Interview 4, 25:00 and 27:50). She had attempted, through her volunteer work and friendships with international students, to develop her communication skills, which she believed assisted her academic work (Student Interview 4, 38:18). However, her attempts to develop her language skills, with the exception of listening, were not reflected in her Test 2 results.

6.2.2e Case Study 5 - Dian

Student with greater improvement in Speaking than in Listening, Reading and Writing

Dian was a 25-year-old postgraduate student from Indonesia. He had been in Australia for one year and four months, and was enrolled in a Master of Commerce.

As shown in Table 26, Dian’s IELTS exit test results had improved by half a band overall. He had improved more in Speaking than in the other three skill areas. His result remained the same for Listening and Writing but improved by half a band in Reading and one band for Speaking. The duration between the tests was two years and five months.

Table 26: Test 1 and Test 2 scores for Dian

Dian had studied English for six years in secondary school, six months at university and three years at a private language school. Even though English was not compulsory, he chose to study it for the purposes of travelling and studying abroad, as well as for employment. He planned to return to Indonesia when he completed his studies. Dian lived on his own.

When he began his course, he stated that he was confident that his English language skills were adequate for university study. Dian took opportunities to talk with native speakers at university and worked in an English-speaking restaurant. This allowed him to: “communicate and listen to the way Australian talk to each other” (Questionnaire - q 23). He took these opportunities because he knew that it would be difficult to “mingle with the natives” (Student Interview 1, 11:00). Dian also watched the Australian Government-funded television station (ABC 1) as he found it easier to understand than the other stations, which he felt tended to speak too fast.

Dian had found it difficult to develop his writing skills during his period of study. This seemed in part because he did not understand how the academic staff wanted the questions answered.

Writing is difficult. If I read something I can understand, and when they make the question, I think the answer is just this and they [academic staff] expect you to explore it more. I can’t. For me that question does not convey more if what I’ve already said. (Interview 1, 32:20)

In addition, he encountered problems using and citing academic articles in his writing.

Very hard paraphrasing something etc. Very difficult…that kind of English is very hard for me; also articles and using them. (Student Interview 1, 28:00)

He felt that he had little support within the university to develop these skills (Interview 4, 36:00). He was disappointed with the support he received at the Teaching and Learning Unit.

[They] don’t help you relate to your main focus in the study. Tend to tell you how to learn - more general. (Student Interview 1, 15:00)

He indicated that there was even less support from departmental academic staff.

From department, I don’t think they are very helpful. Tend to avoid students unless you chase them up after the lecture. (Student Interview 1, 15:00)

Dian may not be aware of how the academic and language staff can support him but his views indicate that he did not consider them useful to him in developing his writing skills. He was able to take opportunities to develop his spoken English through his work and interaction with fellow students, but he was less able to develop his writing skills without support from academic staff, and perhaps this hindered his development in Writing.

6.2.2f Case Study Six - Ollie

Student with lower improvement in Speaking than in Listening, Reading and Writing

Ollie was a 30-year-old postgraduate student from Indonesia. He had been in Australia for one year and four months, and was enrolled in a Master of Applied Commerce.

His test results in Table 27 show an overall score gain of one band. His results indicate that he improved in Listening by one and half bands, and in Reading and Writing by one band. His Test 1 and Test 2 results for Speaking were the same. While Ollie’s Speaking skills were strong, he was selected as a case study because his improvement in the other three skill areas was greater than his Speaking improvement, especially when compared to Carlos, who improved greatly in all the four skill areas over a similar time period. The time difference between Ollie’s Test 1 and Test 2 was 16 months.

Table 27: Test 1 and Test 2 scores for Ollie

Ollie had studied English for three years at secondary school and one year at university. In both of these contexts, English was compulsory. In addition, Ollie had completed an English preparatory course before undertaking university study. He believed that English was important for studying abroad and for employment.

Ollie indicated that he knew it was important to develop his academic English skills while at university. In the English preparatory course, his teachers had identified academic writing as the one area that he needed to improve if he were to succeed at university (Student Interview 8, 25: 30). He took any opportunities available within the university to do this. For example, he actively sought assistance from the Teaching and Learning Unit within the faculty:

I have used mostly the faculty learning unit. Basically for my assignments and whether they understand what I write. I got helpful feedback mainly in writing skills. Sometimes I ask them about vocabulary. (Student Interview 8, 18:00)

The unit helped him structure his academic writing and his use of appropriate vocabulary within an academic context. Ollie’s course required extended pieces of writing for assessment.

I found that for writing and reading, my uni works help a lot because I do a lot of reading and writing for commerce assignment. I’ve been writing so many assignments I believe it really has helped my skills. (Student Interview 8, 15:55)

Ollie was satisfied with his Test 2 results for Reading and Writing, which he believed reflected the opportunities afforded to him within the university to develop these skills. He also believed that his Listening skills had improved because of attending lectures, listening to group discussions and watching television (Student Interview 8, 15:65).

Ollie commented on the importance of developing his Speaking skills but his Test 2 results showed that he had not improved in this area. He stated in the interview that there were few opportunities to develop these skills within the academic program.

With speaking, only a little bit with university. We have discussions in seminar and tutorials, but I only talk a few times in classes. (Student Interview 8, 15:68)

Unlike many of the other students in the study, Ollie had not worked while he studied. He sought opportunities to improve his English skills outside the university by talking to his Australian flatmate and speaking English at church. He thought that it was important to develop his speaking skills for employment purposes. Ollie stated in the interview that speaking skills were more important in the workplace than academic skills such as reading and writing (Student Interview 8, 35:20). However, he spent most of his time speaking in his mother tongue with his friends.

Ollie believed that his Test 2 results did not represent the improvement he felt that he had made during his studies, although he indicated in the questionnaire that his reading and writing skills were stronger that his speaking skills. He certainly placed more emphasis on improving his reading and writing skills during his study. His advice to future international students placed greater emphasis on actively seeking opportunities to develop English language proficiency.

I guess for those who are having difficulties, consult TLU [the Teacher and Learning Unit] as much as possible, and also spend more time with tutors and lectures, and try to mingle with locals as much as possible.

(Student Interview 8, 22:50)

6.2.3 Academic staff interviews

This section reports on the main themes that emerged from the staff interviews regarding educational, personal and social factors influencing international students’ English language improvement. The following analysis categorises the main factors that seemed to influence students’ English language improvement. It is important to stress at this point that these factors seemed to interrelate and merge. This is especially true in discussing the social and personal factors. In the discussion below:

• DP refers to disciplinary academic participants, who are responsible for teaching subjects within the courses available in the faculty
• TLUP refers to the Teaching and Learning Unit participants, who offer English language support within the faculty.

6.2.3a Educational factors

All of the six staff participants interviewed agreed that international students’ English language skills were important for their future employment. Comments ranged from “Language is important for students to find a job” (Staff Interview 1, DP1) to the following comment by TLUP 1:

There was a survey recently with Graduate Careers Australia and they surveyed all the employers of a large sample. They asked ‘what do you value most in a graduate?’ and top of the list, oral, written communication and English, and way down was work experience and academic excellence. (Staff Interview 1, 23:07)

He considered the issue very important for international student graduates, offering the following comment:

I’ve heard anecdotally that some accounting firms prefer to hire non-accountants who are native speakers and train them in accounting than to hire accountants. This is a pretty serious problem. (Staff Interview 1, 24:05)

One participant, DP 2, agreed that employer needs are important and that poor English skills reflect negatively on the university:

If I were an employer and I had a student with really good marks sitting in front of me and they had trouble stringing a sentence together or their resume was full of grammar holes…it would reflect badly on the institution. (Staff Interview 2, 26:50)

The staff participants agreed generally about the importance of international students developing their English language skills during the course. There was less agreement about the type of English language skills students needed to develop. These were defined in broad terms and ranged from communication skills required for employment (TLUP 1, 2 and 3, DP 2) English academic writing skills required for assessment (TLUP 1 and 2, DP 1) or social skills required for group work (TLUP 1, 2 and 3, DP 2 and 3).

For DP1, international students should arrive at university with basic communication skills which they should then develop alongside their local Australian counterparts:

We admit international students who sort of have a baseline level of communication skills which we think of as appropriate for studying at a high level tertiary institution, and then hopefully I thought we should be developing the communication skills of everyone. (Staff Interview 1, 13:34)

However, TLUP 1 felt that it was not a simple matter for international students to develop their language skills in class, especially when they were in classes with students from the same language backgrounds.

Some classes are composed of 100% students from China. How can we expect them to improve when [their] colleagues are compatriots? (Staff Interview 1, 14:25)

The large numbers of international students, often from similar first language backgrounds, appeared to hinder the opportunities for language development within classroom teaching.

One view from TLUP 1 was that some international students did not improve because their English language reached a plateau (Staff Interview 1, 8:23). He also indicated that the lack of writing in assessment tasks within the faculty could hinder students’ writing development.

Students don’t do a lot of writing, in our faculty they do a lot of group work so their writing skills are masked by other people in the group that might be better. I had a student in the other day that had been in here for two years and he told me this is his first individual assignment in his postgraduate degree. So that would have an impact, not getting the individual writing skills early. (Staff Interview 1, 10:32)

DP 1 agreed with the above comments (Staff Interview 1, 10.42), highlighting the lack of writing within the accountancy course, which had large numbers of international students. For DP3, English language grammar was a factor in assessment:

When I first started I used to mark for English, and then after I marked about 10 crappy (sic) English essays they started making the same mistakes. So I gave it away. In general the student who gets a high result has written a good essay. A student who has most of the facts and just poorly presented will be given a lower mark. (Staff Interview 2, 15:04)

TLUP 3 commented that students would be more motivated to develop their English language skills if it was assessed by academic staff:

The students are motivated by the assessment. I don’t want to generalise but if it’s included, often they come in to see me and if they bring in a marking criteria sheet which includes ‘creative expression, fluency’ those types of things as criteria, then they do take it very seriously. They will say to me, “Is my English okay? What can I do?” If it’s not there, you can’t blame them. (Staff Interview 3, 26:40)

The nature of writing tasks and the assessment of English language skills seemed to be an important factor in influencing international students’ language development.

There appeared to be differing views from the participants about where the responsibility lay for English language development of international students. DP1 stated that it was not the faculty’s responsibility, “we certainly shouldn’t be thinking of ourselves as an English improvement service for international students” (Staff interview 1, 13:34).

Some considered it to be the student’s responsibility. DP3 suggested that language development would occur almost by osmosis, as students developed their confidence and were exposed more to the English language environment.

I have taught 2^nd year Environmental Economics and 3^rd year Environmental Economics. You give essays to the 2^nd year students and they lack confidence, they’re in there, how do I write the essay, what’s required etc. But when they get to 3^rd year, they’ve got a lot more confidence and get up and go and do it. So somehow they are maturing. (Staff Interview 2, 8:16)

Others stated that international students did not take the opportunities available to improve their language skills. For example, DP2 argued that the students were given the opportunity to develop their oral communication skills by presenting in class, but that the students did not want to participate:

I teach a masters group that are meant to be the top of the crop. I was gobsmacked at the start of the semester. There was a group presentation and it’s made as non-threatening as possible but it’s part of the assessment. I had three groups of students come up to me saying, “Oh I don’t want to do this because we are non-English speakers”, like they were three years old or something. First time I struck that and have taken the course for several years.

(Staff interview 2, 33:10)

The TLU participants proposed that international students required a more targeted approach, with support from the disciplinary staff in the faculty. TLUP 2 stated there was little support from disciplinary staff in emphasising the importance of language development to the students:

So if it’s strongly encouraged by the head of department, I think there would be a bigger take-up of the undergraduate program than there currently is. It’s really left up to the students to find out on their own, no-one strongly encourages them from people who they see as being the leaders in the disciplinary teaching. (Staff Interview 3, 25:40)

Her colleague, TLUP 3, voiced similar views:

It is important that language is valued and there is opportunity for students to learn. That it is included in the educational program. (Staff Interview 3, 25:10)

As in most Australian universities, English language communication skills are classified as a generic skill at The University of Melbourne. TLUP 2, representing the views expressed by TLUP 1 and 3, argued that generic skills should be taught within the disciplinary content:

One of the [graduate] attributes is communication skills and faculties have to take that seriously and integrate that into the curriculum, not just for ESL students but all students. There should be some improvement in their literacy or communication skills. More so for the ESL students and I think it’s a responsibility to put in some kind of writing program or language development program that’s embedded within the curriculum so that from 1^st year all the way up they are improving both, so they can go out and function in the workplace. They might have a degree but if they can’t function in a workplace what use is that degree?

(Staff Interview 3, 20:50)

Offering a different view about the relationship between language development and disciplinary teaching, DP 3 argued that generic skills were employer focused, while the disciplinary teaching was academic focused:

Well that’s basically what the generic objectives in degrees and subjects are all about. Verbal and written presentation, working in groups…being able to be critical. I read most of those objectives as being pitched towards what employers want rather than what I’d call our academic objectives. (Staff Interview 2, 25:44)

TLUP 3 suggested that students should be better informed about the need to improve their English language skills. Discussing undergraduate students, she argued:

A lot of students have said to me “but why are we worrying about language, we are doing a commerce degree, why does anybody care about what our language is like?”

(Staff Interview 3, 21:58)

Her colleague, TLUP 2, offered a different perspective:

But I’ve had the opposite, particularly postgraduate students. They just look for programs and so their attendance - they are optional programs - and their attendance is so high. They just love any opportunity to develop their language skills and communication skills, maybe they are maturer so they can see the value in that, a lot of them comment they are not here just to do a degree; one of their objectives is to improve English. So they appreciate the opportunities - very small I must say in our faculty - opportunities to improve.

(Staff Interview 3, 22:20)

TLUP 3 observed that postgraduate students valued the importance of English language skills more than undergraduate students because they had experience of the workplace:

I think the postgraduates have been out in the workforce, they’re aware of the power of English in their own countries, and the employment potential. I think for undergraduates it’s get the piece of paper, do the degree and that’s it. (Staff Interview 3, 23:07)

The above responses illustrate the different perspectives of where English language skills should be taught within the faculty. There appeared to be some tension between the disciplinary and TLU participants about where and how international students could develop their English language. They also indicated that students needed to take the opportunities offered within the faculty, although it appeared that some international students might not be aware of why they were required to further develop their language skills or how they could do this. What emerged from the staff interviews was an unclear vision of which language skills students needed to develop, where these skills should be developed and who should be responsible for developing them.

6.2.3b Personal factors

The participants interviewed were mainly in agreement about the personal factors that influenced international students’ English language improvement. Most stated that motivation was the most important factor for student improvement. Typical responses included, for example, DP 2:

But if you’re motivated enough and committed to practicing, I mean you might not make it to the top of the field but you’d probably get better. (Staff Interview 2, 21:30)

His colleague, DP 3, expressed similar views indicating that international students should be motivated enough to place themselves in situations where they have to communicate in English:

If you were a student that came into poor English skills and you started off in an apartment with a whole lot of peers with poor language skills, spoke Mandarin, went over to football and still spoke Mandarin, you’re in a lot of trouble. (Staff Interview 2, 22:04)

TLUP 2 also agreed that students could succeed if they were motivated to do so:

I think motivation…the students’ individual motivation I think is the key factor to whether they make the extra effort to learn the language. (Staff Interview 3, 28:55)

Most of the comments about motivation concern international students taking advantage of the opportunities available, such as language support from the TLU, developing their language skills through their own study, speaking to students in English, and living and socialising with people where English is the main language for communication.

6.2.3c Social factors

In analysing the data, there appeared to be a close relationship between the personal and social factors influencing language improvement. In general, the participants seemed to express the view that if students were motivated then they would seek social opportunities that would assist in their language development.

According to many of the participants, language development would be hindered if students chose to socialise with others from the same language background outside university. For example TLUP 1 observed:

A lot of students, especially from mainland China, hang around with their own ethnic group and it’s not unusual for their English skills to get worse. (Staff Interview 1, 10.30)

However, according to TLUP 3, it could be difficult for students to socialise beyond their first language group because of the large numbers of international students and the students’ preference for mixing with others from similar language backgrounds.

A lot of students say that they came here to mix with Australians, but it’s easier to be with students from their home country. Some try to find Australian friends but there still is that cultural divide. Australians hang out with Australians, and international students hang out together. The more students there are, the harder it becomes, even with mentoring and social programs. (Staff Interview 3, 27:47)

Many of the participants felt that an important influence on students’ English language development was the type of people with whom they socialised. TLU 1 voiced a typical observation made by participants:

Where they live obviously. If they are living with a homestay family obviously their English will improve a lot faster than if they lived with their compatriots. That’s very rare. And also if they do volunteer work or paid work in the community that would have an impact. Whether they make an active attempt to meet with native English speakers - they don’t have to be

Australians, can be Germans, Russians as long as their English is better than their’s it would be helpful. That would involve joining clubs on campus and getting out of their comfort zones, meeting people…actively improving their English literacy. (Staff Interview 1, 15:35)

While most of the socialisation outside the university seemed to be a matter of student choice, the academics interviewed said they tried to provide opportunities for students to socialise within the class. One of the main initiatives of the faculty had been the socialisation tutorial, which DP 1 described:

The faculty has been doing things at undergraduate level to get better integration. A lot of subjects have introduced this socialisation tutorial. We never used to have tutorials in the first week but now we do. They’re completely dedicated to people getting to know each other. Hopefully it gets integration going. (Staff Interview 1, 14:49)

However, TLUP 1 felt that international students tended to avoid socialising with local students because they preferred to be surrounded by speakers of their first language:

They come here to improve English. They know how important it is. It’s the world language, but when they come here it’s a natural tendency to gravitate to one’s own and speak one’s own language and it takes a lot of effort to break that tendency. But it doesn’t help if the whole class composition is made up entirely from your own country. (Staff Interview 1, 21:00)

It appears that while academics can contribute to creating a teaching environment that is inclusive of all, the students also have to take these opportunities, as well as others outside the classroom, both within and outside the university. DP2 shared his experiences:

So personal and social factors do play a part. We’ve got a lot of opportunities to help them in this respect - you know, in encouraging formulation of groups in class - but we can’t totally influence it. Admittedly not all students want to assimilate with students out of their demographic, whether it be age or culture-based demographic. (Staff Interview 2, 23:22)

The implications of the findings from these case studies are examined in the next section.

7. DISCUSSION

7.1 Score gains

The statistical analyses of the Test 1 and Test 2 score data indicated the following.

• The greatest average improvement in Test 2 was in Listening and Reading and the least average improvement was in Writing.
• The average improvement on Listening, Reading and Writing was closely correlated, that is, students tended to improve equally in these three areas.
• The average improvement in Speaking was not significantly correlated with the other three skills, ie students tended to improve more or less well in Speaking than they did in Listening, Reading and Writing.
• Students with a lower average combined result in Listening, Reading and Writing in Test 1 tended to improve significantly more than students with higher results.

The finding that the greatest average improvements in Test 2 were in Listening and Reading, followed by Speaking and then Writing, suggested that it is more difficult to improve in the productive than receptive skills on the IELTS Test.

The fact that the pattern of improvement in Speaking was not consistent with Listening, Reading and Writing was an important finding and one which helped to inform the selection of two of the case studies. As suggested below it then helped to build the model explaining student development in English.

The finding that students with lower initial scores improve more than those with higher scores is also important. Green (2005) suggests that this result generally reflects both an effect of regression to the mean attributable to measurement error and to more rapid progress at lower levels of language proficiency. Regression to the mean predicts that those scoring above the mean in Test 1 will be more likely to score lower in Test 2 and vice-versa. According to Green (2005, p 58) this occurs:

because the proportion of the score that is attributable to (measurement) error may advantage an individual candidate on one occasion and disadvantage them on another. Candidates scoring above the mean at Time 1 are more likely to have benefited from error and those scoring below to have suffered from it. At Time 2, the unlucky candidates see their scores increase and the initially lucky see them fall.

He provides the example of the Writing test where factors unrelated to Writing ability, such as variations in rater harshness or topic familiarity on the part of candidates, may affect their score.

Note, however, that the figures for the standard error of the mean (Tables 7, 8 and 9) are quite low and therefore the effect attributable to regression to the mean may not be especially large in this study.

In addition, as Green (2005) suggests, progress in language learning tends to be more rapid at lower levels of functional ability and this may be reflected in the more rapid progress through the lower IELTS bands. This finding is consistent with the results of a previous study into improvement over the course of 10-12 weeks’ intensive English language instruction before university entry (Elder and O’Loughlin 2003).

It should be remembered that there were two sources of uncontrolled variability in the test score data: a) the amount of time between when students sat for Test 1 and commenced their course of study and b) the amount of time students had been studying their higher education course when they sat for Test 2 according to whether they were undergraduate or postgraduate students (between six months and two and a half years). Obviously these sources of variability reduce the accuracy of the findings in relation to score gains. Given these two different sources of variability and the relatively small sample size (N=63), the statistical findings should be viewed cautiously. They should be read as suggestive of trends rather than as authoritative results.

7.2 Factors influencing score gains

7.2.1 Statistical analyses of the test score data and student questionnaire responses

Of the various factors possibly influencing score gain, only four were found to be statistically significant in the analyses.

• Undergraduate students improved more than postgraduate students.
• The longer the time period since students took Test 1, the more likely they were to improve in Test 2.
• The longer students had spent studying English previously at university, the less likely they were to improve in Test 2.
• Students who had stayed in an English-speaking country at some point in the past were more likely to improve in Test 2.

As suggested above, a limitation of the statistical analyses of score gain in this study was the uncontrolled variability related to when Test 1 was undertaken by the students (any time in the two years before they began their current course) and the length of time between Test 1 and Test 2 (between six months and two and a half years depending on their level of study). The significant results relating to level of study and years since IELTS Test 1 confirm that both factors were indeed important sources of variability. The results for the other two factors - previous English study at university and previous stay in an English-speaking country - which yielded statistically significant results, were difficult to interpret at face value without additional information.

While the quantitative findings from the student questionnaire hinted at important correlations in the development of language skills, they could not tell us what outside factors influenced score gains. To understand these factors, we must turn to the qualitative findings from the questionnaire as well as the interview data.

7.2.2 The case studies and staff interviews

Analysis of the case studies and staff interviews indicated the following.

• International student agency (ie their ability to make choices concerning English language development) is an important factor in influencing English language improvement.
• The degree of English language support sought within the university and the degree of contact with English outside the university appear to strongly influence students’ English language improvement.
• The institutional practices within the Faculty of Economics and Commerce afforded few opportunities for international students to develop their language within the disciplinary teaching and learning context.
• Academic staff lacked the ability to change the institutional practices within the faculty.

What emerges from the analysis of the case studies is that student agency is most important in terms of English contact outside of university and seeking English language support within university, which appears to influence students’ English language improvement. For example, Charmaine and Carlos exercised their agency by actively seeking opportunities to speak English and develop their social English skills, as well as seeking support from the TLU or other English language development programs. These actions seem to have resulted in improvement in their language development. Other students, such as Truc and Daisy, who had few opportunities to develop their social language skills and were not required to write extensively in their course, experienced limited development in their English language during their studies. Dian sought experiences to communicate in English outside the university, while he struggled to understand the writing conventions of his discipline and did not seek support from the TLU or academic staff. As a result, his Speaking score in Test 2 improved more than his Listening, Reading and Writing scores. On the other hand, Ollie prioritised the development of his writing skill and, while he sought some opportunities to speak English outside the university, he preferred the company of his Indonesian friends. For him, his Listening, Reading and Writing scores improved in Test 2, while his Speaking skills remained the same.

What has emerged from the study is the importance of English language contact inside and outside the university to improve students’ language ability. This has been represented in Figure 3. The vertical axis represents English language contact outside the university and the horizontal axis represents academic language support.

The four cells in Figure 3 are exemplified by the different student case studies and, when taken together, indicate the trends that emerged from the study. While the test score analyses for the entire student cohort should be treated cautiously for reasons previously outlined, Figure 3 accurately represents the trends that emerged from the case studies and staff interviews. Further research needs to be conducted investigating international students’ English language development in other university

sites and within different disciplines. The findings from this study may not be generalisable to other disciplines. For example, many of the students from the Faculty of Economics and Commerce were not required to write extensive pieces of writing. However, it is noteworthy that Dian, who was completing a 10,000 word research paper for his course, expressed similar views concerning social language development to other students in the study who were completing smaller pieces of writing. In faculties such as Arts, students may be writing more extensively, but collaborative learning is usually taking place in tutorials, therefore social English language as well as academic disciplinary writing, reading and listening, is important. The findings suggest that research into international students’ English language development needs to consider students’ agency in adapting to their studies, as well as the kinds of support that is available to them within their institutions.

Figure 3: Interrelating factors influencing English language development

Most of the discussion in the wider literature concerning international students in English-speaking universities has centred around issues of their English language skills. There has been extensive research into how international students develop their academic writing skills, which has focused on the influence of cultural background and writing traditions in developing western academic writing (Cadman 1997; Connor 1996; Fox 1994). It has also explored the influence of academic practices within certain disciplines within institutions (Lea and Street 2000; Lillis 2001). Academic writing has been perceived in the research literature as central to international students’ success in higher education.

More recently, research has explored the subjective realities of international students as they adapt to various western-learning environments (Hellstén and Prescott 2004; Kettle 2005; Morita 2004). These studies explore notions of identity and agency as international students adapt to the academic discourse and practices of their disciplinary communities. Learners are viewed as agents in the classroom and their agency is both shaped and constrained by the institutional practices of the university. Many of these studies explore language socialisation within higher education, which “refers to the process by which novices or newcomers in a community or culture gain communicative competence, membership and legitimacy in the group” (Duff, 2007, No. 162: 310). They draw on the notion of communities of practice where social practice is viewed as central to learning. Lave and Wenger (1991) refer to the process by which learners are socialised into a community of practice as ‘legitimate peripheral participation’ (LPP). They argue that peripherality and legitimacy are required to make participation possible. Peripherality provides opportunities for participation that give exposure to actual practice, in the sense that it “offers the student a way of gaining access to the sources of understanding through growing involvement” (Lave and Wenger 1991, p 37). In other words, the pedagogy becomes visible to the learner by allowing them to participate in, and understand, the community of practice. To gain access to the community of practice and have the potential to become an insider, Wenger (1998, p 101) argues that newcomers must be granted enough legitimacy to be treated as potential members. He argues that gaining legitimacy is important, as newcomers to the community “are likely to fall short of what the community regards as competent behaviour”.

The findings in our study suggest that spoken language proficiency is an important factor in international students gaining legitimacy within their disciplinary community of practice. However, this was difficult given that most of the newcomers to the community of practice were international students, requiring socialisation into the academic disciplinary discourse. The academic participants in particular seemed concerned about the lack of participation by international students in group work and oral presentations. The students indicated that this was in part due to their lack of social English skills.

Whereas the focus in many of the studies on language socialisation within higher education has been on academic presentations given in seminars and tutorials, as well as professional learning contexts (Duff, 2008 No. 163), the findings from the study reported here seem to suggest that the students were more concerned about developing their social, rather than their academic, oral English skills. This may in part be due to the lack of focus on oral presentations in the Faculty of Economics and Commerce, as indicated through the student interviews. There appeared to be few opportunities for the students to develop their spoken language skills, as most of the language support offered through the TLU was for written language support. However, the students were required to work in groups and complete group assignments. To gain legitimacy within their community of practice within the university, the students in the study sought to find employment or to place themselves in situations were they could speak English. Students sought to develop these skills outside of the university, indicating that this assisted with their participation within their disciplinary communities of practice.

In the interviews, academic participants indicated that they were not necessarily responsible for the English language development of international students. They believed that the language development would occur naturally over the period of time students spent at the university. In addition, the academics noted that factors such as student motivation to improve their language skills, as well as opportunities to communicate with English speakers, were important in developing students’ English language. The academic participants did not appear to take responsibility for the language learning needs of their international students, stating that personal and social factors influence the students’ language development. The TLU staff seemed to lack significant influence within the faculty to support the students, relying mainly on students individually seeking support from the unit. For the

students who chose to attend the TLU, it appeared that this action positively influenced their language development. However, the TLU staff participants stressed the importance of support from the academics for the TLU programs to ensure that international students were aware of the services and so that language development could be more closely aligned with the disciplinary discourse community.

8. CONCLUSION

The study has sought to investigate IELTS exit score gains in higher education and the factors that influence these for international students for whom English is a second/additional language. Students showed the least improvement in Writing. This may be due to the fact that within the Faculty of Economics and Commerce, depending on the choice of subjects studied, students were not required to undertake extended pieces of writing. This seemed to be the situation particularly for the undergraduate students interviewed. The lack of writing development would be of concern to employers both in Australia and overseas. Furthermore, the study found that Speaking skills could not be predicted from improvement in the other three skill areas. This is due in part to the fact that both the students and the teachers indicated the importance of social connections outside of university to assist the students in developing their spoken skills. Student exposure to English outside the university was dependent on the choices students made about seeking opportunities to develop their spoken language skills.

Part of the reason international students generally seek an education in an English-speaking university is to improve their English language skills, and many employers are concerned about the English language proficiency of international students. The study has revealed that it cannot be assumed that international students who complete their degree in an English-speaking university have developed their language skills in all areas. The findings suggest that most show improvement on one or more areas, and very few students show improvement in all areas, as was the case with Charmaine and Carlos. The factors that influenced their improvement were the high degree of contact with English outside the university and the English language support sought within the university.

The findings from the case studies have important implications for the provisions of English language development programs within higher education. While these programs mainly focus on academic language development, the study indicated that social English language use is important for international students. Students should therefore be given ample opportunity to develop their social, as well as academic, language skills for successful participation in university life and beyond.

On a final note, the introduction of an IELTS exit test upon course completion may have positive washback onto the curriculum by increasing the accountability of faculty staff for both the academic and social language development, and encouraging students to take up their responsibility in developing their language skills. However, because of the limitations of the quantitative score gain data in this study, it was not possible to reach any firm conclusions about the potential of IELTS as an exit test for graduates. More research is needed to validate the use of the IELTS for this purpose. Such research needs to examine whether the Test accurately reflected students’ proficiency at the end of their course of study.

References

• Archibald, A, 2001, ‘Managing L2 writing proficiencies: areas of change in students’ writing over time’ in International Journal of English Studies vol 1, no 2, pp 153-174
• Bachman, L, 1990, Fundamental considerations in language testing, Cambridge University Press, Cambridge
• Berry, V and Lewkowicz, J, 2000, ‘Exit-tests: is there an alternative?’ in Hong Kong Journal of Applied Linguistics, vol 5, no 1, pp 19-49
• Brown, JDH, 1998, ‘An investigation into approaches to IELTS preparation, with particular focus on the academic writing component of the test’ in IELTS Research Reports, Volume 1, IELTS Australia Pty Ltd, Canberra, pp 20-37
• Davies, A, 1992, ‘Is proficiency always achievement?’ in Melbourne Papers in Language Testing, vol 1, no 1, pp 1-15
• Davies, A, 2008, ‘Assessing academic English’ in Studies In Language Testing, Volume 23, University of Cambridge ESOL and Cambridge University Press, Cambridge
• Duff, P, 2007, ‘Second language socialisation and sociocultural theory: insights and issues’ in Language Teaching, vol 40, pp 309-319
• Duff, P, 2008, ‘Language socialisation, higher education and work’ in Encyclopedia of Language and Education, Vol 8: Language Socialisation, eds PA Duff and NH Hornberger, Springer, New York, pp 257-270
• Elder, C and O’Loughlin, K, 2003, ‘Score gains on IELTS after 10-12 weeks of intensive English study’ in IELTS Research Reports, Volume 4, IELTS Australia Pty Ltd, Canberra, pp 153-206
• Green, A, 2005, ‘EAP study recommendations and score gains on the IELTS Academic Writing test’ in Assessing Writing, vol 10, pp 44-60
• Harman, G, 2005, ‘Internationalisation of Australian higher education’ in Internationalising higher education: Critical explorations of pedagogy and policy, eds P Ninnes and M Hellstén, Comparative Education Research Centre, Hong Kong, pp 119-140
• Harré, R and van Langenhove, L (eds), 1999, ‘The dynamics of social episodes’ in Positioning Theory: Moral Contexts of Intentional Action, Blackwell Publishers Ltd, Great Britain, pp 1-13
• Hellstén, M and Prescott, A, 2004, ‘Learning at university: the international student experience’ in International Education Journal, vol 5, no 3, pp 344-351
• International English Language Testing System, 2005, IELTS Handbook, University of Cambridge ESOL Examinations, the British Council and IDP Education Australia
• Kettle, M, 2005, ‘Agency as discursive practice: from “nobody” to “somebody” as an international student in Australia’ in Asia Pacific Journal of Education, vol 25, no 1, pp 45-60
• Morita, N, 2004, ‘Negotiating participating and identify in second language academic communities’ in TESOL Quarterly, vol 38, no 4, pp 573-603
• O’Loughlin, K, 2008, ‘The use of the IELTS for university selection in Australia: a case study’ in IELTS Research Reports, Volume 8, IELTS Australia Pty Ltd, Canberra, pp 145-243
• Rao, C, MacPherson, K, Chand, R and Khan, V, 2003, ‘Assessing the impact of IELTS preparation programs on candidates’ performance on the GT reading and writing test modules’ in IELTS Research Reports, Volume 4, IELTS Australia Pty Ltd, Canberra, pp 237-262
• Read, J and Hayes, B, 2003, ‘The impact of the IELTS test on preparation for academic study in New Zealand’ in IELTS Research Reports, Volume 4, IELTS Australia Pty Ltd, Canberra, pp 153-206
• Stake, RE, 1994, ‘Case studies’ in Handbook of qualitative research, eds NK Denzin and YS Lincoln, Sage, London

APPENDIX 1: STUDENT QUESTIONNAIRE

Office Use Only

Date administered: ____

SECTION A: In this section, we are interested in finding out about you.

Please write your responses in the spaces provided.

1. Family name: ____

2. Given name:

3. Email address: ____

4. University of Melbourne student number:

5. Sex:

6. Date of birth: ____

7. Country of birth: ____

8. Nationality: ____

9. First language:

10. Other languages spoken: ____

11. Current course: Undergraduate

Postgraduate

Name of course: ____

12. Date of IELTS test used to enter the university ____

(If you did not use IELTS to enter your current university course, then you cannot participate in this study)

13. What English courses did you do before coming to Australia?

I studied English at secondary/high school in (name country) ____ for ____ years ____ months.

I studied English at university in ____ for ____ years ____ months.

I studied English at a private language school in ____ for ____ years ____ months.

I had a private tutor in ____ for ____ years ____ months.

14. Why did you learn English before you came to Australia?

(circle ‘ yes’ or ‘ no’ for each of the following statements)

15. How long have you been in Australia?

I have been in Australia for ____ years ____ months.

16. Where are you living at the moment? (Tick one of the following)

How long have you lived there?

17. Have you lived in any other English speaking countries?

(circle ‘yes’ or ‘no’)

YES NO

If yes, how long did you live in those countries?

For example: Country 1 2 years 6 months from 2001 to 2003.

Country 1 ____ years ____ months from ____ to ____.

Country 2 ____ years ____ months from ____ to ____.

Country 3 ____ years ____ months from ____ to ____.

SECTION B: In this section we are interested in finding out about how you are coping with your current course.

Please circle the appropriate response option or write your response in the spaces provided.

18. When you began your course did you think your English was good enough to cope well with your studies in Australia? Yes No Unsure

Please explain:

19. Has your English improved while you have been completing your course? Yes No Unsure

List the factors that you think have helped you to improve your English.

List the factors that you think have prevented you from improving your English.

20. Has the University required you to complete English language subjects during your course? Yes No Unsure

If ‘yes’,

a) what English language subject or subjects?

b) how have these subjects helped you with your English (e.g., listening and taking notes in lectures, reading, writing your assignments)?

21. Have you felt that you needed other English language support to cope with your course (e.g. help with your assignments)? Yes No Unsure

Please explain:

22. Have you been given any additional English language support during your course?

Yes

No

Unsure

If ‘yes’,

a) who thought this would be a good idea? (for example, a lecturer, a tutor, yourself)

b) where did you get the support?

c) to what extent has this support helped you with your English?

d) how has this support helped you with your English (e.g., listening and taking notes in lectures, reading, writing your assignments)?

23. What factors in your life outside the university have helped you improve your English or prevented you from improving your English (for example, talking to your Australian friends)?

How have these factors helped you?

SECTION C: In this section we would like you to add any additional comments you have about the issues covered in this questionnaire.

Please make further comments on matters relating to your English language abilities, your IELTS results and success in your studies.

THANK YOU FOR YOUR ASSISTANCE.

APPENDIX 2: STUDENT INTERVIEW SCHEDULE

Each of the students reads over their questionnaire responses before the interview begins.

Questions:

1. Why did you take part in our study?

2. How do you feel about your latest IELTS results?

3. Do you think they reflect your improvement?

4. What language skills do you think IELTS tests?

5. During your course, what happened inside and outside the university to help you improve your English skills?

Prompts - What services did you use?

• What other things did you do?

6. Do you think your experiences of English are typical of other international students?
7. Can you tell us a story about one of your friends that is different from your own?
8. When you first arrived at university, which English language skills did you consider stronger? And now?
9. What do you see the relationship between developing everyday social and academic English?
10. What advice would you give new international students in your faculty about developing their language skills?

APPENDIX 3: STAFF INTERVIEW SCHEDULE

Background information:

Years teaching at tertiary level

Years teaching at the University of Melbourne?

Subjects taught in the faculty

Estimate of % of international students taught at undergraduate / postgraduate level

Briefly discuss the main findings from IELTS Test 1 and Test 2 scores.

Questions:

How would you explain these results?

Focus on writing -

Focus on listening -

Focus on reading -

Focus on speaking -

What educational, personal and social factors do you think influence whether or not the students improve?

APPENDIX 4: IELTS TEST 1 AND TEST 2 INDIVIDUAL RESULTS

APPENDIX 5: SUMMARY OF IELTS SCORE GAINS FROM TEST 1 TO TEST 2

APPENDIX 6: QUESTONNAIRE DATA

Section A: Question 1 - Question 12

In this section, we are interested in finding out about you.

Section A: Question 13: What English courses did you do before coming to Australia?

Section A: Question 14

Why did you learn English before you came to Australia?

(Circle ‘yes’ or ‘no’ for each of the following statements).

Section A: Question 14 (continued)

Why did you learn English before you came to Australia?

(Circle ‘yes’ or ‘no’ for each of the following statements).

Section A: Question 15 - Question16

15. How long have you been in Australia?
16. Where are you living at the moment? How long have lived there?

Section A: Question 17

Have you lived in any other English-speaking countries?

If yes, how long did you live in those countries?

Section B: Question 18

When you began your course did you think you English was good enough to cope well with your studies in Australia? Please explain. (Students did not add any comments).

Section B: Question 19

Has your English improved while you have been completing your course?

A. List the factors that you think have helped you improve your English.

B. List the factors that you think have prevented you from improving your English.

Section B: Question 20

Has the University required you to complete English language subjects during your course? If ‘yes’,

A. What English language subject or subjects?

B. How have these subjects helped you with your English (e.g. listening and taking notes in lectures, reading, writing your assignments)?

Section B: Question 21

Have you felt that you needed other English language support to cope with your course (e.g. help with your assignments)? Please explain.

Section B: Question 22

Have you been given any additional English language support during your course?

If ‘yes’,

A. Who thought this would be a good idea? (for example, a lecturer, a tutor, yourself) B. Where did you get the support? C. To what extent has this support helped you with your English? D. How has this support helped you with your English (e.g. listening and taking notes in lectures, reading, writing your assignments)?