Predictive validity in the IELTS test: A study of the relationship between IELTS scores and students' subsequent academic performance

Abstract

This study investigated the relationship between the IELTS test and academic outcomes. Specifically, it sought to determine the extent to which the IELTS test predicts the subsequent academic performance, as well as the language difficulties, of international students enrolled in an Australian tertiary institution. It also aimed to investigate whether any of the individual tests of Listening, Reading, Writing and Speaking was critical to academic success. This was researched in three ways: through statistical correlations between students’ IELTS scores and grade point average, student questionnaires and staff interviews.

The IELTS scores of 113 first-year international students from the TAFE and Higher Education sectors of the Faculty of Business of an Australian university were correlated with their first-semester grade point average (GPA). In the total sample, significant correlations were found between the Reading and Writing tests and GPA (.262, .204 respectively). When Higher Education and TAFE scores were looked at separately, only the Reading score remained significant for the Higher Education group. While none of the correlations was significant in the TAFE group, the magnitude of the correlation between the Writing test and GPA (.194) was very similar to that for the total sample, which was statistically significant.

Regression analysis found a small-to-medium predictive effect of academic performance from the IELTS scores for the total sample and the Higher Education group, accounting for 8.4% and 9.1% respectively, of the variation in academic performance. The Reading test was found to be the only significant predictor of academic performance in the total sample and Higher Education group. IELTS was not found to be a significant predictor of academic performance for the TAFE group.

The qualitative data on students’ and staff perceptions of the relationship between English language proficiency and academic performance corroborate to some extent the statistical findings, particularly in relation to the Reading, and to a lesser extent, the Writing tests and the skills they represent. However, while the Listening test was not significantly correlated to academic performance, students and staff from both TAFE and Higher Education highlighted the importance of listening skills in first-semester study. Both staff and students were generally positive about students’ ability to cope with the language demands of their first semester of study. Aside from language, staff also saw sociocultural and psychological factors such as learning and educational styles, social and cultural adjustments, motivation and maturity, financial and family pressures to have an influence on the academic outcomes of international students in their first semester of study.

1.0 Introduction

The International English Language Testing System (IELTS) is one of several English proficiency tests used by tertiary institutions in Australia, Britain, Canada and New Zealand to assess the English language proficiency of international students from non-English speaking countries applying for a course of study. In these tertiary institutions a specified minimum score on IELTS or alternative proficiency measure is a pre-requisite for entry into a course. The setting of such minimum scores rests on the widely accepted assumption that a certain level of language proficiency is necessary for successful academic performance in an English-medium tertiary course. The issue of the predictive validity of a test such as IELTS is a crucial one, as these tests serve a gate-keeping role for tertiary institutions. Over the last ten years, this has been an issue of growing importance as a result of the increased enrolments of international students at institutions around Australia. However, despite increased international enrolments and the crucial role these tests play, there is little clear evidence so far of the exact nature of the relationship between international students’ scores on proficiency tests such as IELTS and their subsequent academic performance in tertiary courses.

A number of studies have researched the relationship between English language proficiency and the academic success of international students in different contexts (see Graham 1987 for a review of research). In these studies there is little agreement so far about the relationship between scores on screening tests for English language proficiency and students’ subsequent academic performance. The conflicting results and inconclusive evidence from these studies can be attributed to problems with:

• defining and measuring English language proficiency,
• defining and measuring academic success,
• the large number of uncontrolled variables involved in academic success or failure (Graham 1987).

In these predictive validity studies, English language proficiency is usually measured by scores on various commercial tests such as TOEFL and IELTS, or by locally-based, institution-specific assessment. Proficiency is therefore defined in terms of performance on a particular test. As each test is different, the definition of proficiency will necessarily vary from test to test (Graham 1987), making comparison between studies difficult. However, Graham (1987) points out that this may not be a major problem, as many studies have found high correlations between the various tests of proficiency commonly used in these contexts.

In many predictive validity studies the criterion most often used for judging academic success is the first-semester grade point average (GPA). It has been argued that GPA does not take into account the number of subjects attempted by a student (Heil and Aleamoni 1974 in Graham 1987), nor does it take into account the nature and demands of different types of subjects. However, in support of the use of GPA in these studies, a large scale study of 2,075 foreign students in the University of California, Los Angeles (UCLA) found that GPA was the best predictor of students’ subsequent academic performance (Sugimoto 1966 in Graham 1987). Other studies, including more recent studies on the predictive validity of IELTS (Bellingham 1992, Cotton and Conrow 1997, Davies and Criter 1988, Elder 1993, Ferguson and White 1993, Fiocco 1992, Gibson and Rusek 1992), have used different measures of academic performance. These include staff perceptions of a student’s performance, student self-ratings, pass/fail and permission to proceed to the next semester. As a student’s GPA does not provide a complete measure of their performance, it has been suggested that more than one measure of academic performance should be used in predictive validity studies (Jochem et. al. 1996 in Cotton and Conrow 1997).

The length of time between measures taken of English language proficiency and academic performance allows for differential rates of learning and a multiplicity of variables to influence outcomes (Davies and Criter 1988). A range of other variables has been reported in the literature as having a possible influence on academic achievement. These include: a student’s area of study (Light, Xu and Mossop 1987), cultural background and country of origin (Wilcox 1975 in Graham 1987), whether they are graduate or undergraduate students, international students or residents, personality and attitude (Ho and Spinks in Graham 1987), motivation, homesickness, attitudes to learning, adjustment to the host culture (Gue and Holdaway 1973 in Graham 1987), and age and gender (Jochem, et.al. 1996 in Cotton and Conrow 1997). In the Australian context, financial difficulties, family pressures to perform well, and amount of preparation before a course have also been identified (Burns 1991 in Cotton and Conrow 1997). Such factors are often impossible to control in these studies and can also limit the degree to which generalisations and comparisons can be made.

Despite the lack of comparability, conflicting results, and differing contexts of these predictive validity studies, a number of commonalities are emerging in the findings on predictive validity studies on the IELTS tests. Firstly, there seems to be growing evidence that the lower the English language proficiency, the greater an effect this has on academic outcomes (Elder 1993, Ferguson and White 1993). Secondly, there seems to be more likelihood to finding positive relationships between proficiency and academic performance when the variable of area of study is controlled (Bellingham 1993, Davies and Criter 1988, Elder 1993, Ferguson and White 1993). Lastly, any positive relationships found between proficiency measures and academic achievement tend to be weak. This may be because academic performance is affected by many other factors aside from language. It may also result from the limited range of proficiency scores in these studies, as students with very low scores are generally not admitted into university.

1.1 Research Objectives

The present study was undertaken to investigate the predictive validity of IELTS for first-year international students enrolled in the Faculty of Business at RMIT University in Melbourne in 1997. The Faculty of Business was chosen because business courses attract the highest number of international students at RMIT and at other tertiary institutions. At RMIT there are over 1,300 students enrolled in this Faculty’s Technical and Further Education (TAFE), undergraduate and postgraduate programs. The study was proposed as a pilot to investigate the academic outcomes of international students enrolled in other faculties of the university.

The research questions were:

1. To what extent does the IELTS test predict the subsequent academic performance of international students in this Faculty as measured by their grade point average? Are any of the individual tests of Listening, Reading, Writing and Speaking critical to academic success or is the overall band score sufficient?

2. What are international students’ perceptions of the relationship between their own English language proficiency and academic performance? For these students, what are the key factors which have the most effect on their own academic performance? To what extent do these perceptions corroborate the more objective correlations undertaken between IELTS scores and academic grades?

3. What are academic staff’s perceptions of the relationship between international students’ English language proficiency and academic performance? For these academic staff, what are the key factors which have the most effect on students’ academic performance? To what extent do these perceptions corroborate the more objective correlations undertaken between IELTS scores and academic grades?

The major part of the study focuses on investigating the predictive validity of the IELTS test in this particular context. However, staff and students’ perceptions were also sought to gain a closer and more personal participant perspective, and gain further insights on the relationship between English language proficiency and academic outcomes.

The following section provides an outline of the method and data collection procedures used in the research study. Section Three reports on and discusses the quantitative findings of the predictive validity of IELTS, followed in Sections Four and Five by the qualitative findings and discussion on student and staff perceptions of the relationship between their own English language proficiency and academic performance. Sections Six and Seven provide some conclusions and recommendations.

2.0 Method

This section outlines, in turn, the stages of data collection for each of the following sets of data collected for the research:

1. The IELTS scores and first-semester academic results of all first-year international students in the Faculty of Business’ TAFE and undergraduate, Higher Education (HE) sectors who had been identified as having enrolled with an IELTS score to fulfil the entry requirements
2. Student questionnaire data collected from the above-identified population
3. Data from interviews conducted with academic staff.

2.1 Students’ IELTS Scores

With the approval of the Faculty of Business and the University’s International Services Unit, those 1997 first-year international students enrolled in undergraduate and TAFE Business courses who had gained entry with an IELTS score were identified. There was some difficulty identifying these students from RMIT International Student Records as, contrary to expectations, students’ English entry levels were not included in the central database. It was therefore necessary to manually sift through the individual files of first-year 1997 international students from this Faculty. A total of 113 students were identified. This represents fewer than 10% of international students enrolled in the Faculty, a lower figure than expected. In terms of the courses they represented, the students were distributed across a wide range of the Faculty’s courses, as Table 1 shows:

Table 1: Distribution of students by course

2.2 Students’ Academic Scores

Once this population was identified, the first-semester academic results of the 113 students were obtained from Central Administration. The majority of Higher Education results were in numerical marks; however, a small number were letter grades only. For the TAFE students, all the academic results were in letter grades only. In order for comparisons to be made, both sets of results were coded into scores of 0 - 5 (See Table 2). In coding the scores it was decided that results such as W and WI (different types of withdrawals) would be awarded a ‘0’ as they were qualitatively different to a N/NN (fail) score, which was awarded a ‘1’. DNS (‘Did not sit’) and RWT (‘Result withheld’) grades were awarded a ‘1’, i.e., the equivalent of a ‘fail’ rather than a ‘withdrawn’.

Table 2: Coding of numerical marks and letter grades

2.3 Calculation of Grade Point Averages

The grade point average was then calculated for each subject. The distribution of academic performance on all subjects is presented in Appendix 4.1. Although the number of TAFE and Higher Education students was almost the same, TAFE students took about three and a quarter times more subjects in total (754) than Higher Education students (229). Furthermore, a sizeable percentage (18%) of all subjects taken by TAFE students resulted in a Withdrawn grade (either W or WI), whereas none of the Higher Education students withdrew from a subject. Because of this discrepancy, two measures of Grade Point Average (GPA) were calculated in order to examine the possible confounding effect of this fact:

• The Grade Point Average score obtained by including withdrawn subjects in the calculation: this is referred to in the following as GPA0; and
• The Grade Point Average obtained by excluding any subjects with a W or WI grade: this is referred to in the following as GPA.

Because the GPA0 and GPA scores for the Higher Education sample are the same (ie., these students did not withdraw from any subjects) the results for this sample are the same for all analyses on the two ways of calculating Grade Point Averages. The results of analyses for the total sample however will differ.

2.4 Student Questionnaires

A student questionnaire was devised (Appendix 4.2) which sought to elicit students’ perceptions of the adequacy of both their IELTS scores and general English language proficiency for academic performance in their first semester of study. Specifically, the questionnaire sought information on the following:

• whether the students had received further tuition in English after taking the exam and before commencing tertiary study,
• whether they thought their English had been adequate for study in the first semester of their course,
• whether they thought they needed a higher IELTS score for the first semester of their course, and in which areas,
• which area(s) of language they thought presented difficulties,
• whether particular methods of assessment had presented varying degrees of difficulty,
• which subject was perceived to have been the most difficult and which the easiest, and why,
• the amount of concurrent English language support they had received in their first semester of study.

The design of the questionnaire was reviewed by a number of people including an academic with a specialisation in statistics in educational research. A staff member from the TAFE sector of the Faculty and language teachers ensured that the questions were unambiguous. A number of revisions were made subsequent to these consultations.

An accompanying letter was written which stated the aims of the study. Students were advised that completing the questionnaire was not compulsory. The students who had been identified in the first set of data were then sent the questionnaire.

2.5 Staff Interviews

An interview schedule was designed which sought academic staff’s perceptions of the adequacy of first year international students’ IELTS scores and general English language proficiency for academic performance in their first semester of study. Specifically, the interviews sought information on the following:

• Staff role in relation to international students,
• Staff familiarity with the English language entry requirements set by the Faculty, and in particular, with the IELTS test and the significance of the scores,
• Whether staff thought that the minimum score on any of the IELTS tests (Listening, Reading, Writing or Speaking) was of particular importance for success in attempting the first semester of their subject,
• Whether staff thought that the level of English required of international students for entry into the Faculty was adequate,
• How staff thought international students coped with the linguistic demands of their course in the first semester and the area(s) in which they thought students experienced most difficulty,
• Whether staff thought international students experienced more difficulties than Australian students in their first semester of study,
• Staff opinion on the amount of language support international students should or already received from the Faculty.

Again, a Faculty staff member and language teachers scrutinised the interview schedule but no revisions were necessary. Staff members representing the maximum range of courses from both the Higher Education and TAFE sectors were contacted for their availability to be interviewed. Nineteen staff members were interviewed: nine from TAFE, eight from HE and two staff members involved in international student support:

Table 3: Distribution of staff interviewed by course

Interviewees were given the questions in advance of the interview in order to allow them to consider their answers. Interviews took place over the space of three weeks, and each interview lasted around 25 minutes. The interviews were audio-taped and later transcribed for analysis.

Results and Discussion

In these sections the results from the three sets of data collected - correlations between IELTS scores and academic grades, student questionnaires, and staff interviews - will be reported on and discussed in turn.

3.0 The Relationship between IELTS Scores and Academic Grades

This section will firstly report on and discuss the distribution and range of IELTS scores and grade point averages, then report on and discuss the correlations and regression analysis performed on these two sets of measures.

3.1. IELTS scores

In the Faculty of Business, the minimum overall IELTS score for Higher Education is set at 6.5, with a minimum of 6 for any individual band. For TAFE the cut-off score is set at 5.5 overall, with a minimum of 5 for any individual band. Despite these requirements, the range of scores for IELTS was greater than expected. The scores ranged from 3.5 - 7.5 for overall bands, and 3 - 9 for individual bands. Table 3 shows the number and percentage of students in the sample who achieved scores below the minimum set for both TAFE and Higher Education:

Table 4: Number and percentage of IELTS scores that fell below the minimum requirements set by the Faculty

There are a number of possible reasons as to why there was a surprisingly high number of students who gained entry into the course with lower than the required minimum scores. Firstly, it is quite likely that these students were able to satisfy other criteria about which we have no information. Students may have, for example, completed a 10-week EAP or Bridging course after sitting their IELTS test and prior to commencing their studies, or studied in an English-speaking country for a period of time before commencing their course. This additional information may have either been missed, or missing, when going through students’ individual files.

Appendix 4.3 shows the distribution of IELTS individual and overall scores in both the TAFE and Higher Education sample. In general, the mean scores were higher than the required minimum in the TAFE group than the mean scores in the Higher Education group. However, from the standard deviation values, the TAFE and Higher Education samples were comparable.

3.2 Grade Point Average

There were significant differences between TAFE and Higher Education students in their grade point average (See Appendix 4.4 for distribution of GPA0 scores). For GPA0 (ie., including ‘0’ or withdrawn scores), the TAFE mean of 1.94 (SD = 0.94) was significantly smaller ( $t_{(111)} = -4.09$ , p < 0.001) than the Higher Education mean of 2.63 (SD = 0.85). When withdrawn subject scores were removed from the calculation in the TAFE sample, the resultant GPA mean score of 2.20 (SD = 0.80) still remained significantly smaller ( $t_{(111)} = -2.77$ , p = 0.007) than the Higher Education mean.

The reasons for this difference in mean scores are not transparent. It may be that the much greater number of subjects TAFE students attempt in one semester results in poorer performance in each subject. It is also possible that the difference is due to different ability levels or academic standards of TAFE and Higher Education students.

It is also of interest to note that of the 58 students in TAFE, 34 failed at least one subject, and there were at least eight students who failed or withdrew from 50% of their subjects. In Higher Education, 20 out of 55 students failed one subject, but only one student failed two subjects, and another failed three. As already mentioned, there were no withdrawals in this group.

3.3 Correlations between IELTS and Grade Point Average

Correlations of GPA0 and GPA scores with the four IELTS individual and overall band scores for TAFE and Higher Education students and for both groups together are given in the Table 5:

Table 5: Pearson correlation coefficients for IELTS individual and overall band scores and GPA scores.

Only a minority of the correlations achieved statistical significance beyond the 0.05 level. In the total sample, the Reading and Writing scores correlated positively with GPA0 and GPA. When Higher Education and TAFE were looked at separately, only the Reading score remained statistically significant for the Higher Education group. While none of the correlations for TAFE students was statistically significant, it is worth noting that the size of the correlation for Writing in the TAFE group (W/GPA0 = .206, W/GPA = .194) is very similar to that for the total sample (.250**, .204*) which achieved statistical significance.

As far as the magnitude of the correlations, it is also of interest to note that while for TAFE students correlations for the Writing test were much higher than correlations for the Reading test, the opposite can be observed with the Higher Education results, where correlations for the Reading test were much higher than the Writing test.

Correlations for the GPA0 scores were marginally higher in the main with the IELTS scores than were those for GPA scores---for all intents and purposes, the two methods of calculating grade point averages produced equivalent results in this instance. Hence, while regression analysis was performed on both GPA0 and GPA scores, only the GPA0 scores are reported on in the following section.

3.4 Regression Analysis

The regression model was not significant for predicting GPA0 ( $F_{(4,53)} = 0.65$ , p. 0.629) scores in the TAFE group as can be seen in Table 6. In addition, none of the four IELTS individual scores was a significant predictor of GPA0 scores. Both the results are not surprising, given the non-significant correlations for the TAFE group in Table 5.

Table 6: Linear regression coefficients for academic performance on four IELTS individual tests in TAFE sample

The regression model was not significant beyond the 0.05 level for predicting academic performance ( $F_{(4,50)} = 2.39$ , p. 0.063) in the Higher Education group (Table 7). However, it does predict 9.1% of the variation in GPA0 scores (adjusted $R^{2} = 0.091$ ), corresponding to a small-to-medium predictive effect of academic performance from the four IELTS individual tests (Cohen, 1992). Furthermore, as can be seen in Table 7, the IELTS Reading score was a significant predictor of GPA0 scores.

Table 7: Linear regression coefficients for academic performance on four IELTS individual tests in Higher Education sample

In the total sample, the regression model was significant for predicting GPA0 $(F_{(4,108)} = 3.58, p=.0009)$ scores. The model accounts for 8.4 per cent of the variation in GPA0 scores (adjusted $R^{2} = 0.084$ ), corresponding to a small-to-medium predictive effect of academic performance from the four IELTS individual tests (Cohen, 1992). Table 8 contains the results of the same linear regression model being fitted to the total sample:

Table 8: Linear regression coefficients for academic performance on four IELTS individual tests in total sample

For the total sample, the only significant predictor of academic performance among the four subtests was the Reading test (Beta = 0.263 for GPA0). The significant correlations between the Writing test and academic performance that were observed in Table 5 are no longer found in the regression model once the effect of the Reading is also taken into account. That is, the association between Writing and GPA0 mostly reflects common variation between Reading and Writing, rather than anything unique to Writing itself and academic performance.

3.5 Discussion

The data analysis found a small-to-medium predictive effect of academic performance from the IELTS scores for the total sample and Higher Education sample, accounting for 8.4% and 9.1% respectively, of the variation in academic performance. In both these samples, the Reading test was the only significant predictor of academic performance. IELTS was not found to be a significant predictor of academic performance for the TAFE sample.

These results are similar to the Validation project for the ELTS (the precursor of IELTS) conducted by Davies and Crier (1988) on 720 subjects. Davies and Crier concluded that the contribution of language proficiency to academic outcome is about 10% (a correlation of 0.3). Positive correlations were also found in the IELTS predictive validity studies of Ferguson and White (1993), Elder (1992) and Bellingham (1993). These studies share with the present study a relatively homogenous group of students in the sample compared with other predictive validity studies of IELTS which found no positive correlations. These common findings lend support to earlier findings by Light, Xu and Mossop (1987) who found that the relationship between English language proficiency and academic outcome varied according to students’ area of study.

Other studies which have found no significant correlations have also had a more limited range of IELTS scores than in the present study. The positive correlations in this study may be partly attributable to the wider range and high frequency of IELTS scores lower than 6 in the sample. There seems to be growing support for the view that it is at lower language proficiency levels where significant correlations will be found with academic outcomes (Bellingham 1993, Graham 1987, Elder 1992).

The magnitude of the correlation between Writing and GPA0 in the TAFE sample, while not statistically significant, suggests that this is an area which would benefit from further examination. It is possible that a larger sample size would reveal significant correlations. It may also be that writing skills play a more important role in the overall assessment of students at the TAFE level than at the Higher Education level, particularly if the assessment is examination-based.

As reported, the magnitude of correlations between Writing and GPA in the TAFE group was much larger than between Reading and GPA, while the opposite was observed in the Higher Education group. Further investigation using larger samples would be useful to clarify this incongruence in the results for the Higher Education and TAFE sectors. It may be that, as suggested in the data on staff perceptions in this study, a more complex level of interpretative reading is required of Higher Education than of TAFE students. For TAFE students, on the other hand, writing may be a more important skill.

It appears that the Speaking and Listening tests have no association with GPA for either TAFE or Higher Education. In some ways, it is not surprising that the Speaking and Listening tests were not significant predictors of academic outcome, as assessment practices in these courses do not emphasise these skills. However, the data from both academic staff and student perceptions in this study stresses the importance of listening skills, particularly in lecture situations, for coping with the demands of their course. It may be that, as the Listening test does not test academic listening skills, it does not accurately predict the kind of listening skills that students in these Business courses are required to develop during the first semester of their course.

4.0 Student Questionnaires

The response to the questionnaire was extremely poor. Of the 113 questionnaires mailed out, only sixteen, or 14% were returned. Of the sixteen students who responded, seven were enrolled in TAFE and nine in Higher Education courses. Although the response was disappointing, those who did respond represented many departments within the Faculty:

Table 9: Distribution of questionnaire respondents by course

4.1 IELTS Scores of Questionnaire Subjects

Seven of the sixteen students had individual band or overall scores less than the required minimum on one or more of the IELTS tests (Appendix 4.5). In the TAFE group only two students (29%) had scores below the minimum; in the Higher Education group (44%), there were four. These percentages indicate that the IELTS scores of the students in this sample were higher than in the population identified in the first set of data (see Table 5).

It is interesting to note that, of the sixteen respondents, three TAFE and four Higher Education students, or 44% of the sample, had received further English instruction in English prior to commencing tertiary studies (Appendix 4.4). This high figure may provide some explanation for the lower than expected IELTS scores in the first set of data, and an indication of the actual numbers of students in the larger population who may have in fact gained entry into their course via means other than their IELTS score.

4.2 Adequacy of English Level and IELTS Scores for First-Semester Study

Most of the respondents were positive that their English was adequate for coping with first-semester study, evidenced in their replies to Questions 3 and 4:

Table 10: Self perceptions of adequacy of English and IELTS scores achieved

Students’ perceptions of the adequacy of their English did not bear any clear relationship to whether or not they had achieved the minimum IELTS entry level scores, ie. had been deemed to have sufficient English to cope with their course. There were students who felt they had adequate English despite not having the minimum entry level scores, and vice-versa.

Given the lower than expected IELTS scores in the data, the generally positive perception of students may seem surprising. However, as mentioned earlier, students may possibly have higher levels of English than their IELTS scores might suggest. Furthermore, the questionnaire respondents were a self-selecting group, and it may be that only those students who were coping reasonably had the time to answer the questionnaire. Thirdly, the questionnaire was distributed to students who were already in their second or third semester of study, when initial difficulties have generally been overcome and whose recollections of perceived difficulties may have receded.

4.3 Areas where a Higher IELTS Score would have been Beneficial

As Table 10 shows, while respondents were generally positive about the adequacy of their English, nearly half of them also believed that a higher IELTS score would have been beneficial in some areas. Listening was the most frequently mentioned by TAFE students, and one TAFE student identified Writing as an area where a higher score would have helped. For Higher Education students, both Reading and Writing were the most frequently mentioned areas. Only one student identified Speaking as an area where a higher score would have helped. Of the seven respondents whose IELTS scores fell below the minimum required, all but one perceived that a higher score in their weakest area would have assisted them.

4.4 Coping With English Language Demands and Methods of Assessment in First-Semester Study

Methods of assessment

Table 11: Self-perceptions of coursework language difficulties in the first semester

Table 11 show the distribution of students’ responses to Question 6. Students’ answers to this question support their general perception of having adequate English for coping with their course, with the majority responding that they had found ‘a few problems’ coping with language demands. While responses for Higher Education and TAFE were similar on the whole, writing and to a lesser extent, listening, appeared to be marginally more difficult for TAFE than for Higher Education students, and assignments more difficult for Higher Education students. Students’ responses in this section did not corroborate with responses to Question 4, as reported in 4.3.

4.5 Subjects Perceived as Easy

The subjects students found easiest seemed to be those in which the English language demands were lower (eg. Computer Foundations, Statistics), those in which students had some prior background, or those in which all students were perceived as being at the same level (eg. Beginner Japanese). Other reasons students gave for finding subjects easy were the teacher and method used, rapport between student and teacher, and interest in the subject.

4.6 Subjects Perceived as Difficult

Students seemed to experience much greater difficulty with subjects that were heavily language-based. For example, seven out of nine Higher Education students found Macroeconomics to be difficult. With the TAFE students, marketing subjects were perceived as the most difficult. The most frequently mentioned reasons given by both groups for difficulties were the lack of prior background and slow reading skills. Other reasons cited were lack of technical vocabulary, inadequate listening and note-taking skills, and/or lack of interest in the subject.

4.7 Concurrent Support Accessed in First-Semester Study

The questionnaire also sought information on the amount of concurrent English support students received during their course. Of the sixteen questionnaire respondents, two (13%) had accessed concurrent support during their first semester. This corresponds accurately with the overall percentage of international students from the Faculty who accessed the university language centre’s concurrent support service in one year. The amount of English language support a student receives is an important variable in predictive validity studies, and it is unfortunate that the sample size of the questionnaire was too small to be able to investigate this in more detail.

4.8 Summary

Overall, the students in this questionnaire sample were positive about their ability to cope with the language demands of first-semester study. However, there was also acknowledgment of some problems coping with writing and listening demands for TAFE students, and assignments for Higher Education students in first semester. Furthermore, in students’ responses to why a subject was difficult, the lack of reading skills and background knowledge were important factors for both TAFE and Higher Education students. These findings corroborate to some extent the findings on the importance of reading and writing in the first part of the study, as reported in 3.1. There was a relatively high number of students in this sample who accessed some form of English language support both prior to commencing study (and after sitting the IELTS exam) as well as during first semester. This suggests that the amount of English support accessed may have been an important variable affecting academic outcomes in the larger sample.

5.0 Staff Interviews

The responses of the Higher Education and TAFE staff in the Business Faculty provided many useful insights into the difficulties they believe international students experience in their first semester in the Business Faculty.

5.1 Familiarity with the IELTS Test

The majority of the staff (15 out of 19) was not familiar with the IELTS test. Many were aware that it was a language test which international students needed to sit to fulfil entry requirements. Most felt that it was the responsibility of the Selection Officer to ensure that the students were linguistically competent to proceed with a tertiary course. In many cases, staff felt that if a student had been admitted to the course, it could be assumed that their English was adequate.

Many staff expressed interest in finding out more about the standard set by the test and its importance as an indicator of the linguistic ability of the students they were teaching. Some felt that an information session would be useful for giving them a clearer perception of their students’ language capabilities, and perhaps assist them in modifying their courses if necessary.

5.2 Importance of Minimum Scores on Listening, Reading, Writing or Speaking Tests for Attempting the First Semester of their Subject

Of all the skills, Listening was seen by the majority of the staff in both Higher Education and TAFE to be the most significant skill in attempting the first semester of their subject, as can be seen in Table 12:

Table 12: Staff perceptions of the importance of minimum scores on Listening, Reading, Writing or Speaking sub-skills for attempting the first semester of their subject

There was a difference between the Higher Education staff and the TAFE staff in the importance placed on writing. Higher Education staff placed Writing as the second most important skill after Listening whereas the TAFE staff placed greater importance on Speaking. Reading was not generally seen as being as crucial, as one staff member explained:

If it’s written material they can take it away, they can take it home, they can get someone else to help them, but with Listening, where you either get it or you don’t get it because it’s gone. That’s where I think they get into the greatest problems.

5.3 Adequacy of English Entry Levels as Set by the Faculty

The majority of staff (13 out of 19) felt that the level of English, while not ideal, was sufficient to enable the majority of the students to be successful. This generally positive perception accords with that of the students’ perceptions in the questionnaire sample, as reported in Section 4. In discussing this question, staff brought up a number of issues in relation to the delivery of courses to international students. One issue was that of maintaining standards. Marrying the needs of international students while maintaining the integrity and standard of their courses posed a dilemma for many staff. One staff member felt that modifying courses may be cheating both the Australian and overseas students:

International students want a body of knowledge, but also because that body of knowledge is taught in English and because that body of knowledge is taught in an English-speaking country, and because that body of knowledge is taught in an inquiry-based learning situation. Now if you start looking at the content and the mechanisms of the environment and think about modifying them to suit the needs of overseas students …then I feel you’re in danger of setting up a system that doesn’t have integrity. I think you are in danger of setting up a system which doesn’t give them what they bought.

Some staff also discussed the demanding nature of new accelerated courses in the TAFE sector. These courses are increasingly popular with overseas students who are keen to avail themselves of a course that will enable them to finish their degree or diploma in less time and return home or enter the workforce. However, a number of staff had very real concerns about the wisdom of trying to achieve so much so quickly:

Slow down their course - let them do it over three years not two. Year One international students should spend more time doing English or having more free time for study and acculturation.

5.4 International students’ ability to cope with the linguistic demands of their course in the first semester and the area(s) in which they experience the most difficulty

Table 13: Staff perceptions of how international students cope with linguistic demands of their course, and the area of most difficulty

As can be seen in Table 13, of the four skills, listening was again seen as critical to the difficulty experienced by international students in their first semester by the majority of staff. However, in discussing the problems it became apparent that the other linguistic skills were also major areas of concern. All of these difficulties were seen to be compounded by other psychological and sociocultural problems which affect students adjusting to a new country and to different educational expectations.

5.4.1 Listening

The skills involved in gaining information from a lecture - listening, interpreting, summarising and internalising information - makes this the skill which most academic staff believed was critical to success. This can be made more difficult by complex concepts and the use of technically difficult vocabulary during the lecture. They felt that students took down notes verbatim and felt secure knowing that they had a copy of the lectures, but they failed to realise that in the Australian context the lecture was only the introduction to learning. Even when a staff member made their lecture notes available, it was not sufficient to ensure comprehension of the material covered.

If a student has not followed a lecture, it is often difficult for them even to frame the right questions to ask the lecturer afterwards. Because students had not fully understood what was being said, they would pick up key words and repeat them in a manner that illustrated that they had no understanding of the context or meaning of their utterances. Or worse still, lecturers felt, they would carefully learn by rote the partially understood information and use it incorrectly for their final assessments.

Staff felt that tutorials also presented students with listening problems. Some lecturers believed that in the tutorial situation, the second language student is at an enormous disadvantage. In a tutorial, ideas may come from more than one person, a train of thought is frequently interrupted, and ideas and sentences are not always completed. What should be the chance to consolidate learning and to clarify issues raised in the lecture, becomes an extremely threatening situation.

5.4.2 Reading

Reading was only mentioned directly by two of the staff. However, the relevance and importance of reading in every aspect of learning arose incidentally in almost every interview. The very active way reading is used in lectures and tutorials to support argument and theory posed a problem for many international students. Without prior reading, many lectures can be incomprehensible. However, students often find the amount of reading difficult to cope with. Furthermore, the language of textbooks can often be difficult for second language students to penetrate. Without this background knowledge, they may not be able to follow the theory or argument being presented in the lecture or tutorial. These perceptions about the importance of reading and prior knowledge were very similar to those of students in the questionnaire sample.

Australian students who have had to learn to cope with the demands of the VCE ¹ gain research skills which they later employ in their tertiary studies. The ability to rapidly scan an index or search a book for relevant information instead of reading it all saves an enormous amount of time. International students, on the other hand, cannot be assumed to have these kinds of skills when they begin their tertiary studies.

Another reading skill discussed by staff was the ability to interpret examination questions accurately. Many staff felt that international students had difficulties interpreting questions correctly or perhaps they had never seen the style of questions given in the exam, and were therefore at a loss how to answer them. Questions were left unanswered or not even attempted. Some lecturers commented that a lot of time was being spent prior to exams going over the exam format, explaining what was required and explaining the use of multi-choice questions. Students needed to be taught how to read the question and answer what was asked for, not simply focus on a key word and assume the rest of the question. In some courses, continuous assessment had been introduced to minimise the distress of final exams and to provide practice in tackling examination questions.

5.4.3 Writing

The difficulty of writing and meeting accepted criteria and realising the expected outcomes pose considerable problems for overseas students. Writing under examination conditions was particularly difficult:

The level of grammatical accuracy in examination answers was a dilemma for some members of staff. Some believed that, if the answer could be understood, albeit with some effort, and the information was correct, it should be judged as correct. Whereas others argued that if there were ‘35 grammatical errors in this paper, (they were) not going to pass it’.

Assignments were seen to be not as difficult by staff, as students had the opportunity to discuss problems in class with the lecturer and with other students, and were able to seek help individually. They were able to clarify the meaning of assignment questions and the work requirements before it was assessed.

5.4.4 Speaking

Only four of the teaching staff perceived speaking to be important. One felt that if a speaking problem existed, it affected every aspect of a student’s ability to gain the maximum from the course:

A study of the relationship between IELTS scores and students’ subsequent academic performance

If there is a speaking problem, it brings out a shyness to communicate with the lecturer, to get help or to communicate what they’re really struggling with.

Some staff felt that the reluctance to speak felt by some international students was also due to a fear of losing face in front of their peers and teachers. This can be compounded by an inability to express ideas easily and respond quickly when their arguments were countered. The increasingly large tutorial groups made this a continuing problem as there was not time to encourage the reticent to speak.

Oral presentations were seen by some lecturers to be particularly problematic for international students:

Presentation skills are culturally proscribed … look people in the eye, give your own opinion and speak up. To many cultures this is arrogant and rude.

Even when the conventions were explained, lecturers felt that it was one thing for students to know what was correct and quite another to put something culturally foreign into practice.

5.4.5 Non-linguistic factors which affect international students’ ability to cope

Motivation

Some staff mentioned motivation as enormously important for the students who were having difficulty. Once a student fell behind, the workload could become crippling. It was only the highly motivated student who could put in the effort to overcome initial failure or inability to cope.

Maturity

Many lecturers felt certain that the more mature international students found their entry into the tertiary world much easier. Many of the older students have already obtained their first degree in their native language and can use their knowledge to benefit in the second language environment.

Educational Background

Lecturers pointed to the often vastly different expectations of teacher and student and the problems associated with this mismatch. The need to be independent and creative learners who are able to form their own opinions and to argue the validity of their stance may appear extremely arrogant to students who are used to seeing the text and the teacher as providing the definitive answers. Accepting this style of learning is made more difficult when students are unsure of their language:

…to argue with different points of view, to speak up in large groups, to be vehement about the right to express a different opinion, to criticise other people’s theories constructively, even to disagree with a printed text may require a mental leap.

Some lecturers felt that those who have already had some experience of the learning culture of the Australian classroom through ELICOS courses were better able to cope because of a greater familiarity with learning practices and teaching styles. Such students also had an enviable confidence with language brought about by everyday use of English, not only academically but socially.

Social adjustment

Many of the staff acknowledged that social adjustment was a major area of difficulty. There was amongst all the staff an awareness that these students were living far away from home, in a culturally problematic environment. Many were working long hours part time to support themselves, living with their peers for the first time, cooking, cleaning, shopping and being desperately homesick. Staff were also aware of the family pressures faced by students whose family’s life savings were being used to fund their studies, and the terrible burden of responsibility to succeed and justify the sacrifices that have been made on their behalf.

5.4.6 International students’ difficulties in comparison with Australian students

The majority of staff (15 out of 19) believed that international students experienced more difficulty than Australian students in their first semester of study. Many lecturers felt that the effort required for international students to achieve the results they want required a much greater effort than for Australian students. For example, some international students spent hours in the library without achieving much because they did not have the study skills to deal with the style of learning that their course assumed.

Staff also discussed the difficulties international students faced with different types of subjects. Students doing more practical subjects, or subjects which they have already studied in their first language were perceived to have less difficulty than those who had undertaken subjects where they were trying to respond to complex language in which the body of knowledge is being conveyed. These perceptions again, correspond closely to students’ perceptions of reasons for the difficulty of subjects.

International students had more difficulty acclimatising to the academic structure, the different style of teaching (‘We speak, other cultures read books’). Language played a large part in both the educational and social stress first year international students experience. However, international students also had to deal with the lack of networks of support, the need to exercise greater independence academically, the isolation from other students that comes from linguistic and social differences, and all these factors combine to make it an extremely stressful period. As one lecturer put it,

Semester One is tough.

Despite all these problems, the attitude and diligence most international students bring to their studies ensures that most of them do succeed:

They make up for the disadvantages with sheer hard work.

5.4.7 Amount of language support international students should or already receive from the Faculty

The majority of staff interviewed felt that more concurrent English language support was needed by international students in the first semester:

Table 14: Staff perceptions of whether concurrent support was needed in the first semester

Although language was seen to be of paramount importance, and concurrent support desirable for students, the general opinion was that it should not be seen as a faculty responsibility to deliver this concurrent support. The reason given for this was often the added cost to the faculty that this would bring, at a time when resources were already tight. Most of the staff, but not all, were aware of the concurrent support that was already available from other units within the university, including the English language centre. It was felt to be the responsibility of the individual student to access this support when they needed it.

The issue of whether the concurrent support received by international students should be credit-bearing was also discussed by some. Staff commented on the difficulty of getting students to attend English classes which were not a credited part of their course. They felt that it would be an added burden, not assistance for students, if attendance were made compulsory. The lack of time to access the support that was already available was felt to be the main problem for international students.

5.5 Summary

Staff interviewed for this study perceived English language proficiency to be an important factor in academic performance. Generally, they were positive about the adequacy of English language entry levels as set by the Faculty, as well as with students’ ability to cope with the language demands of their courses. However, there was also a keen awareness of students’ difficulties with both language and other areas. Staff highlighted listening to lectures as the most important skill as well as the area of most difficulty for first-year international students. The amount of reading and the skill of interpreting examination questions were also perceived to be important by the majority of staff. Aside from language, a range of non-linguistic factors including motivation, social adjustment, educational background and maturity, were also perceived to be important to academic success.

6.0 Conclusions

This study investigated the extent to which the IELTS test predicted the subsequent academic performance of first-year international students from the Faculty of Business of a large Australian university. Correlations and regression analyses were performed to measure the predictive effect of students’ individual and overall band scores on the IELTS test and their first-semester grade point average.

The data analysis found a small-to-medium predictive effect of academic performance from the IELTS scores for the total sample and Higher Education group, accounting for 8.4% and 9.1%, respectively, of the variation in academic performance. The Reading test was found to be the only significant predictor of academic performance in both the total sample and Higher Education group. None of the IELTS tests was found to be a significant predictor of academic performance in the TAFE group, although the magnitude of the correlation for the Writing test was similar to that of the total sample which achieved statistical significance, suggesting a strong relationship which warrants further investigation with a larger sample size.

The subjects in this study made up a relatively homogenous group compared with other predictive validity studies which found no positive relationship between English language proficiency and academic outcomes. The positive findings in this study lend support to the view that area of study is an important variable to take into account in predictive validity studies (Bellingham 1993, Davies and Criter 1988, Elder 1993, and Ferguson and White

1993). While positive, the correlations found in this study were relatively weak, and support similar findings in other IELTS predictive validity studies.

It has been suggested that it is at the lower levels of proficiency where language becomes an important factor in predicting academic outcomes (Elder 1993, Ferguson and White 1993, Graham 1987). The high frequency of IELTS scores below the minimum set for admission which was found in this data may partly account for the positive correlations found.

Student and staff perceptions of the relationship between English language proficiency and academic outcomes were also investigated through student questionnaires and staff interviews. This was done to gain a closer participant perspective of the key factors which are perceived to influence the academic outcomes of first-year international students. The findings from the qualitative part of the study corroborate to some extent the link between reading and writing and academic performance found in the first part of the study.

Both staff and students were positive on the whole about the adequacy of students’ language skills for coping with the demands of first-semester study. The TAFE students perceived writing and secondly, listening to present the most problems. For the Higher Education group, assignments presented the most difficulty. For these students, the most difficult subjects were those that were heavily language-based, required prior knowledge and a lot of reading. The student questionnaire data also provides a possible explanation for the unexpected wide variance of IELTS scores found in the first set of data. The data also suggests that the amount of English language assistance received both after the IELTS test and prior to commencing study, as well as concurrently during the course, are possibly key intervening variables which warrant further investigation.

In the staff interviews the language skills perceived to be the most important for academic success were listening to lectures and interpretative reading. Aside from language, staff also saw sociocultural and psychological factors such as learning and educational styles, social and cultural adjustments, motivation and maturity, financial and family pressures to have an influence on the academic outcomes of international students in their first semester of study.

7.0 Recommendations

The predictive validity of proficiency tests such as IELTS is an important and complex issue which needs ongoing research. The following recommendations are made based on the findings of this study:

• The IELTS Reading, and to a lesser extent, the Writing scores should be given special consideration in the selection process of international students applying for tertiary Business courses.

• Information on entry-level English language proficiency measures and alternative criteria should be incorporated into universities’ student databases, if they are not already, in order to facilitate ongoing monitoring of the appropriateness and comparative effectiveness of the set admission criteria.

• Predictive validity studies of this kind could be replicated in other faculties to more accurately determine the relationship between proficiency and academic success in differing contexts.

• Information seminars on IELTS and other entry-level English language criteria used for admission could be provided to staff to give them a clearer idea of their students’ language competencies and potential areas of difficulty.

• A research network could be set up by IELTS Australia and UCLES between past, current and prospective researchers into IELTS to facilitate information-sharing and discussion of relevant issues, problems and findings.

Studies of this kind, despite their limitations, are important in helping to build a more comprehensive picture of the exact nature of the relationship between students’ English language proficiency and their subsequent academic performance. They also help build awareness of other key influences on academic outcomes, as well as of the factors which need to be taken into consideration with regard to the admission criteria, course delivery and assessment of international students.

Acknowledgments

The researchers wish to thank the following people for their assistance with this project:

• Paul Dudgeon who was the statistical consultant for the project
• Sally Armstrong, Professor Patrick Griffin, Kathryn Hill, Associate Professor Brian Lynch, and Janne Morton for advice on a range of issues
• staff and students at the Faculty of Business of RMIT University who gave of their time to respond to the questionnaires and participate in the interviews
• The Centre for English Language Learning and the Faculty of Business of RMIT for their assistance and support with the research.

The researchers would also like to thank IELTS Australia for their support with this project.

References

• Bellingham, L. (1993) The relationship of language proficiency to academic success for international students. New Zealand Journal of Educational Studies 30, 2: 229-232.
• Burns, R. (1991) Study and stress among first year overseas students in an Australian university. Higher Education Research ad Development 10, 1: 61-77 in Cotton, F. and F. Conrow (1998). (See below.).
• Cohen, J. (1992) A power primer. Psychological Bulletin 112: 155-159.
• Cotton, F. and Conrow, F. (1998) An investigation of the predictive validity of IELTS amongst a group of international students studying at the University of Tasmania. IELTS Research Reports 1998 Vol 1. EA Journal Occasional Paper 1998 Vol. 1.
• Davies, A. and Criter, C. (1988) ELTS Research Report 1 (I) ELTS Validation Project Report. London: British Council and Cambridge: UCLES.
• Elder, C. (1993) Language proficiency as a predictor of performance in teacher education. Melbourne Papers in Language Testing 2, 1: 68-85.
• Ferguson, G. and White, E. (1992) A predictive validity study of IELTS - a report commissioned by UCLES. Institute of Applied Linguistics, University of Edinburgh.
• Fiocco, M. (1992) English proficiency levels of students from a non-English speaking background: a study of IELTS as an indicator of tertiary success. Unpublished research report. Perth: Curtin University of Technology.
• Gibson, C. and Rusek, W. (1992) The validity of an overall band score of 6 on the IELTS test as a predictor of adequate English level appropriate for academic study. Unpublished M.A. thesis. Macquarie University, Sydney.
• Graham, J. (1987) English language proficiency and the prediction of academic success. TESOL Quarterly 23: 505-521.
• Gue, L. and Holdaway, E. (1973) English proficiency tests as predictors of success in graduate studies in education. Language Learning 23: 89-103 in Graham, J. (1987) (See above).
• Ho, D. and Spinks, J.A. (1985) Multivariate prediction of academic performance by Hong Kong University students. Contemporary Educational Psychology 10: 249-259 in Graham, J. (1987) (See above).
• Jochem, W., Snippe, J., Smid, H and Verweij, A (1996) The academic progress of foreign students: study achievement and study behaviour. Higher Education 31: 325-340 in Cotton, F. and Conrow, F (1998) (See above.).
• Light, R.L., Xu, M., & Mossop, J (1987) English proficiency and academic performance of international students. TESOL Quarterly 21: 251-256.
• Sugimoto, R. (1966) The relationship of selected predictive variables to foreign students’ achievement at the University of California, Los Angeles. Dissertation Abstracts 2865A in Graham, J. (1987). (See above).
• Wilcox, L.O (1975) The prediction of academic success of undergraduate foreign students. Dissertation Abstracts International 35, 6084B. in Graham, J. (1987) (See above).