Assessing the impact of IELTS preparation programs on candidates’ performance on the General Training reading and writing test modules

Abstract

The motivation for the study on which this paper is based came from several concerns regarding the consistently poor performance of Fiji candidates in the IELTS Reading and Writing modules. There are several factors that could account for this poor performance, with a major factor being inadequate test preparation. In this study, we carried out an initial assessment of English language skills (in the form of a pre-test) of a sample of potential IELTS candidates. These candidates participated in 30 hours of intensive preparation classes focusing especially on the General Training (GT) Reading and Writing test modules. At the conclusion of the preparation program the candidates did an exit test (the post-test), provided written feedback on the program, and gave oral interviews. The pre- and post-test performance and interview data were analysed quantitatively and qualitatively. The findings revealed that there was significant gain in the scores of candidates in the Writing module. The gain in the Reading module was not as dramatic. The researchers highlight the influence of several factors such as time, motivation, anxiety and the nature of the skill itself on the candidates’ performance during the preparation program, and in the actual IELTS exam. This paper raises issues related to candidate perceptions, the impact of intensive course preparation, and the implications of such preparation on pre-test English language competence.

The team is grateful to Dr France Mugler for her invaluable assistance with the analysis of the data.

1.0 Introduction

Recent political events in Fiji have resulted in a dramatic increase in the number of people - particularly skilled tradespeople - wishing to take the IELTS test for migration purposes. A significant number of these people take the test with very little preparation, or awareness of what is required of them. In the time that IELTS has been offered at The University of the South Pacific (USP) (the first test was conducted in January 2000), the majority of our candidates have not done well on the Reading and Writing modules, despite achieving quite good results on the Listening and Speaking modules. We at USP have become increasingly aware of the fact that a significant number of our IELTS test candidates attempt the test on a number of different occasions without having done anything to prepare themselves in the interim. Not surprisingly, they see little change in their test results. This prompted us to examine effective approaches to IELTS preparation for what is essentially a very clearly defined group of test-takers. Consequently, the primary aim of this project was to investigate how participation in an IELTS preparation program that targets the GT Reading and Writing modules affects candidates’ levels of pre-test English language competence.

We anticipated the following research outcomes:

• Raised awareness of IELTS test demands: We were keen for our test-takers to be better informed about, and better prepared for, the demands of the IELTS test. These people needed to have much clearer expectations of the levels of English language competence they needed to demonstrate, and the specific test tasks they were required to complete within the set time limits.
• Improved performance on the GT Reading and Writing modules: Many of our candidates were being let down by their very limited reading strategies, lack of familiarity with reading topics outside their context, and lack of awareness of specific writing conventions.
• Enhanced quality of IELTS preparation programs: We wanted to do more than just provide our prospective IELTS test candidates with information about the test. Our aim was to provide quality preparation that gave these people opportunities to develop familiarity with the IELTS test task types, and to practise specific techniques to enable them to complete these tasks to the best of their ability.

2.0 Background

Australia and New Zealand now require migrants in the skilled category to show formal evidence of their English proficiency - a clear example of what Elder and O’Loughlin (Unpublished Report 2002) refer to as the powerful role of IELTS as a gatekeeper. Fiji has seen a dramatic increase in the number of candidates sitting for the IELTS test. Although IELTS testing in Fiji started fairly recently (see above), the general observation so far is that candidates perform better in the Speaking and Listening modules than in the Reading and Writing.

There are several factors that could account for the generally poor performance of Fiji candidates in the Reading and Writing modules. One factor is the lack of, or inadequate, test preparation. In Fiji, most IELTS candidates fall in the skilled migrant category; they are trades-people and para-professionals who have been in employment for extensive periods of time. They have not had any systematic classroom instruction since they left school, and for some this is as long as 20 years ago.

Another factor that could account for the poor performance is the amount and type of exposure to the English language these people experience. For almost all our IELTS candidates, English is their second or third language. Their exposure to reading and writing in English is limited but they have been functioning in the English language for daily interaction at their workplace for many years. Despite this, they lack confidence in their own English language competence and so they come to the test with markedly increased levels of anxiety. Many are totally overwhelmed by the content as well as the mechanics of the test.

Added to this is the effect of the test environment. Many of our candidates are not at all familiar with the USP campus. They feel daunted by the physical scale of the university and by what the campus represents in Fiji society. By establishing some contact with the university through attendance at and participation in a preparation program, it is possible that these negative impressions of the test environment can be diminished.

Building on the factors discussed above, this study assumed that candidates exposed to a period of focused test preparation would tend to perform better in the IELTS Reading and Writing modules than those who do not have such preparation opportunities. Although a number of institutions in Fiji have recently begun to offer limited tuition to intending IELTS candidates, there are no sustained long-term preparation programs offered, or programs focusing on specific aspects of the test. Many unsuccessful candidates return to sit the IELTS test after three months without any pre-test support program, finding themselves in a similar predicament as they were in for their first attempt.

3.0 Literature review and rationale for investigation

Although several studies have been done focusing on the impact of the IELTS test on educational processes and society in general (Milanovic and Saville 1996; Horak 1996; Saville 1998), there has been very little study on the actual impact of pre-test preparation on IELTS performance. In a study looking at the effect of background disciplines on IELTS scores, Celestine and Su Ming (1999) found no significant difference between the results of students who attended preparation and those who did not. They attributed this to two main factors: the duration of the preparatory session may have been too short; and IELTS is a proficiency test, therefore, the amount of preparation would not make any significant difference to the scores attained.

In contrast is the research conducted by Elder and O’Loughlin (Unpublished Report 2002), who worked with a sample of 112 students enrolled in a three month intensive English language program to examine the relationship between this intensive study and IELTS band score gains. Their results showed that the students made variable progress, with an average gain of approximately half a band overall. Interestingly, the most dramatic gains were in listening, while least change was seen in reading. It was also evident that the more proficient students at the beginning of the program were less likely to show marked improvement in test scores.

Furthermore, according to UCLES Research Notes (www.ucles.org, retrieved 2002) research and development focusing on the productive skills of speaking and writing have been a major focus.

The research described above not only shows that there is room for further work in the area, but it also provides a useful frame for a study in the South Pacific context. There are additional variables, which need to be considered with our clearly defined test-taking population. These include language use and functions, cultural factors, affective factors, the testing environment, and test mechanics. All of these play a significant role in the outcomes of any language proficiency testing, but we are keen to pursue their impact within our context.

4.0 The study

In the following sections we provide details on the design and implementation of the preparation program, the study methods and the approach taken to our data analysis.

4.1 The preparation program

Before detailing the methods used in our study, it is necessary to contextualise our study by outlining our approach to setting up the preparation program itself. Consequently, the following section outlines the identification of the training centres, the organisation of the program, the selection of program materials and the program staffing.

4.1.1 Identification of training centres

The IELTS test preparation training program was held at three different locations, covering the major population centres in Fiji: Suva - the national capital; Lautoka - a large sugar-production town on the western side of the main island (four hours by road from Suva); and Labasa - the main town on the second largest island, a forty minute flight from Suva. These locations were selected not only because they are the larger business centres in Fiji, but the University of the South Pacific also has its Extension Subcentres in these areas. The selection of these centres was also underpinned by our desire to facilitate the candidates’ access to the program.

4.1.2 Preparation program design

The format and content of the activities in the preparation classes were based on IELTS specimen materials and other appropriate materials prepared by the tutors (see 4.1.4 below for information on tutors). The 30-hour program was held over three consecutive weekends (five hours on each of the two days) at the three centres. The time allocation was decided upon after consultation between the centres, ourselves and a random sampling of prospective program participants. The preparation program also included teaching and learning activities focusing on the IELTS Speaking and Listening modules as we felt that it was important for the participants to be exposed to the test in its entirety. A copy of the timetable for the IELTS Preparation Program is included in Appendix 1.0.

The program was taught using a variety of approaches and activities. The participants were involved in seminars, group work, independent activities, collaborative pair-work, take-home tasks and group discussions. The tutors kept cumulative records of the participants’ performance in the range of tasks which they were required to complete.

4.1.3 Preparation materials

The materials for the preparation program were taken from various IELTS preparation course books as well as selected English language texts (a list of sample materials is included in Appendix 2.0). These materials were very carefully selected by the tutors. Efforts were made to take into account the background of the candidates, and the specific demands of the IELTS examination as perceived by these candidates. As the focus of our study was on reading and writing, the majority of the practice materials focused on these two skills. For listening, exercises (and audio-tapes) from IELTS listening practice texts were used.

4.1.4 Staffing

Three tutors (two females and one male) were nominated to teach at the three centres. Two of the tutors are certificated IELTS examiners and the third is an IELTS invigilator at the USP IELTS Centre. All three tutors are of similar age, they have comparable qualifications (in that all three have postgraduate qualifications in Applied Linguistics and TESOL), and they are all experienced teachers of English for Academic Purposes (EAP). Additionally, all three have experience in tutoring small groups of candidates preparing for the IELTS test. The tutors were rotated between the three test centres ensuring that there was consistency in content as well as in teaching style.

4.2 Study methods

Our study was conducted within the framework provided by the implementation of the IELTS Preparation Program. The following section provides details on our questionnaires and interviews, the pre- and post-tests used, and the control group which was set up to parallel the main experimental group.

4.2.1 Initial questionnaires

Participants in the preparation program were required to complete an initial questionnaire, which asked for the following information: their personal details, their profession, years of schooling, their reasons for taking the IELTS test, a self-rating of their skills and the ways in which they use English on a daily basis. A copy of this questionnaire is included in Appendix 3.0.

4.2.2 Candidate interviews

At the end of the preparation program, a number of the candidates participated in an interview with a member of the project team. This interview was based on open-ended questions designed to obtain the candidates’ self-assessment of the skills acquired over the duration of the program and their views on the preparation program. The interviews in Suva were done face-to-face, and the Labasa interviews were done over the phone.

4.2.3 Pre-test

Before embarking on the preparation program, all candidates were required to take an “entry test” designed to parallel the IELTS GT Reading and Writing test modules. The test took two hours and candidates worked with six reading texts (in three sections), for which there were 40 questions, and they also produced two pieces of writing. It is important to note here that two different scoring criteria were used for marking the reading and writing tasks. The reading scores were a straight mark out of 40 while the writing tasks were marked in bands ranging from 0 to 9 (similar to IELTS bands). When marking the writing tasks, the tutors used the following broad marking criteria:

• content: number of ideas, development of ideas, relevance, number of words
• organisation/structure: logical sequence, appropriate links, coherence
• presentation: accuracy of expression, spelling, grammar and vocabulary.

4.2.4 Post-test

At the end of the 30-hour preparation program the candidates sat an exit test, which was similar in format, topic content and task design to the pre-test (but was taken from a different source). This test was used to determine whether there had been any tangible gain in the candidates’ performance on the reading and writing tasks. Comparisons, based on various statistical analyses, were made between the achievements of the candidates on these two tests.

4.2.5 IELTS test comparison

Within our sample group there were a number of participants who had taken the IELTS test before. For these people, we were able to consider not only the changes, if any, in their performance on the pre-test and post-test, but also their IELTS performance on two separate occasions.

4.2.6 Control group

Our control group consisted of nine candidates selected randomly from a list of candidates registered for the IELTS test at the USP centre in Suva. Ideally, we would have liked a bigger control group but logistically this was not possible because participation in the three research tasks was purely on a voluntary basis. The control group completed the pre-course questionnaire and the pre- and post-course tests, but not the 30-hour preparation program. The results obtained on these tests were analysed alongside the results from the preparation program group.

5.0 Study participants

In the following discussion, we outline how we selected the program participants and we provide some biographical information on this group.

5.1 Selection of preparation program participants

A total of 62 candidates participated in the 30 hours of intensive IELTS pre-test preparation. The candidates were randomly selected from those intending to take the IELTS (GT module) test at the USP test centre in the near future. This sample included people taking the test for first time, as well as people who have sat the test more than once (21% of our sample had taken the test at some time during the past year). The selection of candidates was open in the sense that no restriction applied to the candidates’ age, gender, ethnicity or educational background.

5.2 Participant details

Most of our program participants were male (60%) and 45% were in the 35-50 age group. In terms of ethnicity and first language, 79% had Hindi as their first language; 10% were Fijian; and Burmese, Japanese, Gujarati and Samoan backgrounds were also represented in our sample. Approximately one-third of our sample was from Suva, the capital, with around 25% being based in Labasa, the main town on the second largest island of Fiji. There was quite a range of professional backgrounds evident in our sample, including the following:

• 26%: trades (includes electricians, mechanics, carpenters, tailors)
• 18%: education (includes lecturers, teachers, education officers, students)
• 10%: management (includes managing directors, credit managers)
• 7%: health (includes doctors, nurses)
• 6% finance (includes bank officers, financial controllers).

In terms of education, half of our sample had completed high school. 16% had vocational Diplomas, 16% had a Bachelor’s degree, and two of our participants had a Master’s degree. 21% of the group were currently studying for further qualifications.

The candidates’ reasons for doing the IELTS test were:

• for migration purposes: 53%
• for further studies: 15%
• to assess own level of English: 11%
• for professional registration: 5%.

5.3 Candidate profiles

We have selected two candidate profiles to include here to enable readers to more fully engage with the candidates with whom we were working. The names of these candidates have been changed.

5.3.1 Manju

• Country of birth: Fiji Islands
• Age: Early 40s
• Level of education: Form 5/Year 11
• Highest exam: New Zealand School Certificate
• IELTS exam: First attempt

Having left school after Form 5, Manju married at an early age and was self-employed as a tailor. She lives in an extended family and speaks mostly Fiji-Hindi at home. However, as her customers are from other races, Manju communicates in English from time-to-time. Some of her reading is in English as she is a keen follower of current affairs so she reads one of the local English newspapers daily. Her leisure reading materials include popular romance novels. She also has access to the local TV station, which airs mostly American, New Zealand, and Australian programs.

Manju was taking the IELTS test for migration purposes. Her reason for doing the preparation program was to improve her chances of passing the test. She proved to be a keen, motivated candidate who was always punctual for the tutorials. Having been away from the learning environment for so many years (her eldest child is a Year 12 student), she was initially a little reserved in class. However, she soon began volunteering answers. As the sessions progressed, she gained more confidence and was an excellent participant in the speaking sessions and during group work.

Manju’s test scores

*There were two writing tasks, similar to the IELTS exam.

At the beginning of the preparation program Manju showed an average ability in reading (with a score of 26 out of 40 in the pre-test). Her reading score did not improve by the end of the preparation program - in fact, it dropped by three marks (23 out of 40). In the post-test, her writing task scores improved by one band.

When compared with the majority of our GT candidates, Manju performed very well in the IELTS test: she received a 6.5 in Reading and 8 in Writing. The latter is very much an above-average score for our candidates. During the preparation program, Manju showed a lot of interest in the writing tasks and seemed to be more focused on this aspect of the program compared to the reading tasks and exercises. Her final score in writing is evidence of that focus. Manju herself was very pleased with her progress. In a post-test interview she mentioned that the preparation program had given her useful information on the IELTS test, in particular its content. It also gave her the opportunity to assess her strengths and weakness (pre- and post-tests) and during the tuition program she was able to work purposefully on her areas of weakness.

5.3.2 Ashok

• Country of birth: Fiji Islands
• Age: Mid 30s
• Level of education: Form 4/Year 10
• Highest exam: Fiji Junior Examination
• IELTS Exam: First attempt

Ashok works as an electrician with the Government Public Works Department in his town. Soon after leaving school, he began this job as an apprentice and has been working there for the last 15 years. He has also acquired his trade skills certificates for the job during these years. Ashok works in an environment where there are other Indo-Fijian and Fijian workers. Most of the time he uses Fiji Hindi to speak to his colleagues. He rarely needs to use English to communicate with his colleagues as he is fluent in Fijian. He lives in a rural area where there is no electricity. The family uses benzene light and hurricane lanterns at night. Ashok confesses that he rarely reads. The only things he may read are the shopping lists prepared by his wife, labels, and instructions on bottles. He has no intrinsic interest in reading. His work keeps him away from home for long hours. He has to work overtime and off-site as well. The lack of good lighting facilities at home also discourages him from reading. He does not have a TV at home so does not watch television programs. So he has very little exposure to English language.

Ashok had to sit for IELTS examination for migration purposes. From the beginning, Ashok came with the preconceived idea that he would find IELTS difficult. He used to sit in the back row during the tutorial sessions, next to his friend who was a relatively active participant in class. He would leave much of the discussion, explanation and clarification to his friend. Ashok rarely participated actively in group sessions or volunteered answers. The speaking prompts were difficult for him. At the beginning, he found it very difficult to plan, organise, and to write in class. As the lessons progressed, he was able to appropriately apply some of the knowledge and skills learnt in the sessions and his writing began to show creativity. In reading, however, he showed little improvement.

Ashok’s test scores

These results show a slight improvement in reading in the post-test but Ashok did not obtain his required score in the final IELTS test. Ashok’s major weakness was his negative attitude. He entered the program with the attitude that he would find it difficult. He was also very shy and reserved in class and would participate only on invitation from the tutor. Towards the end of the preparation program he began to realise that he had the potential to improve, and that there were others in the class like him, but he needed a lot more time, more practice and perhaps a more positive attitude to help him gain a better result. In the post-test interview, he confessed that he should have been more proactive in class. He wished the preparation program had been longer and less intensive. Unlike other candidates, he felt he grasped skills much more slowly and therefore needed more time. However, he feels he is much better equipped now to attempt the test and felt he would perform better if he sat for the test a second time.

6.0 Limitations of the study

During the course of the study a number of constraints to the interpretation of our data set became evident. Sixty-two candidates registered for the preparation program. However, at the end of the preparation program, only 48 candidates’ results could be used as valid results. There were several reasons for this. At the participant recruitment stage, several people were unclear about the module they wished to take. This study was restricted to the General Training module, but when the preparation program started, we were surprised when several Academic candidates withdrew on the first day. A few others who were Academic chose to complete the program but they were excluded from our statistics. In addition, the day of the post-test happened to be a religious holiday for some candidates in Labasa and Lautoka, therefore some candidates who had completed the program were absent from the post-test. They too were excluded from the data set. Hence, at the conclusion of the preparation program, our data set had decreased to only 48 complete pre- and post-test results.

The 30 hours of tuition was offered at a very small fee and therefore it is possible that the participants’ level of commitment is questionable. For IELTS tuition, candidates would normally be paying five times the amount they paid for this preparation program.

7.0 Study data

7.1 Questionnaire data

The pre-test questionnaire items for the 48 candidates were coded and analysed using the SPSS statistical package. Most of the questionnaire items were categorical items such as age, occupation and education level and these were put through simple frequency counts and correlations. The means, standard deviations, median minimum and maximum changes were calculated for the pre- and post-test results to compare the gains. The Wilcoxon sign rank test was done to see if any change was significant.

Two areas of information from the initial questionnaire are presented below to provide additional context for the data. As with the two candidate profiles provided above, we hope that this information provides a useful lens through which to view the effects of participation in the preparation program.

7.1.1 Self report 1: Use of English

The questionnaire asked for information on where and when the participants regularly used English. The following results were obtained:

• 77% used English at work
• 11% used English when socialising, spending time with friends
• 8% used English for their studies
• under 2% used English at home.

We were very interested to see that a high percentage of our participants described using English for reading and writing every day. 75% told us that they read texts such as newspapers, magazines, work-related texts and ‘story books’ every day; 65% needed to write official letters and reports every day.

7.1.2 Self report 2: Self-assessment of current English language competence

The candidates’ self-assessments surprised us, as the majority of them reported feeling very positively about the same skills that we had identified as being their areas of weakness. Although only $29\%$ of our sample cited reading as their strongest skill (mainly due to work requirements), reading was high on most people’s lists, in terms of being good or very good. The following percentages of our sample identified these skills as being good/very good.

• Writing: 87.1%
• Reading: 80.7%
• Speaking: 80.6%
• Listening: 77.5%

7.2 Interview data

Six illustrative extracts from this data set are provided below. These extracts provide information on the program participants’ backgrounds and their overall response to the test preparation tasks they completed.

I am a tailor working at home. For more than 10 years I have not done any reading or writing. The only writing I do is my client’s name and address and their orders (clothes).

The preparation program was very helpful because after many years I got the opportunity to do some reading and writing. It was also very helpful because I had no idea what IELTS test was like. After the program I was confident to do the test and I am happy with the results. I think without the tutorial I would be very nervous to do the test. (Usha)

From the day I heard I have to do the IELTS test, I was very nervous because I did not know anything about the test. I was a bright student at school but for more than 13 years was out of the school system. I was feeling OK about the writing but I was scared of the reading because I have been doing little or no reading.

The preparation program was good for me because it gave me ideas about the test. I knew what type of reading and writing tasks to expect. I also got ideas about how to tackle the questions in the reading and also how to organise my essay. (Vinod)

Because of the training I got to know more about the IELTS test. The best sessions were the ones on writing. I don’t write essays and the training gave me an idea of what to do. It taught me how to write and organise points and to give my opinions. Everything was well organised but I think there should have been more reading exercises which I could have used for revision. (Nishad)

The training was very helpful. Before that I did not know much about IELTS, the content and the structure. I feel that if I didn’t do the training I may not have done so well in the IELTS test. The content was sufficient and well organised, but I feel we should have spent more time on the Listening to become more familiar with the different speaker’s voice and speed. The reading classes were all right but I feel that the training and the test content were a little different. But the training gave me a good idea of what is involved in reading. (Arif)

I have problems in writing essays because I don’t have to do them at work. So I was happy that we had good training and I feel my writing got better. I liked the way the classes were held. I enjoyed the group work we had to do because I was not so shy when I had to speak in front of everybody. The teachers were very friendly and taught us well. For me the reading was hard and I am happy we had the training because it helped me in the IELTS test. (Mohammed)

I found the classes very useful. I work in a big company where I use English all the time but I know I have problems in grammar some times. The writing classes helped me to improve here and made me think carefully and plan my writing. For me, it was very useful to know how to analyse the tasks in writing because once I know how to do this, my essay and letter writing will improve. The reading material was a little difficult because the time given for each was short, but because we were taught how to look for the answers, I did improve here. I feel that if there were more classes on reading, I would have done much better. (MS)

7.3 Test results

The complete results of our statistical analyses are included in Appendix 4.0.

7.3.1 Experimental group

Anticipated research outcome 2: Improved performance on GT Reading and Writing

Table 1 below provides data on the performance gains on the reading and writing sub-tests. The differences in means, and the maximum loss and gains between the pre- and post-tests are presented. The results show that in Writing Task 1 there is an average gain of half a band (0.4) and for Writing Task 2 the gain is almost one band (0.9). For both the writing tasks, there is a maximum gain of three bands.

Table 1: Experimental group

The mean change was much greater for reading (1.04) than for writing. The maximum gain on the reading (14 points, which constitutes more than a 25% increase in performance) is indeed dramatic, however it is matched by a similar maximum loss (12 points).

7.3.2 Control group

The Control group results show a higher mean for reading than that of the experimental group: (experimental group: R1=23.6, R2=24.6; control group: R1=27, R2=28). However, these scores need to be viewed with some caution as it is possible that they were dragged up by a particularly strong performance in only one or two individual candidates.

On the other hand, if the candidates in the control group are a better group as far as reading is concerned, then the lack of tangible improvement would appear to reflect Elder and O’Loughlin’s study (2002), where the maximum change in performance experienced by more proficient students was much lower. Similarly, the writing tasks of our control group showed no change, ie. the means for both the pre- and post-test writing are exactly the same (5.9 and 6.0). Table 2 below summarises the minimum and maximum changes in test scores for this group.

Table 2: Control group: Minimum and maximum change in performance

The control group in this study was fairly small (nine). Ideally, they should have been drawn from a group with comparable scores on the pre-test but, as discussed above, this was not possible at the time.

7.3.3 Further comparison of experimental group and control group

The differences in means between the pre- and post-tests are consistently smaller for the control group than for the experimental group. For the control group, the average gain in both the writing tasks was less than half a band (0.33 and 0.11). The maximum gain in the post-test writing tasks was one band. Table 3 below presents these comparisons.

Table 3: Comparison of experimental group (EG) and control group (CG)

The Wilcoxson test of significance shows that the gain in the post-test is highly significant particularly on Writing Task 2 (Wilcoxson, tied Z-value - 0.4586, tied P-value < 0.0001) and Writing Task 1 (Wilcoxson, tied Z-value -3.375, tied P-value <0.0001). The gain is less significant for reading (Wilcoxson, tied Z-value -1.450, tied P-value 0.1455). Complete results are given in Appendix 4.

With the control group, there is a maximum gain of four points for reading and one band for writing. With any test situation, it is expected that there will be some gain due to the familiarity with the test the second time around. Candidates in the experimental group were not only familiar with the test when they took it for the second time, but they had also been through a number of preparation tasks. It may be possible that the gain is so much larger in this group because of their very recent involvement in the program.

8.0 Discussion and implications

The most significant (and professionally rewarding) finding from this research was that, for this group of candidates, participation in a preparation program did appear to make a positive difference to their IELTS test performance. As was expressed in Elder and O’Loughlin’s study (2002), we feel that this is valuable information for institutions such as USP, to pass on to prospective IELTS test candidates, clearly with certain caveats.

One aspect of our results was surprising and deserves mention here. We observed a number of significant tangible gains in candidate performance after far fewer hours of test preparation than the usual claims of around 100 hours required for a candidate to improve by one band. However, we feel that this cannot be accepted at face value. We are concerned about questions of retention and attrition of knowledge, for example, how durable is this gain in performance? The post-test was done soon after the preparation program and it would be very interesting to find out how these candidates would perform if they were retested at a later date.

A finding that reflected earlier research (eg. Elder and O’Loughlin 2002) was that there was less tangible gain in the reading than in the writing. We wonder why reading proficiency is more difficult to influence. Could the candidates’ general background knowledge have had an effect on their reading comprehension? Or can it be attributed, perhaps, to different time allocations in the preparation timetable? If so, then there are implications for the design of future preparation programs. It may also be due to the inherent differences in how fast or slow tangible progress can be made in the different skill areas for different test candidates. There is clearly a need to investigate further the development of reading skills at various proficiency levels for specific reading needs.

The nature of the gain in performance we were investigating also needs to be examined. For example, in the IELTS writing test, performance is assessed in terms of bands. These descriptive bands are wide, which means that a candidate may ‘improve’ between the pre- and post-test within that band; however, the width of the band does not allow us, or the candidate, to detect the shift in performance. Even so, a number of these candidates did show significantly increased performance in writing, particularly in Task 2, which is arguably a more complex task.

9.0 Conclusion and future directions

IELTS testing and preparation programs are fairly recent phenomena in Fiji. This was the first study done in this context, and we were encouraged by the very positive outcomes we obtained. For a significant number of our candidates, their participation in the preparation program did appear to result in clear gains in test scores within this set timeframe.

However, there is more to effective test preparation and enhanced performance than apparently improved test scores. We see an obvious need for further studies and a combination of data to accurately determine whether preparation programs ‘work’ or not, and what precise factors affect specific aspects of candidate performance. As indicated above, the significant gains in performance in this study should not be accepted at face value. It is possible that an extended study, which involved an examination of environmental context factors such as language input, schooling experiences, motivation and familiarity with the test, combined with the data of this particular study would provide a richer evaluation of the effectiveness of proficiency test preparation programs.

The candidates in our sample came from what may be seen as an impoverished target language background. Although English is the medium of instruction in schools, the question of the actual amount of appropriate exposure to the language remains. The preparation program on which this study was based focused on the discrete skills of the IELTS exam, and not on the sustained and varied interactive input that is needed to increase proficiency in a language.

Chandra Rao, Kate McPherson, Rajni Chand, Veena Khan

Bibliography

• Anderson, J. C. and Hamp-Lyons, L. (1996). ‘TOEFL Preparation courses: a study of washback’. Language Testing, Vol. 13: 280-296.
• Brown, J. D. (1998). ‘Does IELTS preparation work? An application of the context-adaptive model of language program evaluation’. IELTS Research Report, Vol. 1: 20-37.
• Celestine, C and Su Ming, C. (1999). ‘The effect of background disciplines on IELTS scores’. IELTS Research Report, Vol. 2: p.48.
• Coomber, J. (1997). ‘Are test preparation programs really effective? Evaluating an IELTS preparation course’. Dissertation for MA, Institute of Language, Retrieved from the University of Surrey URL, March 2002.
• Elder, C and O’Loughlin, K. (2002). ‘Investigating the relationship between Intensive English Language study and Band Score Gain on IELTS’. Unpublished report.
• Horak, T. (1996). ‘IELTS Impact Study Project’. Unpublished MA Dissertation, Lancaster University.
• Long, M. H. (1984). ‘Process and product in ESL program evaluation’. TESOL Quarterly, Vol. 18: 409 - 425.
• Milanovic, M and Saville, N. (1996). ‘Considering the impact of Cambridge EFL Examinations’. UCLES Internal Report. Retrieved www.cambridge-efl.org.
• Rudman, H. C. (1989). ‘Integrating testing with teaching’. Practical Assessment, Research and Evaluation, Vol. 1 p. 6.
• Saville, N. (1998). ‘Predicting impact on language learning and the classroom’. Internal draft for IELTS Impact Study, UCLES. Retrieved from Cambridge EFLOn-line.

APPENDIX 1.0

IELTS preparation program timetable

APPENDIX 2.0

Sample materials

Cameron, P. and Todd, V. (1996). Prepare for IELTS. Sydney: Insearch Language Centre and International Programs.

Cameron, P. and Todd, V. (2001). The New Prepare for IELTS. Sydney: Insearch Language Centre and International Programs.

Cameron, P. and Todd, V. (2002). The New IELTS Speaking Test Supplement to the New Prepare for IELTS Preparation Course. Sydney: Insearch Language Centre and International Programs.

Hogan, M. J., Campbell, B., Todd, V., and Perret, G. (1991). Prepare for IELTS (Practice Tests for Module C [Humanities] and General Training Module). Sydney: Insearch Language Centre and International Programs.

Jakeman, V. and McDowell, C. (1999). Insight into IELTS CUP.

Jakeman, V. and McDowell, C. (2001). IELTS Practice Tests Plus. Pearson Education Limited and Longman.

Maggs, M. M. (1982). Identifying Main Ideas. Stokie: Voluntad Publishers.

O’Sullivan, K., Lindeck, J. (2000). Focussing on IELTS Reading and Writing Skills. Sydney: NCELTR.

Pejovic, V., Nicklin, M., Read, P. (1997). IELTS Preparation and Practice. Indonesia Australia Language Foundation and Oxford University Press.

Sahanaya, W. and Lindeck, J. (2002). IELTS Preparation and Practice Listening and Speaking (Second Edition). South Melbourne: Oxford University Press.

Watcyn-Jones, P. and Allsop, J. (1990). Test Your Prepositions. Penguin.

APPENDIX 3.0

Initial questionnaire

THE UNIVERSITY OF THE SOUTH PACIFIC

IELTS RESEARCH PROJECT 2002

Candidate questionnaire

A. PERSONAL DETAILS

1. Name: ____________________ 2. Age: ________
2. Male/Female: ____________________
3. Occupation: ____________________
4. Nationality: ____________________
5. Address: ____________________

B. LANGUAGE EXPERIENCE

7. What is your first language? ____________________

8. What other languages do you speak besides English? ____________________

9. I use English (please tick)

• At work: ____
• At home: ____
• When socialising with friends: ____
• During religious ceremonies: ____
• Other (please specify): ____________________

10. How often do you read in English? (please tick)

• Everyday: ____
• Sometimes: ____
• Never: ____

11. What kinds of things do you read? (tick)

• Newspapers: ____
• Magazines: ____
• Story books: ____
• School texts: ____
• Other (specify): ____________________

12. How often do you write in English? (please tick)

• Everyday: ____
• Sometimes: ____
• Never: ____

13. What kind of writing do you do? (tick)

• Letters to friends: ____
• Official letters: ____
• Reports (at work): ____
• Other (specify): ____________________

14. How do you assess your English language skills? (please tick in appropriate box).

15. Give one reason why ____ is your strongest skill.

    ____________________

16. Give one reason why ____ is your weakest skill.

    ____________________

C. EDUCATIONAL DETAILS

17. Highest level of education: ____________________
18. Year that you left school: ________
19. Are you studying currently? If yes, what? ____________________

D. IELTS DETAILS

20. Have you ever taken the IELTS exam? YES/NO When? ____________________

21. If YES, what was your band score?

• Overall: ____
• In reading: ____
• In writing: ____

22. What is your reason for taking IELTS? ____________________

23. Have you had any IELTS preparatory tuition previously? YES/NO

24. IF YES, when and for how long? ____________________

Thank you for taking the time to complete this questionnaire. Please return the questionnaire to your tutor.

APPENDIX 4.0

Complete results

A4.1 The experimental group

Descriptive Statistics

Wilcoxon Signed Rank Test for writing 1, pt writing 1

Wilcoxon Rank Info for pre test reading, post test reading

Wilcoxon Signed Rank Test for pre test reading, post test reading

Wilcoxon Rank Info for writing 1, pt writing 1

Wilcoxon Rank Info for writing 2, pt writing 2

One case was omitted due to missing values.

Wilcoxon Signed Rank Test for writing 2, pt writing 2

One case was omitted due to missing values.

A4.2 The control group

Wilcoxon Signed Rank Test for CG writing 1, CG pt writing 1

Wilcoxon Signed Rank Test for CG pre test reading, CG post test reading

Wilcoxon Rank Info for CG writing 1, CG pt writing 1

Wilcoxon Signed Rank Test for CG writing 2, CG pt writing 2

Wilcoxon Rank Info for CG writing 2, CG pt writing 2

Wilcoxon Rank Info for CG pre test reading, CG post test reading