Developing the assessment literacy of IELTS Test users in higher education

Abstract

The rapid growth of IELTS has resulted in a growing number of people providing information about the IELTS Test, setting standards, interpreting scores and advising test-takers. This project examined the assessment literacy needs of university test users (including admissions, marketing, academic and English language staff), how well these needs are being met and what other approaches could be adopted to meet these needs. The study took the form of a “proactive evaluation” (Owen, 2006), which included:

• online survey and face-to-face interviews to investigate the assessment literacy needs of IELTS Test users at two Australian universities and how well these needs are met by current resources
• discourse analytic study of the IELTS Guide (2009)
• comparative evaluations of different IELTS Test resources and the institutional sections of the IELTS, TOEFL and PTE Academic websites
• a review of best practice in staff online training programs.

The survey and interview findings indicated that the IELTS Test was mostly needed for advising prospective students about English language entry requirements and making admissions decisions. To these ends, test users were mainly focused on four topics about the IELTS Test: the minimum IELTS scores for entry to courses at their university, the different components of the IELTS Test, how long IELTS scores are valid, and the relationship between IELTS scores and other evidence of English proficiency accepted by their university. The survey and interview results also indicated that the needs of the IELTS Test users were reasonably well met. Most mainly accessed their institution’s English language entry regulations and, to a lesser extent, the IELTS official website, for information about the IELTS Test. After reading the IELTS Guide (2009), all survey respondents generally found it to be informative. However, some believed that it could have included more information about the meaning and interpretation of IELTS test scores. A discourse analytic study of the IELTS Guide suggested that it had more of a marketing - than educational - emphasis, which may limit its usefulness as a training document. The comparative evaluations of different IELTS Test resources and the institutional sections of the three different test websites suggested the IELTS website was an informative resource, although some of its content and user-friendliness could be improved. The most popular alternative choice for learning about the IELTS Test in the survey and interviews was online tutorials, an approach that has not been used to date by the IELTS partners. A detailed example of best practice in online training programs is provided. Finally, recommendations are made for developing the assessment literacy of IELTS Test users.

1 Background and rationale

1.1 Test users and IELTS

In recent years there has been a growing interest in studying the use of the IELTS Test in higher education institutions. A number of important studies have been funded by IELTS partners including Deakin (1997), McDowell and Merrylees (1998), Hagan, Starfield and Coleman (2003), Rea-Dickins, Kiely and Yu (2004), and O’Loughlin (2008).

Coleman, Starfield and Hagan (2003) examined the attitudes of both students and staff (administrative and academic) towards IELTS in three institutions in Australia, China and the UK. They found that, while all participants in the study were generally positively disposed towards the Test, overall, students were more knowledgeable about the Test and more convinced that the institution’s IELTS entry level was appropriate for the course of study they were undertaking. Staff generally felt that the IELTS scores should be higher and that the English language ability of many students was not adequate for their chosen course of study. Perhaps the most disconcerting finding was that the university staff (administrative and academic) in the three participating institutions demonstrated low understanding of the meaning of IELTS scores.

In her doctoral research project, Banerjee (2003) examined the use of proficiency test scores, including IELTS, in the selection of postgraduate degree courses at a UK university. She found that the selection of international students at the University of Lancaster was a complex, holistic decision-making process based on the recommendation of an academic staff member taking into account a wide range of criteria. Yet, like Coleman, Starfield and Hagan (2003), Banerjee found that academic admissions officers were not very knowledgeable about the meaning of IELTS and proficiency test scores. In a study conducted at the University of Bristol, Rea-Dickins, Kiely and Yu (2004) also found that university admissions staff were not always sufficiently knowledgeable about the meaning of IELTS test scores. They argued for stronger training of admissions tutors in order that they become better informed about the meanings of IELTS score profiles. This included awareness of and access to the IELTS website.

In the Australian university context, O’Loughlin (2008) examined the use of IELTS within a large faculty of a leading university. The study reported variable levels of knowledge about IELTS (both of the Test and the scores it produces) among university staff (and students), including a lack of understanding as to what different IELTS scores imply about a student’s language ability, their readiness for university study and their need for further English development. It also noted the prevalence of ‘folkloric’ beliefs among staff about English language proficiency and IELTS, some with a firmer basis in reality than others.

1.2 Educating IELTS Test users

Educating test users, such as admissions, marketing and academic staff, is a high priority if the IELTS Test is to be used appropriately for the purposes for which it was intended. This relates to the central question of the Test’s validity. As Messick (1996) suggested, the validity of a test hinges critically on the interpretation of test scores and the uses to which they are put. No matter how psychometrically sound a language test might be, the meaning and use of test scores is ultimately determined by test users. Within a university context, test users include the staff who develop entry policies, market, recruit, select, teach and support international students in a university context. Previous research (cited above) suggests that such people may have only limited understanding of IELTS including its purpose, format and content as well as the meaning and interpretation of test scores.

1.3 Assessment literacy

The term ‘assessment literacy’ has recently been taken up by the language testing profession to refer to the understandings about language assessment that various people - such as test developers, assessors, test users and teachers - need to acquire to develop, score, interpret and improve classroom-based assessments. As Taylor (2009, p 25) suggests:

… an appropriate level of assessment literacy needs to be nurtured not just among engineers and technicians who are actively involved in test development or research activities, or even among applied linguists and language teachers involved in delivering language education, but much more broadly in the public domain if a better understanding of the function and values of assessment tools and their outcomes are to be realised across society.

Taylor (2009, p 30) argues that the familiar term ‘test wiseness’ could be vested with new meaning to refer to the assessment skills, knowledge and principles which various stakeholders need to acquire to ensure the valid and ethical use of a test. However, the type and level of test wiseness needed by different IELTS stakeholders may be quite different. Just as the needs and learning experiences of language teachers may be different from language testing specialists and, therefore, should be studied in their own right (see, for example, O’Loughlin, 2006) so the perspectives of test users such as university staff warrant investigation (Shohamy, 2001). Moreover, test users are a very diverse group of individuals: in the university context, some of them may require only a limited understanding of the test such as how scores are reported and how to interpret them in relation to university entry.

2 Aims of the study

The study built on a previous project (O’Loughlin, 2008) in seeking to investigate the assessment literacy needs of IELTS Test users in higher education. The earlier study concluded that there was a clear need for IELTS Test users to be better informed. However, it was also noted that the question of how much and what type of knowledge could be sufficient is pivotal to understanding the needs of different test users. In the university context, test users include academic and admissions staff involved in advising and selecting prospective students, marketing and recruitment staff who provide advice to prospective students about course entry requirements, and academic and support staff who teach international students.

Developing the assessment literacy of such IELTS Test users may involve informing them about issues such as the purpose and content of the Test, the meaning of Test scores, the appropriateness of cut-off levels, its validity, reliability and predictive power and its comparability with other accepted forms of evidence of English proficiency. Another fundamental question here is how this knowledge might be best communicated.

The key research questions addressed in this study therefore were:

1. What are the assessment literacy needs of IELTS Test users in higher education?
2. How well are these needs currently being met?
3. What other approach(es) could be adopted to meet these needs?

3 Context of the study

3.1 Research sites

The study examined the assessment literacy needs of academic and non-academic staff at two large metropolitan Australian universities, hereafter referred to as University A and University B.

University A is a leading higher education institution. University B is a dual sector - higher and vocational education - institution. In both universities more than 25% of the student population has been international over the last five years. The IELTS Test is a well-established form of evidence for student selection at both institutions.

3.2 Targeted participants

The study aimed to recruit volunteer participants working in a range of roles across each university including admissions, marketing, academic, and both pre-course and in-course support language staff. Knowledge of the IELTS Test was identified as relevant to each of these work areas on the basis of a previous study (O’Loughlin, 2008) and through an updated analysis of the current uses of the Test in recruitment, selection and teaching at University A and University B. It was anticipated that some staff in each work area would be currently using IELTS test scores directly in their work, while others would be using them more indirectly or not currently at all. It was decided, however, that no staff who volunteered would be excluded from the project since their readiness to participate indicated that the IELTS Test had some relevance to their work.

4 Methodology

4.1 General approach

The study took the form of a ‘proactive evaluation’, a form of program evaluation (Owen, 2006). Proactive evaluations are particularly useful in reviewing and improving current practices, in this case developing test users’ assessment literacy. They position the researcher or evaluator as an adviser, “providing information about the extent of the problem that policy should address, or what program format is needed” (Owen 2006, p 41). The evaluation, therefore, provides advice to organisations about new directions for the work they undertake. In this case, the evaluation aimed to inform the work of the IELTS partners in developing the assessment literacy of university staff who directly or indirectly use IELTS test scores. Proactive evaluations typically include needs assessment, analyses of current practices (where they already exist), syntheses of relevant literature and reviews of exemplary practice. This study employed all of these approaches.

4.2 Data collection

The study collected data on both the ‘subjective’ needs of IELTS Test users, ie those identified by the study participants themselves, and their ‘objective’ needs, ie those identified by other parties such as, in this study, the IELTS partners and the researcher. The researcher and research assistant carefully investigated the specific objective needs of IELTS Test users by referring to the IELTS Handbook (2007), the IELTS Guide (2009), the IELTS official website and the IELTS Scores Explained DVD (2009), as well as the websites of the two universities targeted in the study. This analysis identified five main purposes for using the Test (see Figure 4) and 15 specific areas of information about the Test (see Figure 5).

4.2.1 Survey

An online survey powered by Survey Monkey™ (http://www.surveymonkey.com) was completed by all volunteer participants (refer Appendix 3). All respondents answered Questions 1-3 and 8-13. Only those who identified themselves as IELTS Test users in Question 3 completed Questions 4-7.

Question 8 and 9 required respondents to read and then evaluate the IELTS Guide for Educational Institutions, Governments, Professional Bodies and Commercial Organisations (2009) hereafter referred to as the IELTS Guide.

4.2.2 Interviews

Semi-structured interviews were conducted with participants who volunteered to be interviewed after the completion of their surveys using an interview template (refer Appendix 4).

4.2.3 The IELTS Guide

As it specifically targets test users in educational institutions, the IELTS Guide was a major focus of this study. As well as featuring in the survey, it was later evaluated from a discourse analytic perspective and also compared to other IELTS resources.

4.2.4 Test resources

The different sources of information currently available to IELTS Test users were collected for evaluation (the IELTS Guide, the IELTS website, the IELTS Scores Explained DVD (2009) and the websites of Universities A and B).

The sections on the IELTS Test, TOEFL (Test of English as a Foreign Language) and the PTE (Pearson Test of English) websites for institutional users were accessed for evaluation.

4.2.5 Online training programs

Information about online staff training programs was collected and an example of a best practice program used in a university context was examined.

4.3 Procedures

The study included the following phases:

Phase 1: Relevant permissions obtained from Universities A and B

Phase 2: University ethics approval obtained

Phase 3: Survey and interview template developed

Phase 4: Survey and interview template piloted

Phase 5: Survey and interview template reviewed in light of pilot study

Phase 6: Volunteer staff participants recruited at the two universities (N = 84)

Phase 7: Survey completed online

Phase 8: Surveys analysed

Phase 9: Semi-structured interviews conducted with volunteer staff who had completed the survey (N=19)

Phase 10: Staff interviews analysed

Phase 11: IELTS Test resources currently used by university staff evaluated

Phase 12: Institutional sections of websites of IELTS, TOEFL and PTE Academic evaluated

Phase 13: Current approaches to educating university staff by IELTS partners reviewed

Phase 14: Best practice in university staff training programs reviewed, especially online training programs

Phase 15: Final report completed

4.3.1 Pilot study

Prior to the main stage of the study, a small-scale pilot study was conducted with a small group of staff (nine staff members) from one of the faculties in University A, from which the participants for the main study would not be recruited in June 2010. The main aim of the pilot study was to develop, trial and refine the instruments to be used in the main study. During the pilot study, an online survey powered by Survey Monkey™ and an interview template were first drafted, revised and then administered to the nine staff. The group of staff in the pilot study comprised three admissions, two marketing, one language and three academic staff members. The group provided a representative sample of the participants planned for the main study. The participants in the pilot study were asked to respond to the questions in the survey and then to provide feedback on the format of the survey, the wording of individual questions and items in the survey, the clarity of the survey and the time they spent completing the survey. Participants’ responses were analysed, their feedback was evaluated and revisions were made to the survey based on participants’ responses and feedback.

The revisions made in the survey included major and minor rephrasing of questions and items, re-ordering of the questions and the removal of one question. The rephrasing of the questions and items were done to enhance their clarity and precision. The re-ordering of the questions aimed to refine the logical development of the survey. The overall format of the survey was also redesigned, to create obligatory and optional sections. This was based on the feedback from some participants (mainly academic staff) who considered parts of the survey less relevant to their roles.

Following the administration and analysis of the pilot survey data, short interviews were planned with four participants who volunteered to be interviewed using a pilot interview template. The four participants included one academic, one admissions, one marketing and one language staff member. The interviewees were initially shown a copy of their completed surveys to refresh their memories so that they could better expand on and clarify their responses. The main aim of the interviews was to gain further insights into and extend participants’ survey responses. The major revisions made to the interview template and the interview process after the pilot study were based on feedback from the participants, the researcher and research assistant. In particular, the questions in the interview template were better aligned with the main questions in the survey and the allocated time for each section of the interviews was revised to reduce the overall length of the interview.

4.3.2 Main study

The main study began in August 2010. The surveys and interviews were completed by early December 2010. Recruitment of participants took place in two phases. First, a list of staff from University A and University B was made after contacting senior managers in relevant work areas at both universities. The list included staff members from admissions, marketing, academic, English language preparation and in-course support. An invitation to participate in the research project was sent to each person on this list via email (refer Appendix 1). Participation in the project was voluntary.

Those individuals who responded in the affirmative were sent an email containing a personalised identification code and a link to the plain language statement and consent information (see Appendix 2) on a secure university webpage. The webpage was designed to direct the participants to the online survey powered by Survey Monkey™ (see Appendix 3).

Respondents indicated their consent to participate in the study in Question 1 of the survey and then answered the remaining questions. A total of 84 participants (43 participants from University A and 41 participants from University B completed the survey. In the final question of the survey (Question 11) the participants were invited to volunteer for a follow-up interview. The completed surveys were then collected via Survey Monkey™. The survey responses were coded and entered into a database. The survey data were analysed in terms of raw numbers and percentages for the two universities, both combined and separately. Responses to each question from the survey were then graphed. Some of the data was also narrative in character (eg optional further comments on selected survey items). This data was coded and thematically categorised.

Following completion of the surveys, staff interviews were conducted. Semi-structured interviews were held with 19 volunteer participants (10 from University A and nine from University B). The staff interview template was used to conduct the interviews (see Appendix 4). The sample included admissions, marketing, academic, and language staff members. The interviews were conducted on an individual basis. As in the pilot study, participants were shown their completed surveys to refresh their memories and to provide a starting point for the interview. The interviews themselves allowed opportunity for both clarification and extension of these responses. Field notes were taken during the interviews and audio-taped recordings were made of the interviews for subsequent analysis.

Other work in the main study was conducted between January and March 2011. The IELTS Guide was examined by the researcher and the research assistant from a discourse analytic perspective, in terms of its educational and marketing functions. Comparative evaluations were also undertaken of:

a) different IELTS resources (including the websites of Universities A and B, the IELTS website, the IELTS Guide and the IELTS Scores Explained DVD) b) the institutional sections of the IELTS, TOEFL and PTE Academic websites.

Finally, current trends in online staff training programs were identified and we analysed in detail an example of a best practice online program used in a university context.

4.4 Methods of analysis

The survey responses were analysed in terms of percentages for the two universities, both combined and separately. Each interview was summarised (see Appendix 5) and then the main themes were identified. Salient quotations which reflected these themes were then transcribed and coded in terms of whether they were a) representative of all 19 interviewees, b) representative of a particular sub-group eg marketing or academic staff or c) individual comments.

The comparative evaluations of the different IELTS Test resources and the institutional sections of the IELTS, TOEFL and PTE Academic websites were undertaken through ratings of how informative they were on a range of topics and their user friendliness. Finally, current trends in online staff training programs were identified, and the best practice example of such a program was completed by the researcher and research assistant and its features described in detail.

5 Results

5.1 Participant information

A total of 84 staff from the two universities agreed to complete the online survey (refer Appendix 1). Responses to Question 2 and 3 provided important background information about the participants.

5.1.1 Work areas of participants

In Question 2 of the survey, respondents were asked to specify their work area. Of the 84 participants, there were 26 academic staff, 22 admissions staff, nine marketing staff, 15 English language staff and two other staff. Forty-three of the participants (51%) worked at University A and 41 (49%) worked a University B. Figure 1 below provides a breakdown of the work areas of participants from University A and University B in terms of percentages.

Work Areas of Participants

Figure 1: Work areas of participants at University A and University B (%)

Figure 1 shows that the four main groups represented at both universities were academics, admissions, marketing and English language staff.

5.1.2 Use of the IELTS Test by research participants

Question 3 of the survey asked participants “Do you currently use the IELTS in your job?” - 50 of the total 84 (60%) participants nominated themselves as current users of the IELTS Test. Figure 2 below shows the relative percentages of users and non-users at each university.

Do you currently use the IELTS in your job?

Figure 2: Comparison of participants at University A and University B as users and non-users of the IELTS Test (%)

Table 1 below provides a breakdown of the survey respondents into IELTS Test users/non-users and their various work roles at the two universities. In retrospect, it would have been preferable to have had more direct users of the IELTS Test involved in the study but the respondents consisted only of those who chose to participate. Nevertheless, there was a reasonable distribution of the five categories of staff, both users and non-users of the IELTS Test, at the two universities

Table 1: Survey respondents (N= 84)

A total of 19 of the 84 survey respondents volunteered to take part in a follow-up interview. Table 2 below provides a breakdown of these interviewees at each university in terms of their work roles.

Table 2: Interview participants (N=19)

This breakdown indicates that overall, the 19 interviewees provided a good representation of the 84 survey respondents.

5.2 Research questions

The results for each of the three research questions are reported below.

5.2.1 Research Question 1: What are the assessment literacy needs of IELTS Test users in higher education?

The data used to collect information about the needs of the participants for this first research question was drawn from Questions 4 and 5 of the survey and the associated interview questions. These questions were answered only by the 50 respondents who identified themselves as IELTS Test users in Question 3.

Question 4 asked: “What do you need the IELTS Test for in your current job? Please tick ALL responses that apply.” Figure 3 shows the overall results. The two main purposes identified by the respondents were a) advising prospective students about English language entry requirements and b) making student admission decisions. It is noteworthy that only five respondents used the IELTS Test to set cut-off levels for university entry. This may indicate that either the survey respondents were not sufficiently senior to do such work or that setting and revising minimum entry requirements was not frequently undertaken at either university.

What do you need the IELTS test for in your current job?

Figure 3: Uses of the IELTS Test in the workplace (N=50)

Figure 4 below compares the uses of the IELTS Test at universities A and B in percentages. The only significant difference is that a greater percentage of IELTS Test users at University A provided advice to prospective students about English language entry requirements than at University B.

What do you need the IELTS test for in your current job?

Figure 4: Comparison of the uses of the IELTS Test at University A (N= 23) and University B (N=27)

Question 5 on the survey asked the 50 respondents who were test users “What specific information about the IELTS Test do you use in your job? Please tick ALL responses that apply.” Fifteen topics were listed as options. These were identified by the researcher and research assistant as potentially most relevant to the work of university staff on the basis of information about the Test given on the IELTS official website (www.ielts.org), the IELTS Handbook (2007) and the IELTS Guide (2009). These topics were, therefore, included as the core informational needs of university staff. Figure 5 below provides an overview of the results.

What specific information about the IELTS test do you use in your job?

Figure 5: Information about the IELTS Test used in the workplace (N=50)

Figure 6 shows that the results for University A and University B were very similar for each of the 15 topics listed. More than 50% of the IELTS Test users at both universities reported using information on four of these topics: the minimum IELTS entry scores for entry to courses at their university ; the different components of the IELTS Test ; how long the IELTS scores are valid; and the relationship between IELTS test scores and other evidence of English proficiency accepted by their university.

In the interviews, participants were asked whether any of the other types of specific information that they did not initially select in Question 5 of the survey might be helpful in their jobs. Ten of the 15 IELTS Test users answered “no” to this question because the other types of information were not relevant to their jobs. For example, one suggested that the information she had not selected “…would not be relevant because I operationalise a policy which is already in place” (#449 Marketing, UNI B).

Another stated that “… a lot of these things [ie specific information about IELTS] I don’t know, but it wouldn’t make a difference [if I did]” (#644 Admissions, UNI A).

The other five test users suggested they might use some of the information they had not initially selected in Question 5. For example, one participant suggested she would use one piece of information about verification of the Test Report Form she found in the IELTS Guide after reading it (#346 Marketing, UNI B). In more general terms, another participant indicated she would be personally interested to know more so that she could “explain to students to allay some of their concerns” about the Test and the minimum IELTS entry requirements of her university (#531 Marketing, UNI A).

What specific information about the IELTS test do you use in your job?

Figure 6: Information about the IELTS Test used at University A (N= 23) and University B (N= 27)

In summary, the IELTS Test users participating in this study mainly needed the IELTS Test for advising prospective students about their English language entry requirements and making admissions decisions. They were mainly focused on four topics: the minimum IELTS entry scores for entry to courses at their university; the different components of the IELTS Test ; how long the IELTS test scores are valid; and the relationship between the IELTS test scores and other evidence of English proficiency accepted by their university. These results reflect the fact that university staff are primarily concerned with information about IELTS test scores which is needed for the selection of international students at Australian universities as it is currently conducted. On the other hand, it is noteworthy tha test users made little use of information about such topics as how the overall IELTS band score is calculated and how the test scores are reported. This kind of information is currently not needed to make selection decisions.

5.2.2 Research Question 2: How well are these needs currently being met?

This question was addressed from several different perspectives: a) the responses to Questions 6-9 of the survey and related questions in interviews; b) a discourse analytic study of the IELTS Guide (2009); and c) comparative evaluations of both existing IELTS resources and the institutional sections of the websites of three prominent English proficiency tests - IELTS, TOEFL and PTE Academic websites.

Survey and interview data

Question 6 (like Questions 4 and 5) in the survey was solely directed to the 50 respondents who were test users. It asked them “Which sources of information about the IELTS Test do you mainly use in your job? Please TICK all responses that apply.” Figure 7 below shows the overall pattern of responses.

Which source of information about the IELTS test do you mainly use in your job?

Figure 7: Sources of information about the IELTS Test accessed by users (N=50)

The results show that the majority of users (84%, ie 42/50) accessed their institution’s English language entry regulations which are available on each university’s website. Slightly less than half of them (42%, ie 21/50) used the IELTS official website. Only six of the 50 users indicated that they accessed the IELTS Guide which can be downloaded from the IELTS website as well as in hard copy. None of the respondents in this study used the IELTS Scores Explained DVD. Figure 8 below shows the breakdown of these figures for University A and University B.

Of the 15 IELTS Test users interviewed, nine said they accessed their institution’s English language entry regulations which were available on their websites, mainly for checking information about the minimum acceptable IELTS scores for particular courses. For example, one marketing staff member said she used “the institution’s English entry language regulations” on her university’s website basically for “double checking against my general knowledge” about IELTS when questions were asked about specific courses (#530 Marketing, UNI A).

Nine IELTS user interviewees said they were aware of one or more of the other sources of information listed and the remaining six had no knowledge of them. Only two of them knew about the DVD but had not used it. Six of them said that they would ask colleagues in the language centre or IELTS Test centre attached to their institutions if they required additional information about the IELTS Test. For example, one marketing staff member said she did not know about the sources listed in the survey other than her own university’s English language regulations and said she “would ask for information from colleagues” in their affiliated language centre if needed (#531 Marketing, UNI A).

Figure 8: Sources of information about the IELTS Test accessed by users at University A (N= 23) and University B (N= 27)

Most interviewees suggested that there needed to be links to the IELTS official website from their respective institution’s website where information about its English language entry regulations were given. However, subsequent checking showed that both university websites did include these links, although they were perhaps not sufficiently prominent. Finally, one marketing staff member suggested that more information could be included on her institution’s website to make it “more user-friendly to students by giving information about the IELTS Test, how to practise and prepare”(#530 Marketing, UNI A).

Question 7 on the survey asked the 50 IELTS Test users “How useful is the information you currently access about each of the following aspects of the IELTS Test? Please tick the appropriate number on the scale from 0-3, 0=Not Useful, 1= Slightly Useful, 2=Useful and 3 = Very Useful”. The aspects listed were the same as those in Question 5. Figure 9 summarises the combined results.

On the one hand, 60% (30/50) of respondents saw the information provided about the minimum IELTS test scores for entry into specific courses at their university as being most useful. On the other hand, 48% of IELTS users (24/50) considered that the information about a) how candidates can prepare for the Test and b) the IELTS Test centres and how to register for the IELTS Test was not useful. Information about the security of the IELTS Test administration and report forms were not considered useful by 44% of users (22/50).

How USEFUL is the information you can currently access about each of the following aspects of the IELTS test?

Figure 9: Usefulness of the information accessed about the IELTS Test (N= 50)

The common theme in the interviews was that if a selected topic was marked as not or slightly useful, it was because the interviewees did not use the information or saw it as irrelevant to their job. One academic staff member involved in selection explained that information about certain aspects of IELTS (eg how the different components of the IELTS Test are scored) was not useful. He indicated that knowing about them would make no difference to the selection process as it currently operated where decisions are based purely on scores and that his judgment was therefore not required. Comparing past practices to those of the present day he suggested that:

…you would work a lot harder and try to know more about these things when I was able to exercise some judgment in borderline cases, but once people are beholden to numbers and that judgment goes out of the window, why do you have to bother knowing about things…with IELTS; the way the university does it, it’s just magic numbers. (#357 Academic, UNI B)

This comment underscores a widespread problem with current selection practices where selection staff only need to check whether an applicant has achieved the minimum acceptable IELTS scores to enter their course. As he suggests, they are not required to exercise any judgment and, therefore, do not require any real understanding of the IELTS Test.

Questions 8 and 9 of the survey required all 84 participants to read the IELTS Guide (2009) before responding. Question 8 asked “How informative is each section of the IELTS Guide. Please tick the appropriate number on a scale from 0-3 (0= Not informative, 1= Slightly Informative, 2 = Informative and 3 = Very Informative)”. Figure 10 summarises the results of Question 8 for Universities A and B combined.

The IELTS Guide was generally seen as informative. The sections (with page numbers in the IELTS Guide) considered informative or very informative by more than 65% of participants were:

What does the test involve? (p 10), (88% - 74/84)

• The quality test (pp 4-5), (80% - 67/84) Candidates with special requirements (p 15), (80% - 67/84) Test results and validity period (p 8), (80% - 67/84)
• Using IELTS test scores (p 8), (79% - 66/84) The international test (pp 3-4), (77% - 65/84)
• Test scores (p 8), (77% - 65/84 respondents) Test results you can trust (pp 6-7), (75% - 63/84)
• Next steps (p 15), (70% - 59/84)
• How can IELTS help you? (p 15), (68% - 57/84) Why IELTS? (p 15), (66% - 55/84)

Most of the interviewed participants (13/19) viewed the IELTS Guide as a useful introduction about the IELTS Test for a general audience but felt that it did not always meet their specific needs as university staff. For example, 58% (11/19) of the participants pointed out that the two sections “Using IELTS test scores” (p 8) and “Test scores” (p 8) lacked sufficient detail about the Test for educational providers. One interviewee said that:

to be honest, I am not sure who it’s exactly aimed at…for me [as a staff member in an educational institution] it doesn’t give enough details…government and commercial organisations may find it useful…but again for me what’s important is what does that band score look like? What can a person do [at the level of proficiency indicated by the band scores]? (#368 Other - Student Services, UNI B)

Some also felt that the Guide lacked support for some of its claims in sections such as ‘The International Test’, ‘The Quality Test’ and ‘Continual Research-Driven Improvement’. One admissions interviewee believed the section ‘Why IELTS’ could be more informative if it were more “research-driven and detailed” (#636 Admissions, UNI A).

Another language teacher interviewee suggested that:

…perhaps there could be some evidence of that [in the IELTS Guide]or links to evidence or research that has been done to show how accurate IELTS is…perhaps compared with TOEFL or the new test, Pearson, that is coming out. That might support what is written here to say…convince or persuade that it is the best test to use. (#377 Language Staff, UNI B)

How INFORMATIVE is each section of the IELTS Guide?

Figure 10: Informativeness of the IELTS Guide (N = 84)

Some interviewees also expressed a need for clearer descriptions of score profiles in the IELTS Guide. The IELTS Handbook (2007) contains descriptors for each overall band score but these are lacking in the IELTS Guide. For example, one academic interviewee said: “…to me what I would find interesting would be a page published with examples of writing performances …it’s very nice for academics to see that” (#357 Academic, UNI B). Another suggested that the band scores and labels (eg Expert User, Competent User) needed to be “linked to a website where more detailed descriptions of band scores could be accessed” (#358 Academic, UNI B).

Asked whether the IELTS Guide is an educational or marketing document, 90% (17/19) of the interviewees saw it as a mixture of both, although they differed in which they thought was the stronger orientation. Some stressed its educational orientation. For example, one admissions interviewee suggested that “it is well-presented, well written, quite accessible and easy to pick up when you need information… the graphics, the diagrams are reasonably good…it’s not text heavy” (#356 Admissions, UNI B). A marketing staff member saw the IELTS Guide as overall educational and informative, though she believed it contained “pretty basic information…[is] not text-rich or evidenced, there is not much detail…no data to support what they are saying…the impression you get is that IELTS did not have to try too hard” (#630 Marketing, UNI A). Other interviewees, however, focused more on the Guide’s marketing function. For example, one marketing interviewee said that the IELTS Guide was “clearly promotional, but in a reliable way…the stuff about global recognition; it’s valid info…not a spin…it’s reliable and informative” (#531 Marketing, UNI A). Another interviewee stated “that’s how it presents IELTS…it’s selling…it’s [in] the choice of language, [eg] ‘expert raters, accessible and convenient, the results you can trust’, it’s very ad speak” (#369 Language Staff, UNI B). In a more critical vein, another said, the Guide could be improved if the marketing material was removed: “it is informative as it is…it would be better if they take the ad[vertising] part out” (#369 Language Staff, UNI B). There is a risk that a guide which seeks to simultaneously inform IELTS users about the Test and promote it to them will fail to educate them not only about its strengths but also, the limitations inherent in all language tests.

Question 9 in the survey asked all 84 respondents “What other information do you think is missing from the IELTS Guide?” This was an open question allowing them to write comments. A third of the comments (12/36) recorded focused on the need for more information about the meaning of IELTS test scores and how to interpret them. This suggests that there is insufficient information about these important aspects on pp 8-9 of the IELTS Guide. Relevant areas of suggested improvement for the IELTS Guide included: the inclusion of recommended individual, as well as overall, band scores; examples of what different overall and individual band scores mean (such as samples of writing scores in the 5.0, 6.0 and 7.0 band score ranges); information about the security and reliability of the Test, and its possible limitations; and more evidence to support claims that IELTS is the best test. In the interviews, there were also some interesting points made about the need for better information to interpret jagged or inconsistent score profiles of test-takers. One interviewee said “I would like to see something in here about jagged profiles, because that’s a term that’s used a lot…” (#369 Language Staff, UNI B). Arguing for the inclusion of samples of test performance to assist with the interpretation of test scores, an academic interviewee involved in selection suggested that:

it would be lovely to see a person with [an overall band score of] 7.5 or 8 with a band of 5 or 5.5 in one area. Then you would be able to show samples of that to your PVC [Pro-Vice Chancellor] and say “why wouldn’t I take that student?” It would mean something, because the PVC probably knows less than me about this.”

(#357 Academic, UNI B)

This comment emphasises how important it is for the IELTS partners to provide detailed information about the meaning of overall and individual test scores and thereby encourage more valid and ethical interpretation of test scores in student selection, especially where human judgment informs the final decision.

The IELTS Guide (2009): A discourse analytic perspective

Given that interviewees indicated that the IELTS Guide was both an educational and marketing document, it was decided to examine the balance between these two orientations in the IELTS Guide from a discourse analytic perspective. The method used for the analysis was informed by a genre analytic model employed by Osman (2008) in an analysis of university brochures in Malaysia. Following Osman, these generic moves were identified within sections of the IELTS Guide:

a) Identifying the Test (providing basic introductory information about the Test) b) Describing the Test (highlighting its coverage of the relevant language skills) c) Establishing the credentials of the Test d) Justifying the Test (outlining the benefits of using the Test) e) Locating the Test (highlighting its global availability) f) Soliciting response (providing contact information) g) Targeting the market (identifying the specific users of the Test).

The seven moves above were linked to the two broad communicative functions of providing ‘neutral information about the Test (ie educating) and promoting the Test (ie marketing). The first two moves (a and b) mainly serve an educational purpose in that they aim to provide disinterested information about the Test. The other five moves (c to g) mostly serve a marketing purpose in that they aim to more explicitly (as in move c) or implicitly (as in move e) promote the Test and/or to portray a corporate image of the Test. However, the educational and marketing functions are not mutually exclusive, but instead are realised across a continuum from predominantly educational moves to predominantly marketing ones in the IELTS Guide. For instance, the text on pages 4-5 of the IELTS Guide (ie The Quality Test) has mainly an educational purpose in so far as it informs readers about the characteristics of the test materials, assessment processes and examiners. But the extensive use of adjectives and qualifiers lends stronger weight to the promotional/marketing function of the text. This is illustrated in the following example where such words and phrases are underlined.

The most effective way to assess speaking skills is through direct interaction with the test-taker.

IELTS is at the cutting edge of English language testing…

Although IELTS has benefited from decades of progressive change, we have always maintained our core commitment to assessing all four language skills - reading, writing, listening and speaking - to the highest of standards…

Examiners are recruited, trained and monitored in line with the highest quality standards…

Also notable in this the same section are the references to aspects of test quality which help to promote the IELTS Test further. These references are shown in the following example where the relevant words and phrases are again underlined

…we have always maintained our core commitment to assessing all four language skills - reading, writing, listening and speaking - to the highest of standards…

Candidate performances in the Writing and Speaking components are assessed by qualified examiners rather than computers…

They work to clearly defined criteria and are subject to extensive and detailed quality control procedures…

This analysis of the IELTS Guide revealed that of the 25 instances of moves made, only seven served a predominantly educational purpose. The other 18 moves mainly served a marketing purpose. Moreover, the marketing functions within these 18 moves are better highlighted than the other seven moves through specific textual and design features in the IELTS Guide. For instance, a move aimed at ‘soliciting response’ is marked in the text by bold fonts or inserted in a text box. These features add further weight to the promotional function of the IELTS Guide. The detailed analysis of the IELTS Guide is included as Appendix 6.

The stronger marketing orientation of the IELTS Guide potentially limits its usefulness as a training document. A stronger educational focus would help to ensure that it provides the accurate and disinterested information that the IELTS Test users need to have about the IELTS Test, including guidelines for the valid and ethical interpretation and use of test scores.

Evaluation of IELTS resources

The focus in this section was on how informative the current IELTS resources are for university staff. It also evaluates these resources in terms of user-friendliness.

Following the analysis of survey data and the interviews, an evaluation was made of how informative (ie relevant and detailed) each of the resources were about 16 different topics (see Table 3 below). Topics 1-15 are the same as those used in Question 5 of the survey but here they are ordered from ‘most used’ to ‘least used’ as indicated by the responses to this question (see Figures 5 and 6 above). An additional topic, ‘Guidelines on standard-setting’, was added to the list for this evaluation. The five IELTS resources evaluated included:

• the sections of the two universities’ websites (accessed on 17 February 2011) which provide information about language proficiency entry requirements
• the section for institutions on the IELTS official website (http://www.ielts.org/institutions.aspx, accessed on 16 February 2011)
• the IELTS Guide for Stakeholders (2009)
• the IELTS Scores Explained DVD (2009).

These resources were rated in terms of the quality of information they provided about each of the 16 topics using a scale from 1-5 with 1 = not informative, 2= slightly informative, 3= informative, 4= very informative and 5 = highly informative. The researcher and the research assistant independently carried out these ratings. The results of this initial assessment were then reviewed in two follow-up discussion sessions and the ratings were finalised. An average score for each of the five resources was also calculated. Table 3 shows the results.

Overall, the most informative resource was the IELTS website (section for institutions) with an average of 4.0 (ie very informative), and individual scores ranging from 3-5 across all topics except for number 12 (How candidates can prepare for the IELTS Test). All of the information about this topic was located in the section for test-takers on the IELTS website. It would be useful to have a link to this information in the section for institutions for staff who need to advise prospective students about preparing for the IELTS Test. The least informative resources overall were the University A and University B websites with an average of only 1.7. However, they were obviously not exclusively designed to provide information about the IELTS Test. The two resources - IELTS Guide for Stakeholders and the IELTS Scores Explained DVD - had average ratings of 3.0 and 3.2 respectively.

The IELTS website was the most informative for topics 1-4, which were also the most important ones for more than 50% of respondents in Question 5 (see Figures 5 and 6 above). In relation to topic 1 (The minimum IELTS test scores for entry into specific courses), the IELTS Guide and the DVD scored the lowest (2 and 1 respectively). The IELTS Guide provides only general guidance on what minimum scores can be set for some broad study areas. The DVD provides information on standard setting procedures, though without any recommendations on minimum scores for specific courses. For topic 4 (The relationship between IELTS test scores and other English evidence), the University A and University B websites both scored 5 while the IELTS website scored 3. Although the IELTS website contains information about how test results are linked to the Common European Framework Reference (CEFR), there is no information about how scores compare with other proficiency test scores, pathways or English study. The IELTS Guide and the DVD both scored only 1 as they contain no information about the relationships of IELTS Test results to other evidence of English proficiency.

Table 3: Informativeness of the IELTS Test resources

A further dimension of the resource evaluation was to assess how user-friendly the resources were. This criterion included ease of access and browsing, and features such as design, layout and indexing, which facilitate locating information in a resource. Table 4 below shows the results of this evaluation.

Table 4: User-friendliness of IELTS Test resources

The University A and University B websites were rated as highly user-friendly (with a score of 5), the IELTS Guide as very user-friendly (with a score of 4), the IELTS official website as user-friendly (with a score of 3) and the IELTS Scores Explained DVD as slightly user-friendly (with a score of 2). What makes the University A and University B websites highly user-friendly is that the information about English language minimum entry requirements is directly accessible on a single webpage. The users are directed via links on the homepages of Universities A and B to the webpages containing information on entry requirements.

The IELTS official website is a generally user-friendly resource as it is clearly divided into four sections (Test-takers, Institutions, Researchers and Teachers) on its homepage. The sections under the Institutions tab are further categorised and include a FAQ section which makes finding information easier. However, it also has a few notable weaknesses. Despite its pleasing and facilitating design features (such as use of colours, separate tabs for different sections) browsing back and forth between pages and finding documents can be difficult, at times, since information in the four different sections of the site sometimes overlap. A search tool within the website could make locating specific information easier. Also, the dates of updates are not clearly shown although there is a date mentioned in the copyright section of the website which generally applies to all the information.

The IELTS Scores Explained DVD is the least user-friendly resource mainly due to variations experienced in the functionality of its auto-run interface when the DVD is played on different machines. While the auto-run interface functions well on some computers and is compatible to different operating softwares and makes it overall easy to find documents and information, on some machines it becomes dysfunctional, making the manual search of documents an arduous task.

For a more detailed analysis of the IELTS Test resources refer Appendix 7.

Comparative evaluation of IELTS, TOEFL and PTE Academic websites

Following the evaluation of IELTS resources, a further comparative evaluation was undertaken of the institutional sections on the IELTS, TOEFL and the PTE Academic websites in terms of how informative and user-friendly they each were about the test.

For this evaluation, the institutional sections of each website were closely examined:

• IELTS http://www.ielts.org/institutions.aspx accessed on 16 February 2011
• TOEFL http://www.ets.org/toefl/institutions accessed on 23 February 2011
• PTE Academic http://www.pearsonpte.com/PTEAcademic/Institutions/Pages/home.aspx accessed on 24 February 2011.

They were each evaluated in terms of the quality of information provided on 16 topics which, with the exception of the last one (Guidelines on standard setting), were adapted from Question 5 of the survey and appear in the same order as the topics in that question.

The institutional sections of the three websites were rated (using the same scale as in evaluation of the IELTS resources in the previous section) from 1-5 with 1 = not informative, 2= slightly informative, 3= informative, 4 = very informative and 5 = highly informative. The researcher and the research assistant made separate initial evaluations of the websites. The results of this initial evaluation were then reviewed in two follow-up discussion sessions and the scores assigned were mediated. An average score was calculated for each website. Table 5 shows the results of this evaluation.

Overall, the section for institutions on the TOEFL website scored the highest with an average of 4.6. The IELTS and PTE websites each score an average of 4.1. Despite scoring 4 or 5 for 11 of the 16 topics, the section for institutions on the IELTS website scored 3 or lower for five of the topics and was less informative than the TOEFL website for four of the topics. A closer look at the scores in Table 5 shows that the IELTS website scored the lowest (3/5) among the three websites for topic 14 (Relationship between test scores and other evidence). While the TOEFL website provides information about how TOEFL test results compare with the IELTS Test and the CEFR (Common European Framework Reference) and the PTE website provides a comparison tool for PTE, IELTS, TOEFL and CEFR, the IELTS website only links test results to the CEFR.

Another topic on which the IELTS website scored lower than TOEFL and PTE was number 10 (Meaning of test scores). The TOEFL and PTE Academic websites provide detailed descriptors for both the global scores and individual scores on each of the four skills of Reading, Writing, Listening and Speaking. However, the IELTS website provides brief, general descriptions for the Overall Band Scores, and more detailed descriptors are only available for Speaking and Writing.

For topics 12 and 16 (Validity and reliability of test scores and Guidelines on standard setting), the IELTS website along with the PTE Academic scored 3 which is lower than TOEFL. Unlike the TOEFL website, the IELTS and PTE Academic websites provide no information on standard error of measurement. This is an extremely important piece of information for the valid interpretation of all proficiency test scores. The information provided on the IELTS website about standard setting is mainly general. For more detailed information, test users are directed to an order form for the purchase of the IELTS Scores Explained DVD. However, the TOEFL website provides Helpful Tips and Score Comparison Tables, in addition to directing users to a Standard-setting CD_ROM for purchase.

The IELTS and PTE Academic websites only scored 1 for topic 3 (Test preparation) compared to a score of 4 for the TOEFL. While the TOEFL website provides a list of resources for preparation which are available for purchase in a sub-section entitled TOEFL Test Preparation, the IELTS and PTE Academic websites’ institutional sections provide no information in this regard (although there is information about preparation in the test-takers section of the IELTS website as previously noted). For more detailed comments on the evaluation of the above websites, see Appendix 8.

Table 5: Informativeness of the institutional sections of the IELTS, TOEFL and PTE Academic websites

The institutional sections of the IELTS, TOEFL and PTE websites were also evaluated in terms of their user-friendliness. As in the evaluation of the IELTS resources, this criterion included ease of browsing, and features such as design, highlighting information and links, and internal search tools which facilitate locating information on a website. Table 6 shows the results of the evaluation.

Table 6: User-friendliness of the institutional sections of the IELTS, TOEFL and PTE Academic websites

Of the three websites, the IELTS website scores the lowest despite its generally pleasing design mainly due to a lack of a search tool within the website. This makes navigating the site and searching for downloadable material cumbersome, even confusing at times.

In summary, the results indicated that the needs of the IELTS Test users in this study are currently reasonably well met. Most rely on their institutions’ English language entry regulations and, to a lesser extent, the IELTS official website, for information about the IELTS Test. They considered that the most useful information provided to them relates to the minimum IELTS entry scores for entry to specific courses, followed by the relationship between IELTS test scores and other evidence, and the international and local recognition of the IELTS Test.

Not many IELTS Test users had mainly relied upon the IELTS Guide prior to the study. All of the survey respondents (IELTS Test users and non-users alike) generally found the IELTS Guide to be informative or highly informative. A number of respondents believed that more information was needed in the IELTS Guide about the meaning and interpretation of IELTS test scores. The majority of interviewees considered that the IELTS Guide had both an educational and marketing orientation, although they differed about which was the stronger orientation. A discourse analytic study of the IELTS Guide suggested that overall it had more of a marketing - than educational - purpose which is potentially problematic in terms of providing test users with accurate and ‘objective’ information about the Test. A comparative evaluation of the websites of University A and University B, the IELTS official website, the IELTS Guide and the IELTS Scores Explained DVD in relation to a range of key topics indicated that the IELTS website was the most informative in terms of relevance and detail but that the two university websites were more user-friendly than the three IELTS resources. Finally, a comparative evaluation of the institutional sections of the IELTS, TOEFL and PTE Academic websites suggested that the IELTS website was slightly less informative overall than the TOEFL website. The IELTS website was also less user-friendly than the other two websites mainly because of its lack of an internal search tool.

5.2.3 Research Question 3: What other approach(es) could be adopted to meet these needs?

The question was addressed through Question 10 of the survey, related questions in the interviews and a review of best practice in staff training programs.

Survey and interview data

Question 10 in the survey asked all 84 respondents “What other ways of learning about the IELTS Test would be useful? Please tick all responses that apply.” Figure 11 below summarises the results for Universities A and B combined.

What other ways of learning about the IELTS test would be useful?

Figure 11: Other useful ways of learning about the IELTS Test (N= 84)

Figure 12 below provides a breakdown of the responses to Question 10 for Universities A and B.

Figures 11 and 12 show that the most popular alternatives to the IELTS Guide are (in order):

a) online tutorials b) information sessions c) workshops.

It is unclear why there was a much stronger preference for workshops at University B. In the interviews, the majority of staff (63%, ie 12/19) preferred these ways of learning about the IELTS Test to the IELTS Guide. The IELTS Guide was considered a static document and it is not clear if, and how frequently, it is updated. The main reason given by the interviewees who most preferred online tutorials was the ease of access and control in terms of how they could use them. One interviewee said:

…I hate going to meetings and information sessions or workshops. And multimedia products are usually too glitzy; don’t run [on all machines], get outdated and are expensive to keep upto-date. I can do it [an online tutorial] in my own time. (#357 Academic, UNI B)

Another reinforced this point suggesting that: “I can take my time going through online tutorials, I can do it when it’s convenient for me…” (#652 Academic, UNI A). In terms of time management, one interviewee suggested that:

…people want to stop and start to do it in their own time: people are time-poor: you want people to get engaged with it when they want to engage with it, when their energy is there, when their interest is there…if there are imposed workshops or information sessions it often detracts from the impact (#369 Language Staff, UNI B).

What other ways of learning about the IELTS test would be useful?

Figure 12: Other useful ways of learning about the IELTS Test at University A (N=41) and University B (N= 43)

In relation to the possible content of online tutorials, another interviewee advocated:

… the whole thing: videos, the whole package, things that you can print out: for example you might have a quiz at the end or you might have the opportunity for people to ask a question and it’s answered somehow. (#369 Language Staff, UNI B)

A further advantage of online tutorials was seen to be the potential links to other resources about the IELTS Test:

…if you are on a website and there is an online tutorial, it can refer you to other areas: to academic articles, to research, to interviews, it can be a very rich environment.

(#369 Language Staff, UNI B)

However, a small minority of interviewees were not in favour of online tutorials. One of them said that, “you have a lot of online things: and you wouldn’t do them…unless they are mandatory, people wouldn’t do them” (#630 Marketing, UNI A). The interviewees who preferred information sessions and workshops highlighted their interactive nature as the chief advantage. For example, one interviewee said that they provided “the opportunity of face-to-face questions and answers, especially for teachers who do not have a background with IELTS”(#577 Language Staff, UNI A). Another suggested she preferred information sessions because she “could ask questions and they could be interactive” (# 531 Marketing, UNI A). Those interviewees who favoured workshops saw them as an opportunity for staff to deepen their understanding of IELTS test scores. One interviewee suggested that:

it depends on whether we want to give more information to people about, for example what a 4 band score looks like…it [a workshop] might be suitable to give samples to people…people like heads of schools, I think they should know what it means…

(#368 Other - Student Services, UNI B)

Those who preferred online tutorials considered that information sessions and tutorials were not very time efficient. For example, one said that “… workshops may take half a day and you could have told me the information in half an hour” (#652 Academic, UNI A).

An alternative to the options listed in Question 10 mentioned by two of the interviewees was the opportunity for staff to sit a free IELTS Test. One of them believed that:

[sitting the test] would help an assessor to have more empathy towards the student…it would come through and make the students feel confident you know what you’re talking about.

(#356 Admissions, UNI B)

The other suggested that this strategy would help to dispel some misconceptions about the IELTS Test on the part of some test users.

I am not sure what admission and other staff members know about IELTS…I think it [sitting the test] would let them learn more directly about some of those basic things, like there is no Pass and Fail in IELTS…there is a lot of myths and misconceptions like that…IELTS in a way has become a magic word. (#577 Language Staff, UNI A)

Finally, five interviewees who worked in marketing and admissions stressed that academic staff, in particular, needed to know more about the IELTS Test. For instance, one marketing staff member argued that:

I don’t think it’s so much us needing information. I think it’s more the academics needing information, because they’re the ones who are teaching students, they are the ones who know that the faculty has set a [particular entry] level…but what does that mean when it comes to teaching students, where are all the holes? (#530 Marketing, UNI A)

This comment also raises the question of whether particular sub-groups of test users in Australian universities need different kinds of information about the IELTS Test in Australian universities. It may be desirable for the IELTS partners to consider developing separate resources for admissions, marketing and academic staff.

Online training programs

The IELTS partners’ current approaches to developing assessment literacy include sponsoring of national and international conferences, brochures, stakeholder information sessions and roundtable discussions, e-newsletters, IELTS Scores Explained DVD, the IELTS website which includes analyses of candidate performance and downloadable documents such as the IELTS Guide for Stakeholders (Louise Dunn, IELTS Australia, personal communication). The one form of education not currently used is online tutorials.

The findings in this study (see Figures 11 and 12 above) indicated that online tutorials were the most preferred option for learning about the IELTS Test. Online tutorials and online programs are being used widely by businesses and organisations across the globe mainly because of ease of access and delivery, and their cost efficiency. As McGee (1999, p 101) suggests:

Web-based training is an increasingly popular option for businesses. Depending on the solution, web-based courses, accessed over the Internet or intranets using browsers, can be taken anywhere, at any convenient time. Within technology-based training, web-based training is the fastest-growing segment. One reason companies implement web-based training is because they have the infrastructure to support it already in place.

Online training programs for adult trainee staff need to have certain features to enhance and guarantee their effectiveness. The design of online training programs is a complicated and intricate task when such programs are to be used by employees who may have little time or motivation to engage with them in busy workplaces. Cercone (2008) considers the design of online training courses for adult learners and suggests that such courses should be designed in such a way that they:

• connect new knowledge to prior learning
• maintain collaboration and social interaction
• promote a self-reflective environment
• include current or immediate applications
• advance self-regulated learning.

Example of best practice: Workplace Discrimination and Harassment - Legal Compliance course

Among the currently used online staff training programs, the course ‘Workplace Discrimination and Harassment - Legal Compliance’ is an example of best practice used in a university context. Only staff at the University of Melbourne have access to this program so it is not available to readers of this report. Nevertheless, its characteristics and features can be identified and are described below.

The course is example of an online training course designed specifically for all university staff. It is an 80-minute program designed to educate staff about issues to do with anti-discrimination legislation, legal rights and responsibilities, to familiarise them with inappropriate behaviour in the workplace, and provide them with information and guidelines on how to prevent unacceptable behaviour. The course is made up of five modules. Each module focuses on a specific aspect of discrimination and harassment. The modules are: Legislation and Responsibilities, Unlawful Discrimination, Work-life Balance, Harassment, and Resolving Issues. A Statement of Achievement certificate is awarded upon successful completion of a final test.

Upon logging into the course a Welcome Note describes tips on how to proceed, how to browse through the pages, how to exit and how to resume after logout. An overview of the course is given at the start outlining the learning objectives and duration. Various tools are incorporated into the program, including a notepad for making notes or typing answers to self-assessments, a virtual coach who provides explanations or refreshers when requested by the trainee, self-assessment components, and a suggestion box. The program provides session time out, instructions for ongoing access to the program and a final test. Before starting the five modules, a detailed overview of each module becomes available by clicking on an icon. Each module has the following components:

• a concrete real-life example of one or more cases related to the main topic of the module
• the overall objectives of the module
• a general description of topic of the module (and, where necessary, links to extra information both within the course and externally through links to other websites)
• the option of access to more specific and localised material pertaining either to the immediate issue discussed (eg relevant legislation) or the trainee’s individual context (eg Australian State and Territory legislation)
• examples of real-life situations and precedents
• a self-assessment component.

The pattern of progression within each module gradually narrows down from a general focus to workplace-specific explanations and then to specific real-life cases. On pages where issues are defined or explained, a photo of a virtual coach (who is selected out of four coaches by the trainee before starting the modules) accompanies the explanations. The photo functions as a link to more detailed or specific information with embedded internal and external links to make access to further information possible. There are also optional links to specific real cases. On some pages, after a topic has been explained, the trainee is asked a question eliciting their idea of what the issue discussed might more specifically relate to. The trainee can write the answers in a notepad box and further explanations or examples related to the question become available when the answers are submitted.

Each module concludes with a self-assessment component. After this has been completed, the correct answers can be accessed. For each question the page number of relevant information is provided. The trainee is given a score with a brief description on performance (eg ‘Good. You’ve understood this module and are now ready to proceed’).

The course concludes with a formal test. Upon successful completion, a short survey asks the trainees to evaluate the course, and a certificate is issued which can be printed out. The trainees are also informed that their access to the course will be maintained for later reference.

The course is a successful example of an online training program. Its virtues include clarity, engaging design, balance of formal and friendly tone, control for trainees over their learning and time, and concrete evidence of achievement. The course is clear and accessible in that:

• it outlines main goals
• it clarifies learning objectives
• it establishes the benefits of doing the course for the trainee
• the explanations are not too detailed or lengthy, and where more information might be required, links to further information are provided
• the learning objectives are specific and aligned with the overall training goals.

The course is well designed so that browsing through the pages is straightforward and links are clearly highlighted. The visual presentations are both clear and engaging: for instance, different aspects of work-life balance are represented in a photo with a female employee juggling glass and rubber balls.

There is also a mix of different learning methods which is both engaging and suited to different learning styles. For example, trainees are strongly involved in learning by giving them the option to access localised information, asking them to give their judgment about cases and requiring them to self-assess their knowledge. Short tests, both between and at the end of explanations, help trainees recall what they have learned and to draw on their previous knowledge. Also, trainees have control over their access to extra information to the extent that they can sometimes choose whether to read it.

Where real-life examples are given, the trainee is given the option to know about the views of both or all parties involved. This not only engages the trainees, it also gives them the chance to better understand the complexity of the issues related to discrimination and harassment.

The tone is both formal and friendly. The course personalises the learning process by giving the trainee the option of selecting a coach whose photo appears on many pages across the modules, as well as by making it possible for the trainee to be able to contact the course coordinator. There is also the use of friendly expressions in the welcome note, ie ‘Last but not least: enjoy the course!’ Issuing a certificate gives the trainee a sense of attaining a specified and tangible outcome.

In summary, the most popular choice by participants in this study for learning about the IELTS Test was online tutorials, an approach which has not been used to date by the IELTS partners. Interviewees who shared this preference indicated that online tutorials provided them with the greatest ease of access and control in terms of use. They also considered that the content and format of online tutorials could be potentially more engaging and that better links could be made to other test resources. Other advantages of online tutorials were found in the training literature. Finally, an example of best practice in online training programs was described in detail. It provides an excellent illustration of how such a program can be used to very effectively educate a wide range of university staff in a highly innovative and engaging format.

6 DISCUSSION

To recap, the survey and interview findings indicated that information about the IELTS Test was mostly needed for advising prospective students about English language entry requirements and making admissions decisions. To these ends, test users were mainly focused on just four topics about the IELTS Test: the minimum IELTS scores for entry to courses at their university; the different components of the IELTS Test; how long IELTS test scores are valid; and the relationship between IELTS test scores and other evidence of English proficiency accepted by their university.

The results also indicated that the needs of the IELTS Test users were reasonably well met. Most mainly accessed their institution’s English language entry regulations and, to a lesser extent, the IELTS official website, for information about the IELTS Test. After reading the IELTS Guide (2009), all survey respondents generally found it to be informative. However, some believed that it could have included more information about the meaning and interpretation of IELTS test scores. A discourse analytic study suggested that the IELTS Guide had more of a marketing - than educational - emphasis which potentially compromises its accuracy, comprehensiveness and ‘objectivity’. The comparative evaluations of different IELTS resources and the institutional sections of the three different test websites suggested the IELTS website was an informative resource, although some of its content and user-friendliness could be improved. Finally, the most popular alternative choice from the survey and interviews for learning about the IELTS Test was online tutorials, an approach which has not yet been used by the IELTS partners.

From one perspective, the implications of the main findings in this study are relatively straightforward. The IELTS partners might produce mostly online educational materials, including a tutorial, that build the assessment literacy of test users on a limited number of topics, so that the test users can fulfil their main roles in advising prospective students and making admission decisions within the system of international selection that currently exists in Australian universities. Other resources should be produced for marketing purposes only.

Yet, from a different perspective, there are questions to be addressed about current admission policies and practices in Australian universities, which, in turn, raise issues for what assessment literacy is now and what it might become in the future. Currently, university applicants are assessed in a simple, lockstep manner. They must first meet the academic and, secondly, the English entry requirements of a particular course at the university of their choice. As far as English proficiency tests are concerned, they must have obtained the minimum acceptable scores and there is usually not flexibility around this decision. This process explains the relatively limited assessment literacy needs of the participants in this study. However, given the limitations and “inevitable uncertainty” (Spolsky, 1995, p 358) of all proficiency test scores in terms of their validity and reliability, they should be carefully interpreted in relation to other relevant information about applicants such as their age, first language and language learning history to ensure the appropriate interpretation and use of test scores. Such a shift in approach by Australian universities requires a major change in selection policy and procedures. The IELTS partners are clearly aware of this important issue. The IELTS Handbook (2007, p 5), in particular, suggests that:

… receiving institutions should also consider a candidate’s IELTS results in the context of a number of factors including age and motivation, educational and cultural background, first language and language learning history.

This crucial recommendation correctly implies that IELTS minimum entry scores should not necessarily be rigidly applied in the decision-making process but, instead, should be interpreted in relation to these kinds of individual factors. However, this guideline does not appear in the IELTS Guide, a much simplified and more promotional version of the IELTS Handbook, nor is it elaborated in any of the other IELTS Test resources. The IELTS partners need to provide more detailed advice to institutions to encourage them to adopt this approach. This is a crucial area for further development of test resources.

Taylor (2009, p 30) argues that “test wiseness” (the assessment skills, knowledge and principles which various stakeholders need to acquire to ensure the valid and ethical use of a test) lies at the heart of assessment literacy. If admissions staff were required to make informed, holistic judgments about the language proficiency of international applicants, then they (and the academic, marketing and English language staff who would need to understand the nature of these judgments) would have to be much better informed about the IELTS Test and, in particular, the appropriate interpretation and use of test scores. Such decision-making would be undoubtedly more complex but ultimately more valid and ethical.

7 Recommendations

In light of this study it is recommended that the IELTS partners:

• develop the assessment literacy of IELTS Test users and other university staff so that the meaning and limitations of proficiency test scores are better understood, thereby enhancing their appropriate interpretation and use in situ
• guide admissions officers to make informed, holistic decisions about the English language proficiency of international applicants, considering IELTS test scores in the context of relevant individual factors, such as their age, motivation, educational and cultural background, first language and language learning history
• provide advice to a) marketing and administrative staff who advise prospective applicants about their institution’s minimum entry scores and b) academic and support staff who teach students about this more complex decision-making process
• create online tutorials to supplement the currently available information sessions and workshops in order to build higher levels of assessment literacy among relevant university staff
• consider the possibility of developing tailored informational resources to meet the different needs of admissions, marketing and academic staff.

The development of enhanced assessment literacy along these lines has the potential to greatly strengthen the interpretation and use of IELTS test scores in the Australian university context.

Acknowledgements

I am grateful to IELTS Australia for funding this study and the university staff who participated in the study. I also gratefully acknowledge Reza Tasviri for his valuable assistance with the data collection, data analysis and preparation of this report, and Basil Alzougool for his work on the design of the survey and analysis of the results.

References

• Banerjee, JV, 2003, Interpreting and using proficiency test scores, unpublished PhD thesis, Department of Linguistics and Modern English Language, Lancaster University
• Cercone, K, 2008, ‘Characteristics of adult learners with implications for online learning design’ in AACE Journal, 16, 2, pp 137-159
• Deakin, G, 1997, ‘IELTS in context: Issues in EAP for overseas students’ in EA Journal 15, 2, 7-1
• Hagan A, Starfield S and Coleman D, 2003, ‘The attitudes of IELTS stakeholders: Student and staff perceptions of IELTS in Australian, UK and Chinese universities’, IELTS Research Reports Volume 5, IELTS Australia, Canberra, pp 160-207
• Hyatt, D and Brooks G, 2009, ‘Investigating stakeholders’ perceptions of IELTS as an entry requirement for higher education in the UK’, IELTS Research Reports, Volume10, ed J Osborne, IELTS Australia, Canberra and British Council, London, pp 17-68
• IELTS, 2009, IELTS guide for educational institutions, governments, professional bodies and commercial organisations, accessed 15 May 2010 from http://www.ielts.org/default.aspx
• IELTS, 2009, IELTS Scores Explained CD/DVD ROM, IELTS partners, University of Cambridge ESOL Examinations, Cambridge
• IELTS, 2011, IELTS official website: Section for institutions, accessed 16 February 2011 from http://www.ielts.org/institutions.aspx
• McDowell, C and MerryLees B, 1998, ‘Survey of receiving institutions’ use and attitude to IELTS’, IELTS Research Reports, Volume 1, ed S Wood, ELICOS/IELTS Australia Pty Ltd, Canberra, pp 116-139
• McGee, MK, 1999, ‘Train on the web’ in Information week, 718, pp 101-105
• Messick, S, 1996, ‘Validity and washback in language testing’ in Language Testing 13, pp 241-56
• O’Loughlin, K, 2006, ‘Learning about second language assessment: insights from a postgraduate student online forum’, University of Sydney Papers in TESOL 1, 1 pp 71-85
• O’Loughlin, K, 2008, ‘The use of IELTS for university selection in Australia: A case study’, IELTS Research Reports Volume 8, ed J Osborne, IELTS Australia, Canberra, pp 145-242
• Osman, H, 2008, ‘Re-branding academic institutions with corporate advertising: a genre perspective’, Discourse & Communication, 2, 57 pp 57-77
• Owen, J, 2006, Program evaluation: Forms and approaches, Allen and Unwin, Sydney
• Pearson, 2011, Pearson Test of English official website: Section for institutions, accessed 24 February 2011 from http://www.pearsonpte.com/PTEAcademic/Institutions/Pages/home.aspx
• Rea-Dickins, PR Kiely, R and Yu, G, 2007, ‘Student identity, learning and progression: The affective and academic impact of IELTS on “successful” candidates’ in IELTS Research Reports Volume 7, IELTS Australia, Canberra and British Council, London, pp 59-136
• Taylor, L, 2009, ‘Developing assessment literacy’ in Annual Review of Applied Linguistics 29, pp 21-26
• Shohamy, E, 2001, The power of tests, Longman, London
• Spolsky, B, 1995, Measured words: The development of objective language testing, Oxford University Press, Oxford
• TOEFL, 2011, TOEFL official website: Section for institutions, accessed 23 February 2011 from http://www.ets.org/toefl/institutions

Appendix 1: Participant recruitment email

Dear [participant],

I am writing on behalf of Dr. Kieran O’Loughlin from the Melbourne Graduate School of Education to invite you to participate in a research project funded by IELTS Australia entitled “Developing the assessment literacy of IELTS Test users”.

The project aims to explore the information needs of university staff in relation to the IELTS Test and how university staff (both professional and academic) might improve their understanding of the IELTS Test so that they can a) provide accurate advice to prospective students about university English language entry requirements and how to meet them b) deal confidently with the university’s English requirements in the admissions process.

Participation involves completion of a short online survey (for less than 30 minutes) and possibly, a follow-up interview (for proximately 30 minutes).

Will you please advise me if you are willing to participate?

Thank you in advance.

Warm regards

Reza Tasviri

Research Assistant & PhD Candidate

Melbourne Graduate School of Education

The University of Melbourne

Email: tasvirir@unimelb.edu.au

Appendix 2: Plain language statement and consent information

PLAIN LANGUAGE STATEMENT

Researcher:

Dr Kieran O’Loughlin

Melbourne Graduate School of Education

Ph: 83448345

Fax: 83448612

Email: kjo@unimelb.edu.au

Project Title:

DEVELOPING THE ASSESSMENT LITERACY OF IELTS TEST USERS IN HIGHER EDUCATION

Introduction

Your name and contact details have been provided by senior managers at your university as a potential participant in this study. The researcher would like to invite you as someone who has experience with and/or uses the IELTS test scores to take part in our research project. The aim of the study is to investigate the information needs of university staff in relation to the IELTS Test and how these needs might best be met.

What will I be asked to do?

If you agree to participate, you will be asked to contribute in two ways. First, the researcher will ask you to complete an online questionnaire about your information needs before and after reading the IELTS Guide. The IELTS Guide is an information booklet about the IELTS Test and will be provided to you through a link provided in the online questionnaire. It will take you no longer than 40 minutes to complete the online questionnaire. Second, at a later date, you may also be asked to take part in an audio-recorded interview. At the end of questionnaire you will be asked if you are willing to be interviewed at a later date to discuss your responses to the questionnaire in more detail. With your permission, the interview will be tape-recorded so that the researcher will have an accurate record of what you say. The interview will take no longer than 30 minutes.

What are the benefits of me participating in the study?

Participation in this research will directly benefit you by enhancing your understanding of the IELTS Test and the scores it generates, as well as your ability to use and provide information or advice about the IELTS test scores in your work. It will also contribute to the training of future test users. The findings of this research will therefore assist IELTS Australia in its efforts to educate test users about the IELTS Test.

How will my confidentiality be protected?

The researcher intends to protect your anonymity and the confidentiality of your responses to the fullest possible extent, within the limits of the law. Your name and contact details will be kept in a password-protected computer file separate from any data that you supply. In the final report, you will be referred to by a pseudonym. Any references to personal information that might allow someone to guess your identity will be removed; however, you should note that as the number of people to be interviewed is very small, it is possible that someone may still be able to identify you. The data will be kept securely in the Melbourne Graduate School of Education for five years from the date of any publication, before being destroyed.

How will I receive feedback?

Once the final report has been accepted by IELTS Australia, a brief summary of the findings will be sent to participants who wish to receive it. It is also possible that the findings will also be presented at academic conferences.

Will participation prejudice me in any way?

Please be advised that your participation in this study is completely voluntary. Should you wish to withdraw at any stage, or to withdraw any unprocessed data you have supplied, you are free to do so without prejudice. Your decision to participate or not, or to withdraw and the answers you provide to the questions in the online questionnaire and the interview, will be completely independent of your professional role and duties. The researcher would like to assure you that your responses to the questions in the online questionnaire and the interview will be used for research purposes only and that you will not be identified in any reporting of the results.

Where can I get further information?

Should you require any further information, or have any concerns, please do not hesitate to contact the researcher using the contact details given above. Should you have any concerns about the conduct of the project, you are welcome to contact the Executive Officer, Human Research Ethics, The University of Melbourne, on ph: 8344 2073, or fax: 9347 6739.

How do I agree to participate?

If you would like to complete the online questionnaire, please indicate that you have read and understood this information by answering Question 1 in the online questionnaire to which you will be directed. The final two questions in the online questionnaire will ask you if you are willing to participate in a follow-up interview and interested to receive a copy of the research findings.

Please proceed by reading the consent information and follow the link provided to access the online questionnaire:

Consent information for persons participating in a research project

By answering Question 1 in the online questionnaire:

1. You consent to participate in this project, the details of which have been explained to you, and you have been provided with a plain language statement.

2. You understand that your identity will only be known to the researcher and the research assistant.

3. You understand that your participation will involve completing an online questionnaire and possibly participating in an interview.

4. You indicate your consent to participate in this project by clicking on the “Consent” button in Question 1 in the online questionnaire.

5. You indicate your willingness to participate in an interview and consent to the interview being audio-taped by answering “Yes” to Question 11 in the online questionnaire.

6. You agree that the researcher may use the results as described in the plain language statement.

7. You acknowledge that:

• the possible effects of completing the online questionnaire and participating in the interview have been explained to your satisfaction
• you have been informed that you are free to withdraw from the project at any time without explanation or prejudice and to withdraw any unprocessed data you have provided
• the project is for the purpose of research
• you have been informed that the confidentiality of the information you provide will be safeguarded subject to any legal requirements
• you have been informed that with your consent the interview will be audio-taped
• you have been informed that your completed online questionnaire and audio-taped interview will be stored at the Melbourne Graduate School of Education and then destroyed after five years
• your name will be referred to by a pseudonym in any publications arising from the research
• you have been informed that a copy of the research findings will be forwarded to you. Should you be willing to receive it, please indicate your interest by answering YES to Question 12 in the online questionnaire.

Please proceed with clicking on the link below to access the online questionnaire:

Click here to access the survey

Appendix 3: Survey

Appendix 3: Survey questionnaire

Appendix 4: Interview template

Semi-structured Interview Questions

General Probes [repeated for each question in the questionnaire]:

Can you explain/elaborate on your answer?

Why did you give that answer?

Additional Question-Specific Probes

Question 2.

Can you tell me about your role?

Question 4.

How specifically do you use information about the IELTS Test in your job?

Question 5.

Would any of the other types of specific information that you did not select be helpful in your job in any way? How would they help?

Question 6.

How do you use your specified source(s) of information in your job?

Do you know about the other source(s) of information listed?

Are you aware of any other source(s) of information about IELTS not listed here?

How do you think your specified source(s) of information could be improved?

Question 7.

Do you want to say anything more about the options you have ticked Not Useful (/Slightly Useful)?

Question 8.

[About the options ticked Not informative/Slightly informative]

How do you think those sections of the IELTS Guide could be improved?

Overall, do you see the IELTS Guide as an educational or more of a marketing document?

How do you think the IELTS Guide could be improved in general?

Question 9.

(if Q. Answered)Why do you think this information could be important to your job?

Question 10.

Would you talk about why you have chosen these options?

Why/How would the option you have ticked be helpful?

Would those formats be more helpful than the IELTS Guide?

Ending Probe

Is there anything else you would like to say about how your understanding of IELTS could be improved?

Appendix 5: Interview summaries

Interview 1: Marketing Staff #345

Interview 2: Marketing Staff #346

Interview 3: Admissions Staff #356

Interview 4: Academic Staff # 357

Interview 5: Academic Staff # 358

Interview 6: Other Staff (Student Services) #368

Interview 7: Language staff # 369

Interview 8: Language Staff #377

Interview 9: Marketing Staff #449

Interview 10: Marketing Staff # 530

Interview 11: Marketing Staff #531

Interview 12: Other Staff # 564

Interview 13: Language staff #570

Interview 14: Language Staff #577

Interview 15: Marketing Staff # 630

Interview 16: Admissions Staff # 636

Interview 17: Admissions Staff # 644

Interview 18: Academic Staff #652

Interview 19: Academic Staff #662

Appendix 5 page facsimiles

Appendix 5: Interview summaries (PDF pages 49-67)

Appendix 6: Results for discourse analysis of the IELTS Guide (2009)

Appendix 6: Results for discourse analysis of the IELTS Guide (2009) (PDF pages 68-72)

Appendix 7: Results for the evaluation of the IELTS Test resources

Rating scale: 5 = highly informative; 4 = very informative; 3 = informative; 2 = slightly informative; 1 = not informative.

Rating scale: 5 = highly user friendly; 4 = very user friendly; 3 = user friendly; 2 = slightly user friendly; 1 = not user friendly.

Appendix 7 page facsimiles

Appendix 7: Results for the evaluation of the IELTS Test resources (PDF pages 73-77)

Appendix 8: Results for evaluation of IELTS, TOEFL and PTE Academic websites

Rating scale: 5 = highly informative; 4 = very informative; 3 = informative; 2 = slightly informative; 1 = not informative.

Rating scale: 5 = highly user friendly; 4 = very user friendly; 3 = user friendly; 2 = slightly user friendly; 1 = not user friendly.

Appendix 8 page facsimiles

Appendix 8: Results for evaluation of IELTS, TOEFL and PTE Academic websites (PDF pages 78-80)

Author biodata

Kieran O’Loughlin

Kieran O’Loughlin is Senior Lecturer and Coordinator of the Master of TESOL program in the Melbourne Graduate School of Education at the University of Melbourne. He has many years of experience as an ELT teacher, manager, teacher educator and researcher. Kieran offers a range of subjects in the Master of TESOL including Second Language Assessment, Teaching English for Academic Purposes, Teaching English Internationally, and Literature in Second Language Education. He also supervises postgraduate research students. Kieran has published widely, particularly in the field of second language assessment.