The cognitive processes underlying the academic reading construct as measured by IELTS

Abstract

This study, building on CRELLA’s 2006/07 IELTS funded research (Weir et al this volume), clarifies further the links between what is measured by IELTS and the construct of academic reading as practised by students in a UK university by eliciting from IELTS candidates, by means of a retrospective protocol, the reading processes they engage in when tackling IELTS Reading tasks. The study provides grounded insight into the congruence between the construct measured by IELTS and that of academic reading in the target domain.

1 RATIONALE

If they are to provide a useful service to receiving institutions, language tests that address the English language proficiency of overseas students must reflect the demands of the academic courses these students are going to follow. Providers of international examinations have a responsibility to provide valid information for stakeholders and to demonstrate the qualities of their offerings. This two-part project explores the basis for the validity of the IELTS Reading Test in terms of its relationship to the academic reading practices of students at a British university.

Little research is available on the relationship between the IELTS Reading module and academic reading in situ. This study, building on CRELLA’s 2006/07 IELTS funded research (Weir et al this volume), clarifies further the links between what is measured by IELTS and the construct of academic reading by students in a UK university by eliciting from IELTS candidates, by means of a retrospective protocol, the reading processes they engage when tackling IELTS reading tasks.

Weir et al (this volume) carried out a survey-based IELTS research study which sought to

establish the nature of academic reading activities performed across a range of courses with particular reference to contextual parameters and cognitive processing, and provide initial data on the relationship(s) between the IELTS reading module and reading in an academic context

investigate problems experienced by students with respect to these parameters and determine the extent to which any problems might decrease the higher the IELTS band score obtained before entry.

This first-phase study focused on the cognitive processing involved in academic reading, specified under a variety of contextual parameters in the target domain. This was considered a logical first step, providing the necessary empirical basis for a subsequent investigation of the cognitive processes involved in taking the IELTS Reading module. Not least, it would help establish the categories of description that we might ask candidates to apply to their IELTS test taking experience.

The current study thus constitutes the second phase of our linked research agenda for the validation of the IELTS reading component. We identify through participant retrospection the range of cognitive processes students employ when they are performing the various tasks in the IELTS Reading Test. This will provide grounded insight into the congruence between the construct measured by IELTS and academic reading practices in the target domain.

2 LITERATURE REVIEW

In earlier frameworks of reading, especially in those that take into account the purposeful and strategic activities of readers, several types of reading are specified (see Khalifa and Weir, forthcoming and Weir et al, this volume, for a full description of these). In general terms, the reading types covered are expeditious reading, i.e. quick, selective and efficient reading to access desired information in a text (scanning, skimming and search reading), and careful reading, i.e. processing a text thoroughly with the intention to extract complete meanings from presented material. The multiple reading models that are now acknowledged in the second language literature suggest that reading for different purposes may engage quite different cognitive processes or constellations of processes on the part of the reader.

Khalifa and Weir (forthcoming) capture the elements deemed important in earlier frameworks and account for the interactions between reader purpose, cognitive processes and knowledge stored in long-term memory (see Figure 1 below). They hypothesise that difficulty in reading is a function of both the level of processing required by reading purpose and complexity of text. In its present form, following Urquhart and Weir (1998), the Khalifa and Weir framework is a conceptualisation of reading skills on multiple dimensions; both expeditious versus careful and local versus global.

In developing reading tests, as well as ensuring the contextual appropriateness of the test tasks, we advocate a cognitive processing approach designed to model what readers actually do when they engage in different types of reading. The principal concern in this study is a comparison between participants’ processing of IELTS Reading Test items and the mental processes readers employ in comprehending texts when engaging in different types of real life reading.

Khalifa and Weir (forthcoming) outline the cognitive processes contributing to reading according to purpose and their model is summarised in Figure 1 below. The left hand column specifies the metacognitive activity of a goal setter because, in deciding what type of reading to employ when faced with a text, critical decisions are taken on the level(s) of processing to be activated in the central core of the model. The various elements of this processing core are listed in the middle column. Processing at a variety of levels might be initiated by decisions taken in the goal setter. Reading is divided into four levels including careful local within sentences, and careful global across sentences (the mental model), text (the text model) and multiple texts levels (the documents model).

It is argued that the goal setter in the left hand column is critical because decisions taken about the purpose for reading will determine the relative importance of these levels (mental model, text, documents) in the central processing core when carrying out a reading activity.

The various exponents of these two dimensions are listed in the model below and then described briefly. A full description is available in Khalifa and Weir (forthcoming), but we offer here a brief outline of key elements in the model to contextualise the design of our retrospective protocol form.

Figure 1. Cognitive processing in reading (Khalifa and Weir forthcoming)

Urquhart and Weir’s (1998) distinctions between global/local and careful/expeditious are of particular importance to the design of the form used in this study and we will briefly describe them here. Global comprehension refers to the understanding of information beyond the sentence, including main ideas, the links between ideas in the text and the way in which these are elaborated.

Local comprehension concerns the understanding of propositions within the sentence (individual phrases, clauses and sentences). In the model above, local comprehension involves word recognition, lexical access and syntactic parsing and establishing explicit propositional meaning at the phrase, clause and sentence level.

Careful reading involves extracting complete meanings from text, whether at the global or local level. As noted above, this is based on slow, careful, linear, incremental reading for comprehension. Expeditious reading, in contrast, involves quick, selective and efficient reading to access relevant information in a text.

In careful global reading the reader may try to identify the main idea(s) by reconstructing the macro-structure of a text. Logical or rhetorical relationships between ideas are represented in complexes of propositions (see Vipond 1980), often represented by the writer by means of paragraphing; global reading involves attempting to reconstruct these complexes.

The distinction across types of careful reading reflects the real life reading processes in academic settings generally. Readers find themselves having to read and learn from a whole text as well as integrating information from various texts especially for the preparation of assignments. It is clear from the brief definition of the frameworks above that careful reading as an umbrella term encompasses processing at sentence, multisentence, text and multi-text levels.

In the past, models of reading have usually been developed with only careful reading in mind (see, for example, Hoover and Tunmer, 1993; Rayner and Pollatsek,1989). However, careful reading models have little to tell us about how skilled readers cope with other reading behaviours such as skimming for gist (Rayner and Pollatsek 1989, pp 477-478). Carver (1992) and Khalifa and Weir (forthcoming) suggest that the speed of reading is important as well as comprehension. In relation to reading for university study, Weir et al (this volume), found that ‘for many readers reading quickly, selectively and efficiently posed greater problems than reading carefully and efficiently’.

Three types of expeditious reading are distinguished in the model: scanning, skimming and search reading. Scanning is a form of expeditious reading that occurs at the local level. It involves reading highly selectively to find specific words, figures or phrases in a text. Skimming is generally defined (Urquhart and Weir 1998, Weir 2005) as reading quickly by sampling text to abstract the gist, general impression and/or superordinate idea: skimming relates exclusively to global reading. In academic study contexts, readers may try to establish the macro-structure of a text and the discourse topic (see Kong 1996) by skimming, using careful global reading to determine how the ideas in the whole text relate to each other and to the author’s purpose.

Unlike skimming, search reading involves predetermined topics. The reader does not necessarily have to establish a macro-propositional structure for the whole of the text, but is, rather, seeking information that matches his/her requirements. However, unlike scanning (where exact word matches are sought) the search is not for exact word matches, but for words in the same semantic field as the target information. Search reading can involve both local and global level reading. Where the desired information can be found within a single sentence the search reading would be classified as local and where information has to be constructed across sentences it would be seen as global.

The different types of reading that readers might choose to carry out (the left hand column of the model), the different levels of processing that might be activated (the central column), and the knowledge base necessary to successfully complete an assigned reading task (the right hand column) provide us with the theoretical framework on which our retrospection protocol form is based. The form is thus intended to elicit from participants taking the IELTS Reading Test how their approach to reading the texts and responding to the tasks presented to them reflects the model of cognitive processing in Figure 1.

We will briefly review the case for the use of protocol analysis in establishing test validity and examine its history before describing in more detail the instrument developed for our study.

2.1 A processing approach to defining reading

It is common for language testers to adopt what has been called a subskills approach, based on the assumption that it is possible to target particular types of item or test task to specific types of reading so that one item might target the ability to understand the meaning of an individual word in a text and another might target the ability to extract the overall meaning of a text within a very limited time frame (skimming). Alderson and Lukmani (1989) have questioned the feasibility of classifying reading test items in this way on the grounds that ‘expert judges were unable to reach agreement on which subskills individual items were addressing. However, Weir & Porter (1994, p7) responded that ‘a growing body of literature suggests that it is possible with clear specification of terms and appropriate methodology for testers to reach closer agreement on what skills are being tested’. The body of literature the authors referred to includes Bachman et al (1988), Teasdale (1989), Lumley (1993), and Weakley (1993). Alderson also now appears to have revised his earlier position, adopting an approach for the DIALANG project in which individual items are said to test identifiable skills (Alderson 2005, pp 125-137).

The debate over subskills centred on the ability of expert judges to arrive at a consensus about what was being tested and the essential role of the candidate was largely overlooked. The majority of studies paid surprisingly little attention to the cognitive processing required for candidates to carry out test tasks. Alderson (2000, p 97) argues that,

The validity of a test relates to the interpretation of the correct responses to items, so what matters is not what the test constructors believe an item to be testing, but which responses are considered correct, and what process underlies them.

In other words, to clearly establish the trait that has been measured we need to investigate the processing necessary for task completion.

2.2 Protocol analysis

A process-oriented approach to defining reading activity in language tests seeks an experimental method which permits comment on the actual reading process itself. Verbal report is a widely used experimental procedure where participants describe the linguistic process which they are engaged in and the results are often known as protocols. The approach is not new. Thorndike (1917) in looking at ‘reading as reasoning’ investigated what students were thinking whilst answering comprehension questions in a test. More recently Anderson et al (1991), Block (1986), Crain-Thoreson et al (1997), Nevo (1989), Perkins (1992), Phakiti (2003), and Weir et al (2000) provide descriptions of protocol-based studies in reading. Such studies can cast illuminative light on whether the different types of reading that have been proposed do in fact instigate the different processing activities that have been shown to obtain in normal processing in reading outside of tests.

Methodological advances in language testing in the 1980’s saw researchers such as Alderson (1990a and 1990b) advocating the importance of gathering information on test-taking processes as part of construct validation and the use of introspective data to throw light on the nature of the trait under consideration.

For discussion of the methodology of protocol studies see: Cohen (1984 and 2006) on Verbal Reports for investigating Test-Taking; Gass and Mackey (2000) for a useful theoretical and practical account of verbal protocol analysis; Ericsson and Simon (1993) on the use of protocol analysis to investigate cognitive processing; Green (1998) on verbal protocol analysis in language testing research; Pressley and Afflerbach (1995) on verbal protocols for reading; and Stratman and Hamp-Lyons (1994) on concurrent think-aloud protocols.

With respect to using students’ introspective data as a method of investigation in reading research, most of the studies carried out in testing reading research using introspection techniques imply the existence of subskills: Hosenfield (1977), Cohen (1984), Grotjahn (1987), Fieldmann and Stemmer (1987), Nevo (1989), and Anderson et al (1991) to name but a few.

A further concern is that, because of the intensive nature of verbal protocol research, which requires participant training and may generate a very large quantity of data for each individual, studies typically involve no more than a handful of participants. In the current study we set out to triangulate the detailed protocol data we had obtained in the Weir et al study (this volume) with less nuanced data elicited from a much more extensive group of participants.

3 METHODOLOGY

3.1 Research instruments

In the previous IELTS funded study in this project, Weir et al (this volume) established that typical sequences of reading activities associated with student assignments often involved expeditious reading followed by selective and intensive careful reading with information then being integrated into the students’ developing understanding of the focal topic.

In the current study we set out to learn which activities and sequences typified reading for the purpose of taking an IELTS Academic Reading test. Initially, we drew on an element of the earlier study (Weir et al this volume) involving the elicitation of both qualitative and quantitative data from a small group of four participants on how IELTS Reading Tests might be approached. One (EAL) respondent provided the following general description of how he combined careful and expeditious reading types in approaching the IELTS Reading Test tasks:

I usually read the texts carefully from the beginning to the end initially then I go to the questions. I can answer some questions without having to read the text again. If not, I usually remember the place where the info necessary for the answer is located and go there usually by scanning which may be followed by some careful reading.

This careful-expeditious-selective strategy pattern contrasts with the expeditious-selective-careful strategy suggested as a common response to academic assignments by the responses to the questionnaire on academic reading outlined above and may point to important differences in how candidates approach IELTS texts and how students approach reading for study purposes.

Weir et al (this volume) identified potential limitations of the IELTS Reading Test as a reflection of academic reading skills in four areas. There was said to be a lack of items requiring

expeditious reading skills

integration of information beyond the sentence level

information at the level of the whole text

information accessed across texts

In the current study we sought data on whether the strategies reported by the earlier small focus group were reflected in the wider IELTS test taking population. Participants were given one part of an IELTS Reading Test and responded to a brief retrospective protocol form concerning the types of reading they had employed.

We selected two tests from the Weir et al (this volume) study for this purpose. These comprised two Academic Reading Tests taken from Cambridge Practice Tests for IELTS: Volume 2 (Cambridge University Press 2000). The IELTS partners do not release retired IELTS forms for research purposes, but the material appearing in these books is developed by Cambridge ESOL, the IELTS partner responsible for test production, using their standard IELTS test production procedures. It conforms to the IELTS specifications and is therefore representative of genuine IELTS test material. The selected tests appear as Tests E and F in the Weir et al (this volume) study and were selected on the basis that both

included only question types still used in the current IELTS Reading Test format (www.ielts.org), but provided a variety of these

included a range of items that had been identified in the Weir et al (this volume) study as requiring both explicitly stated and implicit information located across sentences as well as within sentences

included items that had been identified by Weir et al (this volume) as motivating expeditious as well as careful reading types

had not been identified in the previous study as having any characteristics that would make them atypical of IELTS texts (see Weir et al this volume for the range of textual measures used).

Each IELTS Reading Test may involve a different combination of item types. The ten broad categories of item type used on the test are listed on the IELTS website (www.ielts.org) with links to further information about each. The list is reproduced below (the seven item types found in the two tests included in this study are marked ). Each item type is glossed with a brief explanation of the skills being targeted, based on information found on the IELTS website:

Type 1 Multiple Choice

Multiple choice items are used to test a wide range of reading skills. They may require the candidate to have a detailed understanding of specific points or an overall understanding of the main points of the text.

Type 2 Short-answer questions

Type 3 Sentence Completion

Candidates are asked to complete the sentence in a given number of words taken from the passage or from a list of possible options.

Type 4 Notes, Summary or Table/Flow-chart Completion

This task type often relates to precise factual information, and so is often used with descriptive texts.

Type 5 Labelling a Diagram

Type 6 Choosing Headings for Paragraphs or Sections of a Text

It is used with passages that contain paragraphs or sections with clearly defined themes

Type 7 Locating Information

It may test a wide range of reading skills, from locating detail to recognising a summary or definition etc.

Type 8 Identification of Writer’s Views/Claims or of Information in a Text

Tests the candidate’s ability to recognise particular points of information conveyed in the passage. It can thus be used with more factual texts

Type 9 Classification

Type 10 Matching

This task type is designed to test the candidates’ ability to recognise opinions or theories.

The full IELTS Academic Reading Test has three parts. Each Test Part has one input text and may include up to four sections or sets of items of the same format. For example, Part 1 of Test E has three sections made up of ‘Yes/ No/ Not Given’ items, multiple choice items and summary completion items. The full test has 40 items (with 13 in the first two parts and 14 in the last) and takes a total of one hour to administer. For the purposes of this study each Test Part was administered separately with a time limit of 20 minutes. Participants were then given a further ten minutes (or longer if required) to complete the retrospective questionnaire. The test was administered in this way to allow time for participants to complete the retrospection form and review their answers within a typical 40 minute class and to avoid overburdening them with having to complete the questionnaire in addition to the demands of a full three-part IELTS test.

A breakdown of the item types found in these two tests is given in Table 1. Both tests included here involve mainly selected response items with Type 8: Identification of Information in a Text and Type 1: 4-Option Multiple Choice items making up between them the majority of items on Test E (10 and 15 respectively of the 40 items) and Type 8: Identification of Information in a Text and various forms of text-to-list matching (Type 7: Locating Information, Type 6: Choosing Headings and Type 10: Matching) making up the majority on Test F (14 and 20 of 40). Six items on Test E (short answer questions) and six on Test F (summary completion) involve a constructed response format, although the participants are able to choose words found in the passages to complete both of these tasks.

Table 1. Test parts and item types included in this study

To investigate the reading types employed by participants responding to the tests, we developed a questionnaire form modelled on the earlier (Weir et al this volume) survey of students engaging in academic reading at the University of Bedfordshire. This form was intended to be used by participants as a retrospective protocol immediately following administration of a part of an IELTS Reading Test.

The retrospection form (see the example in Appendix C) was designed to address the choices between reading types (see above) made by participants as they encountered the reading texts and items in IELTS. Questions about the participants’ background and previous experience of IELTS (Age, Gender, First Language, Nationality, Date of most recent IELTS test, IELTS Reading score, and Intended university subject) were included on the answer paper (Appendix B).

The three sections of the retrospection form were as follows:

1 Sequence of reading activities

Each IELTS text is accompanied by 13 or 14 items and these are usually divided into between two and four item sets (groups of items, each with a different question format such as multiple choice or gapfilling). This section of the questionnaire sought information on whether participants were reading the text before looking at each item set and whether they were using expeditious or careful reading when doing so.

The three choices given for each Test Section were; a) read the text or part of it slowly and carefully (careful reading)/ b) read the text or part of it quickly and selectively to get a general idea of what it was about (expeditious reading - skimming)/ c) did not read the text.

2 Strategies for responding

This section sought information on how participants read to find the answers to each item. Here the focus was on establishing the processes that participants engaged in to locate the correct answer to each individual item. These processes might include lexical matching between words in the question and words in the text, using knowledge of discourse conventions to select the relevant part of the text or integrating information from the text with prior knowledge about the topic.

The twelve items were as follows:

match words that appeared in the question with exactly the same words in the text (local - scan reading based on word recognition)

quickly match words that appeared in the question with similar or related words in the text (local - search reading based on lexical access)

look for parts of the text that the writer indicates to be important (global, text level)

read key parts of the text such as the introduction and conclusion (global, selective reading at text level)

work out the meaning of a difficult word in the question (local, word recognition)

work out the meaning of a difficult word in the text (local, word recognition)

use my knowledge of vocabulary (lexical knowledge)

use my knowledge of grammar (syntactic knowledge)

read the text or part of it slowly and carefully (careful reading, establishing propositional meaning: global or local)

read relevant parts of the text again (careful reading- global or local)

use my knowledge of how texts like this are organised (text structure knowledge)

connect information from the text with knowledge I already have (general/ topic knowledge)

3 Information base for the response

This section sought information on where participants felt they had found the necessary information to enable them to answer each question. They were asked to indicate whether they had found the necessary information:

within a single sentence (propositional level)

by putting information together across sentences (mental model level)

by understanding how information in the whole text fits together (text level)

without reading the text (general/ topic knowledge)

Or, alternatively, whether they ‘could not answer the question

The instructions explained that all items allowed for the selection of more than one of the response options. This provision was made so that complex and recursive response strategies could be at least partially captured by the questionnaire.

After passing through several iterations within the research team, the form underwent trialling with a small focus group of three IELTS participants who reported back to the researchers on their experience. Revisions were made to the content and format to make the retrospection form more accessible to language learners before it was used with larger numbers of participants (see the discussion of changes relating to the operationalisation of inferencing below). To reflect the different numbers of items and of item sets associated with each of the texts, a separate form was prepared for each of the six IELTS Test Parts used in the study. An example is provided as Appendix C.

3.2 Participants and settings

Participants included some 352 learners on IELTS preparation, university pre-sessional and advanced general English classes in the UK and Taiwan over the period July to October 2007. Although we would not claim that these learners are a stratified random sample of the global IELTS test taking population, the groups were, as well as being accessible to our research team, the kinds of learner for whom the IELTS academic modules are intended (they were mostly preparing for higher academic study).

Learners were each given one of the six IELTS Test Parts we had identified for the purposes of the study and these were administered in class by their teachers. The teachers explained what the students had to do and an instruction sheet was provided for each participant (Appendix A). Immediately following the administration of the questionnaire and collection of answer papers, the teachers were free to review the answers and to discuss reading approaches with their students on the basis of their questionnaire responses. Table 2 here describes the participants by number and institution.

Table 2. Participants by institution

4 ANALYSIS

In our data analysis, we generated descriptive statistics for preview reading, response strategy and location of necessary information by Test Section and compared the patterns of response across these both by participant reading ability and by item type. We also compare the findings of the current study with the outcomes from Weir et al (this volume).

For the purpose of comparing the approaches to reading adopted by higher and lower ability participants, we divided the participants into three groups according to their total scores as a measure of reading ability. IELTS Test Parts vary only a little in overall difficulty across forms. As a result, we felt that using the same score ranges across Test Parts as a basis for categorisation would provide a crude but adequate indication of overall reading ability for the purposes of this study.

In dividing the participants by level, we employed three broad categories: 0 to 5, 6 to 8 and 9 and above points. These categories are (based on the equivalences published at www.ielts.org) roughly indicative of IELTS band scores of 5.5 and below, 6.0 and 6.5 and above respectively.

Using these score categories, we carried out chi-square analysis and analysis of variance as appropriate to explore whether reading ability had any significant (p<.05) effect on preview reading, response strategy use or locating necessary information.

5 RESULTS

Figure 2 below shows the distribution of scores on each Test Part. Mean scores ranged from 6.99 on Test Part E2 to 8.14 on Test Part F2. This places the majority of participants on all test Parts at around the equivalent of a band 6.0 level for IELTS Reading. The mean score for participants worldwide is 6.04 for females and 5.90 for males (www.ielts.org).

Figure 2. Total Scores by Test Part

Table 3 displays the numbers of participants by first language and gender. 78.9% of the participants were L1 speakers of Chinese with 4.3% Arabic and 4.0% Thai speakers making up the next largest L1 groups. 4.8% of participants did not respond to this question. IELTS no longer publishes information on the proportion of speakers of specific L1s among the worldwide candidature, but we would assume that the study population includes a relatively high proportion of Chinese speakers. The majority of our participants (58.8%) were women. This compares with 51.3% of the IELTS Academic Module candidates worldwide (www.ielts.org). Participant ages ranged from 14 to 57 with a median age of 22 years.

Table 3. Participants by first language and gender

In the following sections we describe the responses to the three sections of the reading protocol form in turn looking both at overall response patterns and at responses to each Test Section. We also examine differences between higher scoring and lower scoring participants.

5.1 Text preview

The first section of the protocol form asked participants about whether they read the text before looking at the tendencies:

read the text or part of it slowly and carefully

read the text or part of it quickly and selectively to get a general idea of what it was about

did not read the text.

Each Test has three Parts, each with its own text. Sets of questions associated with each text may follow different formats. Each Part includes at least two sets of questions, referred to here as Sections. Participants were asked to indicate whether or not they read the text before looking at the questions in each of these Sections. The results are summarised in Figure 3 below. Note that only one Test Part (E2) included more than 3 Sections. As E2.4 is made up of a single item, it is not included in Figure 3.

Test Section 1

Test Section 2

Test Section 3

Figure 3. Text preview by Test Section

An analysis of participant responses to the first section of the protocol form revealed the following (Figure 3):

A majority of participants chose (b) read the text through quickly and selectively before reading each the questions for each Section; skimming the text without specific questions in mind: 61% did this before reading the questions in the first Section, 55% before reading the questions in the second Section and 46% before reading the questions in the third Section.

22%, 26% and 36% for reading slowly and carefully (a)

17%, 19% and 18% for not reading the text before attempting the questions (c).

Although, as we see from these data, a majority of participants read quickly and selectively before approaching the questions, on the third Section of Test E Part 1, more participants read slowly and carefully before looking at the questions (a - 40%) than quickly and selectively (b - 35%) with 25% not previewing the text (c) before reading the questions for this section. The increase in the number of participants who did not preview this section probably reflects the position of the task - the last of three tasks relating to the same text. Participants may either have felt they did not need to read the text again before addressing the questions or perhaps may have felt under time pressure as they approached the end of the time available. Conversely, a large proportion of participants may have found they had enough time available to read through the text again carefully before attempting Section 3.

When the protocol data were compared with IELTS Test Part scores, participants who did not preview the text tended to have higher scores than the quick and selective pre-readers, who in turn tended to have higher scores than the slow and careful pre-readers. Chi-square tests comparing the pre-reading choices of low (5 and below), mid (6 to 8) and high (9 and above) scorers on the tests confirmed that these differences were significant (p<.01) across all Sections within the Test Parts.

Figure 4. Total scores by test preview

Figure 4 again refers to the three Sections (sets of questions of the same format) within the Test Parts. Pre-read 1, 2 and 3 refer to whether and how participants read the relevant text before reading the questions in each Section. The Figure provides a comparison between participants at the three levels of total score: those scoring five or below, 6 to 8 or 9 and above on the relevant Part of the test. The charts indicate that participants at the highest level were less likely than lower scoring participants to read the text before the questions (although a majority even of these higher level participants did preview the text quickly and selectively). It may be that the higher-scoring participants did not need to spend as much time on previewing the text in order to respond successfully. Certainly a strategy involving reading the questions first before turning to the text is closer to the expeditious reading behaviour reported by most undergraduates in the Weir et al (this volume) study.

5.2 Test response strategies

Figure 5. Response strategy use by score level

Figure 5 indicates that strategies 1, 2, 3, 4, 9 and 10 were all relatively popular, being selected at least once by over 60% of participants. Strategies 5, 6, 7, 8, 11 and 12 were less so, each being selected by less than 40% of participants.

Across Test Parts the most popular test strategy was 2 - quickly match words that appeared in the question with similar or related words in the text. This emerged as the most frequently endorsed item on ten of the fifteen Test Sections with 83% of participants reporting using this strategy at least once. 10 - read relevant parts of the text again was also popular, appearing as the most popular choice on two Test Sections and being selected at least once by 77% of participants. 3 - look for parts of the text that the writer indicates to be important was the most popular strategy on another two Test Sections and was selected at least once by 76% of participants. 4 - read key parts of the text such as the introduction and the conclusion and 12 - connect information from the text with knowledge I already have were equally the most popular on one section. The least popular strategy was 8 - use my knowledge of grammar which was chosen for one or more items by 26% of participants.

* The effect for ST5 was negative i.e. greater use of ST5 was associated with lower scores

Table 4. Analysis of variance: Response strategy use, all participants

One-way analysis of variance (Table 4) comparing the three groups of test-takers on strategy use (including all test sections) indicated significant differences (p<.05) by level on three strategies: strategy 2, strategy 5 and strategy 10. The significant differences on strategies 2 and 10 suggest that higher scoring participants were more likely to use an approach combining search reading with careful re-reading of relevant sections of text. Such an approach parallels the most widely adopted approach to academic reading taken by participants in Weir et al (this volume) and may suggest that the more successful participants are approaching the IELTS tasks in a similar way to students reading for an assignment. The significant difference for strategy 5 presumably reflects the greater difficulty that low scoring participants have with word recognition.

The picture of reading in response to IELTS test items that emerges is consistent with the general approach to academic reading reported by student readers in the Weir et al (this volume) protocol study: quick and selective search reading followed by intensive careful reading of relevant text parts.

5.3 Test response strategy use by Test Section

To explore the implications of task type for response strategy use, we compared the responses by Test Section.

Table 5 shows patterns of text preview, response strategy use and locating information across sections. Where mean scores for a strategy use in a Test Section are above a threshold value (see Table 5), these are identified in the table. Graphs displaying these data in more detail are provided in Appendix D below.

Comparing strategy use by Test Section reveals some clear differences between the sections (see Table 5 and Appendix D), while patterns of strategy use were loosely associated with item type. Two sections (E1.2 and E2.1) included items of Type 1 (Multiple Choice). E1.2 and E2.1 yielded mutually consistent patterns of strategy use with the five most popular strategies occurring in the same order of preference on both; 2 (match related words), 1 (match exact words), 3 (look for parts of the text that the writer indicates to be important), 10 (read relevant parts of the text again) and 9 (read slowly and carefully) It is notable that this tem type, together with sentence and summary completion, was particularly associated with the direct word matching strategy (1). However, there were also differences in how participants responded to the multiple choice tasks. On E1.2 the information required was most often reported within the sentence while in E2.1 it was found across sentences.

Test E, Part 1, Section 1 (E1.1); E2.2, Test F 1.1 and Test F2.2 all involved IELTS item Type 8, ‘Identification of Writer’s Views/Claims or of Information in a Text’ with a selected response, True/ False/ Not Given format. All involved widespread use of strategies 2 (match related words), 9 (read slowly and carefully) and 10 (read relevant parts of the text again) and participants most often reported locating the necessary information across sentences followed by across the text as a whole (see Table 5 and Appendix D).

Response strategy 3 (look for parts of the text that the writer indicates to be important) was the most popular selection on F1.1 (Appendix D). This strategy seems particularly well-suited to Text F1 as it includes subheadings that might have helped to signpost where relevant information was to be found (the other section in which response strategy 3 was the most popular - F3.1 - involved matching subheadings to paragraphs).

PR1 read the text or part of it slowly and carefully PR2 read the text or part of it quickly and selectively to get a general idea of what it was about PR3 did not read the text. ST1 match words that appeared in the question with exactly the same words in the text ST2 quickly match words that appeared in the question with similar or related words in the text ST3 look for parts of the text that the writer indicates to be important ST4 read key parts of the text such as the introduction and conclusion ST5 work out the meaning of a difficult word in the question ST6 work out the meaning of a difficult word in the text ST7 use my knowledge of vocabulary ST8 use my knowledge of grammar ST9 read the text or part of it slowly and carefully ST10 read relevant parts of the text again ST11 use my knowledge of how texts like this are organised ST12 connect information from the text with knowledge I already have L1 within a single sentence L2 by putting information together across sentences L3 by understanding how information in the whole text fits together L4 without reading the text L5 could not answer the question

Table 5: Text preview, response strategy and locating information by Test Section

The two Test Sections involving partially constructed responses - the selection of a word or words from the passage to complete sentences or summaries of the text (E3.1 and F1.2) - both involved a high proportion of strategies 2 (match related words) and 1 (match exact words) followed in popularity by 10 (read relevant parts of the text again) and 7 (knowledge of vocabulary). Necessary information was located within a single sentence. A third section involving summary completion (E1.3), but with a selected response format, was identified with the use of information distributed throughout the text and with strategies 2 (match related words), 10 (read relevant parts of the text again) and 9 (read slowly and carefully.)

Both E3.2 and F3.2 involved Type 10 (Matching) items and also produced a broadly similar pattern of strategy use. On section E3.2 strategy 10 (read relevant parts of the text again) was most popular, followed by 2 (match related words), 9 (read slowly and carefully), 3 (look for parts of the text that the writer indicates to be important) and 7 (knowledge of vocabulary.) On F3.2, strategy 2 was the most popular, with 10 second and 9 third. Strategy 1 (match exact words) was in fourth place and 3 in fifth.

Necessary information was most often reported as being found across sentences in E3.2, but within sentences in F3.2. This reflects differences between the items in the two sections. E3.2 provides paraphrases of facts and opinions expressed by the writer and these cannot be answered through exact word matching. F3.2 on the other hand requires matching of the names of systems described in the text (fingerprint scanner, voiceprint etc.) to groups of people (sports students, welfare claimants). The necessary information is explicitly stated in one or two sentences of the text. For example, the sentence ‘In some California housing estates, a key alone is insufficient to get someone in the door; his or her voiceprint must also be verified’ allows the participant to match item 39 ‘home owner’ to option D, ‘voiceprint’.

5.4 Analysis of Variance

Having found indications of a relationship between strategy use and item type, we explored whether strategy use had an impact on participants’ scores on each Test Section. Using one-way analysis of variance we compared the three student groups’ use of strategies on each test section. Significant (p<.05) results were found for one or more strategies on the following Test Sections (Table 6).

The effect for ST7 was negative i.e. greater use of ST7 was associated with lower section scores

Table 6. Analysis of Variance: Test response strategy by Test Section

Strategy 2 (match similar words), the most popular strategy overall, was associated with success on E2.1 (4-option Multiple Choice) and F3.2 (Matching). The answers to F3.2, which also yielded a significant effect for strategy 1 (exact word match) as we have seen above, involved explicitly stated information at the sentence level with cues provided by exact or near-exact matches between answer options and words in the text. E2.1 also yielded a significant effect for strategy 9 (slow careful reading) suggesting that similar word matches (perhaps between the phrase the youngest readers in item E17 and beginner readers in the text for example) might have served as a precursor to more careful and intensive reading in identifying the correct answers, which were mainly said to be found across sentences.

It is interesting that strategies 3 and 4 emerged as significant (p<.05) on sections F1.1 and F3.1, both sections on which strategy 3 (look for parts of the text that the writer indicated to be important) was the most popular strategy and strategy 4 (read key parts of the text such as the introduction and conclusion) also ranked among the five most selected strategies. This suggests that the more successful participants on these sections were able to make use of information at the text-level in arriving at a correct response.

General topic knowledge appears to have been beneficial in responding to F1.1. Further investigation indicated that it was the results on items 3 and 4 that were particularly affected by background knowledge. These two items were 3. Just over half the nurses in the 1986 study believed that management understood the effects of shift work on them and 4. The Canadian study found that ‘illness in the family’ was a greater cause of absenteeism than ‘work to do at home’. It may be that participants with some experience of a working environment were better able to predict the answers (No and Yes respectively.) This would appear to indicate a potential vulnerability in these two items that is highlighted by this retrospection exercise.

Strategy 8 was associated with higher scores on both E2.3 and E3.2. Although the formats were different, both of these Test Sections involved matching summaries to information or views given in the text. In E3.2 a four-way choice is offered between hopes expressed by the writer, fears expressed by the writer, fears expressed by others and facts reported by the writer. Grammatical knowledge may have proved useful in helping the participants to recognise that all of the hopes are expressed through ‘will’ constructions, the fears through ‘may and the facts through is + to be.

5.5 Location of necessary information

Participants most often reported finding the information necessary to respond to the tasks by putting information together across sentences (2). This was selected most frequently on nine of the fifteen Test Sections and was chosen at least once by 89% of participants. 1 (within a single sentence) was the most popular selection on four Test Sections and was chosen at least once by 76%. 3 (by understanding how information in the whole text fits together) was the most frequent selection on one section and was the second most popular choice overall, being chosen at least once by 82% of participants. The fourth and fifth options, 4 (without reading the text) and 5 (could not answer the question) were not often selected on any Test Section, but were selected for one or more items in total by 26% and 27% of participants respectively. Of the 234 answers reportedly given without reference to the text, 92 (39.3%) were correct. 19 of these occurred on Section F2.2 (against 13 incorrect guesses) and a further 9 (against 5 incorrect guesses) on another section with Type 8 items (Identification of writer’s views/claims or of information in a text -Yes/No/Not Given): E2.2. This suggests that Type 8 items may be particularly vulnerable to guessing - a point underlined by the discovery that the researchers were also able, without reading the texts, to give the correct answer to those items that had yielded more correct than incorrect test-taker guesses.

Only a handful of items involved more than five participants reporting that they were unable to find an answer: Test E, items 11 (7 participants unable to find an answer, item facility [p =.35]), 12 (6 participants, p=.28), 32 (6 participants, p =.48) and 38 (6 participants, p =.46) and Test F item 13 (9 participants, p=.30). As the low item facility values above suggest, these were all among the more difficult items in their sections and most occurred towards the end, suggesting the effects of time pressure. The exception, item E32, occurred in section E3.1 (Sentence Completion) and required participants to find a second success of genetic research in finding the cause of disease (cystic fibrosis). The level of confusion that is suggested by the high number of participants unable to find an answer may be attributable both to the constructed response nature of the item (participants needed to refer to the passage for an answer, not to a list of given options) and to a lack of independence in the item, which seems to require that participants should identify the first success - muscular dystrophy (the answer to Item 31) - before being able to recognise the second.

In eleven of the fifteen sections, the results of the current study were broadly consistent with the small-scale protocol study on the same test materials conducted by Weir et al (this volume) (see Table 7 below), although the participants in the earlier study were generally more likely to report finding information within sentences and, unlike many of those in the current study, did not find information to answer questions by drawing on the text as a whole (except in responding to item E26 [Section E2.4] - choose a heading for the text).

* In these two cases the participants found answers to near equal numbers of items in a section from information located within the sentence and from information distributed across sentences

Table 7 Location of necessary information: comparison with findings reported in Weir et al (this volume)

This tendency for the participants in the current study to report drawing on more of the text than those in the earlier research, using information across sentences and across the whole text in responding, is also reflected in the four sections where the results were discrepant: E1.3, E2.1, E3.2 and F1.2. The differences may be explained at least in part by proficiency level, the Weir et al (this volume) participants being native speakers of English or language learners with a higher level of proficiency than most of the current participants.

These discrepancies may also have been an artefact of the research method as participants in the earlier study were asked not only to decide whether an answer was to be found within or across sentences, but also whether the information was explicitly stated or implicit. The explicit: implicit distinction was dropped in the current study because of the impracticality, revealed through piloting, of operationalising it sufficiently clearly for participants to use at distance. The distinction between implicit and explicit information may have led the earlier (Weir et al this volume) participants to report finding the necessary information within the sentence even where this required bridging inferences based on other parts of the text.

This may be illustrated by item F1.1-2. The item requires participants to identify (Yes/ No Not Given) whether the following proposition is supported in the text ‘Nurses in the Prince William Hospital study believed that there were benefits in taking as little sick leave as possible’. To answer, participants would need to relate information about the study (given in the statement, ‘The study reported here was conducted in the Prince William Hospital in Brisbane, Australia’) to the reported attitudes concerning sick pay (given in the two sentences, ‘A prevalent attitude amongst many nurses in the group selected for study was that there was no reward or recognition for not utilising the paid sick leave entitlement allowed them in their employment conditions. Therefore, they believed they may as well take the days off - sick or otherwise.’) Participants in the earlier study agreed that they had found the necessary information within the sentence. The theme of the study at the Prince William hospital, which had also appeared in item 1, could, for these participants, now be treated as ‘given information’ and could perhaps be inferred in responding to item 2. The key sentence presenting new information was, ‘they believed they may as well take the days off - sick or otherwise’ and this may have provided enough to support a correct response to item 2. Without the distinction between implicit and explicit information, the largest group of current study participants (48%) reported finding the necessary information across rather than within the sentence (selected by 30%).

The clear discrepancy between the findings relating to E1.3 may point to alternative approaches to this item type (Type 4 summary completion with selected response). As this section involves completing a summary of the text, it is unsurprising that the participants in the current study tended to find answers by drawing on the text as a whole. However, the answer options are generally identical to or closely related to words in the text (e.g. question option - honesty and openness/ text - honest and open; question option - social record/ text - social record.) This implies that a direct word matching strategy starting from the answer options and focusing on sentence level propositions might, together with a degree of inferencing, have supported success on this section for the Weir et al (this volume) participants, although strategy 1 (exact word matching) was not a particularly popular choice on this section for participants in the current study.

6 Conclusion

Weir et al (this volume) reported that: the ‘… major focus of the IELTS test appears to be on careful reading whereas the survey data reported here suggest that for university students expeditious skills and strategies are just as critical for academic study and in a number of cases more problematic for both L1 and L2 students.’

This was followed by a call for an extensive protocol based study of the cognitive processing of students taking the IELTS Reading Test to illuminate whether this was the case.

The current study provides clear evidence that, for most participants across the different task types, expeditious reading in fact plays an important role in the way they seek to answer the questions. We found that, consistently across Test Sections, the majority of participants chose to read the text through quickly and selectively before reading the question. The most popular test strategy was 2 - quickly match words that appeared in the question with similar or related words in the text. This emerged as the most popular selection on ten of the fifteen Test Sections with 83% of participants reporting using this strategy at least once.

However, this does not imply that expeditious reading is tested separately from careful reading in IELTS, but rather that the two appear to be integrated. Two key strategies that were noted in the earlier study were also prominent in participant self reported behaviour in this. 10 - read relevant parts of the text again appears as the most popular choice on two Test Sections and was selected at least once by 77% of participants. 3 - look for parts of the text that the writer indicates to be important was the most popular strategy on another two Test Sections and was selected at least once by 76% of participants.

The picture of reading in response to IELTS test items that emerges is consistent with the general approach to academic reading reported by student readers in the Weir et al (this volume) protocol study: quick and selective search reading followed by intensive careful reading of relevant text parts.

It is also clear from the protocol data that IELTS participants have extensive scope for careful reading. Because IELTS includes 13 or 14 questions relating to each short text, there are opportunities to read the text or parts of it several times in finding the information necessary to respond. The longest text here (E3) has 1,034 words (including the title and glossary) and the shortest (E2) has 586. If a participant were to spend about one third of the available time reading the questions and writing the responses, he or she would still only need to read at the very slow rate of around 50 to 75 words per minute in order to read through each text once. IELTS allows for very intensive careful reading of material that we have already seen is probably of only modest difficulty when compared with the introductory undergraduate readings described in Weir et al (this volume).

Earlier concerns by native speaker informants in Weir et al (this volume) relating to the number of items that seemed to focus on the sentence level were lessened. Participants most often reported finding the information necessary to respond to the tasks by putting information together across sentences (2). This was selected most frequently on nine of the fifteen Test Sections and was chosen at least once by 89% of participants. There is some evidence that there may nonetheless be a high proportion of test items where the answer can be found within one sentence. 1 (within a single sentence) was the most popular selection on four Test Sections and was chosen at least once by 76% of participants.

Fears that IELTS was not addressing understanding at the whole text level also appear to be ill grounded. 3 (by understanding how information in the whole text fits together) was the most frequent selection on one section and was the second most popular choice overall, being chosen at least once by 82% of participants. This points to the value and necessity, in addition to expert judgement, of using protocol studies as a means of establishing what participants themselves perceive they are doing when they respond to the tasks.

The demonstrated relationship between the adoption of certain strategies and success on various items indicates the critical importance of ensuring that there is a clear match between the strategies that are being elicited by items in a test and the construct that is being measured. Most formats in IELTS emerge from this study in a positive light in this respect but there must be some concern about Type 8 items (Identification of Writer’s Views/Claims or of Information in a Text -Yes/No/Not Given): E2.2, which may be particularly vulnerable to guessing.

Unfortunately for the test developer, it is also apparent that the relationship between item type and response strategy may not be straightforward. Certain item types do appear to provoke the use of certain strategies; multiple choice and summary completion are associated with direct word matching, for example. On the other hand, task type is not a very reliable predictor of patterns strategy use. Some Test Sections employing Yes/No/Not Given items encouraged the use of expeditious reading strategies such as looking for parts of the text the writer indicates to be important or reading key parts (F1.1), while others do not (F2.2). Some response strategies were common across Test Sections (particularly the lexical relatedness strategy 2 ‘quickly match words that appeared in the question with similar or related words in the text’ and careful reading strategies 9 (read … slowly and carefully) and 10 (read relevant parts of the text again). While this suggests that IELTS does involve the use of expeditious reading strategies on the part of participants, this is almost invariably associated with careful reading of relevant passages. Perhaps the only means of testing expeditious reading is to enforce time constraints; Section E2.4 is the only section that appears to encourage expeditious reading without careful reading. This section includes a single item and occurs at the end of a Test Section, suggesting that time constraints were likely to have played a part in determining participant response strategies.

In brief, the researchers recommend that the IELTS partners should consider the following.

As part of the pre-testing process, make routine use of response protocols to investigate how test-takers respond to test tasks - response strategies cannot be assumed from item type

Ensure that each test form includes a variety of task types that are likely to require both expeditious and careful reading and that involve both global and local information processing

Give close attention, in trialling, to the possibility of guessing correct answers, particularly to Yes/No/Not Given items

Impose time constraints on part of the test to encourage the use of the expeditious reading strategies necessary for university study

References

• Afflerbach, P, and Johnston, P, 1984, ‘On the use of verbal reports in reading research’, Journal of Reading Behaviour, vol 16, no 4, pp 307-321
• Alderson, J C, 1990a, ‘Testing reading comprehension skills (Part One)’, Reading in a Foreign Language vol 6, no 2, pp 425-438
• Alderson, J C, 1990b, ‘Testing reading comprehension skills: getting students to talk about taking a reading test (Part Two)’, Reading in a Foreign Language vol 7, no 1, pp 465-503
• Alderson, J C, 2000, Assessing Reading, Cambridge University Press, Cambridge
• Alderson, J C, 2005, Diagnosing foreign language proficiency: the interface between learning and assessment, Continuum, London
• Alderson, J C, and Lukmani, Y, 1989, ‘Cognition and Reading: cognitive levels as embodied in test questions’, Reading in a Foreign Language vol 5, no 2, pp 253-270
• Anderson, N, Bachman, L, Perkins, K, and Cohen, A, 1991, ‘An exploratory study into the construct validity of a reading comprehension test: triangulation of data sources,’ Language Testing 8 (1), 41-66
• Bachman, L, Kunnan, A, Vanniarajan, S, and Lynch, B, 1988, ‘Task and ability analysis as a basis for examining content and construct comparability in two EFL proficiency test batteries’, Language Testing, vol 5, pp 128-159
• Block, E, 1986, ‘The comprehension strategies of second language readers’, TESOL Quarterly vol 20, no 3, pp 463-94
• Carver, R, 1992, ‘Reading Rate: theory, research and practical implications’, Journal of Reading vol 36, no 2, pp 84-95
• Cohen, A, 1984, ‘On taking language tests: What the students report’, Language Testing, vol 1, pp 70-81
• Cohen, A, 1986, ‘Mentalistic measures in reading strategy research: Some recent findings’, The ESP Journal, vol 5, no 2, pp 131-145
• Cohen, A, 2006/7, ‘The Coming of Age of Research on Test-Taking Strategies’, Language Assessment Quarterly vol 3, no 4, pp 307-331
• Cordon, L, and Day, J, 1996, ‘Strategy use on standardized reading comprehension tests’, Journal of Educational Psychology vol 88, pp 288-95
• Crain-Thoreson, C, Lippman, M and McClendon-Magnuson, D, 1997, ‘Windows on comprehension: reading comprehension processes as revealed by two think-aloud procedures’, Journal of Educational Psychology, vol 89, no 4, pp 579-591
• Ericsson, K, and Simon, H, 1993, Protocol Analysis: verbal report as data, MIT Press, Cambridge, MA
• Faerch, C and G, Kasper (eds) , 1987, Introspection in second language research, Multilingual Matters, Clevedon
• Fieldmann, U, and Stemmer, B, 1987, ‘Thin Aloud A Retrospective Da___ in C-Te___ Taking: Diffe Languages Diffe Learners- Sa Approaches?’, in C Faerch and G Kasper, eds
• Field, J, 2004, Psycholinguistics: the key concepts, Routledge, London
• Gass,S M, & Mackey, A, 2000, Simulated Recall Methodology in Second Language Research, Lawrence Erlbaum, Mahwah, N J
• Green, A, 1998, Verbal protocol analysis in language testing research, Cambridge University Press, Cambridge
• Grotjahn, R, 1987, ‘On the methodological basis of introspective methods’, in C Faerch and G Kasper (eds)
• Hawkey, R, 2006, Impact theory and practice: Studies of the IELTS test and Progetto Lingue 2000, Cambridge University Press, Cambridge
• Hoover, W A., and Tunmer, W E, 1993, ‘The components of reading’ in G B Thompson, W E Tunmer and T. Nicholson, eds, Reading acquisition processes WBC Print Ltd, Great Britain, pp 1-19
• Hosenfield, C, 1977, ‘A preliminary investigation of the reading strategies of successful and nonsuccessful second language learners,’ System, vol 5, no 2, pp 110-123
• Khalifa, H and Weir, C J, (forthcoming) Examining reading studies in language testing, Cambridge, UCLES/Cambridge University Press, Cambridge
• Kong, D, 1996, ‘An empirical study of the strategies adopted by EFL readers while reading EAP materials for main ideas-implications for teaching and testing reading’, unpublished MA TEFL Dissertation, CALS, University of Reading
• Lumley, T, 1993, ‘The notion of sub-skills in reading comprehension tests: An EAP example’, Language Testing, vol 10, pp 211-234
• Nevo, N, 1989, ‘Test-taking strategies on a multiple-choice test of reading comprehension’, Language Testing vol 6, no 2, pp 199-215
• Perkins, K, 1992, ‘The effect of passage and topical structure types on ESL reading comprehension difficulty’, Language Testing vol 9, no 2, pp 163-172
• Phakiti, A, 2003, ‘A closer look at the relationship of cognitive and metacognitive strategy use to EFL reading achievement test performance’, Language Testing, vol 20, no 1, pp 26-56
• Pressley, M, and Afflerbach, P, 1995, Verbal protocols of reading: the nature of constructively responsive reading, Hillsdale, NJ, Lawrence Erlbaum
• Rayner, K, and Pollatsek, A, 1989, The psychology of reading, Prentice Hall, Englewood Cliffs, NJ
• Stratman, J, and Hamp-Lyons, L, 1994, ‘Reactivity in concurrent think-aloud protocols’, in Speaking about Writing: Reflections on Research Methodology, ed P Smagorinsky, Sage, CA, pp 89-112
• Teasdale, A, 1989, ‘Introspection and judgemental approaches to content validation: a study using the test in English for educational purposes’, unpublished MA Dissertation, University of Reading
• Thorndike, E, 1917, ’Reading as reasoning: a study of mistakes in paragraph reading’, Journal of Educational Psychology vol 8, pp 323-32
• Urquhart, A, and Weir, C J, 1998, Reading in a second language: process, product and practice, Longman, London
• Vipond, D, 1980 ‘Micro- and macro-processes in text comprehension’ Journal of Verbal Learning and Verbal Behaviour, vol 19, pp 276-296
• Weakley, S, 1993, ‘Procedures in the content validation of an EAP proficiency test of reading comprehension’, unpublished MA TEFL Dissertation, CALS, University of Reading
• Weir, C J, 2005, Language testing and validation: an evidence based approach, Palgrave, Basingstoke
• Weir, C J, Hawkey, R, Green , A, Devi, S, and Unaldi, A, (this volume) ‘The relationship between the academic reading construct as measured by IELTS and the reading experiences of students in the first year of their courses at a British university’, IELTS Research Project Report
• Weir, C J, Yang, H, and Jin, Y, 2000, An empirical investigation of the componentiality of L2 reading in English for Academic Purposes, Studies in Language Testing 12, UCLES / Cambridge University Press, Cambridge,
• Weir, C J and Porter, D, 1994, ‘The multi-divisibility or unitary nature of reading: the language tester between Scylla and Charybdis’, Reading in a Foreign Language, vol 10, no 2, pp 1-19.

APPENDIX A: INSTRUCTIONS TO PARTICIPANTS

Instructions

You will have 30 minutes to do the test and fill out the questionnaire.

Please answer the test questions on the ANSWER sheet provided. After answering each question, please fill out the QUESTIONNAIRE for that question.

Questionnaire Section 1

In this section of the questionnaire, please describe what you did before you read the test questions.

For example, if you read the text or part of it slowly and carefully before reading questions 1 to 6 of the test, you should tick the box on the right like this:

Questionnaire Section 2

After answering each question on the test, please turn immediately to the questionnaire and tick the sentences (1 to 12) that describe what you did when you answered the test question. Then go on to the next test question, and repeat the same procedure until you have answered all the questions.

For example, immediately after answering question 1 if you matched words that appeared in the question with exactly the same words in the text, you would tick sentence 1 under Q1. If you also worked out the meaning of a difficult word in the text, you would also tick sentence 6:

Sentences 13 to 17 are about how you found the answer to each question. If you found the answer within a single sentence, you would tick sentence 13.

Thank you very much for your cooperation.

APPENDIX B: EXAMPLE ANSWER PAPER

Name: …

Age: … Gender: □ Male □ Female

First Language: …

Nationality: …

Date of most recent IELTS test: …

IELTS Reading score: …

Subject you intend to study at university: …

APPENDIX C: EXAMPLE PARTICIPANT RETROSPECTION FORM

Name: …

Section 1

Tick the sentence that best describes what you did.

Questionnaire Section 2

Tick any sentence that describes what you did when you answered each question on the test. You may tick more than one sentence for each question on the test.

APPENDIX D: TEXT PREVIEW, TEST RESPONSE STRATEGY USE AND LOCATING INFORMATION BY TEST SECTION

E1.2 Type 1: 4 Option Multiple Choice Text Preview

Response Strategy

Locating Information

E2.1 Type 1: 4 Option Multiple Choice Text Preview

Response Strategy

Locating Information

E3.1 Type 3: Sentence Completion Text Preview

Response Strategy

Locating Information

PR1 read the text or part of it slowly and carefully PR2 read the text or part of it quickly and selectively to get a general idea of what it was about PR3 did not read the text. ST1 match words that appeared in the question with exactly the same words in the text ST2 quickly match words that appeared in the question with similar or related words in the text ST3 look for parts of the text that the writer indicates to be important ST4 read key parts of the text such as the introduction and conclusion ST5 work out the meaning of a difficult word in the question ST6 work out the meaning of a difficult word in the text ST7 use my knowledge of vocabulary ST8 use my knowledge of grammar ST9 read the text or part of it slowly and carefully ST10 read relevant parts of the text again ST11 use my knowledge of how texts like this are organised ST12 connect information from the text with knowledge I already have L1 within a single sentence L2 by putting information together across sentences L3 by understanding how information in the whole text fits together L4 without reading the text L5 could not answer the question

E1.3 Type 4: Summary Completion - select from a list answer Text Preview Response Strategy Locating Information Test E1.3

Test E1.3

Test E1.3

F1.2 Type 4: Summary Completion - select words from the passage Response Strategy Locating Information

Text Preview Test F1.2

Test F1.2

Test F1.2

F3.1 Type 6: Choosing Headings (for Paragraphs or Selections of a Text) Response Strategy Locating Information

Text Preview Test F3.1

Test F3.1

Test F3.1

PR1 read the text or part of it slowly and carefully PR2 read the text or part of it quickly and selectively to get a general idea of what it was about PR3 did not read the text. ST1 match words that appeared in the question with exactly the same words in the text ST2 quickly match words that appeared in the question with similar or related words in the text ST3 look for parts of the text that the writer indicates to be important ST4 read key parts of the text such as the introduction and conclusion ST5 work out the meaning of a difficult word in the question ST6 work out the meaning of a difficult word in the text

ST7 use my knowledge of vocabulary ST8 use my knowledge of grammar ST9 read the text or part of it slowly and carefully ST10 read relevant parts of the text again ST11 use my knowledge of how texts like this are organised ST12 connect information from the text with knowledge I already have L1 within a single sentence L2 by putting information together across sentences L3 by understanding how information in the whole text fits together L4 without reading the text L5 could not answer the question

E2.4 Type 6: Choosing a Heading (for Whole Text) Text Preview Response Strategy Locating Information

Response Strategy Locating Information

F2.3 Type 7: Locating Information Text Preview

Response Strategy Locating Information

F2.1 Type 7: Locating Information Text Preview

PR1 read the text or part of it slowly and carefully PR2 read the text or part of it quickly and selectively to get a general idea of what it was about PR3 did not read the text. ST1 match words that appeared in the question with exactly the same words in the text ST2 quickly match words that appeared in the question with similar or related words in the text ST3 look for parts of the text that the writer indicates to be important ST4 read key parts of the text such as the introduction and conclusion ST5 work out the meaning of a difficult word in the question ST6 work out the meaning of a difficult word in the text

ST7 use my knowledge of vocabulary ST8 use my knowledge of grammar ST9 read the text or part of it slowly and carefully ST10 read relevant parts of the text again ST11 use my knowledge of how texts like this are organised ST12 connect information from the text with knowledge I already have L1 within a single sentence L2 by putting information together across sentences L3 by understanding how information in the whole text fits together L4 without reading the text L5 could not answer the question

E1.1 Type 8: Yes/No/Not Given Text Preview Response Strategy Locating Information Test E1.1

Test E1.1

Test E1.1

Response Strategy Locating Information

E2.2 Type 8: Yes/No/Not Given Text Preview Test E2.2

Test E2.2

Test E2.2

Response Strategy Locating Information

F1.1 Type 8: Yes/No/Not Given Text Preview Test F1.1

Test F1.1

Test F1.1

PR1 read the text or part of it slowly and carefully PR2 read the text or part of it quickly and selectively to get a general idea of what it was about PR3 did not read the text. ST1 match words that appeared in the question with exactly the same words in the text ST2 quickly match words that appeared in the question with similar or related words in the text ST3 look for parts of the text that the writer indicates to be important ST4 read key parts of the text such as the introduction and conclusion ST5 work out the meaning of a difficult word in the question ST6 work out the meaning of a difficult word in the text ST7 use my knowledge of vocabulary ST8 use my knowledge of grammar ST9 read the text or part of it slowly and carefully ST10 read relevant parts of the text again ST11 use my knowledge of how texts like this are organised ST12 connect information from the text with knowledge I already have L1 within a single sentence L2 by putting information together across sentences L3 by understanding how information in the whole text fits together L4 without reading the text L5 could not answer the question

E2.2 Type 8: Yes/No/Not Given Text Preview

Response Strategy

Locating Information

E3.2 Type 10: Matching Text Preview

Response Strategy

Locating Information

F3.2 Type 10: Matching Text Preview

Response Strategy

Locating Information

PR1 read the text or part of it slowly and carefully PR2 read the text or part of it quickly and selectively to get a general idea of what it was about PR3 did not read the text. ST1 match words that appeared in the question with exactly the same words in the text ST2 quickly match words that appeared in the question with similar or related words in the text ST3 look for parts of the text that the writer indicates to be important ST4 read key parts of the text such as the introduction and conclusion ST5 work out the meaning of a difficult word in the question ST6 work out the meaning of a difficult word in the text

ST7 use my knowledge of vocabulary ST8 use my knowledge of grammar ST9 read the text or part of it slowly and carefully ST10 read relevant parts of the text again ST11 use my knowledge of how texts like this are organised ST12 connect information from the text with knowledge I already have L1 within a single sentence L2 by putting information together across sentences L3 by understanding how information in the whole text fits together L4 without reading the text L5 could not answer the question

Author biodata

Cyril Weir

Professor Cyril Weir has a PhD in language testing, is the author of Communicative Language Testing and Understanding and Developing Language Tests and is the co-author of Examining Writing, Evaluation in ELT and Reading in a Second Language. In 2005 he published Language Testing and Validation: an evidence-based approach. Taught short courses and carried out consultancies in language testing, evaluation and curriculum renewal in over fifty countries world-wide. Current interests include academic literacy and test validation.

Roger Hawkey

Dr Roger Hawkey has a PhD in language education and assessment, is the author of two recent language test-related books, Impact Theory and Practice: Studies of the IELTS test and Progetto Lingue 2000 (2006) and A Modular Approach to Testing English Language Skills (2004). English language teaching, program design and management posts and consultancy at secondary, teacher training and university levels, in Africa and Asia, Europe and Latin America. Research interests include: language testing, evaluation and impact study; social, cognitive and affective factors in language learning.

Anthony Green

Dr Anthony Green has a PhD in language assessment. Is the author of IELTS Washback in Context (Cambridge University Press) and has published in a number of international peer reviewed journals including Language Testing, Assessment in Education, Language Assessment Quarterly and Assessing Writing. Has extensive experience as an ELT teacher and assessor, contributing to test development, administration and validation projects around the world. Previously worked as Cambridge ESOL Validation Officer with responsibility for IELTS and participated as a researcher in IELTS funded projects in 2000/1, 2001/2 and 2005/6. Current research interests include testing academic literacy and test impact.

Sarojani Devi

A postgraduate student at the University of Bedfordshire, currently investigating the academic reading of first-year undergraduates at a British university.