The relationship between the academic reading construct as measured by IELTS and the reading experiences of students in their first year of study at a British university

Abstract

This study investigates the academic reading activities and problems of students in their first-year of study at a British University, and compares the emerging model of academic reading with an analysis of the reading construct as tested in the IELTS Reading Module. The contextual parameters of the reading texts of target students are reviewed and a comparison made with those performance conditions obtaining for reading activities in the IELTS test. The extent to which any problems in reading might decrease the higher the IELTS reading band score obtained before entry is investigated.

Author biodata

Cyril Weir

Professor Cyril Weir has a PhD in language testing, is the author of Communicative Language Testing and Understanding and Developing Language Tests and is the co-author of Examining Writing, Evaluation in ELT and Reading in a Second Language. In 2005 he published Language Testing and Validation: an evidence-based approach. Taught short courses and carried out consultancies in language testing, evaluation and curriculum renewal in over fifty countries world-wide. Current interests include academic literacy and test validation.

Roger Hawkey

Dr Roger Hawkey has a PhD in language education and assessment, is the author of two recent language test-related books, Impact Theory and Practice: Studies of the IELTS test and Progetto Lingue 2000 (2006) and A Modular Approach to Testing English Language Skills (2004). English language teaching, program design and management posts and consultancy at secondary, teacher training and university levels, in Africa and Asia, Europe and Latin America. Research interests include: language testing, evaluation and impact study; social, cognitive and affective factors in language learning.

Anthony Green

Dr Anthony Green has a PhD in language assessment. Is the author of IELTS Washback in Context (Cambridge University Press) and has published in a number of international peer reviewed journals including Language Testing, Assessment in Education, Language Assessment Quarterly and Assessing Writing. Has extensive experience as an ELT teacher and assessor, contributing to test development, administration and validation projects around the world. Previously worked as Cambridge ESOL Validation Officer with responsibility for IELTS and participated as a researcher in IELTS funded projects in 2000/1, 2001/2 and 2005/6. Current research interests include testing academic literacy and test impact.

Aylin Ünaldi

Dr Aylin Ünaldi has a PhD in Applied Linguistics-Language Testing (Bogazici University, Turkey) and an MA in Applied Linguistics (University of Reading, UK). She has experience in foreign language teaching, teacher training, test development and validation. Her research interests include language test validation, academic literacy and reading into writing as an integrated academic skill. She is currently studying academic literacy for second language learners.

Sarojani Devi

A postgraduate student at the University of Bedfordshire, currently investigating the academic reading of first-year undergraduates at a British university.

1 AIMS OF THE PROJECT

The aims of the project were:

• to establish the nature of academic reading activities performed across a range of courses at a British university with particular reference to contextual parameters and cognitive processing
• to investigate problems experienced by students with respect to these parameters
• to provide initial broad spectrum data on the relationship(s) between the IELTS Reading Module and reading in an academic context
• to determine the extent to which any problems in reading might decrease the higher the IELTS Reading band score obtained before entry

2 RATIONALE

It is critical that receiving institutions can depend on the results of language tests as valid indicators of the English language proficiency of students with respect to the academic courses they are going to follow. Hawkey (2006, p 126) finds concern from receiving institutions both with international students’ academic reading problems and with some of the ways in which reading is tested by IELTS.

In the academic context, a high premium is placed on students being able to extend their knowledge beyond what is learnt in their university classroom context. To succeed in this, students need to read to learn (McClellan 1997). They must use an appropriate combination of the skills and strategies that are required for the different purposes of reading in tertiary level study. Enright et al (2000) assert that this will involve processing beyond the level of searching for information and basic comprehension of main ideas in a text and require an understanding of how information in a text as a whole is connected, and how to integrate information from across a variety of texts for use in written assignments or exam essays. The extent to which these purposes are required in tertiary level study and the extent to which IELTS can predict any problems in fulfilling them are in need of investigation.

A review of the literature indicates that, to date, no serious studies appear to have been undertaken in which the focus is on the contextual parameters and cognitive processing involved in academic reading (see Weir 2005), and the symmetry of these with the IELTS Reading Test. In the context of linking students’ academic reading activities and problems, with the IELTS test, research into reading under the joint British Council-IDP IELTS funded research program has so far been limited. Only two of the studies since 1995 have had an exclusive focus on the IELTS Reading Module. Further research such as the present study is clearly still needed

The aim of this study is to investigate the academic reading activities and problems of students (both undergraduate and postgraduate) in their first-year of study at a British University, then to compare an emerging model of academic reading with our analysis of the reading construct as tested in the IELTS Reading Module. This survey of the theoretical and empirical research on reading will thus focus on the nature of reading comprehension, including its cognitive processes, skills and strategies, and then review various models of reading to take account of these elements. Relevant contextual factors such as the reading texts of our target students will then be discussed and a comparison made with those performance conditions obtaining for reading activities in the IELTS test.

3 READING

The traditional approach to reading adopted by psychologists, language testers and teachers, is based on a slow, careful, incremental view of reading for comprehension. In contrast to this orthodoxy, Weir (1983) provided survey data suggesting L2 readers have particular problems in expeditious reading, ie quick, selective and efficient reading (see further below) in the target language. Given the expectation that students need to understand the whole domain of knowledge covered by their degree programme, this entails processing large amounts of text (paper- and web-based) expeditiously (that is quickly, selectively and efficiently) as a precursor to the careful reading which takes place once relevant information has been located (Urquhart and Weir 1998). As Weir (1983) and Weir et al (2000) showed, careful reading ability is not sufficient in itself for academic study.

We advocate a four-cell matrix which distinguishes systematically reading level from reading type, a distinction now significant in many of the reading studies and models in the field. The matrix accounts for key areas in this review of the relevant reading literature. In its distinction between careful and expeditious reading, the issue of the range of purposes, strategies, skills and processes involved in reading is raised. Taking account of recent work in the field we include in the careful reading cell the activities of careful reading to understand the way ideas are connected in the whole of a text and the integration of information across texts for the purposes of completing written assignments and/or exam essays building on the work of Enright et al (2000). With regard to reading purpose, Jordan (1997) similarly makes the connection between academic reading and the writingbased tasks or activities, for assignments, dissertations, projects or reports, for which the reading is often a preparation.

Table 1: Types of Reading

This framework assumes a multi-componential model of reading and its assessment. In the identification of both a global and a local level at which the reading strategies, skills and processes may operate, the question of the place and role of linguistic elements associated with reading performance is raised. The extent to which the test or the reality requires students to comprehend information within and beyond the sentence is a key issue (see Alderson 2000). The themes and elements informing this matrix are pursued below.

3.1 Careful and expeditious reading: processes and problems

Careful reading is characterised as identifying lexis, understanding syntax, seeking an accurate comprehension of explicit meaning and making propositional inferences. These take place at a local or a global level, i.e. within or beyond the sentence right up to the level of the complete text.

Recent research (e.g. Cohen and Upton 2006; Rosenfield et al 2004; Hawkey 2006), as well our initial experience with our project students themselves (see pilot study questionnaire responses in Study One below), indicates that careful reading alone is an inadequate construct for the students targeted by our research.

Khalifa and Weir (forthcoming, and see their proposed model of the reading process in 3.2 below) suggest, in their review of the literature on examining reading, that the significant drawback of many process-based models of reading, as well as many of the earlier componential models of reading (Coady 1979; Bernhardt 1991) is that they are nearly all premised on a careful reading model and do not take sufficient account of the different purposes of reading. They cite Hoover and Tunmer (1993), who observed that their notion of the simple view “assumes careful comprehension: comprehension that is intended to extract complete meanings from presented material as opposed to comprehension aimed at only extracting main ideas, skimming, or searching for particular details” (p 8). They also refer to Rayner and Pollatsek (1989, p 439) who stated that for most of their account of the reading process they are focusing on the skilled, adult reader reading material of the textbook variety. They admit that careful reading models have little to tell us about how skilled readers can cope with other reading behaviors such as skimming for gist (Rayner and Pollatsek 1989, pp 477-478). Most of these reading models therefore fail to describe the processing experience of skilled readers in real life reading activities.

The actual academic reading demands faced by our target students are likely to involve expeditious as well as careful reading (see Weir 1983). Urquhart and Weir (1998) use the term “expeditious reading” to describe “how readers process texts quickly and selectively, i.e. expeditiously, to extract important information in line with intended purposes” (ibid, p 101). The construct includes a range of reading types (Urquhart and Weir, ibid), abilities (Enright et al, ibid., Cohen and Upton, ibid), micro-skills (e.g. Munby, 1978), skills (eg, Levine et al 2000), strategies (eg, Purpura 1998). These overlapping terms exemplify the “fair amount of confusion” in the literature noted by Urquhart and Weir (ibid) in the labeling, and perhaps the conceptualisation, also of elements in the reading activity. Weir et al (2000, p 19) distinguish between skills as text-driven, largely subconscious linguistic processes involved in reading, and strategies as reader-driven purposeful and conscious aspects of reading.

Expeditious reading would appear likely to include, for new university students, skimming, search reading, and scanning. Skimming is generally defined (eg, Munby 1978; Urquhart and Weir 1998; Levine et al 2000; Weir 2005) as reading to obtain the gist, general impression and/or superordinate main idea of a text. The reader asks: “What is this text as a whole about?”, while avoiding anything which looks like detail. For Urquhart and Weir (1998) the defining characteristics of skimming are (a) the reading is selective, with sections of the text either omitted or given very little attention; (b) an attempt is made to build up a macrostructure (the gist) on the basis of as few details from the text as possible. The reader is trying to reach the top level structure of a text, that is, the discourse topic.

For Urquhart and Weir (1998) search reading involves locating information on predetermined topics. The reader wants information to answer set questions or to provide data for example in completing written assignments. It differs from skimming in that the search for information is guided by predetermined topics so the reader does not necessarily have to establish a macropropositional structure for the whole of the text.

Unlike in careful reading, Urquhart and Weir (ibid) argue that in expeditious reading, the linearity of the text is not necessarily followed. The reader is sampling the text, which can be words, topic sentences or important paragraphs, to extract information on a predetermined topic in search reading or to develop a macrostructure of the whole text as in skimming. The process can be top-down when the reader is deciding how to sample the text and which part(s) of the text to be sampled; it can also be bottom-up when the reader’s attention is on the sampled part(s) of the text.

Scanning involves reading selectively, to achieve very specific reading goals, e.g. finding the number in a directory, finding a particular author’s name. The main feature of scanning is that any part of the text which does not contain the pre-selected symbol(s) is dismissed. It may involve looking for specific words/phrases, figures percentages, names, dates of particular events or specific items in an index at the local word level. Rosenshine (1980) defines it as involving recognition and matching.

The types of reading summarised in the matrix above will not necessarily be associated with particular types of text. Students may be scanning books, journals (hard copy or online), newspapers or websites, or they may be skimming them or reading them carefully according to their reading purposes, not because of the types of text concerned. Clearly, our collection of data on the nature and the problems of the academic reading activities of the students across their different courses must cover all their reading sources (see Levine et al 2000), not just hard copy.

There is evidence that L1 as well as L2 academic readers have problems (e.g. Weir 1983; Urquhart and Weir 1998). Many universities, including the university at which this study is carried out, offer support programs for both. But the research in the literature often indicates marked difference between the problems faced by L1 and L2 university students (eg, Cohen and Upton 2006; Tercanlioglu 2004). Tercanlioglu suggests that L1 students use metacognitive strategies more frequently in their academic reading where ESL students may have to spend much of their available processing capacity on decoding information. The metacognitive strategies referred to here, as in educational psychology, are strategies we exercise consciously involving the active control over the cognitive processes engaged in learning. Livingston (1997) cites planning how to approach a given learning task, monitoring our comprehension, and evaluating our progress toward the completion of a task as examples of metacognitive strategies.

3.2 Models of reading

The relevance to our study of relationships between the academic reading construct as measured by IELTS and the reading experiences of students in the first year of their courses at a British university has already involved us in a consideration of reading types, levels, strategies, skills, sub- or micro-skills, processes, needs and purposes and, now, metacognition. This suggests the need to consider models of academic reading in order to frame the study and clarify relationships between the key constructs. The right model would help identify the most appropriate combinations of processes, skills and strategies to be employed for the different types and purposes of reading to achieve effective comprehension of texts from a range of sources.

The models of reading in the literature tend to be categorised under generalised labels. A brief survey of these is helpful in informing the model to be developed in this study even though, as might be expected, the labels and constructs involved overlap and are not used consistently.

Perhaps the most fundamental consideration in the development of a model of the academic reading of new students across fields of study is the componentiality of reading. As Weir et al (2000) ask: Can reading be broken down into underlying skill or strategy components for the purposes of teaching and testing? (p. 14). The discussion above already suggests that it can, but the reading research nevertheless includes examples of what Weir and Porter (1994) refer to as ‘unitary’, bi-divisible and ‘multi-divisible’ models of the reading construct. They cite empirical studies supporting the single factor hypothesis including Lunzer et al (1979), Rosenshine (1980) and Carver (1992). Schedl et al (1996), in their TOEFL research report on the dimensionality of the TOEFL reading comprehension items, also support the existence of a general reading ability and the essential uni dimensionality of the TOEFL Reading test, although they accept that there may be a second factor relating to text content or position.

Weir et al (2000) suggest that part of the reason for the uni-componential view of the reading construct is that product-based studies of reading test scores typically use factor analysis. Factor analysis is all about reduction, and may be somewhat insensitive to subtle differences such as those across related reading skills and processes. Factor analysis may thus tend to show apparently different reading skills behaving in similar statistical ways. This may be taken to imply that there is one broad ability of reading rather than a range of skills and strategies involved in the activity. However, more process-oriented studies, as already implied above, clearly suggest the reading construct has more than one dimension. Note the bi-divisible views of reading cited in Weir et al (ibid), including Carver (1992), Guthrie and Kirsch (1987), where the two components appear to be reading competence and vocabulary, the latter rather counter-intuitively separated from the essentially uni-dimensional construct of reading competence. The Schedl et al (ibid) model of the TOEFL reading test above may also be considered bi-dimensional.

Componential models of reading with two dimensions would also, however, appear less in tune with recent applied linguistic developments than conceptual multi-dimensional models. The current focus is on defining ESOL learner and user communicative needs in the interests of transnational education and employment mobility and the consequent focus on specifying and assessing language proficiency levels foreign language (cf the Common European Framework of Reference for Languages, Council of Europe, 2001).

Nor would models with a small number of broad categories of sub-components be in accordance with current trends. Coady’s (1979) three-component (conceptual ability, language proficiency, background knowledge), and Bernhardt’s (1991) language, literacy and knowledge model are revealing, process-based and, in the case of Bernhardt, include metacognitive strategies such as goal-setting and comprehension monitoring. But in the current era, with its increasing demand for evidence-based validation of multi-skill language assessment and proficiency specifications for key stakeholders, reading skills need to be described in comprehensive, multicomponential target language domain terms

As the matrix above already suggests, with its careful and expeditious reading cells, each operationalised through a range of skills at both local and global levels, reading is indeed a complex construct. Grabe and Stoller (2002) support this view and classify reading processes into higher and lower-level processes. The lower-level processes include word recognition (lexical access), syntactic parsing, semantic proposition formation and working memory activation. The higher-level processes comprise the formation of a text model of comprehension, a situation model of reader interpretation, background knowledge use and inferencing, and executive control processes these latter appearing to be similar to metacognitive strategies.

The bottom-up vs top-down distinction in models of reading, with their implications for related approaches, are also worth brief consideration in our development of an appropriate model for university student reading and its assessment. Bottom-up models tend to operate in terms of a hierarchical written text, from grapho-phonic, phonemic, syllabic, morphemic, word, to sentence levels right through to text level. Readers are assumed first to process “the smallest linguistic unit, gradually compiling the smaller units to decipher and comprehend the higher units (eg sentence syntax).” (Dechant 1991). Top-down processing involves the general and domain specific knowledge that readers can employ to predict text meaning and sentences and words within a text (see Bernhardt 1991).

There are also hybrid reading models combining the reasonable insights of both the bottom-up and top-down models. The interactive reading model (eg, McCormick 1988), developed further by Kintsch (2004) in his construction-integration model of text comprehension, emphasises the reader-driven, purposeful and conscious aspects of reading noted above (and in Weir et al 2000). Further acknowledgement of the reader role in reading is provided in the interactive-compensatory model of Stanovich (2000), which suggests that a specific weakness of a reader in a particular skill may be made up for by strengths in others.

Our early pilot questionnaire to some of the student population from which our final samples will be drawn suggests that the students themselves appear to see their own academic reading as multi-dimensional (see Study One below). Findings indicate that a key problem is to cope with the heavy reading load, under time pressure. The students accept that the appropriate reading processes, strategies and skills are important, and have interesting ideas about what good academic reading may involve, although there is not much evidence of systematic application of optimal strategies and skills.

From the evidence of theoretical and empirical research involving models of reading, and given the needs of our study, it is likely that the appropriate model developed will be a multi-dimensional dynamic model of reading, taking account as far as possible of global and local levels of reading as well as the metacognitive strategies, the skills and the processes involved in understanding texts from various sources, for various purposes.

The model of reading developed will, as suggested above, also have to take account of the model represented by the IELTS Test Reading Module. The IELTS Handbook for 2005, although it is, like the latest IELTS Website, somewhat short on construct specification, appears to imply a multi-dimensional model of reading even though a single band score is awarded for reading. Under the task types listed as used in the Reading Module, are those that require test-takers to complete notes, summaries, and a range of iconic presentations (diagrams, flow-charts, tables) using what they have read. They are also expected to identify information in the text, identify writers views or claims, summarise paragraphs or text sections. A variety of text sources are used in the test including magazines, journals, books. One may infer that, though test users have only a single reading module band score on which to make judgments on candidates’ reading proficiency, a range of reading skills have been measured.

Alderson (2000) proposes that part of the problem in actual testing practice is that numerous reading skills probably exist, but are difficult to test separately. Weir & Porter (1994, p 7) take a different view and state that “a growing body of literature suggests that it is possible with clear specification of terms and appropriate methodology for testers to reach closer agreement on what skills are being tested”. The body of literature the authors referred to includes Bachman et al (1988), Teasdale (1989), Lumley (1993), Weakley (1993) and Buck and Tatsuoka (1998). Khalifa and Weir (forthcoming) point out that in the recent DIALANG project (see Alderson 2005); individual items are now also viewed by Alderson and his colleagues as being associated with identifiable skills.

Alderson’s (2000) earlier reservations not withstanding, Koda (2005) feels that the successful identification of specific components that contribute to reading ability is an important paradigm in the current reading research literature. A componential approach based squarely on a sound theory of processing can be useful in that it provides insight into potential components in reading ability which require our attention if we are to approximate to a valid construct of reading in our reading tests.

Oakhill and Garnham (1988, p 48) query whether, without any theoretical grounding, the tests of these different comprehensions are of any value for diagnostic assessment. They also feel that the problem is that much of the research has focused on product rather than process in reading. Khalifa and Weir (forthcoming) similarly point out that what was largely absent in the componential approach in the past (leaving aside the later process-oriented studies) was any serious attempt to relate components to a model of reading ability. They argue that this may stem from an earlier preference for a posteriori statistical analysis of construct in the testing community as against an a priori approach concerned with both the theoretical underpinnings of a test’s construct before it is administered and its contextual validity.

Khalifa and Weir argue (op cit) that “the main criticism of the product-based, a posteriori, statistically driven approach is that it was not usually based on a sound analysis of salient cognitive processes. Furthermore, by its nature, it told us little about what is actually happening when a reader processes text. Further insight may be possible if we attempt to go deeper and examine as far as is possible the actual processing that goes on during reading activities. If we can identify skills and strategies that appear to make an important contribution to the reading process, it should be possible to test these and use the results for reporting on reading proficiency (see Urquhart and Weir 1998, Weir et al 2000, and Shiotsu 2003 for a further discussion of these issues).”

Khalifa and Weir go on to suggest that “…in our search for differentiated skills and strategies we need to turn to the theory of what it means to comprehend.” Grabe (1991) offers a list of component skills in reading on the basis of reading theories (as against an earlier reliance on armchair intuition):

automatic recognition skills (see Perfetti 1997),

vocabulary and structural knowledge (see Bachman 1990 on grammatical competence, Perfetti 1997 on syntactic parsing, and word representation knowledge),

formal discourse knowledge (see Koda 2005),

general and domain knowledge (see Carrell 1983 on formal schemata, Anderson and Pearson 1988 on content schemata, and Kintsch 1998 on domain knowledge),

identifying central ideas of a text ideas (see Oakhill and Garnham 1988 and Baumann 1986)

inferencing skills (Chikalanga 1990, 1992),

metacognitive knowledge (Urquhart and Weir 1998 and Weir et al 2000)

skills monitoring (see Carrell et al 1988).)

Perfetti (1997) adds proposition integration as part of building text comprehension and the development of “an accurate and reasonably complete text model of comprehension”. The work of Enright et al (2000) supports this. Khalifa and Weir (op cit) also point to the need to process and integrate information from several texts in a related field for many readers and suggest:

The cognitive construction of intertextuality offers a useful heuristic for looking at reading into writing at an advanced level and it extends our view of reading beyond the act of comprehension of a single passage.

Having accepted in principle the value of a componential approach, empirical enquiry into the reading activities of university students should help us better ground any argument for the cognitive validity of the tasks IELTS employs in its reading tests. By more closely relating putative skills/strategy components to a cognitive model of reading we may be able to better ground what IELTS is testing.

In this recent framework (see Figure 1 below) developed by Khalifa and Weir (forthcoming) there is a synthesis of existing views on cognitive processing that takes into account the research evidence on componentiality as well as considering the various models that have been proposed to explain reading comprehension (see above for our discussion of these). Khalifa and Weir comment that: “in the left hand column we include the metacognitive activity of a goal setter because, in deciding what type of reading to employ when faced with a text, critical decisions are taken which affect the level(s) of processing to be activated in the central core of our model. The various elements of this processing core in the middle column are thus initiated in accordance with decisions taken in the goal setter. The components of the knowledge base required for text comprehension are included in the right hand column”

This literature review of theoretical and empirical research on reading framed our study of relationships between the reading experiences of students in the first year of their courses at the University of Bedfordshire and academic reading as measured by the IELTS test. The view of reading arising from this work is mirrored in the model developed separately by Khalifa and Weir for the Cambridge ESOL constructs volume, (Khalifa and Weir, forthcoming). This model of processing at various levels in L2 together with our literature review and the data from our open ended pilot questionnaire (see Appendix 2 and Section 5 below) proved useful in the development of our main study questionnaire on student reading activities and on the problems students encounter in their academic reading. The questionnaire provides the main database in this study.

Figure 1 A model of reading (Khalifa and Weir forthcoming)

3.3 Context validity

A central assumption in Weir’s (2005) test validation model is that cognitive processing always occurs within and is significantly affected by a context. Weir’s context validity relates the features of the task to the language in the text that must be processed if the task is to be completed successfully.

If test task performance is to be used to support inferences about performance in the wider domain of real-world tasks it is essential that both target reading activities and test tasks be described in terms both of cognitive processes and of contextual parameters.

Similarly, Bachman and Palmer (1996) argue that situational and interactional authenticities are essential features of useful test tasks. These factors in judging a test’s usefulness (concerning the extent to which test tasks reflect salient contextual parameters and cognitive processes engaged by test-takers) are analogous to Weir’s contextual and cognitive validities. It is widely accepted that, given the constraints imposed by testing conditions on contextual features (such as the time available to complete a task), full situational authenticity is generally unrealistic for language assessments. However, contextual features of a test ought to reflect as many of the relevant features of the target reading activity as possible. The literature on the textual parameters that are potential sources of text complexity is daunting and we will only scratch the surface of it here before identifying those parameters that appear to be both useful and applicable for our study.

Bachman et al’s (1988, 1995) test comparison studies involve such textual properties as the nature of text, length, vocabulary, grammar, cohesion, distribution of new information, type of information, topic of discourse, rhetorical organisation and illocutionary acts. Freedle and Kostin (1993, see also Freedle 1997), in a detailed analysis of reading comprehension item difficulty, take into consideration vocabulary, concreteness/abstractness, subject matter, coherence, length of various segments such as word, sentence, paragraphs as text related variables. Fortus et al (1998) investigated length, number of negations, number of referential markers, vocabulary, grammatical complexity, abstractness, topic, rhetorical structure as textual variables contributing to the level of difficulty of reading comprehension items. Enright et al (2000) identify two groups of salient textual features to operationalise in test texts: grammatical/discourse features and pragmatic/rhetorical features. Alderson et al (2004) include text source, authenticity, discourse type, domain, topic, nature of content, text length, vocabulary and grammar as relevant features for text analysis. Khalifa and Weir (forthcoming) suggest that linguistic demands of task input - reading texts in this case - can be explained in terms of lexical and structural resources, discourse mode, functional resources, content knowledge and writer-reader relationships.

The text linguistics literature on complexity also identifies certain factors as important contributors to the level of difficulty, or ease with which a text can be processed and offers certain methodologies for evaluating this. Readability formulae such as Dale and Chall and Flesch (in Gervasi and Ambriola 2002) involve the calculation of word and sentence length and identification of specific vocabulary. Although in several studies readability formulae are criticised as being inadequate to reveal textual complexity (see, for example, Masi 2002 and Gervasi and Ambriola 2002), they still form the basic aspects in more recent and detailed analyses of textual complexity. Masi (2002) suggests that, together with linguistic and quantitative factors of word and sentence complexity, other semantic and syntactic factors such as structural embedding, content, background knowledge of the reader, the type and genre of text, should also be taken into account to reveal a more reliable and predictive measure of text complexity. The latter, however, are hardly measurable by automatic procedures such as computerised calculation.

From the picture emerging above, it was possible to identify a group of criterial features that suggested themselves as useful for the analysis of texts for testing purposes. We were careful to select features which facilitated as quick and unambiguous an analysis as possible, i.e., they could be practically used by judges in a short time but with a high consensus. The ‘contextual parameters analysis scheme’ developed for and used in this project (see Figure 2 below) thus involves the parameters listed in Figure 2 below, where we draw on contextual parameters most likely to have an impact on reading test performance.

CONTEXT VALIDITY

Figure 2: Context validity parameters addressed in this study

Using this framework as our informing source, we will explore salient parameters of context validity in terms of task setting and task linguistic demands (input and output)

Alderson (1996) and Nuttall (1996) have argued that a long text is required for candidates to skim for main ideas, scan for specific information, make relevance judgements and distinguish between main points and subsidiary details. As Alderson et al (2004) have pointed out in relation to the Council of Europe Framework of Reference, distinctions between long and short texts are generally inexplicit, nor is not clear how long a text or what time constraints would need to be imposed to reflect successful skimming, scanning and relevance judgements in academic reading.

It may also be that shorter texts, of the kind used in IELTS, may encourage candidates to engage word-level rather than text-level operations since the memory load involved in processing relatively short texts makes this feasible, although time constraints might serve to offset this effect. This will be investigated in the studies below.

A number of researchers and commentators (Perera 1984; Urquhart 1984; Weir 1993; Alderson 1996; Nuttall 1996; Shiotsu 2003) have identified potential sources of difficulty arising from the linguistic elements in a text. They suggest that structural, lexical, and conceptual difficulty strongly influence the ease with which a text can be read.

The emergence of computer-assisted analysis of extensive language corpora has facilitated the use of word lists to inform language test development and validation. Of particular value to IELTS are academic word lists that identify words used more commonly in academic than in other contexts, particularly the sub-technical vocabulary that occurs across disciplines (Campion and Elley 1971; Coxhead 2000). It would be encouraging to see that IELTS texts reflect the occurrence of such words in academic texts sourced from the university.

Work undertaken by Alderson and Clapham (1992) at the inception of IELTS pointed to a very close relationship between a test of grammar and the IELTS reading component. Indeed the relationship was so close that a decision was taken to eliminate the grammar test from the IELTS battery. Similarly, Shiotsu (2003) explored components likely to affect reading test performance for Japanese undergraduates and found that syntactic knowledge played a central role. Shiotsu and Weir (2007), using structural equation modelling, demonstrated the relative importance of syntactic over lexical knowledge in accounting for variance in tests of reading with candidates from a variety of language backgrounds.

Texts with less complex grammar tend on the whole to be easier than texts with more complex grammar Berman (1984) investigated how opacity and heaviness of sentence structures could result in increased difficulty in processing. Again, this suggests that a valid test of academic reading should reflect the syntactic features likely to be encountered in academic texts.

The effect of the use of cohesive devices on comprehension is less clear-cut. While Alderson (2000) notes that an absence of cohesive devices does not seriously damage comprehension when the topic is relatively familiar to readers, it has been argued that explicit cohesive devices help in establishing textual coherence (Goldman & Rakestraw 2000) and that their lack inhibits the recall of texts, being indicative of a less successful mental representation (Ehrlich 1991).

Urquhart (1984) and Barnett (1989) suggest that rhetorical features should be considered together with sentence-level features in estimating text difficulty and this view is supported by available research. Studies investigating the effects of textual organisation on text difficulty (see for example Meyer & Freedle 1984; Carrell 1984; Goh 1990) suggest that differences in rhetorical organisation do affect processing. All three studies found that problem-solution, comparison, and causation structures resulted in better recall than classification or description structures. Koda (2005) cites a number of studies reporting the positive effects of improving text structure and the benefits of explicit training in coherence on comprehension and memory. Freedle (1997) finds that texts subjectively judged to be high in coherence yield easier main idea reading comprehension items. Rhetorical features should therefore be a further consideration in the selection of texts for tests of academic reading.

Genre is explained by Weigle (2002, p 62) as the expected form and communicative function of the written product. Genre is generally understood to encompass ‘salient features and conventions which are shaped by communicative purposes’ (Hyland 2002, p 62). It is evident from the literature that specific genres will involve specific conventional features (lexico-grammatical, semantic, and discoursal) which are likely to impinge on the text processing of readers (Bhatia 1997; Hyland 2000). It would seem logical to suggest that if texts to appear in a test are sourced from academic contexts they are likely to share lexical, syntactic and discourse features with texts encountered at a university.

The following genres, identified through the development of the student questionnaire (see Studies 1 and 2 below), are seen as relevant to the present analysis:

text book

magazine and newspaper article

research/academic journal article

report.

Rhetorical task refers to ‘the primary intent of the author’ that guides the reader in understanding the text (Enright et al 2000 p 20). Enright et al (2000) suggest a three-way classification of rhetorical tasks (which they term ‘pragmatic features’):

Exposition informs the reader. It may involve descriptions, comparisons, contrasts, explanations and elaborations.

Argumentation/persuasion/evaluation supports a point of view with reasons, evidence and analysis of an opponent’s errors in reasoning. Vocabulary might reflect attitude or perspective and it may be personal in tone. It differs from a balanced, unbiased stance.

Historical biographical/autobiographical narrative tells a story with a defined setting and episodes.

Pattern of exposition refers to ‘subcategories of exposition’ (Weigle 2002 p 62), or a specific pattern a writer employs to communicate. Although a single text may include a number of rhetorical moves, it is the overall theme or main point that is targeted through this feature (Enright et al 2000, p 23). The following patterns are suggested as being worthy of investigation in the literature:

Definition/description/elaboration, providing full definitions of concepts, describe unfamiliar terminology, elaborate on terms specific to the discipline and clarify specific uses of the terminology

Illustration involves providing examples or a short anecdote to fully describe an abstract concept.

Classification involves grouping several items together according to similar features or principles, showing how discrete items belong to a larger group.

Comparison/Contrast involves designating distinctions among concepts, particularly regarding their similarity and dissimilarity.

Cause and Effect involves analysing causes and effects in relation to an overall point.

Problem/Solution involves describing a problem or a series of problems then proposing a solution, which will have a plausible, salutary effect on a course of action.

Justify as used here is similar to the category of Analysis used by Enright et al (2000). Texts in this category provide evidence to justify a point of view.

Rhetorical Organisation refers to the extent to which there is an explicit pattern of topic progression through the text. Such progression might be signalled by headings, topic sentences and discourse markers.

The contextual parameter of content knowledge in the socio-cognitive framework proposed by Weir (2005) and shown in Figure 2 above suggests that the relationship between the candidate’s pre-existing knowledge and the propositional content of a text will affect the way it is processed. Nuttall (1996) puts forward the widely held view that, all else being equal, the greater a reader’s knowledge of the topic of a text, the easier it should be to process. This has been an area of debate for IELTS since its inception as the five academic subject specific modules inherited from the ELTS test were reduced first, in 1989, to three and finally, in 1995, to one.

The decision to abandon subject specific modules was taken on the grounds that there was only very limited evidence that it had any effect on text difficulty. Tan (1990) and Clapham (1996) both investigated the effect of content familiarity on candidates’ performance without finding significant effects on test scores (although Clapham does note an effect for the most specific texts in her corpus). However, Khalifa (1997) made the contrary finding that familiarity with the topic of text can be a good predictor of difficulty. Alderson (2000) also acknowledges the facilitating effect of familiarity with the subject matter and Urquhart and Weir (1998) warn against the danger of using insufficiently specialised texts. It has been suggested that, in order to minimise effects of topic familiarity, test tasks should be based on materials sourced from a variety of academic subject areas (Enright et al 2000).

The concern with the contextual parameter of nature of information is with the extent to which the information in a text is concrete (i.e. concerning observable, concrete phenomena) or abstract i.e. (concerning unobservable phenomena such as social institutions) or, at a higher level of abstraction, metaphenomenal (Moore and Morton, 2000) (concerning theoretical treatment of abstract phenomena). Different levels of abstraction may, of course, be found within a single text.

Alderson et al (2004, p 127) see this as a useful feature to consider in estimating text difficulty in relation to the Common European Framework of Reference. Information that is more abstract may prove to be more difficult to process and so divert cognitive resources from language processing. At the same time abstract information often implies a linguistic complexity that may further stretch the L2 reader’s resources.

Studies such as Steffenson, Joag-Dev and Anderson (1979), Chihara et al (1989), Al-Fallay (1994) and Sasaki (2000) have provided evidence that cultural knowledge plays an important role in text comprehension. In these studies, certain ‘key’ words - proper nouns, words describing institutions and words that reflected unfamiliar cultural practices - were changed into words that would be more familiar for the participants. For example, in Chihara’s (1989) and Sasaki’s (2000) studies, which used the same texts, Joe was changed to Hiroshi, state to preflecture and a mother hugged rather than kissed her son because these changes were felt to reflect a Japanese rather than an American cultural context for the narrative. The resulting texts, because they appeared more familiar to the participants, resulted in higher scores on a cloze test based on the passage. In this study the judges were asked to look for words that might be associated with a specific culture, including references to;

names for specific people, places and products (Harvey et al; Rice Krispies; the city of Chicago).

specific historical events or periods (the Norman Conquest, football-related violence in the 1970s);

local institutions (the probation service, the House of Lords);

locally familiar objects (breakfast cereals; sharp suits)

locally situated social practices (window shopping; children in the classroom undertaking problem solving activities in pairs)

idiomatic language including culturally specific references (milestone research; professional soap boxes)

The approach currently taken by the IELTS designers is to avoid content that is dependent on knowledge eithe of specific discipline areas or of particular cultures. Item writers are advised to reject texts that might be too technical for the general reader or that assume knowledge specific to certain cultures.

With regard to the contextual parameter of reader-writer relationship, Nystrand (1989) states that meaning is created between the participants of a discourse and resides in the expectations and assumptions of both the reader and the writer of each other. Writing, rather than being an isolated individual action, involves the endeavours of both the reader and the writer and is shaped through mutual assumptions involved in the understanding of rhetorical situations (Hyland 2002, p 35). Any act of writing is charged with assumptions about the participant relationships and how these are carried out in culturally and institutionally legitimate ways (ibid p 69). Hyland (2002, p 72) states that “Managing social relationships, then, is crucial in writing as a text communicates effectively only when the writer has correctly assessed both the readers’ resources for interpreting it and likely response to it. This is, in part, achieved through the use of metadiscourse.”

Metadiscourse is a term which describes a range of lexical items (words and expressions) whose main function is to enhance communicative efficiency in two main ways: by streamlining the inference process involved in figuring out the relation between parts of the text and the context (including the co-text) and by establishing and managing the rapport between the communicator and the audience.

Discourse relationships are shaped by the writers’ choice of specific rhetorical devices. As one important element of reading texts, reader-writer relationship in both undergraduate and IELTS reading texts will be analysed in Section 9 according to the type and frequency of metadiscoursal features listed by Hyland (2005).

Our review of the literature relevant to our research project has covered key insights into reading processes, skills and strategies and reviewed approaches to the modelling of reading to take account of these, in the context of the reading needs, purposes and problems of our target students. The contextual validity of reading test has been discussed with reference to task setting and task linguistic demands ,with particular reference to linguistic, discoursal and rhetorical features. A key focus of the review has been to the way that the reading module in the IELTS test addresses the testing of academic reading and we will analyse further the nature of the IELTS reading model in our continuing study of contextual and cognitive parameters below.

The main study questionnaire items we developed for the next stage of our project were principally concerned with investigating the degree to which the students perceived themselves as carrying out the variety of

operations in reading suggested by our literature review and a pilot open-ended survey (see Appendix 1), and the problems they encountered with these and a variety of contextual parameters.

The main survey is an attempt to establish the components of reading in an academic context and to identify particular operations and performance conditions where students have problems. These data will enable us to examine IELTS to see the extent to which these components are covered and in particular the extent to which attention in IELTS is paid to the problematic operations and conditions.

Our data for this comparison will be firmly rooted in the theoretical literature but also the construct will have been further grounded in relation to responses of real students studying at one English university.

4 RESEARCH METHODOLOGY

The study employs a mixture of qualitative and qualitative methods as appropriate.

Critical review of documentation and published literature relating to the nature of reading in an academic context and the problems encountered by overseas students in coping with this

Self report by students on

• the cognitive processing and performance conditions encountered in academic reading

• the difficulties occasioned by these

Analysis of level of problems experienced in reading reported by students (i.e. with various activities and performance conditions) in relation to level of IELTS test performance

Investigation of 42 IELTS testlets (14 complete reading tests in all) through application by expert judges of a descriptive framework of expeditious and careful reading strategies to each item in each testlet

Investigation of 42 IELTS test texts (14 complete reading tests in all) and 14 extracts from core undergraduate textbooks through application by expert judges of a descriptive framework of textua parameters to each text

Further details of particular methodologies employed will be provided in respect of each of the studies reported below.

5 STUDY 1: OPEN-ENDED PILOT QUESTIONNAIRE ON ACADEMIC READING ACTIVITIES

5.1 Introduction

The pilot questionnaire (see Appendix 1) was trialed in several iterations. It asked for biodata, then responses to 13 open-ended questions, and in its final form was administered in April 2006, and elicited 77 responses. The questionnaire data analysed here were intended to inform further data collection approaches and content, in particular the IELTS Academic Reading Project online structured survey administered between 1 June and 31 October 2006.

The sample for the piloting operation was obtained from a range of student types (undergraduate/postgraduate, 1st /2nd/3rd year and home /international students). Despite its opportunistic nature interesting pilot data emerged on the reading sources, purposes, strategies experiences and difficulties of University of Bedfordshire students from a range of backgrounds and fields of study. Lessons were learnt for the content, design, wording and administration of the Academic Reading Project online structured questionnaire which is the focus of section 6 of this report.

Given the richness of the responses to the open-ended questionnaire, they were analysed qualitatively using key word and topic counts, with direct quotation indicated appropriately. The summaries are in terms of descriptive statistics.

Key points from the analysis of all pilot questionnaire responses are now summarised.

5.2 Reading source types

Responses confirm that books remain the key source of students’ academic reading, but with journals also prominent and a fair number of students doing around half their academic reading online. The pilot study students offer insightful comparisons between book and online sources of information indicating, for example, that:

books offer a wider range of sources and more to understand

print sources may provide deeper information

print materials tend to be first choice

online (OL) sources may be for interest but not suitable for assignments

OL reading complements, follows up print reading

The Web with its wide range of information, can offer explanations, clarifications, of questions raised from reading of books

useful and convenient to have some journals online, but often limited access

prefer to print out OL information (2), less comfortable reading from screen

“don’t use OL so much because can’t scribble, highlight, take notes so conveniently”;

online sources less reliable, credible than books, journals

The pilot questionnaire data provide further evidence that assignment reading is a multi-source task. 34% of our 65 responses on the item claimed as many as 10-19 sources for an assignment, fairly evenly divided between books, journal articles and websites, although books were more often the main source of reading than the other two.

It is appreciated, of course, that decisions on what to read are not always the students’ own. Table 2 summarises responses to the item on how the pilot sample students decide what to read for their courses.

Table 2: Student decision influences for their reading

Most of the sample students (77%) did not distinguish between their approaches to reading from different information sources. In the 17 responses who did distinguish, however, the following points were made.

5.3 Reading approaches

Close to the heart of the research question of our Project are the students’ responses to the instruction When you have decided what to read, describe how you read it. Most responses indicate strategies to identify reading focus according to assignment and the materials needed. A key word count from the 78 responses to this item indicates the following approach foci:

Table 3: Strategies adopted to read assignment information

We note that students do not always appear to distinguish between “scan”, meaning to locate specific information, topic, point, and “skim” meaning read for gist, general impression, both strategies presumably needed in reading for assignments. Typical sequences of reading action associated with student assignments were:

“Run through to see if suitable for me,(then) read properly”, read cited parts again

“Read only sub-titles and main issues”

“headings, bullet points then, if they are worthy, read the whole article”

read relevant sections + skim others

read chosen area, sub-heads, relevant information

skim and use index

Note here the students’ reference mainly to expeditious rather than careful reading. These responses, central to our research, inform and are pursued in, the online questionnaire

Most of the sample students (77%) did not distinguish between their approaches to reading from differen information sources. In the 17 responses which did distinguish, however, the following points were made:

Reading books:

locate relevant material using title (2), index (2), chapter titles (2), summary

Reading articles:

since articles are shorter, browse-reading most relevant sections

check abstracts, introduction, discussion

articles are harder

highlight (2) then go back over (2)

Reading books and articles:

read books more thoroughly (2), articles selectively

read whole articles (2), books selectively (2)

skim books, but articles require more thought, processing

reading approach depends on prior knowledge rather than materials type

Again some of these approaches are the focus of further attention in the online questionnaire.

Of the 70 responses to the question on whether students’ reading approaches were the same when reading for assignments as for examinations, 34 (48%) said yes. Respondents claiming different approaches suggested the following, many of the strategies and problems specified being relevant to our research focus on reading purposes, strategies and difficulties, and to our focus on IELTS as an exam in its own right. We note reference by the pilot study students here to both expeditious and careful reading.

Reading for exams:

“exams require triggers with which you apply theory to questions”

not everything selected for exams is in books, easy to find most you need on Net

reading for exams requires more depth (4), more thorough readings (2) critical evidential approach (2), more breadth or topic coverage (6), less detail (2); more specificity, detail (2); more reading to clarify problem topics (2)

reading for exams is already identified, involves only reading specified chapters and handouts, preparing and memorising (2) for predicted exam questions, essays (4)

more hard work, tension with exams;

in exams read everything through, then start exam

Reading for assignments:

assignment reading takes more time, read twice (2), use more sources (3)

more variety, less detail (3), more detail (1), more specific (1), more general (3) , more depth so more time to master

need to read and make notes from selected relevant material (5)

need to read in order to apply examples

need to skim.

5.4 Reading problems

The open-ended pilot questionnaire pursued further the question of difficulties experienced by the students in their academic reading. The 58% of the pilot sample responding identified the problems summarised in Table 4.

Table 4: Reading difficulties

Very closely related are the pressures identified by 61% of the sample with regard to the academic reading, as shown in Table 5:

Table 5: Reading pressures

5.5 Perceptions of successful reading

A revealing and related question was What do you think a successful reader is at university? The response data were informative for our focus on perceptions of academic reading. 81 features were identified in the 72 responses, as summarised in Table 6.

Table 6: Successful reader characteristics

Interesting verbatim responses included these:

“Someone who understands what’s what and achieves most of the reading suggested and completes some off their own bat”

“selective reader using appropriate techniques with the context and time framework”

“has no difficulty reading books, articles, not only course books but other fields, can skim text and know gist, figure out context and meaning without looking up words”

“enjoys and engages in study reading on a regular basis”

“can organise reading, understand, to represent in their minds corresponding to author’s”

Students’ perceptions of what successful readers do are clearly relevant to our analysis of the reading needs of first-year university students and how these are assessed. The leads provided here by the pilot study respondents inform the main questionnaire in Study 2 below.

5.6 Some general conclusions from the open-ended pilot questionnaire

Reading emerges from the pilot study data as indeed a concern for the students, including the EL1 majority. Their major problem is coping with the heavy reading load, under time pressure. This being so, appropriate reading processes, strategies and skills are important, and accepted as such. The students have some good ideas about what good academic reading should involve. However, there is not a great deal of evidence of systematic application of optimal strategies and skills by the students themselves.

The rationale for the research is supported by responses to the open-ended pilot questionnaire. In spite of the use of an opportunity sample, the data sought and received usefully informed further stages in the IELTS Academic Reading Project. The responses of the students together with our findings from the literature review informed both the content and the wording of the descriptive categories in the final structured questionnaire (see Study 2 below).

6 STUDY 2: MAIN QUESTIONNAIRE SURVEY

Following the analysis of the open-ended pilot questionnaire, and a number of further small-scale piloting stages, the final version of a structured questionnaire was distributed to home and international, undergraduate and postgraduate students at the University of Bedfordshire in May/June 2006 and was also available on the web. When students returned in September 2006, we targeted (electronically through Blackboard) former year 1 students now entering the second year of their courses, especially in those subject areas with low returns so far. For our purposes these were considered students in their first year of study.

The student population was thus sampled opportunistically. The Project team had neither the authority nor the resources to design and implement a stratified random sample. Nor, as indicated above, would the purpose of the study have been served by an experimental research design, with its characteristic control of variables and establishment of experimental and control groups, for before and after measurement of isolated variables. In the sections below, however, the nature of the sample will be described, and the responses of student sub-groups categories within that sample subjected to descriptive and some inferential statistical analysis to contextualize and justify claims made.

Over the period from the June 1 2006 launch of the online and paper-and-pencil versions of the questionnaire, and the closing date of 1 October 2006, 434 students responded on-line and 332 in hard copy format. This high total respondent figure of 766 students is considered adequate for the purposes for which the questionnaire was designed, and the data elicitation methods used. The total population of the University of Luton at the time (the name of the university changed officially to University of Bedforshire in September 2006) was 16,150, including 6550 students in their first and 4400 in their second year.

As noted above and in line with the purpose of the study, the questionnaire was to be administered to students to elicit information and views on their academic reading experiences and the difficulties that they may have encountered in reading for their courses. The focus of the questionnaire was to inform a profile of the students reading experiences in terms of purposes, strategies and difficulties, so that the relationship between this profile and the academic reading construct measured by the IELTS reading module might be investigated. The questionnaire is included as Appendix 2 of this Report.

The survey includes both home and overseas students, undergraduates and postgraduates and students in their second and first year of study at the University across a range of fields of study. Year 2 students were included as a check on whether things altered much in subsequent university study. Responses on key variables are cross-tabulated where such data may inform answers to the main questions this study seeks to answer. Examples of this will be seen in the analyses below, for example, of the reading problems of English as an alternative language (EAL) and English as a first language (EL1) students, or between Year 1 and Year 2 students.

6.1 Gender, age, regional background

Table 7 summarises basic information on the Study 2 survey student sample in terms of gender, age and regional background.

Table 7: Gender, age and regional distribution of the questionnaire respondents

There are significantly more female students in the sample than male, with a 70% to 30 % split. The Higher Education Statistics Agency (HESA) statistics for UK universities in the academic year 2003/04 also notes a preponderance of female students; approximately 58% of undergraduate students then were women, while 42 per cent were men. In our sample, the gender distribution across the Year 1 and the Year 2 students is similar, the Year 1 group showing a 69.9 to 30.1% distribution, Year 2, 71.2% to 28.8%. The table reveals a good spread of participants across the four age groups, with the younger group (age 18-22) predominating, as desired in a study with a focus on first-year students, but with useful sub-samples also in the three broader senior ranges (23-29, 30-39 and 40+). The age ranges for the Year 1 and Year 2 students in the questionnaire sample are almost identical.

The sample population includes British as well as non-British students, in line with the point made in the literature review above, that academic reading problems affect both groups. Of the 62.5% of the questionnaire respondents who are not of British nationality, around 72% are from non-European backgrounds, 28% from Europe. This again compares reasonably well with the HESA figures for non-British UK university students, 64% of whom were from non-European countries, 36.4 from Europe.

6.2 Academic stage

As Table 8 indicates, 84.4% of the Study 2 sample are undergraduates, across years one to three of their studies.

Table 8: Level and stage of studies of the questionnaire respondents

Most (67.7%) of the questionnaire respondents are in Year 1, which is the main focus of our study, but over 30% are in their second year. Comparisons between year one students and their colleagues in year two may provide interesting eventual insights into changes in aspects of their academic reading as time passes, as their experience grows and as the nature of the reading they are called upon to do alters (see further below).

6.3 English language status (gender, regional background, academic stage)

The language background variable is, as indicated in our project aims and the literature review above, a key focus of this research. In the Study 2 questionnaire sample population, no fewer than 43 languages are represented, English (38.9%) and Chinese (38.4%) being easily the most numerous, with European languages other than English also prominent(14.2%). The first and second year student groups are well matched proportionately across first languages, dominated by EL1 (year 1 and year 2 at 36.5% and 44.8% respectively) and Chinese L1 (36.5% and 33.5%).

Of the EAL students, 310 (66.5%) are from outside the UK, 130 (27.9%) from Europe, and 26 (5.6%) from UK. Among the 298 students in the EL1 group, 261 (87.6%) are UK nationals, five European and 32 (10.7%) from outside the UK and Europe.

We might normally have expected more of the EAL students, most of who are from overseas, to have come to Britain for postgraduate studies. But we have already seen that in our sample EAL population a high proportion, 66.2% (n = 310) are between 18 and 22 years old. In fact, a fairly similar proportion of the EAL and the EL1 sub groups, 81.9% and 88.2% respectively, are studying here at undergraduate level. Analysis of the year of study category across our EAL and EL1 groups shows high proportions of students (70.1% and 64% respectively) in their first year of study at the University. Comparisons between our undergraduate and our postgraduate sub-groups are made below where relevant to our main research questions, especially with regard to academic reading sources, purposes strategies and difficulties.

6.4 Subject areas

There is a broad coverage of subject areas across the student sample, as may be seen in Table 9, which again compares the EAL and EL1 sub-groups. The main subjects of the EAL students are Business and Finance; Advertising, Marketing and Public Relations; and Language, Communication EFL and TEFL, each of these subject areas being pursued by more than 15% of the group. The main subjects of our EL1 students are Educations studies, with 22.4% of the sub-group, Social Sciences and Social Work (14.2%) and Sport and Exercise Science (10.5%). Table 9 analyses the subject areas represented across our population sample, with particular reference to the EAL and EL1 sub-groups.

Table 9: Subjects studied by the EAL and EL1 questionnaires respondent groups

The subject areas are fairly evenly shared across the first and second year groups, apart from somewhat larger Year 1 groups in advertising, marketing and PR and HRM, and slightly more prominent Year 2 groups in language and communication (including EFL and TEFL) and Media Arts.

The 766 students in our questionnaire sample are pursuing in the main a similar range of subject areas to those of the overall UK student population as analysed in HESA figures for 2005. These indicate main sub-groups of 22% studying medical and related subjects, 17% education, language and communication; 13% business and administration; 11% maths and the physical and biological sciences; 14% engineering and technology, including computer science (6%).

6.5 IELTS Test Reading Module scores

Figure 3 summarises the scores awarded to the 301 of our sample population who had taken the IELTS test:

Figure 3: IELTS Reading Test scores of the main questionnaire respondents

The analysis shows that the mode band on the IELTS Reading Module was 6.0, the band score awarded to 41.2% of our sample. The reported overall global average IELTS test Academic Reading Module band score for 2002 (ibid. 7) was 5.79. The IELTS Annual Report (e.g. 2003) notes a “minimum of Band 6 in each module” as a common university entrance cut-off band. An IELTS overall band score of 6.0 is, according to the same source, a common English language entrance qualification for undergraduate studies. Note that the IELTS descriptor for Band 6 specifies the following “Competent User” profile: A “generally effective command of the language despite some inaccuracies, inappropriacies and misunderstandings. Can use and understand fairly complex language, particularly in familiar situations”. This characterisation of Band 6 English language competence will be borne in mind when we probe further the reading approaches and problems of our student sample population.

The students scoring 5 or 5.5 on IELTS in our sample were in a Foundation programme for overseas students, which prepares them for entry to certificated degree courses.

The next most frequent band score in our sample was 6.5 (21.6% of the IELTS-taking respondents), closely followed by the 5.5 band (18.9%). The 6.5 average band is often cited as appropriate for entrance to graduate courses, though there is considerable variation in IELTS cut-off bands across different universities, fields of study and levels (IELTS, 2005. Our main questionnaire respondent sample appears to be reasonably near to norm as regards IELTS band scores.

In terms of student year of study, the key central IELTS reading band scores of our Year 1 and 2 respondents are not significantly different, as Figure 4 indicates. Once again, it will be noted, the 6.0 reading band score predominates, with 6.5 and 5.5 the next most common, respectively.

Figure 4: Year 1 and Year 2 IELTS Reading Test bandscores

The IELTS reading score averages of our undergraduate and postgraduate sub-groups were 6.14 and 6.19 respectively.

6.6 The questionnaire Likert scale Items

The analyses of our questionnaire from here on are based on responses to Likert scale items, using a 5-choice degree of agreement format, including one “neutral” option. The scale throughout is: 5 definitely agree, 4 mostly agree, 3 neither agree nor disagree, 2 mostly disagree, 1 definitely disagree. Using the mode (i.e. the most common selection made by respondents) is the most logical way to indicate the response tendency on each item. Adding the number of definitely agree and mostly agree selections is also an appropriate way of indicating the strength of respondents’ agreement or disagreement with items. Differences between groups are investigated through analysis of variance.

Sources of academic reading information

Section 2 of the questionnaire elicits information from the students on the range of information sources on their courses. The responses here are again analysed for the EAL and the EL1 sub-groups. Table 10 summarises student responses on the relative importance of books, journal articles, reports, the Internet, newspapers and magazines in their academic reading. “D” in this and subsequent tables, represents the selection of the definitely agree category on the Likert scale, “D & M”, the sum of respondents’ definitely agree and mainly agree selections. Rank orders (r/o) of the reading agreement strengths are added in parentheses.

Table 10: Sources of information across EAL and EL1 questionnaire respondent groups

Broadly speaking, and in terms both of measurement of responses by the definitely agree choice only, and by the sum of the two positive agreement categories, the order of importance of sources is: 1) books 2) Internet sites 3) journals, 4) reports, 5) newspapers and 6) magazines. Informed by the pilot questionnaire (Study 2 above) a related item on the main questionnaire asks respondents how much reading they actually do online compared with paper print materials. Table 11 summarizes responses for both EAL and EL1 participants.

Amount of Reading done online

Table 11: EAL and EL1 group online reading source proportions

The message of the table here is that the EAL students appear to do rather more of their reading online than do their EL1 colleagues. The mode value for the former group is the 30.3% who do from 41-60% of their reading online, compared with 23.8% of the latter, 30.7% of whom read 0-20% online.

Cross-tabulated reading source data suggest close agreement between first and second year students, except, perhaps, for almost 11% fewer among the Year 2 sub-group definitely agreeing that Internet sources are important on their courses. Table 12 here summarises the perceptions on reading sources across the two sub-groups.

Table 12: Year 1 and Year 2 group online reading source proportions

The range of academic reading sources and the prominent role played by Internet sites in the academic reading of contemporary university students have clear implications for pre-university reading tests such as IELTS. This is a message to which we return later in this Report. In terms of substantive differences of perceptions between our under- and post -graduate sub-samples, it is of interest (as well as and intuitively credible) to note that a high 83.5% or our graduate sub-sample definitely or mostly agreed on the importance of journal articles on their course, compared with 70.7% of our undergraduate sub-group. Similar was the substantially higher proportion of the postgraduate group agreeing the importance of reports on their courses (75.6% of the graduate sample compared with 56.8% of the undergraduates).

6.7 Students’ course reading purposes and how they read for their assignments

We now move into a key area of our research into the reading experiences of students in the first year of their courses at a British university, namely the purposes of their reading, in particular how they read for their assignments.

Again, we shall be looking also for potential differences between EAL and EL1 students, this time with regard to their perceived reading purposes on their courses. If there are differences in agreement across the two categories of student, the implication could be that EAL and EL1 students, because perhaps of their different levels of TL reading proficiency, set out with different purposes as they tackle the reading tasks required by their academic studies. If there appears to be no significant difference in perceived reading purposes across the two sub-groups, it may be inferred that students read with similar purposes whether their language status is EAL or EL1. As indicated above, more significant variables across reading purposes may be the field and/or the year of study.

The rubric for Section 3 of the Study 2 questionnaire is: The following purposes for reading are important on my course. This is followed by four statements of reading purposes which may be characterised, in the terms emerging from our literature review (above), as strategic, global and expeditious reading oriented. Table 13 summarises student responses to the items, again in terms of the EAL and EL1 sub-groups and using the same analytic categories as Table 11 above. The numbering of the reading purposes in the table is that of the original questionnaire (see Appendix 2).

Table 13: Responses on the importance of reading purposes across EAL and EL1 groups

Analysing the Likert scale responses across the 468 EAL students and the 298 EL1 students, the main inference is that all four purposes are important to the students, in mainly similar rank orders. Searching texts for required information appears to be the most important reading purpose on the whole for the students, while the greatest difference between the groups (p<.01) appears to be that it is the EL1 group who emphasise more strongly the importance of the reading purpose basic comprehension of main ideas. There are clear implications here for the testing of reading, on which light may be shed in the accounts in this report of Study 3 on the cognitive parameters in IELTS and Study 4 on IELTS contextual parameters.

Table 14 now presents a re-analysis of the same data on reading purpose, this time to investigate whether there is significant variation across the Year 1 and Year 2 student groups.

Table 14: Responses on the importance of reading purposes across Year 1 and Year 2 groups

Searching texts for required information is again a key reading purpose across the year groups, with the other specified purposes also agreed as important, with no real difference of perception across the Year 1 and Year 2 students. Nor was there substantive difference between our post- and under-graduate sub-samples in terms of the main purposes for their academic reading, all of which were again agreed as important by both groups.

6.8 Reading for assignments

Section 4 of the questionnaire, under the rubric How I read for assignments, includes 16 items, as identified in Table 15 below. The table again summarises responses in terms of strength of agreement with each item across the EAL and EL1 groups, average rank orders, with the final two columns making Year 1: Year 2 comparisons. The table contains a considerable quantity of data which should thus inform the questions our study is asking in order to establish the nature of academic reading activities across a range of courses with particular reference to cognitive processing, contextual parameters and the problems experienced by students with respect to these parameters. There are lessons to be learnt here for the valid design of reading tests for potential university students.

The items in Section 4 are all strategies for academic reading. They include conscious actions by the reader taken before the reading begins (e.g. I think carefully to ensure that I know exactly what I will be looking for before I start reading), and while it is taking place (e.g. While reading I try to relate content to what I know already and judge its value). Most of the strategies specified would seem to relate to expeditious reading at the global level, e.g. I quickly look through the whole of the text for a general understanding ….., I think of key words and quickly look for them or words with similar meanings to check if a text is worth reading; I read critically to establish and evaluate the author’s position on a particular topic). Some of the strategies specified, however, cover careful reading approaches at the local level (ibid.), e.g. I read a text slowly all the way through ….; or at the global level, e.g. I try to understand how the text is organised, how the ideas and details connect with each other.

Table 15: Responses on ways of reading for assignments across EAL and EL1 and Year 1, Year 2 sub-groups

Evidence from these analyses is that the following reading strategies are the most strongly agreed as applied to their academic studies by the questionnaire respondent sample, with differences between the EAL and the EL1, Year 1 and Year 2 sub-groups as stated.

The strategy: remembering where relevant information is or marking its location for later use in writing my assignment, is definitely or mostly agreed by 77.1% of the EAL group and 86% of the EL subgroups, 85.4% or the Year 2 group and 79.2 of the Year 1 students; this strategy has the highest mean rank order of all the strategies specified, although only just higher than:

I think carefully to make sure I know exactly what I’ll be looking for before I start reading, is definitely or mostly agreed by 77.4% of the EAL group and 80.1% of the EL1 subgroups. 80.5% and 76.1% of the Year 1 and Year 2 sub-groups respectively

I think carefully of key words and quickly look for them or words with similar meanings to check if text is worth reading more carefully is definitely or mostly agreed by 70.6% of the EAL group and 81% of the EL1 subgroups, 70.6% and 68% of the Year 1 and 2 groups respectively

I look at the titles or headings of text before deciding to read it quickly is definitely or mostly agreed by 82.1% of the EAL group and 73.3% of the EL1 subgroups. While this strategy receives a strong endorsement from both sub groups, it is more strongly agreed with by the EAL sub-group [p <.01], who actually rate it their most strongly agreed strategy; there also appears to be a difference between the Year 1 group’s top ranking of this strategy and the Year 2 students’ fourth ranking of the strategy (p <.01).

I make notes on relevant points from the text as I go along is also a strategy receiving strong endorsement; as the fifth highest rank-ordered on average, it is definitely or mostly agreed by 66.4% of the EAL group and 78.7% by the EL1 subgroup [significance of difference p < .01]; both the Year 1 and Year 2 groups also see this strategy as important.

A striking feature of the most strongly agreed academic reading strategies here is that all five are clearly dual-oriented strategies in the sense that the reader is pre-specifying or identifying concepts or information for future use as well as current understanding. Such strategies are also, of course, related more closely to expeditious than to careful reading, and are global rather than local in the sense that they appear to involve searching a whole text for relevance to an actual assignment task.

If we also consider academic reading strategies which were not particularly strongly endorsed, we find a tendency for these to be rather less dual-oriented and expeditious-reading related. The following strategies were given the five weakest agreements, in terms of their rank orders

I read a text slowly all the way through even if some parts do not seem relevant to my assignment. This clear example of careful reading is the lowest rated and ranked by all groups (EAL, EL1, Year 1 and Year 2) and as measured by all definitely and mainly agree measures.

I read critically to establish and evaluate the author’s position on a particular topic. This strategy is perhaps, more surprisingly, also low-ranked by all groups. Is this an indication of a lack of requirement or inclination to read critically for writer stance?

I first get an overall meaning of text e.g. by reading the first paragraph and conclusion and first sentence of other paragraphs. This strategy is again ranked low (r/o 13 - 15 across all positive ratings) by both language background and Year sub-groups (although more than 50% of them rate the strategy as definitely or mainly agreed).

I gradually understand what a text is about by reading the sentences slowly and carefully in the order they occur: this strategy is low-ranked (r/o10-15, positive rating % 56.7 and 52.9 for the EAL and EL1 groups respectively, 54.8% for Year 1 students, 56.4% for Year 2); notice how the strategy is rather less weakly rated than the read a text slowly all the way through even if some parts do not seem relevant to my assignment strategy above, both involving slow reading, but the earlier and less agreed strategy not excluding the reading of irrelevant text parts.

I try to understand how the text is organized: how the ideas and details connect with each other; this strategy, similarly ranked to the previous one, may be a semi-unconscious skill rather than a strategy, perhaps therefore under-rated by the students concerned (see below).

The cross-tabulation of the undergraduate: post-graduate sub-groups in terms of assignment reading strategies suggest that the two groups do not differ in many of these. The main exception appears to be looking quickly through the whole text for a general understanding before doing anything else (apparently a stronger tendency for the graduates than the undergraduates ; definitely’ and ‘mostly agree’ percentages 80.7% and 69.7% respectively).

There are certainly interesting issues here in terms of the kinds of reading strategies appropriate for the training and assessment of pre-university students. These are discussed further below.

6.9 Student difficulties encountered when reading for assignments

Now the focus of our questionnaire data moves to the difficulties encountered by the target students when reading for their assignments. Overall, as indicated in Table 16 below, it is the similarities rather than the differences across the EAL and EL1 sub-groups that strike one. Equally closely matched, as also analysed in Table 16, are the reading problems across Year 1 and Year 2 students.

Analysing the Likert scale responses, whether judged in terms of the proportions of definitely agree percentages or the combined definitely and mostly agree categories across the 468 EAL students and the 298 EL1 students, the main difficulties experienced, in rank order, appear to be with:

reading texts where the subject matter is complicated

the time available to do the necessary reading

reading lengthy texts

finding relevant information quickly.

The time and reading load problem, it will be recalled, was already signposted by the pilot questionnaire in Study 1. Also connected with the problem of time, is the difficulty of finding relevant information quickly, which is rated fourth across all categories (i.e. EAL, EL1, definitely agree selections and definitely and mostly agree selections combined. A slight difference is with the EAL group percentage on this difficulty, but their definitely and mostly agree count still ranks it the sixth most prominent difficulty of the 17 problems specified.

The evidence is strong from Table 16 here that time and complicated reading subject matter are the major problems for both EAL and EL1 groups. The two items referring directly to these problems are the highest rated and ranked for difficulty for both EAL and EL1 groups, the EL1 group finding significantly more difficulty (p<.01) with complicated subject matter suggesting that first language status is not necessarily the major issue when the study subject itself is complicated. Both groups, however, agree that lack of background knowledge to understand the reading content is a fairly major problem (41.8% of the EAL group and 35.5% of the EL1 group respectively definitely or mostly agreeing on this, their 7th and fifth most highly ranked difficulty respectively). Related, most likely, to the problem of time is the need to find relevant information quickly, a significant problem for 57% and 59% of the EAL and EL1 groups respectively.

Summarizing ideas from a text in my own words, a skill which, of course, integrates reading with writing and speaking, is rated as definitely or mostly an agreed difficulty by 56.3% and 44.7% of the EAL and EL1 groups respectively the difference in rating strength significant at p <.05.

Potential reading-related difficulties not so highly rated or ranked (though still affecting around 40% of our EAL population) will also inform our analysis below of the appropriateness of IELTS reading tasks. Table 16 suggests the following as among such categories:

relating the content of a text to my existing knowledge: ranked the least or next least of the difficulties of both EAL and EL1 groups

making notes on information I will need: low-ranked as a reading-related skill by both groups, only 26.1% and 19.9% respectively of the EAL and EL1 groups rating it as a definitely or mostly agreed difficulty in reading for assignments.

understanding the text as a whole; how main ideas and details are connected to each other and integrating information from text I am reading with info from other texts I have read: the next lowest ranked difficulties, both with very similar ratings.

The final two columns in Table 16 permit inferences from the Year 1: Year 2 variable on the matter of student problems in academic reading. Here, as in Table 15, Year 1 and Year 2 student responses are compared in terms of percentages of definitely agree plus mostly agree responses and in terms of rank orders of the 17 reading difficulties concerned, these, of course, derived from the Project literature review (see Section 3 above) and the pilot questionnaire open-ended data (Section 5). As signaled above, it is the similarity of the perceptions of the reading problems of the first and second year students which is notable. This suggests that these problems do not disappear with exposure to a wide range of reading in the first year of study.

Table 16: Responses on difficulties encountered with reading for assignments across EAL and EL1 AND YEAR 1, Year 2 sub- groups

Once again, as inferred from the EAL : EL1 group difficulty comparisons above, the same four main problems are identified, in the same rank order by the Year 1 and the Year 2 sub-groups, namely:

reading texts where the subject matter is complicated

the time available to do the necessary reading

reading lengthy texts

finding relevant information quickly.

Note that, once more, the evidence is strong that students find their time inadequate to handle the problems of finding relevant information quickly from long and complicated texts. The factors and tasks here are closely and explicitly related to the expeditious reading construct established in the literature review above, and pursued as a major focus of Study 3 on the cognitive parameters of the IELTS Reading Test and Study 4 on the test’s contextual parameters.

Next ranked by the Year 1 and 2 sub-groups, both EAL and EL1, is the difficulty of words I do not know, though there is an indication (not statistically significant) that this becomes marginally less of a problem in the second year. Such is not, apparently, the case with lack of background knowledge to understand reading content, still a major problem in Year 2, it seems.

There is also evidence in Table 16 that reading-related activities not so highly rated or ranked are shared across the Year 1 and 2 as well as the EAL and EL1 student groups. The following figure as such less problematic academic reading activities:

relating the content of a text to my existing knowledge: ranked the least or next least of the difficulties by both Year groups as they were by EAL and EL1 groups

making notes on information I will need: low-ranked as a reading-related skill by both groups, at only 21.8% and 27.8% respectively of the Year 1 and Year 2 groups rating it as a definitely or mostly agreed difficulty in reading for assignments; the difference between the Year 1 and Year 2 percentages here is significant, however, at p<.01, suggesting that this problem is somewhat more severe for students in their second year

understanding the text as a whole; how main ideas and details are connected to each other and integrating information from text I am reading with info from other texts I have read are again, for the Year 1 and Year 2, the next lowest ranked difficulties, both with very similar ratings, but for both first and second year groups only slightly lower rated in terms of difficulty than reading carefully to understand the main ideas.

The most substantial difference among the academic reading problems between our undergraduate and postgraduate student sub-samples appears to be with difficulty in finding relevant information quickly, a problem for 45.3% of the former group, 31.7% of the latter.

Our findings on the difficulties encountered by students when reading for their assignments must surely be of interest to the designers of tests such as IELTS, which set out to measure and claim as valid indicators of English language proficiency, the IELTS scores of international students seeking to study at British and other EL1 universities. This message is pursued in the next section of our Study 2 report here.

6.10 IELTS Reading Test scores and Student Reading

Given that this research project is examining the relationship between the academic reading construct as measured by IELTS and the reading experiences of students in their study at a British university, indications from the questionnaire data of relationships between higher and lower scoring IELTS test-taker groups and their reading strategies or problems should certainly be of interest. If, for example, significant differences are found between the lower- and the higher-scoring IELTS groups and their responses on a particular academic reading problem, predictive validity could be inferred for the IELTS Reading Test.

Descriptive statistics were run for the 301 IELTS reading test-taker scores in our questionnaire sample, to establish the extent to which there were significant differences across the highest-scoring group (IELTS Reading bandscore 6.5 and above) and the mid- and lower-scoring groups (6.0, and 5.5 or less), data and discussion on these band scores in Figure 4 and accompanying comment above. Table 17 summarises one-way ANOVA results for cases where the differences between high, lower and lowest IELTS-score groups and particular reading problems as identified on the questionnaire, were found to be significant (p<.01).

Noteworthy in Table 17 are the following significant (p<.01).differences, each indicating a relationship between IELTS reading score and a perceived problem with academic reading at the University of Bedfordshire:

between the highest (IELTS 6.5+) and lowest (5.5 or less) groups for difficulties:

• with the time available to do the necessary reading

• understanding a detailed logical argument

between the two higher groups (6.0 and 6.5+) with difficulties:

• reading for basic comprehension of the main ideas in a text

• making notes on relevant points from the text “as I go along”

• lengthy texts

between the highest group (6.5+) and both lower groups (6.0 and 5.5-) for difficulties with:

• unknown words

• making notes on information I will need

• reading carefully to understand the main ideas

• relating the content of a text to existing knowledge

• integrating information from the text I am reading with information from other texts I have already read and between the lowest (5.5-) and middle group(6.0) on:

• gradually understanding what a text is about by reading the sentences slowly and carefully in the order they occur

The inference here is that the IELTS test has, for these takers, predicted significant differences in some of the reading problems faced by the students in our sample with their actual academic reading experience at university. If we look more closely at the problems concerned here, we may note that the problems concerned involve both careful reading (e.g. reading the sentences slowly and carefully in the order they occur) and expeditious reading (e.g. relating the content of a text to existing knowledge), and with both cognitive strategies and contextual factors (e.g. the time available to do the necessary reading).

Table 17: One-way ANOVA statistics indicating student reading problems with a significant (p<.01) difference between students with higher and lower IELTS Reading scores

6.11 Overall difficulties of the four skills in university studies

The final item on the main questionnaire attempted to elicit student views on the relative difficulty for them in their studies of the four language skills. Table 18 summarises responses for EAL and EL1 sub-groups.

Table 18: EAL and EL1 student perceptions of the relative difficulties of the four skills

Of particular interest here is that a lower percentage of EAL students see reading as their most difficult of the four macro-skills, yet a third of these students regard it as their second most difficult skill. There are implications for the IELTS test, especially in the light of IELTS impact study findings (Hawkey 2006, p 122) that the “Reading module is seen as clearly the most difficult of the four IELTS test modules across our candidate and preparation teacher participants”. Is it in fact valid in terms of test construct and content that one of the four test modules should be perceived by test-takers as of a different level of difficulty than the others, if the macro-skill it is testing is not the most difficult in the target academic domain? The EL1 sub-group, it will be noted, also does not see academic reading as their most problematic, though again it is rated the second most difficult skill area

The analysis in Table 18 suggests writing as clearly the perceived most difficult academic language skill for our EAL sub-group, speaking for the EL1 students. Perhaps a psycho- and sociolinguistic factor is operating here. The EAL students may well be thinking of language proficiency problems affecting their assignment and other academic writing. The focus of the EL1 students, on the other hand, may be on the stresses of the kind of “speaking” involved when having to handle an oral presentation.

6.12 Conclusion

In this report of Study 2, we have described our survey sample for the main questionnaire in some detail, with reference across key parameters to the general university population. Data have then been presented and analysed on sources of academic reading information, student course reading purposes and strategies for assignment-related reading, then their perceptions of key academic reading problems. Relationships between the EAL students’ IELTS Reading Test scores and the cognitive parameters of reading were then explored.

The logic of the study should still be clear. Now that we know more, from Study 2, of the actual academic reading sources, strategies and problems of our fair cross-section of University of Bedfordshire students, we may investigate, in Study 3 the cognitive parameters tested by the IELTS Reading Module and, in Study 4, how the texts used in the IELTS Reading Module compare with the kinds of texts that the students actually meet once they are at university.

7 STUDY 3: COGNITIVE PARAMETERS IN IELTS: TEXTS AND TASKS IN THE IELTS ACADEMIC READING MODULE

7.1 Approach and instrumentation

Responses to the main project questionnaire to University of Bedfordshire students were helpful in establishing the nature of academic reading activities and problems across a range of courses. The next logical step in the project design was thus to examine IELTS Reading module tests to evaluate the extent to which they may actually cover reading activities and problems revealed by the student questionnaires analysed in Study 2. This was the aim of Study 3.

The instrument for the analysis of IELTS reading tests was derived from the literature review (above), in particular the reading strategies, skills and processes reported in Section 3.1 above, which discusses the processes and problems of careful and expeditious reading (derived from Urquhart and Weir ibid). Then, in December 2006, all members, staff and students, of the CRELLA Project team participated in a standardisation exercise involving the use of a draft matrix specification of expeditious and careful reading strategies, to be matched against actual IELTS Reading Test tasks. The matrix, as finalised by the standardisation exercise, appears as Table 19 here.

EWS: Explicit within sentence. Establishing basic propositional meaning at sentence level through explicitly stated ideas in the text. Basic comprehension questions are used to assess lexical, syntactic, and semantic abilities and the ability to understand important information presented in sentence- level propositions IWS: Implicit within sentence. Inferencing by creating information, which is not explicitly stated in a sentence. Understanding information in a sentence may require addressing conceptual gaps by constructing a message from both what is explicitly stated and from our stored knowledge. Such inferences are necessary for a full understanding of the sentence. EAS: Explicit across sentences. Establishing meaning through explicitly stated ideas across sentences. IBS: Implicit between sentences. Inferencing meaning which is not explicitly stated between sentences in a text. TM: A text model. Creating a text model. Constructing an organized representation of the text including main points and supporting details; an integrated understanding of how supporting ideas and factual details of the text form a coherent whole. SM: A situation model. Answering questions based on a situation model. Addressing conceptual gaps by constructing a message from both what is explicitly stated and from our stored knowledge. Building a situation model involves the reader forming a representation of the content, relating the contextual information of a text to mental models of corresponding real life situations. Table 19: Finalised reading cognitive parameter matrix and reference key for the analysis of IELTS Reading tests.

In the standardisation operation Project team members used the reading strategy descriptors in Table 19 and the draft test task : reading strategy matrix (as in Table 20 below) to describe the strategies they actually used on an authentic IELTS Reading Test. This was followed by discussion of the experience by the whole team, suggestions for revisions to the forms, and, finally, approval of the revised reading cognitive parameter matrix and reference key (Table 19) and the form in Table 20, which was to be used in phase two of Study 3.

Table 20: IELTS Reading Test task types and the reading strategies to respond to them

In this, three Project members were asked to record independently the reading strategies they employed to respond to each task on a selection of fourteen IELTS Reading Module tests. All the IELTS tests used in the analysis are authentic and now in the public domain. In the account of Study 4 below, the dimensions and other contextual parameters of the IELTS tests are analysed fully, in comparison with typical authentic texts used by students across their main fields of study. We note here merely, from the 14 complete tests selected for our analysis, that:

most complete IELTS Reading Module tests are around 15 A4 pages long

they contain three separate texts on which candidates must respond to common test tasks of the kind specified in Table 20

there are a total of 40 items in each test

each test contains an average of 3458 words to read, including the tasks and rubrics

the average number of words of reading text to read in each test is 2562 words

the average test text is 854 words long (maximum 1063 words, minimum 589 words).

As noted in the Table 20 matrix, the reading test tasks included: the matching of suggested and actual test content; the categorisation of suggested content as Yes / No (or True / False) or Not given in the test text; gap-filling; multiple-choice; table or other iconic completion, and short-answer questions.

The three IELTS Reading Test analysts were informed participants. Two, EAL users, responded to all 42 testlets qua IELTS-takers, and entered in the matrix in Table 20, the cognitive strategies they used in the process, along with any comments they felt relevant to the research purpose. The third team member, an EL1 user, covered the same tests and recorded self-report comments of the process he adopted to complete the reading test tasks. The research approach to Study 3 thus elicited both qualitative and quantitative data on how the IELTS tests concerned were approached.

7.2 Analysis and findings: qualitative

The qualitative comment made by the test-taker-analysts remind us that the very fact that we are reading as test takers may affect the strategies and skills applied to our reading tasks, this a matter to be pursued further in Study 4 below, on the contextual parameters of the IELTS test reading modules. It is appropriate to discuss our qualitative data first here as they provide a general context for the more quantitative analysis of the test-taker reading strategies subsequently analysed. The report-backs are presented verbatim, with interpretative commentary added as relevant to our research questions.

One EAL team member (test taker A below) added to his quantitative analysis of the reading strategies used to take the tests the following general description of how he approached the IELTS Reading Test tasks:

I usually read the texts carefully from the beginning to the end initially then I go to the questions.

I can answer some questions without having to read the text again. If not, I usually remember the place where the info necessary for the answer is located and go there usually by scanning which may be followed by some careful reading. That, I could not make explicit in the analysis.

In revealing that he “usually read the texts carefully from the beginning to the end initially”, the test-taker-analyst makes a point relevant both to our investigation of the cognitive and the contextual parameters of reading and its testing. We recall that university student questionnaire data in Studies 1 and 2 above, suggest strongly that, students, in their actual academic reading lives at university, do not commonly “read a text slowly all the way through even if some parts do not seem relevant to my assignment” (see Table 15 above). In fact this strategy was the lowest rated and ranked by all sub-groups (EAL, EL1, Year 1 and Year 2) responding to the main questionnaire. There thus appears to be some conflict between reading as part of student academic studies and the cognitive parameters of the IELTS Reading Test. Our reader-analyst here may be exhibiting a tendency fo some readers to employ careful reading strategies for the test texts because this a reading test task, with, they can already see, a dozen or more items on it to be answered.

A second Study 3 team member, an EL1 reader, makes the following comments on the approach adopted to the IELTS Reading Module, at least partly, it would seem, because it is a test:

Before reading any questions or, if the “Choose the correct heading for each section from the list of headings below” item is placed before the text, after reading that, I make a decision whether to read through the whole text. But I want to read through quickly so I may skip read while trying to keep the gist and main details, predicting likely key question points as well as looking for points we already know we need, to answer the matching question. Then I search for relevant parts of the passage after reading each question, trying to zero in to decide on whether the detail helps me answer the question concerned.

Note here the problem, inherent in the test format, that the reader’s decision to read expeditiously appears to be inhibited by only partly knowing in advance what information she is seeking, as she appears to apply a combination of skim, search and scan reading strategies to the text. (We return, in our quantitative analysis below, to the issue of partial overlap across the broader expeditious reading strategies).

The EL1 reader’s report back makes points of interest with regard to relationships between test task types and the cognitive parameters of the strategies used to handle them. Such report back, like the students responses to the Study 2 questionnaire items on their actual academic reading, could be helpful in the design and validation of reading tests such as IELTS.

On the IELTS matching task type (eg, Reading Passage 1 has three sections, A - C. Choose the correct headings for each section for from the list of headings below … ) , the reader-analyst report is tentative but critical. Like most report-backs, it is rather complex, but so, as we have seen in section 1 of this report, is the process of reading, as attempts to test it.

Do candidates deal with this task as skim and then select (someone else’s “gist”)? The task should involve expeditious reading, skimming i.e. The reader locates and comprehends information at the overall gist level [Table 19]. The reading should be selective, (i.e. looking for info relevant to the heading given) with sections of the text either omitted or given very little attention (i.e. skip over sentences not appearing to refer to the heading given). We should be making an attempt to build up a macrostructure (the gist) based on as few details from the text as possible.

But, in fact, given we’re looking for three sets of information, and have also to bear in mind three others (the distractors in the test task), are we really likely to be that selective in our reading? How much can we really leave out? Especially given the “tricky” questions with their deliberate overlap across the headings. They are probably “designed” to make candidates have to think “Ah, that bit’s about

disruptive effects of tourism … or is it about the expansion of tourism?”. So a likelihood of reading more carefully than in the similar real-life situation.

In terms of real-life reading purposes and strategies, we the readers know, for our own real purposes, what overall gist we are trying to skim for; or what topics we are search reading for, (in order THEN to read more carefully about); or what specific info we are scanning for (then, perhaps, to read carefully about). It is possible that we are also aware, as we are searching, skimming of scanning, of what info we do not want (like the distractor headings in the IELTS task that are not the right descriptors of the section…). But do we ever really read to select, from a surfeit of closely related topics, one rather than others that summarises a part of a text? Does the task not actually force us into careful reading, section by section, in order to be sure of selecting the right summarising heading? Presumably some candidates would:

read the headings and try to remember their “meaning”, then

read Section A; then compare what they have read with all the headings,

then select. Then repeat this process two more times. Maybe by Section C, they could expeditious read a bit because the options by then are narrower?

This report-back is presented in full because it appears to cover several key problem areas in the cognitive parameters of the IELTS Reading Test, and to offer an explanation of how a reading test like IELTS may push test-takers towards the careful rather than expeditious forms of reading.

The EL1 reader’s further report-back relating the various IELTS task types to cognitive parameters are as follows:

Yes / No (or True / False)or Not given:

I tried not to read closely, but instead to locate key words involved in the task e.g. “cost”,” deserts”, “hill”, “government”, then see if the location was right to answer the question. If so write in the Y, N, NG as appropriate. If not, search-read for another cue / key word that I had thought could be “related vocabulary in the semantic field”.

Filling in blanks/ Table completion:

Here, we need to scan to locate the people/location context, then find the right word (in the text) for the completion. I read the people/location reference and the clause for completion, then scan the text for the context, then the clause content reference, then write in the word concerned.

Multiple Choice

Here I seemed mainly to end up reading for explicit meaning, or implicit meaning within sentence (EWS or IWS in Table 19 above].

This kind of qualitative report-back data, though complex, can usefully inform our view of reading, as well as our attempts to assess proficiency in it.

7.3 Analysis and findings: quantitative

The quantitative analysis for Study 3 is based on the completion, by two informed EAL participants, of the same 14 IELTS tests, including all their 42 texts (at three texts per test), each test with a total of 40 items, and including the seven main task types specified in Table 20 above. The two test-taker-analysts thus covered 560 test items, in the solution of each of which they identified the strategy or skill they applied. As the two total numbers in the right-hand columns of Table 21 are both somewhat just above 560 (at 562 and 585 respectively for test-taker-analysts A and B respectively) the indication is that for a few items the test-takers felt that they applied more than one strategy or skill.

As implied by the qualitative analysis and findings above, we would not predict that the two EAL test-takers would each activate the same cognitive strategies to complete the same tasks. There are several reasons for this, most of these already implied by our review of the reading literature above. Test-takers’ approaches to taking reading tests clearly differ. Thus, so do the meta-cognitive strategies employed (see Section 1 above), including the combinations of skills identified by the report-backs above. What is more, as the definitions used in the Study 3 test-completion exercise (see Table 19) indicate, the three expeditious reading strategies of skimming, search-reading and scanning involve some overlapping processes and actions (see further below). Add to this the effect of individual differences (also see the literature review above) between the two test-taker analysts and considerable variations across the tasks and skills are likely.

What we might well expect from the analysis, however, would be similarities in strategy and skill use in terms of the key distinction made throughout this study, namely between expeditious (skimming, search reading, scanning) and careful reading skills.

Table 21 summarises the responses of the two EAL test-taker analysts to all the reading test tasks of the 14 authentic IELTS Reading Modules attempted, in terms of the types of reading strategies and skills they perceive that they applied.

Table 21: summary of responses of two EAL test-taker analysts to the reading test tasks of 14 authentic IELTS Reading Modules

Some general conclusions

A number of key points of interest may be inferred from the data in the table. There is indeed a difference of balance between the readers across the expeditious reading strategies of skimming, search-reading and scanning Test-taker analyst B appears to do significantly more skimming and scanning than test-taker analyst A, whereas the latter scans and search-reads with similar regularity. But, as our definitions above and their use in the qualitative report-back data both indicate, these expeditious reading strategies do share some elements, for example the aim of locating information quickly, and the likelihood of some consequent careful reading. Some blurring of the edges by readers across the three expeditious strategies is thus possible and acceptable. Furthermore we record above how test taker analyst A read through each text carefully before answering the questions and so removed any necessity subsequently for obtaining the overall gist of a passage expeditiously. When he returned to the text scanning and search reading were sufficient to locate information for more careful reading.

The most important finding here in terms of our research questions is the apparent preponderance of careful reading over expeditious reading strategies applied by both test-taker analysts, 77% of the claimed cognitive skills and strategies (883 out of the total of 1154) apparently belonging to the former category. What is more, 634 of the reading strategies applied by the two readers were apparently at the sentence level, compared with 242 strategies applied to items seen as requiring attention beyond the sentence. This indicated imbalance is a matter of potential concern given the findings of Studies 2 and 3, that the students at the University of Bedfordshire, when asked about their actual academic reading purposes and problems, saw reading activities of the expeditious kind as more appropriate to their needs than careful reading skills. The data here suggest that the reading skills and strategies tapped by the IELTS Reading Module test may need further investigation and possible modification to more closely represent the academic reading constructs of university students through texts and tasks that test more extensively students’ expeditious reading skills. The low occurrence of items testing student’s ability to process text beyond the sentence level is also a cause for some concern given the nature of the student reading abilities outlined and empirically supported earlier. The almost complete lack of items at the text level, let alone across texts, must similarly be a cause for concern.

Within the two readers’ careful reading skill use, there is evidence relevant to the test developers, on the line between information that is explicit and information that is implicit. Once again, report-back description is revealing, this time from test-taker-analyst B:

What I understand by explicit is that the answer is directly accessible from the text and may appear in paraphrased form or with synonyms of key words. I take implicit as [that] the answer is not given directly in the passage but is illustrated by the author thru’ examples or style of writing.

This definition from Reader B seems reasonable but again reflects a test-taking perspective, relating to whether an answer to a test question appears in the text as referenced in the question or in some other form. This is not quite the same as the implying defined by the Webster New World Dictionary, that is: “to indicate without saying directly”.

Be that as it may, both test-taker analysts do find cognitive strategies involving implicit meaning at the careful reading level, though in substantially fewer cases (109) than those involving explicit meaning (767), as indicated in Table 21 above.

8 STUDY 4: CONTEXTUAL PARAMETERS

8.1 Focus and methodology

As well as generating data on students’ reading activities and problems they encounter in academic reading and comparing these with the reading activities required in IELTS, we also carried out an initial investigation of the contextual parameters of 14 core undergraduate textbooks at the University of Bedfordshire and compared these parameters with those obtaining in the set of 14 IELTS reading tests supplied by Cambridge ESOL

Examinations and investigated in terms of their cognitive parameters in Study 3.

In cooperation with the University library staff we established core first-year undergraduate texts in each of the areas where large numbers (3000+) of international students are studying in Britain according to the most recent HESA student record data (2004/5). The courses taught at the University of Bedfordshire in these high density areas were as follows:

The selection of the core undergraduate texts in these areas was made on the basis of:

those books which had had the most reservations made for them in the last 3 years and in particular the current year

those books which were taken out the most in the current academic year

confirmation by course leaders of key books for each area

books students considered to be the most important (as established through the pilot questionnaire (see above) and direct inquiry).

For Study 4, forty-two samples of academic text were collected to match the 42 IELTS texts. These comprised three extracts from each of the 14 different text books - sections extracted at random from the opening chapter, the middle and the concluding chapter. These are core texts that undergraduate students are expected to get to grips with during their studies at the University of Bedfordshire. The length of extracts (targeted to be between 500 and 1,500 words) corresponded broadly to the length of the texts included in the IELTS Reading Test

The IELTS texts and the extracts from Bedfordshire academic texts were subjected to a variety of quantitative and qualitative analyses as indicated in Table 22. Measures of the quantitative features listed in Table 22 were obtained through the Web VocabProfile available at http://www.lextutor.ca/ supplemented by analysis through WordSmith Tools (Scott 2006) and text analysis tools packaged with Microsoft Word for Windows. For the qualitative analyses, two expert judges, with doctorates and experience of teaching and test development in the area of academic literacy, employed Likert scales and categorisation tools to evaluate the texts

Measures of vocabulary include word length (number of characters/ word), type-token ratio, lexical density and word frequency levels. Grammatical complexity may be estimated through word/sentence and sentence/paragraph ratios and through the proportion of passive verbs. Summaries of these features are obtained through Web VocabProfiler and text analysis summaries provided through Microsoft Word for Windows. Readability statistics (Flesch Reading Ease and Flesch-Kincaid Grade Level) are also calculated using Microsoft Word: both measures being based on the relative numbers of syllables, words and sentences found in a text.

In investigating discourse mode here we include genre (or text source), rhetorical task, pattern of exposition and rhetorical organisation.

Each judge independently assigned each text to one of the following genres, identified through the development of the student questionnaire used in study 1:

text book

magazine and newspaper article

research/academic journal article

report

The judges also identified the subject area with which each text appeared most closely associated, using the HESA classification of courses of study shown above.

Each text was classified by the two judges according to the following discoursal features discussed above:

Rhetorical task

Exposition

Argumentation/ persuasion/ evaluation

Historical biographical/ autobiographical narrative

Pattern of exposition

Definition/ description/ elaboration

Illustration

Classification

Comparison/ Contrast

Cause and Effect

Problem/ Solution

Justify

The two judges also used five-point Likert scales to make a subjective evaluation of the texts on the following features:

Rhetorical Organisation (1 explicit to 5 not explicit). This is intended to reflect the ease or difficulty with which the overall propositional pattern of the text is likely to be understood by the reader.

Grammatical complexity (1 mainly simple sentences to 5 mainly complex sentences)

Cohesion (1 explicit) to 5 (not explicit). An evaluation of the extent to which relations between the ideas were explicitly marked through reference, conjunctions and connectors.

Content knowledge

Subject specificity (1 general to 5 specific) This involved an evaluation of the frequency of technical vocabulary and the extent to which terms were glossed in the text for the general reader.

Nature of information (1 concrete to 5 abstract). An evaluation of the extent to which the text was concerned with concrete observable phenomena.

Cultural specificity (1 culture neutral to 5 culture specific) This involved an evaluation of the frequency of culture specific content as set out in the literature review above and the extent to which culturally specific references or examples were explained to the general reader.

Table 22 below specifies contextual parameters for the analysis of IELTS texts and extracts from core undergraduate texts used by students at the University of Bedfordshire.

QUANTITATIVE QUALITATIVE

Table 22: Contextual parameters for the analysis of IELTS texts and extracts from core undergraduate texts

A guiding principle in this particular study was to develop a set of methodological procedures that can easily be replicated by IELTS test developers in the future. Item writers need to have as clear an idea as possible of the complexity of any texts (across a range of parameters) where there is a prima facie case for inclusion in terms of length and appropriateness for testing intended skills and strategies. Study 4 was an initial attempt to establish a set of practical and meaningful procedures which might assist in this process. The point of comparison is the texts actually read by first-year students

8.2 Quantitative studies

The results of a one-way ANOVA comparing the IELTS and undergraduate texts on the range of contextual parameters are presented in Table 23 below. There were significant (p<.05) differences between IELTS texts and undergraduate texts for readability measures (Flesch reading ease and Flesch-Kincaid reading level); standardized type-token ratio; proportion of words on the academic word list (AWL); proportion of words appearing on the first 1,000 word frequency and 3,000 word frequency levels and the proportion of infrequent (off list) words. There were no significant differences on any other of the quantitative measures listed in Table 22.

Table 23: Analysis of variance of IELTS and undergraduate text contextual parameters

It is interesting to note that the IELTS texts were estimated both by the Flesch reading ease and Flesch-Kincaid measures to be significantly (p<.05) easier to read than the undergraduate texts. The difference between the means for IELTS and for undergraduate texts was 5 points on the 100-point Flesch reading ease scale or one year in terms of the Flesch-Kincaid grade levels. Figure 5 is a box-and-whisker plot summarising the distribution of Flesch-Kincaid reading levels for IELTS and undergraduate texts. The line in the middle of the boxes represents the median and the upper and lower boundaries of the boxes represent the upper and lower quartiles of the distributions. The figure indicates that the IELTS texts were generally of a similar level of readability to the undergraduate texts, falling within the range of undergraduate text readability. However, one text (Test 8, Text 1), appears as an outlier with a reading grade level of 8. This text, which concerns the construction of Hong Kong airport, has the lowest number of words per sentence of any of the texts analysed and is at the lower extreme for the average number of characters per word (4.5). An implication here may be that using readability formulae could assist the test developers in identifying texts that might fall outside the range of readability typically found in university level texts.

It is also of interest that no IELTS text had an estimated grade level higher than 16, although undergraduate texts ranged as high as 18. This might be taken as a further indication that even the most difficult of the IELTS texts do not reflect the level of the most challenging of the texts that undergraduates might expect to encounter in their first year of study.

Figure 5: Flesch-Kincaid Reading Grade levels of IELTS and undergraduate texts

The type token ratio (TTR) is the ratio of different words (types) to the total number of words (tokens). This represents a simple, if rather crude index of the number of different words the reader will need to know to understand a passage. It is generally recommended that a standardized length of text be used as in calculating the TTR as the length of a passage will affect the figure obtained (Scott 2006). Although standardized measures are not provided through the Web VocabProfiler, they can be obtained through another lexical profiling tool: WordSmith Tools. We used WordSmith Tools to find standardized TTRs based on 250 word sections of text and the results are displayed in Figure 6. It can be seen that the IELTS texts had a significantly higher mean standardized TTR than the undergraduate texts.

Figure 6: Standardized Type-Token Ratio of IELTS and undergraduate texts

The fact that the standardized TTR in the undergraduate texts is noticeably lower than in the IELTS texts could be taken to indicate that candidates in IELTS are exposed to more demanding texts in terms of this ratio. With course books, for example, one is likely to find more repetition of key words so that the reader is able to develop familiarity with these as they progress through the text.

It is also worth noting that the measure of lexical density employed here (the proportion of content words in the text) did not reveal any significant differences between IELTS and the undergraduate texts. This may be taken to suggest that IELTS reading texts in this respect do reflect a similar range of vocabulary to that appearing in undergraduate text books.

IELTS texts included significantly (p<.05) fewer sub-technical academic words and more very frequent words (words at the 1,000 and 3,000 word frequency levels) than the undergraduate texts. The proportion of running words on the academic word list (AWL) in IELTS texts overall was observed to be 7.9%, which is lower than that found in the corpus of academic texts from which the AWL was derived (10.0%), a second corpus of academic texts investigated by Coxhead (2000) (8.5%) and that for the undergraduate texts investigated here (10.3%).

Although the mean proportion of AWL words occurring in IELTS texts was higher than the 4% found by Coxhead (2000) in newspaper texts, the lowest proportion found in an IELTS text (2.2%) was closer to the proportion found in fiction texts (1.4%) and was just over half of the lowest proportion found in any part of an undergraduate text (4.33% for one section of a business studies text book). This IELTS text was taken from Part 1 of the test and concerned the provision of credit for young people in Zambia. The relatively low proportion of AWL words in the IELTS texts may reflect the high proportion of these texts that are sourced from newspapers and magazines (see the discussion of genre below).

The proportion of AWL words varied by IELTS test part, with Part 1 texts having the lowest (7.65%) and Part 3 texts the highest proportion (8.24%) of AWL words. Even in Part 3 of the test, however, coverage of the AWL was lower than in the undergraduate texts.

Figure 7: Comparison between proportion of running words appearing on the AWL in IELTS and undergraduate texts

The findings in relation to the AWL indicate that IELTS texts typically include a similar, if rather lower proportion of sub technical academic vocabulary to the undergraduate texts. Again, investigating coverage of the AWL might assist the test developers in identifying texts that lack representative coverage of sub technical academic vocabulary. This finding appears, like the findings relating to readability, to suggest that IELTS texts may lack some of the features of academic texts that may cause greatest difficulty for students.

The significantly higher proportion of words in IELTS texts at the 1,000 and 3,000 word frequency level may be a corollary of the differences noted in relation to the AWL with a higher proportion of these more ‘general’ words appearing in IELTS texts in place of the subtechnical vocabulary more frequently found in the undergraduate texts.

The undergraduate texts include on average almost four times as many off list words (words that do not appear on the AWL or on any the 15,000 word frequency level list). Items of this nature include proper nouns and acronyms (Marks & Spencer; Charles; Myanmar; the BBC) as well as neologisms and some of the more technical language found in these texts (applet; compurgation; mediastinum; reusability). It is perhaps unsurprising that such words should appear less often in IELTS texts, which are required, as we have noted above, to avoid both cultural and subject specificity.

Although the measures employed here are admittedly crude, it is encouraging for the IELTS academic reading test that they appear to reveal few quantifiable differences between the texts that students might expect to encounter in their first year of study and those used in the test. The main areas of concern indicated here are that the IELTS texts generally include a lower proportion of subtechnical academic vocabulary than the undergraduate texts and that IELTS texts may not fully reflect the level of readability found among the more challenging academic texts that first year undergraduates might expect to encounter. Tools such as the Web VocabProfiler used here might prove useful for the test developers in identifying texts with characteristics that are outside the range typically found in academic text.

8.3 Qualitative data

Following our identification of textual features in the literature review, two judges rated the texts IELTS and undergraduate texts on six criteria: rhetorical organisation, subject and cultural specificity, abstraction, grammatical complexity and cohesion. Rates of agreement between the two judges are shown in Table 24. Rates of agreement were highest for the more readily observed textual features of rhetorical organisation, grammatical complexity and cohesion, but were also considered acceptable for the more subjective features of subject and cultural specificity and level of abstraction. Where the two judges disagreed, the average of the two ratings was used in the subsequent analysis.

Table 24: Rates of agreement between the two judges on textual features

Table 25 below shows the results of the non-parametric tests of difference between IELTS and undergraduate texts. Figure 8 below displays the mean ratings for IELTS and undergraduate texts on each of the six criteria. Results were significant (p<.05) for both subject and cultural specificity. Although the undergraduate texts appeared to involve greater levels of abstraction, the results for this variable were not significant. As noted above in relation to the vocabulary measures, the significant difference between the IELTS and undergraduate texts in relation to subject and cultural specificity no doubt reflects the requirement for IELTS to avoid subject specificity and cultural allusion. No significant differences emerged on the measures of rhetorical organisation, grammatical complexity or cohesion.

Table 25: results of non-parametric tests of difference between IELTS and undergraduate texts.

The cultural specificity found in the undergraduate texts (and reflected in the number of non-technical off list words that occurred therein) could only add to the difficulty of reading them.

This may indicate that lack of background knowledge to understand the reading content is not always a matter of knowledge directly related to the subject, but may also arise from writers’ assumptions about readers’ cultural knowledge. Readers hitherto exposed only to relatively culturally neutral texts of the kind found in IELTS might well find the greater cultural specificity of the undergraduate texts to be a further source of difficulty.

Figure 8: mean ratings for IELTS and undergraduate texts on six criteria.

Table 26 shows the level of agreement between the two judges in assigning the texts to categories for the features of genre, rhetorical task, pattern of exposition and subject area.

Table 26: level of agreement between two judges on features of genre, rhetorical task, pattern of exposition and subject area.

The categorisation of texts by genre is set out in Table 27 below. The categorisation of the undergraduate texts was straightforward as all were text books, but there was some disagreement between the two judges in relation to the IELTS texts. Both agreed that 17 of the texts had been sourced from magazines or newspapers, that seven came from text books and that one was a research article. However, the second judge was less likely to identify magazine or newspaper articles as the source, seeing nine of those so identified by the first judge as coming from text books and a further seven from research articles. Discussion following the categorisation exercise indicated that some texts had been more difficult than others to categorise and that it was not always clear to the judges whether an individual text had been sourced from a research article, magazine article or text book. Although some texts had very obvious journalistic features, such as opening paragraphs that served as ‘attention grabbers’ and one text had the conventional headings of the research article; distinguishing characteristics were not always so easy to locate. A number of texts had little to indicate whether they had been sourced from a newspaper section, from a popular science magazine, from an introductory text book or from a more specialised academic publication. It might be of interest to explore how genre is affected by the editing process through which texts are prepared for inclusion on IELTS. It is possible that changes made to texts might have affected the judges ability to assign them to a genre.

What is clear from the exercise and the subsequent discussion between the judges is that IELTS texts often appear to be somewhat journalistic and that newspaper/ magazine texts are well represented in the test. The main study questionnaire responses indicated that newspapers and magazines may feature as sources in first year academic reading, but books, journals, reports and internet sources were all regarded as more important.

Table 27: Categorisation by genre: Results for judge 1 displayed by row, judge 2 by column

As indicated in Table 28, both judges agreed that most of the texts were expository in nature - 30 of the IELTS texts and 27 of the undergraduate texts. Both judges also agreed that argumentation and historical/biographical texts were also represented among both sets of texts. In terms of rhetorical task there appears to have been a close match between IELTS and these undergraduate texts.

Table 28: Categorisation by rhetorical task: Results for judge 1 displayed by row, judge 2 by column

With respect to pattern of exposition, as analysed in Table 29, the two judges agreed on the classification of 35 of the 42 IELTS texts, but just 26 of the 42 undergraduate texts. Subsequent discussion revealed that IELTS texts were more often felt to reflect one clearly dominant pattern of exposition while the undergraduate texts often involved two or more patterns occurring in sequence. This difficulty may have been caused by the way in which the undergraduate texts were collected; they were extracted from longer texts, often cutting across sections in the textbooks each of which displayed different patterns.

A further challenge for the judges in identifying patterns of exposition was that the categories are not mutually exclusive - definitions and descriptions often include illustration and a problem-solution text may additionally imply cause-effect. Determining which pattern was dominant in each of the texts investigated did not prove to be straightforward.

The analysis suggested that almost half of the IELTS texts displayed problem-solution or cause-effect patterns while the majority of the undergraduate texts involved elaboration. The selection of texts may have contributed to the difference: the opening chapter of an introductory text book often being concerned with elaborating the scope of the subject. On the other hand the brevity of IELTS texts and the high occurrence of newspaper/magazine articles may favour problem-solution and cause-effect patterns of exposition. The use of short texts with relatively clear dominant patterns may also bring its own problems; candidates may not be well prepared to encounter lengthier texts and to cope with transitions and relations between sections that follow different organisational principles.

Table 29. Categorisation by pattern of exposition: Results for judge 1 displayed by row, judge 2 by column

In classifying the texts according to subject area (see Table 30 here) the two judges were in complete agreement in assigning the undergraduate texts to subject area and agreed on 35 of the 42 IELTS texts. A broad range of subject areas were represented among the IELTS texts investigated with social studies, engineering & technology and business & administrative studies emerging as popular topic areas for the test.

Table 30: Classification of IELTS texts by two judges according to subject area

8.4 Conclusion

Overall, this study indicates that the kinds of text used in IELTS are those that introduce academic topics to a general audience, often in the form of articles sourced from newspapers or magazines that present research findings to a general audience These include self-contained reports on developments in science and technology and overviews of academic debates. The IELTS texts often present solutions to problems that are likely to be of interest to the general reader. The advantage of the IELTS approach to text selection is that the texts appearing in the test do, based on the limited corpus explored here, have many of the features of the kinds of text encountered by undergraduates. Although there are minor differences attributable to source (word frequency) and length (TTR), the IELTS texts include a vocabulary and a level of grammatical complexity that would place them within the range of texts encountered in the first year of study.

9 OVERALL CONCLUSIONS

This project has attempted to address the relationship between the academic reading construct in terms of what this means for students in their first year of study at one British University and the construct of reading as operationalised by the IELTS Reading Test. This has been a huge undertaking, equivalent in many ways to at least three joint-funded IELTS research projects, but nevertheless essential if we are to get to grips in the longer term with this very much under researched part of the IELTS battery.

However, despite the time and resources we have managed to allocate to it thanks to the collective effort of colleagues and PhD students at the University of Bedfordshire, it is still very much an initial attempt to map out the field of academic reading and the relationship with IELTS.

Nevertheless, this project does suggest some valuable conclusions and raises a number of interesting pointers for the future research into, and development of, the IELTS Reading Test. These are summarised below

It is encouraging to note that as it stands IELTS candidates at bands 6.5 and above, 6 and 5.5 and below can be distinguished in terms of the perceived reading problems our sample appear to have at these levels. There are significant differences in the self reporting of problems students encounter with a number of important strategies and the higher the band on IELTS reading the fewer the perceived problems.

The major focus of the IELTS test appears to be on careful reading whereas the survey data reported here suggest that for university students expeditious skills and strategies are just as critical for academic study and in a number of cases more problematic for both L1 and L2 students.

A protocol-based study of the cognitive processing of students taking the IELTS Reading Test would illuminate further the extent to which this is the case. Research into comparability of performance on items testing careful and expeditious reading skills and strategies by the target population is also necessary. If a clear need is established to distinguish between the two, it may then be necessary for IELTS to be more proactive in trying to test these expeditious strategies in terms of how the test is structured.

It would also be of interest to explore how texts are shaped and adapted through the item writing process and the implications of this process for the contextual parameters explored here. How might the item writers’ conceptions of the skills being tested through the tasks they set compare with the candidates’ protocol reports?

In terms of contextual parameters, the descriptive framework employed in this study has proved useful in identifying individual IELTS texts with idiosyncratic characteristics that do not match those typically identified with academic text. We feel that this project offers a methodology whereby such disparities might be identified at the text selection stage.

Overall the IELTS texts did generally fall within the parameter ranges exhibited by our small corpus of undergraduate text extracts. However, in relation to a number of contextual parameters there is evidence that the demands imposed by even the most ‘difficult’ of the IELTS texts may fall some way short of those imposed by the most challenging of the academic texts included here.

Research in reading may, like performance testing more generally (McNamara 1995), be likened to opening Pandora’s box. Once it is unlocked a vast array of questions clamour to be answered, some of which will require detailed intensive study on specific areas.

References

• Alderson, JC, 1990a, ‘Testing reading comprehension skills (Part One)’ Reading in a Foreign Language vol 6 (2), pp 425-438
• Alderson, JC, 1990b, ‘Testing reading comprehension skills (Part Two): getting students to talk about a reading test (A pilot study)’ Reading in a Foreign Language vol 7 (1), pp 465-503
• Alderson, JC, 1996, ‘The testing of reading’ in Teaching Reading Skills in a Foreign Language, ed C Nuttall, Heinemann, London, pp 221-229
• Alderson, JC, 2000, Assessing reading, Cambridge University Press, Cambridge
• Alderson, JC, 2005, Diagnosing foreign language proficiency: the interface between learning and assessment, Continuum, London
• Alderson, JC and Clapham, CM, 1992, ‘Applied Linguistics and Language Testing: A Case Study of the ELTS Test’, Applied Linguistics, 13, 2, pp 149-167
• Alderson JC, Figueras, N, Kuijper, H, Nold, G, Takala, S and Tardieu,C, 2004, ‘Specification for item development and classification within the CEF: the Dutch CEFR construct project’, paper presented at workshop on research into and with the CEFR, University of Amsterdam Feb 2004
• Al-Fallay, I, 1994, ‘Limiting bias in the assessment of English as a foreign language: the impact of background knowledge on the proficiency of Saudi Arabian students learning English as a foreign language’, unpublished doctoral dissertation, University of New Mexico, Albuquerque
• Anderson, R and Pearson, P, 1988, ‘A Schema-theoretic view of basic processes in reading comprehension’ in Interactive approaches to second language reading, eds P Carrell, J Devine and D Eskey, Cambridge University Press, Cambridge, pp 37-55
• Bachman, L, 1990, Fundamental considerations in language testing, Oxford University Press, Oxford
• Bachman, L, Davidson, F, Ryan, K and Choi, I, 1995, An investigation into the comparability of two tests of English as a foreign language: The Cambridge-TOEFL comparability study, UCLES / Cambridge University Press, Cambridge
• Bachman, L, Kunnan, A, Vanniarajan, S and Lynch, B, 1988, ‘Task and ability analysis as a basis for examining content and construct comparability in two EFL proficiency test batteries’, Language Testing vol 5, pp 128-159
• Bachman, L and Palmer, A, 1996, Language testing in practice, Oxford University Press, Oxford
• Barnett, M, 1989, More than meets the eye, Prentice Hall Regents, Englewood Cliffs, NJ
• Baumann, J, (ed) 1986, Teaching main idea comprehension. International Reading Association, Newark, Delaware
• Beard, R, 1972, Teaching and learning in Higher Education, Penguin Books Ltd, Harmondsworth
• Berman, R, 1984, ‘Syntactic components of the foreign language reading process’ in Reading in a foreign language, eds JC, Alderson and A, Urquhart, Longman, London, pp 139-159
• Bernhardt, E, 1991, Reading development in second language: theoretical, empirical and classroom perspectives, Ablex Publishing Corporation, New Jersey
• Bhatia, V, 1997, ‘Applied genre analysis and ESP’ in Functional approaches to written text: classroom applications, ed T Miller, USIA, Washington DC, pp 134-149
• Buck, G, and Tatsuoka, L, 1998, ‘Application of the rule-space procedure to language testing: examining attributes of a free response listening test’, Language Testing, 15 (2), 119-157
• Campion, M and Elley, W, 1971, An academic vocabulary list, Wellington, NZCER
• Buck, G, Tatsuoka, K, Kostin, I, 1997, ‘The sub-skills of reading: Rule-space analysis of a multiple-choice test of second language reading comprehension’, Language Learning 47 (3) 423-466
• Carrell, P, 1983, ‘Three components of background knowledge in reading comprehension’, Language Learning vol 33 (2), pp183-207
• Carrell, P, 1984, ‘The effects of rhetorical organisation on ESL readers’ TESOL Quarterly vol 18(3), pp 441-469
• Carrell, P, Devine, J and Eskey, D (eds) 1988, Interactive approaches to second language reading, Cambridge University Press, Cambridge
• Carver, R, 1992, ‘What do standardized tests of reading comprehension measure in terms of efficiency, accuracy and rate?’, Reading Research Quarterly, vol 27 (4), pp 347-359
• Carver, R, 1997, ‘Reading for one second, one minute, or one year from the perspective of rauding theory’, Scientific Studies of Reading, vol 1(1), pp 3-43
• Chihara, T, Sakurai, T and Oller, J, 1989, ‘Background and culture as factors in EFL reading comprehension’, Language Testing, vol 6(2), pp 143-151
• Chikalanga, I, 1990, Inferencing in the reading process, Unpublished PhD thesis, University of Reading
• Chikalanga, I, 1992, ‘A suggested taxonomy of inferences for the reading teacher’, Reading in a Foreign Language, vol 8(2), pp 697-709
• Clapham, C, 1996, The development of IELTS: A study in the effect of background knowledge on reading comprehension, Studies in Language Testing 6. UCLES Cambridge University Press, Cambridge
• Coady, J, 1979, ‘A psycholinguistic model of the ESL reader’ in Reading in a Second Language eds R Mackay, B Barkman, and RR Jordan, Newbury House, Rowley, Massachusetts, pp 5-12
• Cohen, A and Upton, T, 2006, Strategies in responding to the New TOEFL reading, Monograph Series 33, ETS, Princeton
• Council of Europe, 2001, Common European Framework of Reference for Languages: Learning, Teaching, Assessment, Cambridge University Press, Cambridge
• Coxhead, A, 2000, ‘A new academic word list’ TESOL Quarterly, vol 34 (2), pp 213 - 238
• Dechant, E, 1991, Understanding and teaching reading: An interactive model, Lawrence Erlbaum, Hillsdale, NJ
• Enright, M, Grabe, W, Koda, K, Mosenthal, P, Mulcany-Ernt, P and Schedl, M, 2000, TOEFL 2000 reading framework: A working paper, TOEFL Monograph Series 17 ETS, Princeton
• Ehrlich, M, 1991, ‘The processing of cohesion devices in text comprehension’, Psychological Research vol 53 (2), pp 169-174
• Freedle, R and Kostin, I, 1993, The prediction of TOEFL reading comprehension item difficulty for expository prose passages for three item types: Main idea, inference, and supporting idea items, TOEFL Research Reports, No. RR-93-44, Educational Testing Service, Princeton, NJ
• Freedle, R, 1997, ‘The relevance of multiple-choice reading test data in studying expository passage comprehension: the saga of a 15 year effort towards an experimental/correlational merger’, Discourse Processes, vol 23, pp 399-440
• Fortus, R, Coriat, R and Fund, S, 1998, ‘Prediction of item difficulty in the English subtest of Israel’s inter-university psychometric entrance test’, in Validation in language assessment: Selected papers from the 17th Language Research Colloquium, Long Beach, ed A Kunnan, Lawrence Erlbaum Associates, Inc, Publishers Mahwah, New Jersey, pp 61-87
• Gervasi, V and Ambriola, V, 2002, ‘Quantitative assessment of textual complexity’ in Complexity in language and text, ed L Merlini Barbesi, PLUS-University of Pisa, Pisa, pp 197-228
• Goh, S, 1990, ‘The Effects of rhetorical organisation in expository prose on ESL readers in Singapore’, RELC Journal, vol 21(2), pp 1-13
• Goldman, S and Rakestraw, J, 2000, ‘Structural aspects of constructing meaning from text’, in Handbook of reading research, eds M Kamil, P Rosenthal, P Pearson and R Barr, Lawrence Erlbaum Associates, Mahwah, NJ, pp 311-335
• Grabe, W, 1991, ‘Current developments in second language reading research’, TESOL Quarterly, vol 25 (3), pp 375-406
• Grabe, W, 2005, ‘Review: Insights into second language reading http://nflrc.hawaii.edu/rfl/April2006/reviews/grabe.html
• Grabe, W and Stoller, F, 2002, Teaching and researching reading, Longman, London
• Guthrie, J and Kirsch, I, 1987, ‘Distinctions between reading comprehension and locating information in text’, Journal of Educational Psychology, vol 79, pp 220-228
• Hawkey, R, 2006, Impact theory and practice: Studies of the IELTS test and the Progetto Lingue 2000, UCLES Cambridge University Press, Cambridge
• Hoover, W and Tunmer, W, 1993, ‘The components of reading’, in Reading Acquisition Processes, eds G Thompson, W Tunmer and T Nicholson, Multilingual Matters Ltd, Clevedon, England, pp 1-19.
• Hyland, K, 2000, Disciplinary discourses: social interactions in academic writing, Longman, Harlow
• Hyland, K, 2002, Teaching and researching writing, Applied Linguistics in Action Series, Longman, London
• Hyland, K, 2005, Metadiscourse, Continuum, London
• IELTS, 2005, IELTS Handbook, Available from http://www.cambridgeesol.org/support/dloads/ielts_downloads.htm
• Jordan, R, 1997, English for Academic Purposes: A guide and resource book for teachers, Cambridge University Press, Cambridge
• Khalifa, H, 1997, ‘A study in the Construct Validation of the Reading Module of an EAP Proficiency test Battery: Validation from a variety of Perspectives’, Unpublished PhD Thesis, University of Reading
• Khalifa, H and Weir, CJ, forthcoming, Examining Reading, UCLES / Cambridge University Press, Cambridge
• Kintsch, W, 1998, Comprehension: A paradigm for cognition, Cambridge University Press, New York
• Kintsch, W, 2004, ‘The construction-integration model of text comprehension and its implications for instruction, in Theoretical models and processes of reading, eds R Ruddell, and N Unrau, International Reading Association, Newark, Delaware, pp 1270-1328
• Koda, K, 2005, Insights into second language reading: A cross-linguistic approach, Cambridge University Press, New York
• Levine, A, Ferenz, O and Reves, T, 2000, ‘EFL academic reading and modern technology: How can we turn our students into independent critical readers?’ TESL EJ, vol 4 (4)
• Livingston, J, 1997, Metacognition: An overview. Available from http://www.gse.buffalo.edu/fas/shuell/cep564/Metacog.htm
• Lumley, T, 1993, ‘The notion of subskills in reading comprehension tests: An EAP example’ Language Testing vol 10 (3), pp 211-234
• Lunzer, E, Waite, M and Dolan, T, 1979, ‘Comprehension tests’, in The effective use of reading, eds E Lunzer and K Gardner, Heinemann, London, pp 37-71
• Maclellan, E, 1997, ‘Reading to learn’, Studies in Higher Education, vol 22, pp 277-288
• McNamara, T, 1995, ‘Modelling performance: Opening Pandora’s box’, Applied Linguistics, vol 16(2), pp 159-179
• Masi, S, 2002, ‘The literature on complexity’ in Complexity in language and text, ed L Merlini Barbesi, PLUS-University of Pisa, Pisa, pp 197-228
• McCormick, T, 1988, Theories of reading in dialogue: An interdisciplinary study, University Press of America, New York
• Meyer, B and Freedle, R, 1984, ‘Effects of discourse type on recall’, American Educational Research Journal vol 21(1), pp 121-143
• Moore, T and Morton, J, 1999, ‘Authenticity in the IELTS academic module writing test: a comparative study of Task 2 items and university assignments’ in IELTS research reports 1999, volume 2, ed R Tulloh, IELTS Australia, Canberra, pp 64-106
• Munby, J, 1978, Communicative syllabus design, Cambridge University Press, Cambridge
• Nuttall, C, 1996, Teaching reading skills in a foreign language, Heinemann, London
• Nystrand, M, 1989, ‘A social interactive model of writing’, Written Communication, vol 6, pp 66-85
• Oakhill, J and Garnham, A, 1988, Becoming a skilled reader, Basil Blackwell, Oxford
• Perera, K, 1984, Children’s writing and recording, Basil Blackwell, Oxford
• Perfetti, C, 1997, ‘Sentences, individual differences, and multiple texts: three issues in text comprehension’, Discourse Processes, vol 23, pp 337-355
• Purpura, J, 1998, ‘Investigating the effects of strategy use and second language test performance with high and low-ability test-takers: a structural equation modelling approach’, Language Testing, vol 15, pp 339-379
• Rayner, K and Pollatsek, A, 1989, The psychology of reading, Prentice Hall, Englewood Cliffs, NJ
• Rosenblatt, L, 2004, ‘The transactional theory of reading and writing’, in Theoretical models and processes of reading, eds R Ruddell and J Unrau, International Reading Association, Newark, Delaware, pp 1363-1398
• Rosenfield, P, Oltman, P and Sheppard, K, 2004, Investigating the validity of TOEFL: a feasibility study using content and criterion-related strategies, TOEFL Research Report, RR- 71 Educational Testing Service, Princeton, NJ
• Rosenshine, B, 1980, ‘Skills hierarchies in reading comprehension’ in Theoretical issues in reading comprehension, eds R Spiro, B Bruce and W Brewer, Lawrence Erlbaum Associates, New Jersey
• Sasaki, M, 2000, ‘Effects of cultural schemata on students’ test-taking processes for cloze tests: a multiple data source approach’, Language Testing vol 17, pp 85-114
• Schedl, M, Gordon, A, Carey, P and Tang, K, 1996, An analysis of the dimensionality of TOEFL reading comprehension items, TOEFL Research Reports No. RR-95-27, Educational Testing Service, Princeton, NJ
• Scott, M, 2006, Oxford WordSmith Tools 4.0, http://www.lexically.net/downloads/version4/html/index.htm
• Shiotsu, T, 2003, ‘Linguistic knowledge and processing efficiency as predictors of L2 reading ability: a component skills analysis’, Unpublished PhD dissertation, University of Reading
• Shiotsu, T and Weir, CJ, 2007, ‘The relative significance of syntactic knowledge and vocabulary breadth in the prediction of second language reading comprehension test performance’, Language Testing, vol 23(4), pp 99-128
• Skehan, P, 1998, ‘Task-based instruction’, Annual Review of Applied Linguistics, vol 18, pp 268-286
• Stanovich, K, 2000, Progress in understanding reading: Scientific foundations and new frontiers, The Guildford Press, New York
• Steffensen, M, Joag-Dev, C and Anderson, R, 1979, ‘A cross-cultural perspective on reading comprehension’, Reading Research Quarterly, vol 15, pp 10-29
• Tan, S, 1990, ‘The role of prior knowledge and language proficiency as predictors of reading comprehension among undergraduates’, in Individualising the assessment of language abilities, eds J de Jong and D Stevenson, Multilingual Matters, Clevedon, PA, pp 214-224
• Teasdale, A, 1989, ‘Introspection and judgmental approaches to content validation: A study using the Test in English for Educational Purposes’, Unpublished MAAL Dissertation, Department of Linguistics, University of Reading
• Tercanlioglu, L, 2004, ‘Postgraduate students’ use of strategies in L1 and ESL contexts : Links to success’, International Education Journal, vol 5 (4)
• Urquhart, A, 1984, ‘The effect of rhetorical ordering on readability’ in Reading in a foreign language, J Alderson and A Urquhart, Longman, London
• Urquhart, A and Weir, C, 1998, Reading in a second language: Process, product and practice, Pearson Education Ltd, Essex, United Kingdom
• Weakley, S, 1993, Procedures in the content validation of an EAP proficiency test of reading comprehension, Unpublished MATEFL Dissertation, CALS, University of Reading
• Weigle, S, 2002, Assessing writing, Cambridge, Cambridge University Press
• Weir, CJ, 1983, ‘Identifying the language needs of overseas students in tertiary education in the United Kingdom’, PhD thesis, Institute of Education, University of London
• Weir, CJ, 1993, Understanding and developing language tests, Prentice Hall, Englewood Cliffs, NJ
• Weir, CJ, 2005, Language testing and validation: An evidence-based approach, Palgrave MacMillan, Basingstoke
• Weir, CJ and Porter, D, 1994, ‘The multi-divisibility or unitary nature of reading: The language tester between Scylla and Charybdis’, Reading in a Foreign Language, vol 10 (2), pp 1-19
• Weir, CJ, Yang, H and Jin, Y, 2000, An empirical investigation of the componentiality of L2 reading in English for Academic Purposes, Studies in Language Testing, vol 12, UCLES Cambridge University Press, Cambridge

APPENDIX 1: PILOT QUESTIONNAIRE

QUESTIONNAIRE ON ACADEMIC READING ACTIVITIES

Dear Respondent

This study is being conducted by the Centre for Research in English Language Learning and Assessment at the University of Luton. The purpose of this study is to investigate students’ reading activities at the university. Your answers will help us to establish where students might need help with Academic Reading.

We intend to use the information you give us to develop on-line, self access, diagnostic instruments which will be linked to appropriate learning modules on reading skills and strategies. These will help Luton University students improve their reading abilities and make studying here easier and more successful.

All information in this questionnaire is completely confidential and no individuals will be identified. If you do not wish to participate in this study, please do not fill the questionnaire out.

Thank you for your help.

Biodata

1. Age:
2. Sex:
3. Nationality:
4. Undergraduate / Postgraduate
5. Course:
6. Year of Study:
7. First language:
8. If you are an overseas student, please give your IELTS Reading Band

READING RESEARCH PROJECT PILOT OPEN-ENDED QUESTIONS

1. Are course books the most important reading for your studies? Yes/No If No, what are?
2. Do you read online? Yes/No If Yes, how much reading do you do actually on line as compared to printed materials?
3. Do you read for different purposes on your degree course? Yes/No If Yes, what are the purposes?
4. How do you decide what to read for your course?
5. For an assignment how many of each different source (how many books, how many articles, how many internet sites etc) would you normally take information from?
6. When you have decided what to read, describe how you read it?
7. Is your approach the same when you read books, articles, etc? Yes/No If No, what are the differences in how you read?
8. Is your approach the same when you read for assignments as against examinations? Yes/No If No, what are the differences?
9. Do you find anything difficult in what you have to read for your course, e.g. in books, articles, etc? Yes/No If Yes, please give details
10. Do you feel under pressure in your academic reading? Yes/No If Yes, what are the pressures?
11. What do you think a “successful” reader is at university?
12. In this academic year, what was the most important book you read? Please give author and title
13. In this academic year, what was the most important article you read? Please give author, title and journal

APPENDIX 2: MAIN SURVEY STRUCTURED QUESTIONNAIRE

QUESTIONNAIRE ON ACADEMIC READING ACTIVITIES

Dear Respondent

This study is being conducted by the Centre for Research in English Language Learning and Assessment at the University of Luton. The purpose of this study is to investigate students’ reading activities at university. Your answers will help us to establish where students might need help with Academic Reading.

We intend to use the information you give us to develop on-line, self access, diagnostic instruments which will be linked to appropriate learning modules on reading skills and strategies. These will help students improve their reading abilities and make studying here easier and more successful.

All information in this questionnaire is completely confidential and no individuals will be identified. If you do not wish to participate in this study, please do not fill the questionnaire out.

Thank you for your help.

Professor C.J. Weir, Putteridge Bury Room 124 cyril.weir@luton.ac.uk.

Personal details: please tick the appropriate box

Age:

• 18-22
• 23-29
• 30-39
• 40+

Gender:

• Male
• Female

Nationality:

• UK
• EU
• International

Status:

• Undergraduate
• Postgraduate

Subject area: please tick one box

• Advertising, Marketing and Public Relations
• Art and Design
• Biology and Biomedical Sciences
• Business and Finance
• Computing and Information Systems
• Education Studies
• Healthcare (Nursing and Midwifery)
• Human Resource Management
• Language and Communication (EFL and TEFL)
• Law
• Leisure, Tourism and Sports Management
• Media Arts
• Psychology
• Social Sciences and Social Work
• Sport and exercise

Year of Study: 1st 2nd 3rd

First language:

If you are an international student and have taken IELTS, please tick your Reading Band score:

• 9
• 8.5
• 8
• 7.5
• 7
• 6.5
• 6
• 5.5
• 5
• 4.5
• 4

Academic Reading Research Project

For each statement below, show the extent of your agreement or disagreement:

• 5 Definitely agree
• 4 Mostly agree
• 3 Neither agree nor disagree
• 2 Mostly disagree
• 1 Definitely disagree

The following sources of information are important on my course:

The following purposes for reading are important on my course:

• Searching texts to find information I can use in assignments and/or examinations
• Basic comprehension of main idea(s) in a text
• Understanding the meaning of the text as a whole: working out how the main ideas and details in a text relate to each other and to the author’s purpose
• Integrating information from different texts for use in assignments and /or examinations

How I read for assignments:

• I think carefully to ensure that I know exactly what I will be looking for before I start reading
• I quickly look through the whole of a text for a general understanding before doing anything else
• I gradually understand what a text is about by reading the sentences slowly and carefully in the order they occur
• I remember where relevant information is or mark its location for later use in writing my assignment
• I think of key words and quickly look for them or words with similar meanings to check if text is worth reading more carefully
• I look at the titles or headings of a text before deciding to read it carefully
• I first get an overall meaning of the text for example by reading the first paragraph and the conclusion, and the first sentence of the other paragraphs
• If I do not know the meaning of a word in a text, I try to work out its meaning
• I read a text slowly all the way through even if some parts do not seem relevant to my assignment
• I read slowly only those sections of a text I have marked as relevant when going through it quickly before
• While reading I try to relate content to what I know already and judge its value
• I look back at previous parts of the text to check meaning
• I try to understand how the text is organized: how the ideas and details connect with each other
• I make notes on relevant points from the text as I go along
• I integrate information from the text I am reading with information from other texts I have already read
• I read critically to establish and evaluate the author’s position on a particular topic

When I read for assignments, I have difficulty with:

• the time available to do the necessary reading
• reading texts where the subject matter is complicated
• words I do not know
• sentence structures
• finding relevant information quickly
• lengthy texts
• lack of background knowledge to understand the content
• making notes on information I will need
• reading carefully to understand the main ideas
• summarizing ideas from a text in my own words
• understanding a detailed logical argument
• reading critically to establish and evaluate the author’s position on a particular topic
• relating the content of a text to my existing knowledge
• deciding what is important for me and what is not
• reading a text quickly to decide whether I should study it carefully
• understanding the text as a whole; how main ideas and details are connected to each other
• integrating information from the text I am reading with information from other texts I have already read

How much reading do you actually do on line as compared to reading printed out materials?

Please tick one box

Read online

• 0-20%
• 21-40%
• 41-60%
• 61-80%
• 81-100%

Rank the following English language skills in their order of difficulty for you in your university studies (1 = most difficult 2 = second most difficult etc)

• Listening
• Reading
• Writing
• Speaking