'What's your score?': An investigation into language descriptors for rating written performance

Abstract

This study addresses the problem of inconsistency in rating IELTS exams and, further, the need for valid criteria for rating levels of written performance.

Determining written performance involves a series of complex semiotic events or processes. On the part of examination candidates these include the interpretation of prompts and the composition of written responses. On the part of raters, the process includes interpreting criteria for rating, making sense of candidates’ scripts and attributing rating criteria to text features in the scripts. Each of these events contributes to variation in scoring individual scripts. Scoring is a linguistically mediated activity and for this reason I planned the study as an investigation into the language features of candidates’ scripts at different levels of written performance, and whether those language features might delineate performance level.

The data for this study came from students who were non-native speakers of English studying general English and English for academic purposes. The students wrote responses to Tasks 1 and 2 of the IELTS General Training Writing Module. Their texts were graded into three performance levels: basic, intermediate and upper intermediate.

The study examined language features of the subjects’ texts as possible indicators of written language development at different levels of performance. The analysis identified numerous linguistic options writers chose in response to the prompts. Some of these stemmed from misunderstandings of cultural knowledge implicit in the topics of the prompts. Different interpretations of the task resulted in observable differences in responses. However, the analysis revealed that less developed texts expressed the interpersonal function with difficulty, using familiar and personal terminology when more formal linguistic choices would have been appropriate. Lower level texts were limited in the organisation of factual information, which was expressed with limited technical terminology. More developed texts demonstrated fluency through the use of clearly linked concepts. Even though upper intermediate texts contained structural errors, these did not interfere with understanding candidates’ meanings.

The analysis indicates that it is not easy to identify specific lexicogrammatical features that distinguish different levels of performance. It is the sum of language features integrated textually which create successful scripts. This suggests that assessment, which focuses on isolated language elements such as vocabulary or sentence structures, detracts from the semiotic processes of composing texts and interpreting and therefore rating texts. The study proposes that as rating is a complex, meaning-making activity, raters respond first and foremost to texts as a whole rather than to individual components. A possible explanation for this is that raters have expectations which stem from sociocultural conventions in terms of text types and social purposes of texts. Assessment is a response to the conventional combinations of linguistic elements in texts. These elements are numerous and vary considerably across texts.

The analysis of lexicogrammatical elements of texts provides evidence of how social purposes are realised in texts and raises questions about how these influence rating processes. The study suggests further research into the use of holistic descriptors of texts for scoring written texts.

1.0 Context of the study

1.1 Introduction

Assessing written texts or examination papers is a complex activity (Weigle 2002). The examination procedure alone involves tasks and task interpretations, text composition and reading for the evaluation of texts. Each of these activities is a complex semiotic event. Ideally assessment procedures reduce the number of variables in the testing context to achieve maximum validity. However, even with carefully moderated test prompts, equivalent testing environments and selected rubrics used by trained raters, the assessment procedure involves interpretative practices resulting in variations in candidates’ responses to the same prompts, and in scoring variations. Added to this is the elusive nature of quality writing and, in particular, writing for different purposes. These factors suggest that the evaluation of writing, as the process of composing and interpreting texts (which lies at the heart of assessing writing), is problematic.

The present study is an attempt to investigate the role of texts or language in use in assessing writing. It sets out to examine an area that has not received sufficient attention in assessment, which is the role of candidates’ texts in evaluation of written work. What is the relationship between text features and scores? What text features are valued in the rating process? The study considers the linguistic features of texts as components for the construction of meaning and for the delineation of levels of performance. The linguistic features are referred to as language descriptors and elsewhere as discourse features in order to draw attention to the language of texts. General criteria for the evaluation of written performances, such as style and appropriacy, lack specific semiotic realisations in texts. However, an appropriate style is linguistically constituted. This study attempts to explore the possibility of identifying language descriptors which differentiate levels of performance.

The first part of this paper outlines some of the factors that influence the composition and evaluation of test papers. The analysis of test papers which follows seeks to describe some of the discourse features in candidates’ texts which may influence raters’ evaluations. The focus on textual features stems from an increasing interest in discourse and text features in assessing writing (Mickan and Slater forthcoming).

1.2 Research context

A significant area of research in assessing writing is the analysis of textual features of written examination papers and their relationship to test scores. In previous studies Mickan, Slater and Gibson (2001) and Mickan and Slater (forthcoming) raised questions about candidates’ interpretation of prompts, the role of the language choices made by examination candidates in the composition of responses, and about the influence of the lexicogrammatical choices on raters’ evaluation of candidates’ texts. For example, Mickan and Slater (forthcoming), in a study of native speakers of English (NS) and non-native speakers’ (NNS) examination responses to an IELTS Academic Writing Task 2 were compared. They found that writers who controlled the discourse of a topic in appropriate text types produced responses valued by examiners. In those studies questions of language development and descriptors for rating levels of performance were not addressed.

The factors which affect scoring of written performance are numerous. Ishikawa (1995) has noted that both holistic and analytic scoring protocols or scales are problematic for discrimination at a single proficiency level. This may be due to lack of precise meanings of the descriptors used in different scales. Polio (1997), for example, notes that measures of linguistic accuracy in second language writing are often inadequate and notes raters’ disagreement about native like usage. Claire (2001, p. 45) reports teachers, in moderation sessions, debating performance criteria and the meaning of terms such as “appropriately” - terms which Brindley (2001, p. 76) refers to as “impressionistic terminology”, and one of the reasons Brindley (2001) reports inconsistent interpretation and application of performance criteria, even when made extremely clear and precise. Elsewhere Brindley (2000) comments on the “highly individual nature of test-taking processes” (p. 74) and notes the problem of matching particular language skills to particular tasks, and also records the difficulties of raters arriving at common interpretations of content characteristics of papers. Smith (2000), in a study of rater judgements, states that “In terms of interpretation and application of performance criteria, analysis of raters’ think-aloud verbal reports strongly suggested that raters had difficulty interpreting and applying some of the relativistic terminology used to describe performance” (p. 186). However the solution is not necessarily to have more prescriptive performance criteria, as these led to less agreement among raters (Smith 2000, p. 177).

Here is the dilemma for rating quality in examination papers. On the one hand it is difficult to achieve rater consistency, on the other it appears not to be helpful to be more precise in the description of criteria.

This study considers the centrality of the text in the rating procedure. The analysis considers language features which contribute to the text in terms of appropriacy, and which may influence rating procedures. This is consistent with the information provided to candidates of the IELTS test: “Part of the task realisation is to respond appropriately in terms of register, rhetorical organisation, style and content” (Handbook 2001, p. 13). This exploratory study therefore investigated what discourse features in candidates’ texts characterise different levels of written performance.

The theoretical position for the analysis in this study is the function of written texts for the social negotiation and construction of meanings. As part of a culture’s social practices, potential meaning of texts is construed through the combinations of lexicogrammar constituting those texts. Writers formulate their meanings in the particular choices of wordings to fulfil social purposes, and readers create meanings from the written representations in the texts (Mickan 2000). Halliday’s (1985) systemic functional grammar forms the background to this approach to text analysis.

2.0 The study

The general aim of the study was to investigate language features of written texts, which indicate different levels of performance and which might be used for describing levels of performance, and which might contribute to the enhancement of scoring second language writers’ responses to assessment tasks. The study was planned to identify and describe discourse or language descriptors of written performance for rating IELTS candidates’ answers in the General Training Writing Module (Tasks 1 and 2) at different bands.

The study addressed two central questions: What language features indicate written language development at different levels? Could these language features be identified and used to complement the IELTS descriptors for rating of written performance on the General Training Writing Module (Tasks 1 and 2) at different bands?

The specific procedures were:

• to collect a sample of students’ written responses to the two Tasks in the General Training Writing Module written by non-native speakers of English (NNS) in classes in an IELTS Administration Centre;
• to describe the characteristic textual features of the candidates’ answers;
• to identify the linguistic elements which characterise performance levels in the subjects’ texts;
• to propose possible applications of results of the study for the evaluation of written performance at different levels.

The subjects of the study were non-native speakers of English (NNS) studying in the Centre for Applied Linguistics in the University of South Australia (CALUSA), an IELTS Administration Centre. The purpose of the study was discussed with administrators at CALUSA and a meeting organised with prospective students, to explain the study and invite their participation. Students were given information about the study and permission was obtained from volunteers for use of their papers in the study.

The students were attending classes in General English and English for Academic Purposes (EAP) Levels One and Two. The classes represented different levels of proficiency and experience in the English language. The General English class was at a lower level than the classes of English for Academic Purposes. The General English class was also larger than the EAP classes. These classes enabled the collection of texts from students with different English language experiences. As the students were international students their experiences in learning English varied considerably: some students had completed English language programs as part of their formal schooling in foreign language contexts, others had studied intensive English-as-a-second-language programs in Australia.

The students came from different first-language backgrounds, ranging from European, Middle Eastern to East Asian. Forty-eight students registered to participate in the project.

All students sat for the written examination at the same time. This was during their normal lesson period and was seen as part of their instruction. The questions used in the study were taken from the IELTS Handbook (July 2001). Based on information from teachers and from the students themselves, none of the subjects had practised or written essays on the selected topics. The two questions are reproduced below.

IELTS General Training Writing test

Instructions: Please write on the two tasks. Write on separate pages for the two tasks. Write your name at the top of each page.

Writing Task 1: A letter

You should spend about 20 minutes on this task

You rent a house through an agency. The gas heater is not working. You telephoned the agency one week ago and asked the agent to repair the heater. It has not been done.

Write a letter to the agent. Explain the problem and ask them what you want the agent to do.

You should write at least 150 words.

Begin your letter as follows

Dear Sir/Madam

Writing Task 2: Essay expressing your ideas

You should spend about 40 minutes on this task

Hotels, restaurants and businesses do not allow smoking inside. In public places such as airports smoking is banned.

This is a good idea, but it takes away freedom of choice. Some smokers do not like the bans.

Do you agree or disagree with banning smoking in public places?

Give reasons for your answer.

Write at least 250 words.

The examination was supervised by teachers and the researcher. Students wrote their responses in the time allocated for the General Training Writing Test (60 minutes).

2.1 Analysis of texts

The analysis of the texts was done with reference to the information given to IELTS candidates in the General Training Writing section of the IELTS Handbook (2001). Only the texts from candidates with completed texts were selected for analysis. An independent researcher categorised the papers into three levels of performance: basic, intermediate and upper intermediate. The independent researcher was not involved in teaching the students nor in the selection of classes nor in the administration of the tests. None of the papers collected for the study were considered to be of an advanced standard. From the sample of papers, more were assigned basic and intermediate levels, so that there were fewer upper intermediate texts analysed in the study. The assigned levels of performance were not correlated with the class levels as this was not required by the study. Each of the examination texts was evaluated independently without attempting to compare performances of students on different tasks. Although this is an interesting area of research, particularly given the social nature of composing texts, the focus of analysis was on discourse differences between actual texts.

The analysis of the two tasks was from the perspective of the social purpose of the task and the textual features chosen by candidates for doing the task. The complexity and range of texts necessitated a focus on features, which seemed to significantly impact on task realisation. The features were selected based on the notion of text as language in use - ie. as a self-contained unit of analysis (Mickan forthcoming) - and then on the particular linguistic configurations or combinations, which realise the social purposes of the tasks.

Note on transcription of texts: All texts have been transcribed without corrections. The below conventions have been followed.

• [ ] elisions
• insertions > <
• indecipherable ---
• word count according to computer check

3.0 Analysis of Task One texts

The wording of the prompt for Task One and of subjects’ texts is discussed in this section. The prompt for this task was the following.

Writing Task 1: A letter

You should spend about 20 minutes on this task

You rent a house through an agency. The gas heater is not working. You telephoned the agency one week ago and asked the agent to repair the heater. It has not been done.

Write a letter to the agent. Explain the problem and ask them what you want the agent to do.

You should write at least 150 words.

Begin your letter as follows

Dear Sir/Madam

The social purpose of this text is to report a problem in order to obtain a specific service. It is both a complaint and report of a problem, and a repeat request for the problem to be dealt with. The response requires candidates to fulfil multiple functions in the composition of the letter. Although we think of letters as a defined genre, letters perform different social purposes which are construed through the selection and structuring of lexicogrammatical resources. Letters serving different social functions constitute different text types. The successful composition of a letter depends on the selection of appropriate discourse resources to meet the social expectations and requirements of the task. This is a complex socio-cultural activity, as indicated in the texts produced for this study.

3.1 Candidates’ interpretation of the task

The task itself seems straightforward: a previous telephone call did not result in repairs or action, so the need to follow up with a written text for future reference makes good sense. The purpose of the letter was understood by most of the students, regardless of their performance level. However, considerable cultural knowledge was required to understand the context and purpose of the task. This world knowledge included an understanding of the need for heating, the nature of heating and the practicalities of rental properties being managed by agents. For some students from the Asian Pacific region such knowledge would not necessarily be a normal part of their experience, particularly if they have lived in tropical climates or have only recently arrived in a country where the climate necessitates interior heating.

The cultural features intrinsic to prompts affect candidates’ ability to understand the intended purpose of the task and thus their ability to respond appropriately (Mickan and Slater forthcoming). For example, some basic and intermediate-level candidates did not seem to understand what a “gas heater” really is, which is indicated by references such as:

“I can’t have shower, -can’t use hot water to wash dishes in winter.” (15)

“I have to go out for diet every day now!”

“I have not cooked from one week.” (B7)

Candidates’ comprehension of the prompt and the purpose of the letter is clearly crucial for the composition of their responses. The linguistic realisation of the task appeared to vary according to:

• candidates’ familiarity with the expected text-type conventions;
• candidates’ interpretation of the prompt (e.g. what is “the problem” and who is to blame);
• candidates’ use of appropriate or necessary vocabulary and their capacity to handle it, and
• their choice of formality or level of affection expressed towards the agent.

In assessing the letters it might be assumed that the overall impression of appropriateness in interpersonal meaning is influential. In discourse-semantic terms candidates are expected to display the “ability to engage in personal correspondence” (Handbook 2001, p. 13). The level of formality in a business letter is one feature that was handled differently by candidates. Obviously this is a complex matter as the writer, imagining the role of a tenant, needs to take into consideration what position to adopt in the letter. For example, is it more desirable to be friendly and to adopt a personal discourse or is the accepted practice of formality more important?

The possibility of choice in formality resulted in a range of different responses. The task required as a minimum an explanation of the problem and a request for action. This prompted a variety of responses. Candidates represented “the problem” either factually and technically or in emotive or psychological terms. The letters included a range of responses:

• the mechanical/technical problem with the gas heater, i.e. what is wrong with the gas heater, how it does not work, etc;
• the author’s physical/psychological problems caused by the broken heater, ie. what the writers do not like, how they suffer from the heater’s faulty operation;
• the social-interpersonal problem between the author and the agency/agent, ie. what is wrong with the agency’s service or reaction to the previous request, how inappropriate it was, that the tenant is unhappy with the delay in replying.

As a formal letter of complaint, a focus on the factual and technical aspects of the letter may be considered as more appropriate than the expression of personal experiences resulting from the faulty operation of the heater. What is significant for assessment purposes is the relative valuing of different responses.

The request for action by the agent was represented variously. Although most students seemed to interpret the prompt as an instruction to write a request to the agent to come and fix the gas heater, some students included a sentence or two to ask the agent for an apology. Some others included arrangements for an appointment. The social-interpersonal aspects of the letters varied depending on whom the writers thought was responsible, whose problem it was, or what in fact was the problem.

Because the expected text-type is semi-formal or formal correspondence, the expression of interpersonal meanings is significant. Some basic (B) and intermediate (I) level texts contained extreme or excessively emotive and affective expressions. They created the impression that the writers were not familiar with the conventions for this kind of correspondence. Such statements included:

“Hello, Sir Madam. How are you today?” (B3)

“Hello. How are you?” (B10)

“Sorry, sir.” (in the end of text) (B4)

“I’m truly embarrasseed(sic) to trouble you in concerning this matter.” (14)

“I’m sorry to trouble you with some of my problems.”(letter opening) (17)

“please care about me.” (14)

“please don’t hesitate and talk with me.” (13)

“I’m very cold. I’m very sad.” (B10)

“If you were me, what would you think?” (15)

“I can’t bear any more.” (15)

Other expressions introduced a threatening tone, which does not seem to fit the circumstances of an overseas student asking for action to repair a heater, at least not at that particular stage of proceedings, ie. a more formal reminder for action to be taken. It is for example too early to threaten with legal action.

“… I phone law council(sic).” (B9)

“I trusted you, but I disappoint now, I think this is wrong of you …” (B8)

“Do you think this problem will harm the reputation of your company?…How can I trust the service of your company?” (15)

“…your company’s reputation…”(11)

“…show me an appropriate reaction including an apology for missing an appointment and for my inconvenience.” (12)

The question is to what extent such personal expressions impact on the judgement of raters. Are they significant? Are they too personal when judged according to the criteria set out in the IELTS Handbook? These are questions that require further investigation.

Another distinguishing feature in texts is the way writers’ texts “express opinions (views, complaints etc.)” (Handbook 2001, p. 13). Basic-level texts often include frequent uses of evaluative terms such as “good”, “bad” and “wrong” and they often include emotive words such as “sad”, “disappoint(sic)” with frequent lexical exaggeration such as “very” and ”!”.

“The house is very beautiful and wonderful but when I very cold, my gas heater wasn’t working, I was very (?) torable. […] I’m very sad.” (B3)

“I feel very sad. […] I feel very cold.” (B10)

”…I trusted you, but I disappoint now, and i(sic) want to get a good impression about you and agency. I think this is wrong of you and your responsibility, and i(sic) trust you to repair everything by next week.” (B8)

Some intermediate-level as well as some basic-level candidates also used emotive terms to express their opinions and complaints, but utilising a wider range of terms or expressions:

“I’ve been patient to wait”; “I was very frightened”; “It is very dangerous.”; “Please care about me.”; (14)

“I want you to show me an appropriate reaction including an apology for missing appointment and for my inconvenience.”(12)

“Please consider my problem about this house.” (13)

“If you were me, what would you think?”; “How can I trust …?”; “I can’t bear any more.” (15)

“I am very sorry to write this …”; “So I will be very glad if you can repair…” (16)

More advanced-level texts which do not include emotive (ie. apologetic, pleading, angry, threatening) terms to express opinions, views or complaints, give the impression of a writer’s linguistic maturity in handling the text as formal, but nevertheless personal correspondence.

3.2 Features of texts

The IELTS Handbook refers to appropriate “rhetorical organisation”. This is understood as referring to the schematic structure of the text. This is clearly important, for reasons outlined above, as raters appear to evaluate according to expectations of the way a text should be organised. This is the socio-cultural reason for conventions in the organisation of a text. The schematic structure of a text is part of the realisation of the other meta-functions of the text, the interpersonal and ideational purposes of the text.

Almost all of the candidates included minimum required information in their texts in a sequence which corresponded to the task:

1. self-identification: tenant
2. the problem: a faulty gas heater
3. request: action by agent.

However, the representation of the details of these matters varied in ways which seemed to differ according to candidates’ performance levels. Many basic- and lower intermediate-level candidates’ texts used ineffective organisation for the presentation of information. For example:

“Duplication” of explanations, ie. explanation of one’s explanation:

”…I have a problem about the gas heater was not working. I telephoned to the agency at the your company one week ago and I asked your agent to repair the heater in my house. It has not been done.

I arrived there and turned on the heater but it was not working. Although, I telephoned immediately to your company. I forgot her name. And she told me about this problem, it was not problem. Tomorrow, She will tell this problem to customer service. Until today, nobody repaired the heater in my house.” (B1)

Crowding or packing different types of information into one paragraph:

“I want you to come to repair the gas heater because now, it was the winter. Every night, it was very cold and I could not sleep a long time. I think the gas heater did not have any gas. Every times(sic), I called to agency, the operator told me that they would send servicer(sic) to repair the gas heater tomorrow.” (B5)

Inclusion of not-so-relevant expressions in factual explanatory paragraph(s):

“Now, it’s time to go through winter. The weather is becoming cooler and cooler, especially at midnight. I was waken up by the cold air several times. All of these things happened because of the gas heater which hasn’t been working for more than one week. I’m truly embarrassed to trouble you in converning this matter. But I’ve got to do. I have phoned you one week before and specified the bad heater to one of you agents. He promised me…” (14)

On the other hand, some basic-level texts have appropriately organised and composed letters because the authors adopted vocabulary from the prompt, without including additional explanation in their own words. In one text (B2) 31 words out of 66 are in the prompt text.

“Dear Sir/Madam

How are you?

We are borrowing a house through your office.

but the gas heater doesn’t work.

One week ago we telephoned your office and asked to repair the heater.

However, it has not been done yet.

We hope to repair the heater as soon as possible.

Because autumn season will come soon.

I really hope and waiting to repair soon.

your sincially …”

The selection of linguistic expressions has enabled the student to carry out the task. Once the writer starts using her own words, the text loses precision and the meaning becomes less clear.

There is a practical question about the required length of the response (150 words). Is 150 words a reasonable length for the purpose of the letter? Appendix One is a possible draft response. The text consists of 65 words and seems sufficient to fulfil the social purposes of the task. This needs to be considered in the light of general instructions for writing a business letter, which is to stick to the facts and be succinct. It might be misleading for candidates to pad the letter out to 150 words and may have contributed to the range of personal details candidates included in their contributions. It should be noted that the task is described in 50 words and that it contains wordings for doing much of the task. It is possible for a well-briefed writer to utilise the wording of the text as a jumping off board for their own text or, as in the case of the text quoted above (B2), for constructing the major proportion of a response.

A second text (UI 1), on the other hand, contains very similar content and is of a comparable length (71 words in total, 20 words found in the prompt), but the writer utilises different linguistic resources.

“Dear Sir/Madam

I rent a house through your agency. One week ago I realised that heater was not working and I asked for repair. And no-one has come here until now. So, I am writing for ask again for repair. I’d like to remind you I’m not happy with the heater broken because this week has being freezing.

Hoping answer as soon as possible.

Thank you in advance

C”

The writer’s comparatively larger linguistic resources are used in addition to the wordings taken from the prompt text. The letter accomplishes the social purpose for the letter, however it is very brief. With only 71 words the text does not fulfil the quota for the task. From the perspective of the task the question is whether it fulfils the purpose of business correspondence. Technically it does so, with brevity, directness and with appropriate textual selections and constructions.

Students’ building of cohesion in texts has impacted on their construction of meanings. Basic-level texts tend to lack continuity which influences readers’ understanding of the texts. It seems to reflect writers’ limited use of linguistic resources to link related ideas. The letters include textual features such as:

• confusing usage of linguistic “reference” resources (anaphoric terms)
• repetition of key word(s)
• problematic (ie. excessive, irrelevant) use of conjunction words (eg. “and”, “but”, “so”), particularly to begin sentences
• inaccurate use of verb tense and time.

Some of these characteristics are evident in the following texts.

(B5) “Dear Peter

My name is [author’s name]. I rent the house through the agency. But when I arrived there and turned on the gas heater, the gas heater was not working. So I telephoned to agency one week ago and told the agent to repair the gas heater but now it had not been done anything.

I want you to come to repair the gas heater because now, it was the winter. Every night, it was very cold and I couldn’t sleep a long time. I think the gas heater did not have any gas. Every times(sic), I called to agency, the operator told me that they would send servicer(sic) to repair the gas heater tomorrow. Therefore, today or tomorrow there was not anybody to come to repair the gas heater. However I will still wait. Thank you for sending servicer(sic) to repair the gas heater . please…(sic)

Your sincerely”

[author’s name]

Many basic- and intermediate-level authors used the word “But” or “but” in the sentence or clause beginning position, intending to make it serve the function as a problem raising marker.

(B7) “Dear Sir

I telephoned you one week ago. But your answer has been nothing yet. I have not cooked from one week ago. Because I can not use the gas heater. It has been very cold in winter, but I can not use the fun(sic) heater too.

Generally, I think that you have to come to fix the gas heater up three or five days. Because, you rent a house.

I think that you are busy. But I want to come that you fix the gas heater.

If you come to fix. I want to call you. But I am busy everyday too.”

Intermediate-level texts sometimes lack textual continuity, but its occurrence is less common compared to the basic-level texts.

The analysis of the texts as a whole suggests differences between basic and intermediate levels. The texts written by basic-level candidates show limitations in the lexico-grammatical resources used for realisation of the task. The evidence for this includes:

• the texts are visibly short
• there is straightforward transfer of words and clauses from the prompt text. This, however, makes the candidates’ texts look reasonably correct and appropriate
• the texts repeats words and phrases
• the lexical and logical cohesion of the texts is sometimes inappropriate and in other cases ineffective.

The texts written by intermediate-level candidates include some evidence of authors’ more extended knowledge of vocabulary and sentence structures, but it also shows that the writers sometimes struggle with accuracy of expression and textual coherence. Irrelevant or unnecessary elaboration with additional explanations, along with ineffective use of lexical and logical cohesion, sometimes makes the texts look poorly organised and difficult to follow. The next example is of an intermediate level text (14), which illustrates considerable skill in the construction of meanings.

“Dear Sir,

How do you do?

I am your tenent who has been here for two weeks. At first I should thank you for helping me find a comfortable and quiet house to rent. And I really like to live here very much.

Now it’s time to go through winter. The weather is becoming cooler and cooler, especially at midnight. I was waken up by the cold air several times. All of these things happened because of the gas heater which hasn’t been working for more than one week. I’m truly embarrassed to trouble you in concerning this matter. But I’ve got to do. I have phoned you one week before and specified the bad heater to one of our agents. He promised me to repair it at once. So I’ve been patient to wait until promised me to repair it at once. So I’ve been patient to wait until today. I’ve had a bad cold and cough. [If you can’t give me] It’s a very bad thing.

And now I think I want to describe >to you< what’s wrong with my gas heater >again<. When I turn on the switch, I can hear the sound of transferring the gas. It’s normal. And the red light is also flashing. But after waiting for ten minutes, I still can’t feel heat. Maybe sometime I can smell the gas even. I was very frightened >because< [that] the gas [will be] is very dangerous. So I can[t] do nothing but wait for you patiently.

I know you must be busy every day. But please care about me during your busy time. Hope you can check it up soonest and repair it well. --- thank you again. Before you come here, please phone me. The call >number< is 8234 5678. [I’ll] Hope your information [as]coming as soon as possible.

With Regards

Yours sincerely

Z”

5/7/01

This text has 314 words and is longer than suggested in the prompt. It is relatively cohesive, has a range of clause structures, uses relevant terminology and expressions for conveying the point of the letter. Textually the letter is clearly constructed. However, interpersonally it includes elements that sit uncomfortably in a business letter. Ideationally the purpose of the text is apparent, although there is reiteration, with some information repeated unnecessarily.

Upper intermediate level candidates have shown their ability to choose and handle relevant expressions utilising a range of lexicogrammatical resources. Relevant information is compactly organised and presented effectively. Texts are sometimes shorter than the required length, which, however, is perhaps due to the nature of the question topic.

3.3 Discussion of letter texts

Juxtaposing the last two texts illustrates some of the complexities in assigning scores to different scripts. Whilst it is possible to deal with technical requirements such as word counts it is not so easy to evaluate the combination of features which make up a text.

The analysis of selected features of the letters suggests the importance of the interpersonal meta-function, the extent to which the letters are more or less personal or formal. What is not clear is how this dimension is evaluated. The IELTS Handbook makes reference to “semi-formal or formal correspondence” (p. 13). This can be interpreted variously. The Handbook also states, “Part of the task realisation is to respond appropriately in terms of register, rhetorical organisation, style and content” (p. 13). These concepts could be interpreted differently and more importantly there is considerable scope for the linguistic realisation of these in students’ responses.

The linguistic or discourse choices made by writers in these texts may not be indicators of the discourse resources they have acquired. However, the writers’ choices are important for the display of writing ability for assessment purposes. It appears that scoring is primarily based on a candidate’s ability to make appropriate lexicogrammatical selections for the expression of meanings, and on the ability to compose appropriate texts of a general nature. The actual wording of texts may indicate inappropriate decision-making because of the ambiguity of the task or the decisions a writer makes about what kind of letter would best affect a response from the landlord.

What is of interest is that there are quite different ways of writing a text in response to the prompt. Raters respond to the content of scripts (Weigle 2002, p. 70). Content is constructed throughout a text in a variety of lexicogrammatical combinations. How are these accounted for in the assessment procedure? Criteria for judging or scoring texts need to take account of a range of text realisations, for example in the interpersonal meta-function, particularly when letters technically meet the requirements outlined for a task.

4.0 Analysis of Task Two texts

4.1 The task

The following topic was set for students as the second task.

Writing Task 2: Essay expressing your ideas

You should spend about 40 minutes on this task

Hotels, restaurants and businesses do not allow smoking inside. In public places such as airports smoking is banned.

This is a good idea, but it takes away freedom of choice. Some smokers do not like the bans.

Do you agree or disagree with banning smoking in public places?

Give reasons for your answer.

Write at least 250 words.

The socio-cultural context of the task is:

• Field: banning smoking in public places
• Tenor: formal or semi-formal (audience is author-examiners/course tutors)
• Mode: written.

There seemed to be little misinterpretation of the context of the prompt. The prompt included paraphrases for some potentially problematic words such as “public places” and “banning” by providing some examples “hotels, restaurants…” and “not allow…”, which would provide support for comprehending the task.

One basic-level text shows the writer’s misinterpretation of the proposition in the question sentence, “Do you agree or disagree with banning smoking in public places?”.

“My answer is disagree, because smoking is no good for ather(sic) not smoking people. If you smoking person, you must more careful for not smoking people. Maybe when you smoking near any child or baby”. (B3) [underlining added]

A few intermediate level writers included some re-definition of the word “public places”, despite the explanation of the term in the prompt. They appeared not to understand the range of meanings the word normally covers. They wrote, for example:

“Nowadays, most public places are banning smoking and even other kinds of places, which many people are paying for their usage, like restaurant, hotel etc. also are not allowing them to smoke inside.” (12)

“Nowadays, most of the hotels, restaurants and businesses do not allow smoking inside. Besides, smoking is banned in public places such as airports.” (19)

Basic level texts are short and in some cases unfinished, so there is insufficient text for the researcher to look at for analysis of students’ understanding of the prompt. It is not clear whether those texts resulted from poor interpretation of the prompt, or whether they were due to writers’ lack of linguistic resources and socio-cultural experiences.

4.2 The nature of the task

The starting point for the analysis is a consideration of the nature of the text expected in response to the prompt. The Handbook (2001, p. 13) provides the following information for candidates.

“In Task 2 candidates are presented with a point of view or argument or problem.

Candidates are assessed on their ability to:

• provide general factual information
• outline a problem and present a solution
• present and possibly justify an opinion, assessment or hypothesis
• present and possibly evaluate and challenge ideas, evidence and argument”.

The task requests students to agree or to disagree and to give their reasons. What kind of text is expected here? An expected text type seems to match Gerot and Wignell’s (1997) “hortatory exposition”. However, unless a candidate is familiar with the lexicogrammatical conventions of this particular text type, the expression in the prompt “essay expressing your ideas” may seem to allow them a wide range of interpretations, particularly about formality (register) and conceptual depth of content because the set task is categorised as “General Training Writing” instead of “Academic Writing”.

4.3 Discourse semantic analysis

Example (B4)

I think that banning smoking in public places.

Of course I like smoking.

But I can’t disturb to many people.

Smoking is harmful.

Freedom of choice is very import(sic) but we mus(sic) to harmful of many people.

So, smoking must to ban in public places.

Intermediate-level texts use lexicogrammar to organise and sequence the text and to express their opinions. There is use of paragraphing and expressions for staging the text such as ‘first’, ‘second’, and ‘in conclusion’. Despite the authors’ efforts, however, some lower intermediate-level texts often lose consistency or uniformity in argument and turn out to be difficult to follow, as in the example below.

Example (I13)

(298 words)

In this text the staging words and paragraphs do not serve the expected function of schematically organising the argument. What distinguishes intermediate-level texts from more advanced texts is the consistent development of an argument. In some intermediate-level texts, it is difficult to ascertain the author’s position until the final paragraph. With others, such as (I13) above, the author’s position on banning smoking in public places is not clearly stated in the text.

More developed texts are logically and structurally better organised. The arguments are generally consistent throughout the text and thus easier to follow. Although there are still several errors, mistakes, and inaccurate use of expressions, readers are guided in following the argument to the end of the texts.

Example text (UI2)

(240 words, underlines added)

In the text above (UI2), the first and the last paragraphs seem to serve the expected functions of introduction and conclusion. The body of the text is mainly devoted to the articulation and elaboration of points to support the author’s argument.

4.4 Content analysis

The presentation of an argument is built up through stages in the organisation of the text and through other linguistic features such as topic related words or technical terminology and with expressions of appraisal. In formal presentations of an argument some objectivity might be expected. This is established through the use of generic or non-human participants, nominalisation and abstract technical terms. Once a text satisfies a certain degree of development and consistency in organisation, for instance by using staging words, the development of the text depends on wordings for the expression of concepts which carry some objective force.

Basic texts give an impression of subjectivity because of the use of personal participants (the use of the first person), which tends to appear in the clause-beginning position as theme. Nominalisation and abstract technical terms are less likely to be used in such texts.

Example (B4)

I think that banning smoking in public places.

Of course I like smoking.

But I can’t disturb to many people.

Smoking is harmful.

Freedom of choice is very import(sic) but we mus(sic) to harmful of many people.

So, smoking must to ban in public places. (45 words)

Example (B10)

My opinion is agree. Banning smoking in public places.

Because I don’t like smoke. I hate smoke.

[…]

I hope sigaret(sic) disappear in the world.

That’s most better idea. (45 words in total)

Example (B2)

I write down my opinion about banning smoking in public places.

As far as I know it started not to do allow smoking at public places in Japan fifteen or twenty years ago.

I thought this was a good idea when I knew to start not to do allow smoking inside in bilding(sic) and in public places.

Because I was not smoking.

And then I did not like a lots of people who is smoking around me.

Meybe(sic) I think smoke is no good for people’s body.

Some smokers will not think good idea about there. However, for smoking should be have some rure(sic). (104 words)

Intermediate-level texts achieve some objectivity through nominalisation and the use of non-human participants. Technical terminology is also used. However, intermediate level texts are at times ineffective in the choice of relevant linguistic resources, which affects consistency and appropriateness in formality and the degree of objectivity.

Basic level texts exhibit some difficulty with cohesion in the presentation of an argument, as illustrated in the following examples.

Example (B9)

[1] I think If banned smoking in public places. [2] It is appreciate several problem. [3] because some people want smoke any where [4] but If banned smoking in public places some smokers very annoyed and later will smoke hide.

[5] However, If permit smoking any where, [6] also we’ll have big problem. [7] I don’t decide agree or disagree, that banned smoking in public places. [8] but, I think that another good method example, [9] It is equipment smoke restrict area, in hotels, restaurants and airport and so on. [10] I guess, most people like it, also some smokers like it.

In intermediate level texts, there is a greater degree of text cohesion, as in text I 13 below.

There are differences in the presentation of information in texts. More developed texts compress the information through the selective use of topic linked lexis. The use of a range of topic linked vocabulary builds up a writer’s argument. In basic-level texts, fewer content words are included, which limits the texts’ conceptual expansion. The lexical chains in the following texts (B9 and I13) give some indication of the range of technical terminology and its spread through the text.

Lexical chain in text example (B9) - whole text quoted above

Compared to the text above, the next text includes more topic related words showing to some extent the author’s analysis of the topic. However, repeated use of the same or similar content words and expressions displays limited conceptual range. Some key words are left unexplained, which influences comprehension of the text. The text consists of many short clauses of equal value and sometimes combined illogically with conjunctions “and” and “but”, or with a comma. The use of subordinate clauses and modifying phrases is limited. This limits conceptual expansion in the text.

Text (I13)

[1] I thought already about banning smoking in the public places, [2] and I am banning smoking, in public areas. [3] Because there are both nonsmokers and smokers in my society,[4] and we breath same air in the public places. [5] I know we approve the right of smokers [6] but should keep the health of nonsmokers. [7] firstly(sic), we live in together in the society, [8] and are breathing same air, [9] so we need fresh air. [10] If smokers have smoking in the public places, [11] there is full of the pollution air, [12] so nonsmokers upset bad air in the public places. [13] I think this in important problem in smoking. [14] Sometimes we feel pollution air, and tired. [15] There are many reason of pollution, [16] but smoking is important reason in the public places. [17] I don’t say about no smoking, [18] but smokers should reagred(sic) about nonsmokers in the public places, [19] and there are many smoking areas in the public places including ait filtering system. [20] If they want to smoke, [21] go there. [22] Secondly, we should keep our health, [23] but smoking causes cancer lung, [24] and we knew already about it, [25] but don’t regard. [26] Sometimes smokers don’t give to regard nonsmokers. [27] When I go down the street after smokers, [28] I can help breathing smoke. [29] We know passive smoking is more dangerous than smoking, but they didn’t know. [30] thirdly(sic), we educate young people about non-smoking, [31] because we know smoking is bad, [32] of course smoking reduce stress of study, but no good way. [33] We look after young people by beginner. [34] in(sic) conclusion, we know already smoking causes reason of pollution and bad health. [35] Smokers have the right of smoking in the public places, [36] but they recognize the right off(sic) nosmoker to breath flesh(sic) air not pollution, [37?] and keeping good health together. [38] Sometimes, we must help smokers to ban smoking in the public places.

Lexical chain in Text (I13)

More developed texts construct argument compactly, with a variety of topic-related terminology. Compared to the lower intermediate-level text quoted above, the following example (UI3) includes lexical variation, subordinate clauses, and the development of concepts.

Example text (UI3)

[1] There has been always contravacy(sic) [2] when something, [3] which is favoured by a great number of people and not others, become banned. [4] It seems quite fair to ban ciggrette(sic) smoking in public places due to the health hazard involved with smoking. [5] However, the number of smokers is not yet decreasing, [6] it’s rather, increasing. [7] There are also many smokers [8] who claim that the banning smoking in public places violates their “freedom of choice”. [9] Despite such claims, I strongly agree with banning smoking in public places. [10] The following paragraph provides two reasons why I disapprove ciggarette(sic) smoking in public places.

[11] At first, it is very well-known [12] that smoking ciggarette(sic) can cause a various forms of cancer. [13] Not only cancer but there are also greater risks of having cardio-vascular diseases involved with cigarette(sic) smoking. [14] What needs to be focused more is not the fact [15] that smokers can harm themselves by smoking [16] but it is equally important to notice [17] that non-smokers can also suffer from various kind of diseases caused by passive(sic) smoking. [18] It is a great commitment for the society to protect the health of those who chose not to smoke cigarettes(sic). [19] Another point I’d like to make is the right of non-smokers. [20] I strongly believe [21] that it is extremely important [22] that smokers to have their freedom of choice. [23] It is, however, inevitable to argue [24] that non-smokers also have rights to “not being polluted” by passive smoking.

[25] It’s quite apparent [26] that banning smoking is very paradoxic in current society [27] where both smokers and non-smokers must have the freedom of choice as well as equally protected rights. [28] I’d like to conclude [29] that it is our great commitment now to solove(sic) this complex issue in 21st century. (281 words)

Lexical chain in Text (UI 3)

4.5 Comment

The analysis of Task 2 texts reveals the numerous linguistic options writers chose for the composition of the essay. The first issue of importance is the nature of the text type expected in response to the prompt. The task is interpreted differently by students. As there are generally social expectations of discourses appropriate to a particular purpose, it is suggested that the variations in the students’ texts will influence the evaluation process. The question is what text outcomes are expected from an argument text in this instance? Is a more formal type of argument rated as superior to a more personal type of writing? Although some organisational features in the texts distinguish lower intermediate level texts and more developed texts (upper intermediate), it is difficult to find definite distinctions, in terms of schematic structure of text and logical consistency in argument, between the two levels of texts. This is discussed further in the next section.

5.0 Discussion

This study asked two main questions: What language features indicate written language development at different levels? Could these language features be identified and used to complement the IELTS descriptors for rating written performance on the General Training Writing Module (Tasks 1 and 2) at different bands? These questions are discussed in the ensuing sections.

5.1 Features of texts

The exploration of different text features in this study assumes the importance a complete text plays in the evaluation process. The analysis has made an initial examination of some of the features in students’ texts, which may contribute to scoring scripts at different levels. The analysis is by no means exhaustive. A few of the more apparent features have been extracted for analysis. In this section these are summarised.

Appendix 1 is a summary in response to the first question, “what language features indicate written language development at different levels?”. It gives an overview of some of the elements in the texts of students, which seem significant for achieving the social purposes of the tasks. This summary displays the features of texts from a functional perspective, which were successfully or less successfully handled at different levels of performance.

In the data the features, which seem to distinguish basic texts from intermediate, can be nominated, but in the range of texts analysed for this study it has not been so easy to identify particular features which distinguish lower and upper levels of performance. Basic-level texts often showed authors’ struggle with textualisation of factual information (explanation), which seems to be caused mainly by limitations in using a range of vocabulary. Poor control of the interpersonal function in basic-level texts resulted in limitations in the expression of complaint and/or request in correspondence (Task 1). Textualisation of complaints and/or requests in Task 1 was better handled by intermediate-level candidates, but they sometimes lacked lexical variety and consistency in the interpersonal function.

Although intermediate-level texts included a wider range of lexicogrammatical resources, they also showed authors’ difficulty in effective textualisation of factual information. Accuracy and appropriateness in wording and structure were often problematic (Tasks 1 and 2). Basic- and lower intermediate-level texts were often characterised by the use of short clauses joined by “and” and “but” conjunctions. As a consequence, the authors were not as successful in developing their arguments. One of the basic problems in the composition of arguments in Task 2 was the lack of expressions used by authors to communicate a sense of objectivity when presenting their point of view. Excessive use of personal participants, for example, tended to reduce the strength of argument. In Task 2 the organisation of opinion and information with the use of paragraphs and staging words, when incorporated appropriately in the context, was effective.

More developed texts demonstrated fluency in handling the selection of appropriate content words with cohesive and clearly linked concepts. Although the more advanced writers occasionally experienced some difficulty with sentence structure and made some mistakes and technical errors in verb conjugation and single-plural relationships, these did not usually affect readability. These were compensated for, through the use of cohesive elements and technical terminology. On the other hand some of the texts were largely grammatically correct, but did not cover a range of concepts, or did not follow conventional discourse structures, and therefore seemed less persuasive. However, the linguistic fluency created relatively convincing texts.

5.2 Descriptors for assessment

The second focus question of the study was: “could language features described at different levels of performance be identified and used to complement the IELTS descriptors for rating of written performance on the General Training Writing Module (Tasks 1 and 2) at different bands?”. Appendix 1 above depicts in summary form some of the numerous linguistic elements which combine in the composition of texts. Some features seem to be more important than others for successful completion of the task. These may include the use of technical terminology, the tenor of a text and the organisation of a text through the use of cohesive devices. However, as separate linguistic items or features they do not constitute a text. The organic nature of a text means it can not be reduced to a few linguistic resources. Raters respond to the lexicogrammar woven as texts. They have expectations of what texts should do in conventional linguistic terms. For this reason the language features which typically characterise particular text types appear to play a significant role in the rating process.

The sample texts examined in this project reveal a wide range of possible responses. So what text features are influential in the determination of levels of performance and might be used for formulation of rating criteria? This depends on the nature of the text or its social purpose. In the case of the first task it seems that the interpersonal element is significant. For text two a clear expression of a writer’s position seems to be critical: the essay needs a logical organisation appropriate to an argument text, which facilitates reading or understanding the line of argument or point of view of the writer. The instructions for the two tasks allow for a range of possible responses. In both cases the discourse features which may constitute the desired text types have been examined. These could form the basis for the selection of language descriptors for rating performance (Mickan and Slater forthcoming). Lexicogrammatical features have been identified and used for teaching written text types (Cope and Kalantzis 1993, Gerot and Wignell 1994), and recent curriculum frameworks have described features to be used for the determination of levels of performance (Department of Education and Children’s Services 2003). These developments offer lexicogrammatical descriptors which could be tested empirically. The use of such descriptors could contribute to the validity of testing as they are task-specific and offer more precision as criteria for assessment than general descriptors such as style and content.

6.0 Conclusion

This study has compared the language features of written texts in order to characterise different levels of performance on the IELTS General Training Writing tasks. Although certain features have been identified, it appears that the more successful texts integrate different features in different ways to achieve their effect. In reviewing all the texts, what emerges as most significant is the sum of the features of text organisation, conceptual depth and variety of lexicon. Even relatively successful texts do not achieve competence in every feature: they might be well organised but the conceptual analysis or representation might be narrow or limited. Some texts are hard to follow but contain ideas realised through the incorporation of technical terminology or content words. The analysis suggests that it is not so easy to identify particular lexicogrammatical features that distinguish lower and upper levels of performance. A text might be advanced in one feature, such as the schematic structure of the text, but not in another.

Although this study identifies some of the lexicogrammatical features which appear to constitute more mature or developed texts as responses to the two tasks in the General Training Writing module, the analysis does not suggest an instructional focus on discrete linguistic items. Rather it suggests the centrality of the social purpose of each task. The function of the task needs to be understood by prospective candidates so that they construct responses with appropriate lexicogrammatical selections for the realisation of the task. Candidates’ success in the task is affected by how specific and goal-oriented the task is.

A possible difficulty for prospective candidates is the interpretation of advice in the Handbook in the section for the General Training Writing module. On one hand it is suggested that attention be paid to register and rhetorical organisation, style and content, on the other hand candidates attention is drawn to performance descriptors such as vocabulary and sentence structure. The analysis of texts in this

study suggests that a focus on isolated vocabulary and sentence structure distracts from the semiotic purpose of texts. What is significant is how the elements of a text contribute to the fulfilment of the social purpose of the task. What may therefore be crucial for writing prompts is clarity in the social purpose of the tasks set for examination. Candidates need to interpret the prompt in order to make appropriate selections for building information in the text, for the organisation of the text as text type and for handling the level of formality and informality in the text. Candidates appear to require clear indications about such aspects as tenor or level of formality, as appropriate interpersonal selections are important aspects of task realisation.

Scoring texts such as those in the General Training Writing module is a complex semiotic process mediated through a rater’s expectations of an appropriate response. It is apparent from the big differences in the students’ papers that the social purpose of the two tasks can be realised through quite different combinations of lexicogrammatical elements. Raters need to make decisions about which are more appropriate to the achievement of the social purpose of the task. Given the importance of the nature of the discourse that constitutes the social purpose of a text type, it is essential to include such significant elements in the criteria for scoring papers.

The measurement of written language performance requires more than the analysis of individual linguistic or content elements. It is how these are combined to create texts which seems to be important. The study of the discourse of texts focuses on the meaning of the text and the linguistic resources for making meaning. Qualitative analysis of the texts provides some insights into at least some of the characteristics of texts to which raters may respond in making judgments of the quality of written scripts. Those selected for discussion in this study are only a starting point for this type of analysis. Future research will need to take up two implications of this study, apart from the obvious need for further analysis of texts at nine levels of performance to identify clusters of lexicogrammar at different levels of performance. The first is developing task and discourse sensitive criteria for rating scripts and trialing these. The second is exploring, with procedures using verbal protocols, raters’ use of text specific and discourse representative criteria.

This study, in looking at the social purpose for which the writing is done, and in attempting to see the evaluation process from a social function viewpoint, proposes that the effect of the integration of the various components of a text is more important than the individual components themselves. This suggests future research into the way the various components of texts interact and influence raters in their scoring of written texts. Clearly rating is a whole-of-text activity which involves semiotic processes associated with interpreting the text, the prompt and the criteria. We know that raters attend to content (Weigle 2002); that they interpret the representation of criteria in the text differently (Smith 2000); and that they interpret criteria differently (Brindley 2001). What the analysis in this study suggests is that there are nevertheless specific text features that may be significant for scoring written texts.

Acknowledgements

I was able to undertake this study with the assistance of a research grant from IELTS Australia. Thanks to Sasha Hampton, IELTS Program Manager, for her support in the preparation of the final document. Particular thanks to my very able research assistant, Yuhiko Fujiwara for long hours of detailed work. Thanks also to the management and teachers at the Centre for Applied Linguistics in the University of South Australia (CALUSA), an IELTS Testing Centre in Adelaide, in particular to Cathy Coutts, Director of Studies, who helped with the organisation of the data collection. A very special thanks goes to the students at CALUSA for making their fascinating texts available to wider scrutiny. Finally I am grateful for the helpful comments of an independent reviewer.

Bibliography

• Brindley, G. (ed). (2000). Studies in immigrant English language assessment, Volume 1. Sydney, National Centre for English Language Teaching and Research and Macquarie University
• Brindley, G. (2000). ‘Comparing AMEP assessments: a content analysis of three reading assessment procedures’. In Brindley, G. (ed). Studies in immigrant English language assessment, Volume 1. Sydney: National Centre for English Language Teaching and Research and Macquarie University (45-79)
• Brindley, G. (2001). ‘Investigating rater consistency in competency-based language assessment’. In Brindley, G. and Burrows, C. (eds). Studies in immigrant English language assessment Volume 2. Sydney: National Centre for English Language Teaching and Research and Macquarie University (59-80)
• Brindley, G. and Burrows, C. (eds). (2001). Studies in immigrant English language assessment Volume 2. Sydney: National Centre for English Language Teaching and Research and Macquarie University
• Claire, S. (2001). ‘Assessment and moderation in the CSWE: processes, performances and tasks’. In Brindley, G. and Burrows, C. (eds). Studies in immigrant English language assessment Volume 2. Sydney: National Centre for English Language Teaching and Research and Macquarie University (15-57)
• Cope, B. and Kalantzis, M. (1993). The powers of literacy: a genre approach to teaching writing. London: Falmer Press
• Department of Education and Children’s Services. 2003. The South Australian Curriculum Standards and Accountability Framework
• Gerot, L. and Wignell, P. (1994). Making sense of functional grammar. Cammeray, NSW: Antipodean Educational Enterprises.
• Halliday, M. (1985). An introduction to functional grammar. London: Edward Arnold.
• Ishikawa, S. (1995). ‘Objective measurement of low-proficiency EFL narrative writing’. Journal of Second Language Writing (4:253-272)
• Mickan, P. (Forthcoming). ‘Beyond grammar: text as unit of analysis’. RELC Conference proceedings 2001. Singapore: Regional English Language Centre.
• Mickan, P. and Slater, S. (Forthcoming). Text analysis and the assessment of academic writing. Canberra: IELTS Australia
• Mickan, P., Slater, S. and Gibson, C. (2001). ‘A study of the response validity of the IELTS writing subtest’. In R. Tulloh (ed). IELTS International English Language Testing System Research Reports 2000 Volume Three. Canberra: IELTS Australia (29-48)
• Smith, D. (2000). ‘Rater judgments in the direct assessment of competency-based second language writing ability’. In Brindley, G. (ed). Studies in immigrant English language assessment, Volume 1. Sydney: National Centre for English Language Teaching and Research and Macquarie University (159-189)
• Weigle, S. (2002). Assessing writing. Cambridge: Cambridge University Press

Appendix 1.0

Draft Response to Task One

Researcher’s draft response for Task One consisting of 65 words. It is a possible formulation to meet the requirements of the task.

Dear Sir

On Thursday last week I telephoned you about the heater not working in our unit. As it is very cold I requested that you fix it as soon as possible. A week has passed, and I remind you that you promised to get it repaired within one week.

This letter is a request to repair the heater as soon as possible.

Sincerely

PM

Appendix 2.0: Summary of analysis

Textual or discourse indicators of performance levels: More basic -> More developed