IELTS: Student and supervisor perceptions of writing competencies for a Computer Science PhD

Abstract

A PhD in any discipline requires a student to produce a substantial written document, which is then assessed by a group of experts in the specific discipline. In the discipline of computer science, it has often been noted anecdotally that many students struggle with the English writing skill needed to produce a thesis (and other documents, such as scientific papers). English writing skill issues seem particularly acute for students for whom English is not their first language, especially as undergraduate degrees in computer science generally do not require students to undertake significant amounts of English writing.

In this project, we investigated the level of competence in written English that is appropriate for Australian PhD students enrolled in Computer Science. In particular, we sought to determine the appropriate level of writing skill required, how the level of skill may change during the students’ candidature, and the reasons for this change, as perceived by both students and supervisors.

We approached these questions by surveying both students and PhD supervisors from a variety of Australian universities, to determine both their perceptions of the writing skill requirements that are appropriate, difficulties encountered, and support services, in the context of the English language learning background of all participants.

We also analysed the performance of students on a given writing task, which was assessed by experienced PhD Computer Science supervisors, English for Academic Purposes support staff and by an IELTS examiner.

We found insufficient awareness of the writing supports available, a need for writing support targeted at technical writing, and an average supervisor expectation of IELTS 6.5 for writing at PhD commencement.

1 Introduction

A PhD in any discipline requires the production of a substantial written document, which is critically assessed by experts in the field. The combination of depth of research in the discipline and the ability to explain research issues and technical solutions means that PhD graduates are often valued for more than just their discipline knowledge. However, it is our observation that many students struggle to attain sufficient competence in writing research documents, particularly students who do not have English as a first language and who may be from a different academic literacy tradition than those of the Australian academy. This is acute in the field of Computer Science (CS), for which competence in writing substantial documents in English is generally not a significant part of undergraduate training (Gurel, 2010).

The aim of this project is to investigate the writing competence of PhD students enrolled in CS degrees in Australian universities, as perceived by both PhD students and supervisors. We aim to obtain a broad understanding of the factors related to the progress, or not, of PhD students in their competence in written English, particularly for scientific documents, from the commencement to the completion phase of their candidature.

Australian PhD students are generally required to pass several milestones based on written reports, and to present a final thesis and seminar on their work. For this reason, the main focus in this project is on writing competence, rather than other aspects of scientific communication.

PhD graduates are often valued for competence in writing technical documents rather than their discipline knowledge per se. Taylor, Martin and Wilsdon (2010) have identified that 53% of Science PhD students in the UK moved to careers outside Science after graduation. In Australia, the graduate employability statistics for 2014 showed that a lack of communication skills is a major reason (48.6%) why existing positions are not filled, with CS graduates being the hardest to place (53.5%) (Graduate Careers Australia, 2014). Many companies do not employ PhD graduates because they are perceived as being overqualified or being deficient in some important attributes such as working effectively as part of a team (Group of Eight, 2013). There are also significantly high unemployment levels among graduates coming from non-English speaking countries (Arkoudis et al., 2009); the key reasons being graduates’ levels of English language proficiency and workplace readiness (Arkoudis et al., 2009).

Another aspect of the importance of communication skills for postgraduate students is the increasing dependence of research funding on external sources and commercialisation projects (Group of Eight, 2013). Increasingly, the main drivers of research in CS are private enterprises that require researchers to not only show a good return on investment, but to explain, often to non-technical readers, the significance and progress of their research. This makes it increasingly important for postgraduate students to develop communication skills, including writing, during their PhD studies.

Many postgraduate students fail to improve their communication skills significantly over their candidature; a number of factors may contribute to this. Students coming from non-English speaking backgrounds with different cultural norms need to adapt to a new living environment as well as learn new and appropriate use of communication skills. In some cases, this can lead to postgraduate students being unable to comprehend their instructors and classmates in postgraduate courses (Liu, 2011). In addition, non-native speakers sometimes have to pretend to understand the conversational content in exchanges with native speakers, often leading to negative impressions being formed by those teaching them (Terui, 2011). Many such students form their own cultural groups with others from the same country to cope with the feeling of isolation, which further limits the opportunities to interact in English.

Lastly, Australian PhD requirements generally include little or no coursework components, which restrict opportunities for teamwork and social interaction (Group of Eight, 2013).

One significant difference between PhD studies in Australia and several other countries is that a PhD student is generally not required to pass an oral PhD viva examination in order to graduate. This increases the importance of written communication for Australian PhD students, as the examiners of their thesis will not have the opportunity to discuss with the student in person any matters arising from their thesis.

This report is organised as follows. Section 2 discusses the relevant literature. Section 3 outlines the research questions, design and approach to analysis. Section 4 presents the findings. Section 5 discusses the findings in relation to the research questions, prior to concluding in Section 6.

2 Literature

The purposes of a PhD are to develop skills in conducting and presenting research, to add to existing knowledge in a particular discipline, and to integrate into the academic community of their chosen field (Thomas & Brubaker, 2000). That is, unlike undergraduate study, the focus is mainly on acquiring research expertise. Completing a PhD in any discipline, therefore, requires the production of a substantial written document that is critically assessed by experts in the field. Such writing requires students to have gained an understanding of how ideas are presented, debated, and constructed within that discipline (Wingate & Tribble, 2012).

To effectively write, students must understand the expectations and conventions of their academic community (Belcher & Hirvela, 2005). Gaining academic literacy includes learning to write for a specific audience, logical organisation, paragraph-development, writing clarity, sentence structure and grammar (Zhu, 2004). Hence, it is not surprising that both local and international students find it difficult to develop the ability to effectively read, reason, critique, and write in a specific discipline within a short stipulated period (Wingate & Tribble, 2012). It is also important to note that academic writing varies, not only with discipline, but also with genre (Hyland, 2002). Swales and Feak (2000:7) define genre as a “recognized type of communicative event”. Examples include journal papers, grant applications, technical reports and theses. Indeed, within the fields of CS and computer engineering, about 90 writing genres have been identified (Orr, 1999).

The extent of difficulty experienced in gaining academic literacy appears to differ between the sciences and humanities. Undergraduate students in science and technology fields usually receive less practice in writing than students in humanities (Kayfetz & Almeroth, 2008), due to the relative difference in both the amount and type of writing required. Academics in science fields also assign a smaller variety of writing assessments than academics in humanities and social science (Cooper & Bikowski, 2007). CS graduates continue to lack in written communication skills despite strong guidelines by professional bodies to alleviate these problems (Dugan & Polanski, 2006). A survey of undergraduate Computer Science courses found that many do not have stand-alone writing classes and those that do are taught outside of the discipline (Burge, et al., 2012). Taylor and Paine (1993) found that a quarter of students taking a fourth-year CS course had never written a term paper before.

There could be multiple reasons why CS academics hesitate to add more written assignments into their courses. Many academics in CS disciplines regard writing ability to be of secondary importance when compared to those in humanities and social science (Casanave & Hubbard, 1992). The time taken to assess and explain reasons for marks allocated for writing is another reason, and some academics may take the view that teaching effective communication is outside their area of expertise and should instead be the domain of communications and English departments (Burge et al., 2012; Carter et al., 2011). Academics may have little formal training in how to teach writing and view setting writing tasks as taking away class time or adding to their already heavy workload (Taffe, 1989). Lack of experience with writing based pedagogies is another possible explanation for failing to set writing assignments (Tircuit, 2012).

It is also important to note that the ability to construct a written argument in English is an aid for developing and refining ideas. Writing is important in CS not simply for publicising findings, but also because the discipline of writing and refining the text helps to codify and formulate ideas (Zobel, 2004). In recent years, a minimum number of publications in international journals have become a necessary pre-condition for PhD graduation in many universities (Huang, 2010). The increased requirement to publish hampers non-native speakers, who have traditionally perceived English as playing a minor or secondary role in their PhD progress (Huang, 2010). Moreover, in the past when the need for explicit measurement of PhD progress and maximum limits on candidature duration were not strictly imposed, non-native speakers had many years to improve their language skills while they focused mainly on developing their ideas (Huang, 2010). Increasing pressure to publish in high-quality journals, which applies to both supervisors and PhD students, means that supervisors may limit the level of freedom given to non-native English speakers in the preparation of papers submitted for publication (Huang, 2010).

Limiting the opportunities for writing in turn may hamper the self-efficacy of students. A recent survey of international students pursuing a PhD in Australia indicates they perceive their level of writing skill as inadequate, although they believe they are improving over time (Son & Park, 2014). Their feedback shows students want English for Academic Purposes (EAP) programs tailored for their own discipline. It has been posited that explicit intervention by teachers is needed for PhD students lacking writing skill. Students lacking proficiency in English have been shown to benefit when their research training is supplemented with courses designed collectively by discipline specific researchers and EAP practitioners (Huang, 2010).

In recent years, some universities have introduced academic writing courses specifically designed for computer scientists. One such course addresses several common challenges faced by CS graduate students including organisation of content, discussion of data, the use of appropriate details, and transitions (Kayfetz & Almeroth, 2008). Students were introduced to a free-flowing style of writing, and peer editing and group editing were introduced. Such writing exercises can help to complement thesis and journal-writing skill as students in such a setting can be free of any power relationship that exists with a PhD supervisor (Huang, 2010).

Others have formed collaborative teams combining EAP practitioners and practising scientists using a methodology that combines EAP practices and genre analysis (Cargill & O’Connor, 2006). Results from programs using these strategies suggest that the writing skill of PhD students is likely to improve when the expertise of established computer scientists is combined with that of EAP professionals (Wilmot, 2016).

The following section presents the research questions, outlines the design, which uses a mixed methodology, and describes the participants, and approaches to analysis.

3 Context of the study

Our research questions were designed to understand the experience, perceptions, and attitudes of CS PhD students and their supervisors regarding doctoral writing. We were interested to know what writing entry requirements were considered adequate, whether writing improves during candidature, and what type of writing support was most effective.

3.1 Research questions

The study addressed the following research questions and sub-questions.

1. What are the writing skill requirements for success in a CS PhD degree (as perceived by supervisors, students, English language assessors, and student services English language support specialists)? 1.1. What are the main difficulties with writing that CS PhD students experience?
2. How does writing skill change throughout the course of a CS research degree?
3. What are the perceived reasons for variation in English writing skill during a CS PhD Degree? 3.1. What are the opinions and attitudes of CS PhD students and supervisors regarding existing services that support student writing?

The following section articulates the design of the surveys and writing tasks, participant selection, and approaches to analysis.

3.2 Research design

The project used a mixed methodology, employing both quantitative and qualitative analysis of student writing progress from the perception of three key stakeholders: doctoral students, supervisors, and EAP professionals. We designed surveys to capture the experience and attitudes of students and supervisors. To better understand the writing skill level of doctoral students, and supervisors’ expectations of student writing skill, we adopted the analytical judgement standard setting method (Pill & McNamara, 2015), which determines cut-off scores via numerical analysis of panel scores of student work. To learn more about supervisor reasoning regarding the scores they allocated, we qualitatively analysed the transcripts of the discussion of allocated scores to sample pieces of writing. All data was gathered and analysed in 2017.

3.2.1 Surveys

We designed an extensive student survey to capture background information about student gender, age, their first language, length of time spent in an English-speaking country, details of any previous English-speaking tests, such as IELTS, Teaching of English as a Foreign Language, and Cambridge, including their score, location and date the test was administered.

We surveyed their awareness, usage and perceptions of the helpfulness of different types of writing support, such as a university drop-in centre, writing club, journal club, thesis boot camp, writing tutor, writing mentor, and other language services. We asked them to rate their English writing skill for different PhD-related tasks, whether they believed their writing skill had changed during their candidature, and if so, what had contributed to this change. The survey included both open and closed questions to provide more detailed information. The complete question list and responses to the student survey can be found in Appendix A.

We surveyed students and supervisors to gauge both groups’ perceptions of the students’ level of writing competence. The surveys provided data on the perceived level of English writing competence from the perspective of the student and the supervisor. Students were also surveyed to determine the types of support that they found most helpful for the writing requirements during their candidature. Other questions were based on observations found in the literature, such as the nature of the research environment in which the student worked (Gurel, 2010:10; Hellmann 2013:12). We also included questions to determine supervisor attitudes around doctoral writing.

Both surveys were administered online using the Qualtrics tool (www.qualtrics.com), which is a simple, free, easy-to-use web-based survey tool and which is recommended by our institution’s ethics advisory board, and adopted by many Australian universities.

3.2.2 Writing task

The analytical judgement standard setting method adopted requires a panel to examine pieces of writing completed by CS PhD students. The students were asked to complete a survey that included a writing task. The short writing task (see Appendix C) incorporated aspects of IELTS Academic Task 1 (the ability to describe a process; linking devices in the proposal; a suitable range of sentence structures; and evidence of appropriate non-technical vocabulary) and Task 2 (outline of the research problem; discussion; formation of an argument; writing that demonstrates justification), with content generally suitable for a CS doctoral student, that is, it was based on general knowledge and skill they should have after completing a CS undergraduate degree. To simulate normal doctoral writing practice, instead of the artificial nature of a handwritten examination, the students completed their writing online and were permitted to use any resources (except other people) to complete the writing task.

We invited CS PhD supervisors and two EAP professionals to form assessment panels and discuss the student writing. The two EAP professionals both had PhDs (one in the humanities and one in the sciences); they both lectured in support programs designed for doctoral students; and they both were responsible for working directly in supporting doctoral students’ writing (one school based and one working in the university central student language and learning services unit).

The panel assessment data activity operated in three phases. First, using two student sample writing pieces, the panel members assessed the samples for the level required of commencing and completing students respectively, working with a score sheet based on Pill and McNamara’s (2015) standard setting structures (se Appendix D). This asks the panel members to rate the piece of writing on a seven-point scale (with 1 being ‘unsatisfactory’ and 7 being ‘strong’). Panel members were reminded to consider their assessment on writing skill and not research skill; panel members were also encouraged to use the “between” categories freely, such as the “between not yet competent and competent” category 4.

Second, the panel members shared the reasons for their chosen scores, with the option of modifying, or not, their initial assessments, after a period of discussion. Third, each panel member assessed 16 further writing samples, two of which were common to all panel members, and the remaining samples were one of two sets of 14. Half of the panel members received the writing samples in reverse order to the other half, to counter-balance learning and fatigue effects.

The writing tasks were also independently assessed by an experienced IELTS examiner, using the four categories of writing assessment used to create an IELTS writing score: coherence and cohesion; grammatical range and accuracy; lexical resource; and task achievement/response.

The independent IELTS-like writing scores were used as the ‘fair scores’ to determine the cut-off scores, again following the method of Pill and McNamara (2015). We used the same approach for both commencing and completing scores, comparing them to the total writing score from the IELTS assessor. Pearson correlations were also calculated between the mean standard setting scores for each piece of writing and the different IELTS component scores, as well as the complete IELTS writing score. While IELTS scores are not exactly continuous, calculating the mean standard setting score results in a continuous variable; therefore, Pearson correlation was selected.

3.2.3 Participants

The participants included students enrolled in a PhD in CS, and PhD supervisors. Survey participants were recruited nationally. Writing task participants and panellists were recruited from universities in the Melbourne metropolitan area.

3.2.3.1 Student survey

We received 125 sufficiently complete responses from PhD students, 74 (59%) of which were male and 51 (41%) female. The respondents were spread across all stages of PhD study, with 40 (32%) commencing, 51 (41%) mid-way in candidature, and 34 (27%) completing. Seventy-five per cent of the students were between 26 and 40 years of age, with most past IELTS test takers being in this age range (see Figure 1).

Forty-nine per cent of students came from an Australian Technology Network University (ATN), a group of universities that focuses “on industry collaboration, real-world research with real-world impact and produce work-ready graduates to become global thinkers in business and the community” (https://www.atn.edu.au/). A further 26% came from the Group of Eight (G8) universities, which is made up of Australia’s eight leading research universities (https://go8.edu.au/). Most students (99%) attended universities with main campuses located in the major cities, and none attended other private universities.

Most students previously studied CS, with the next most common prior field of study being either Engineering or Information Systems.

Figure 1: Age range of student survey participants, divided between IELTS test takers and non-IELTS test takers

Participants were able to select their language from a list or enter it if it was not listed. The largest first language group was English (37), with significant cohorts having Mandarin Chinese (16), Persian (10), Bengali (8) and Arabic (7) as their first language, and many other languages having only one student selecting them (see Table A6 in Appendix A for the complete list). Most (65) did not speak additional languages other than English and their first language, with the most common other languages being Wu Chinese (13) and Mandarin Chinese (7).

While only 37 respondents stated that English was their first language, 68 said it was the language they were most proficient in for writing. Persian and European language speakers tended to write best in their first language, as did most Mandarin Chinese, Arabic, Vietnamese, and Bengali speakers. Students with nearly 20 different first languages (Bengali and Mandarin being the most common at five respondents each) listed English as their best writing language, but no student with English as a first language was better at writing in a language other than English. For the complete student survey results, see Appendix A.

3.2.3.2 Staff survey

Responses were received from a total of 44 supervisors, made up of 11 females and 33 males. The age range was fairly evenly distributed, with 1 under 30 years of age, 15 aged 31-40, 12 aged 41-40, and 16 over 50 years old. The universities at which they work were predominantly the G8 universities (18) and ATN universities (16), with small numbers from the Independent Research universities (2), regional universities (2) and other publicly-funded universities (5). One supervisor did not nominate a university.

The language questions for supervisors were presented in the same way as those for students. Twenty-seven supervisors identified English as their first language. Others nominated Hindi (3), Chinese Mandarin (2), Persian (2), Vietnamese (2), French, German, Italian, Japanese, Portuguese, Spanish, Swedish and Turkish. Twenty-three supervisors spoke no language other than English. Others included Bengali, Chinese Mandarin, Dutch, French, German, Italian, Japanese, Polish, Portuguese, Spanish, Swedish, and two other languages that were not specified.

Thirty-nine supervisors selected English as the language in which they are most proficient for written tasks, with other respondents indicating Chinese Mandarin, German, Japanese, Persian, and Vietnamese as their most proficient language. There were 12 supervisors whose first language was not English, but for whom English was the language in which they considered themselves most proficient for written tasks.

Thirty-two supervisors considered their English writing ability as highly proficient (Question 10), 11 as proficient and one as adequate. Thirty-nine supervisors had over 10 years of living in an English-speaking country, with 31 of these having over 20 such years. Thirty-eight supervisors had over 10 years of working in an English-speaking country, with 23 of these having over 20 such years.

The range of supervision experience was quite varied, with 12 having supervised no PhD students to completion, 14 having done so for up to five students, and 17 for more than five. The number of PhD theses examined showed similar results (13 with 0 examined, 15 with up to five, and 17 with over five). Most (42) had held a research grant of some kind, with 22 having held prestigious Australian Research Council (ARC) grants (seven with more than five ARC grants, 13 with more than five other grants) - an indicator of success in academia in Australia, particularly for CS. This shows that the survey had good representation across early career, mid-career, and well-established academics. For the complete supervisor survey results, see Appendix B.

3.2.3.3 Writing task

Twelve panel members participated in the analytical judgement standard setting activity in one of three workshops to assess short writing tasks of PhD student participants. The panel included two EAP professionals who had successfully completed a PhD, as well as having many years of experience working with doctoral students in workshops along the themes of conducting literature reviews, critical thinking, and writing a thesis. Also, the EAP professionals established and facilitated peer-to-peer doctoral writing groups, as well as one-on-one engagement, academically supporting students throughout each stage of writing a thesis.

The 10 CS supervisor panel members represented different universities and a mix of gender, seniority, supervision experience, cultural diversity, and English as a first or additional language. An additional panel member participated via an online simulation of the standard setting session; this panel member’s input was included in the quantitative but not the qualitative analysis.

3.2.4 Qualitative analysis

We coded thematically with all qualitative data double coded by two separate researchers. Due to the relatively small amount of qualitative data, a manual approach (as opposed to Nvivo software) was chosen. After each individual coder completed their coding, they compared codes and modified them as necessary to reach agreement. The coding was in two stages. The first stage involved reading and re-reading the textual data to decide on themes. Themes were then consolidated based on the two coders’ discussion. Inter-rater reliability was calculated using Cohen’s Kappa coefficient, and found to be at least 0.86 (0.813 to 0.9 at a 95% confidence interval) for all codes. Where agreement could not be reached, a third team member determined which of the two primary codes was accepted. Schreier (2012:206) argues that where there is disagreement between two primary coders, working with a third coder, ideally with expertise and understanding of the topic as was the case in this analysis, is a feasible and valuable approach to qualitative analysis.

3.2.5 Quantitative analysis

Descriptive statistics were generated for survey questions, and we attempted to discover relationships between key variables. Means were compared using both 95% confidence intervals and effect sizes. Pearson and Spearman correlation coefficients were calculated to determine the relative strength of relationships between aspects of writing skill and past IELTS scores. Pearson correlation assumes the data has a normal distribution and is continuous, which is not quite the case here, since IELTS scores tend to be rounded to the nearest half. The IELTS scale is also unlikely to be linear (interval). Therefore, in addition to Pearson’s correlation, we calculated Spearman’s ranked correlation coefficient, which makes no assumptions about normality, and can be applied to ordinal data.

4 Findings

As described above, we have three main research questions and three sources of information - the student survey, the supervisor survey and the writing task. However, each of the three research questions is related to each of the three sources, and so it is not always simple to separate the material uniquely to each question. When material is relevant to more than one question, we will present it for the earliest relevant question. Therefore, most of our data is relevant to the first research question, and hence this first section will be considerably larger than the following two. In particular, the writing task data will be presented as part of the analysis of the first research question, despite potentially also being relevant to the other two.

4.1 Research question 1: Writing skill requirements

The first research question and subquestion are re-stated below, followed by the findings from student and staff surveys, standard setting and panel qualitative analysis.

1. What are the writing skill requirements for success in a CS PhD degree (as perceived by supervisors, students, English language assessors, and student services English language support specialists)? 1.1. What are the main difficulties with writing that CS PhD students experience?

4.1.1 Student survey

4.1.1.1 English language experience of participants

To provide context for student perception of writing skill requirements, we asked about their prior English language experience and skill. In addition to asking which languages the participants spoke, we asked about their time in an English-speaking environment with the following questions:

1. How many years have you lived in an English-speaking country?
2. How many years have you studied in an English-speaking country?
3. How many years have you worked in an English-speaking country?

Participants were asked to select one of the ranges as shown in Table 1. For each question, there are two clear peaks, with one being at the ‘>20’ response and the other at the median (shown in bold type), indicating those who probably grew up in an English-speaking country versus those who did not. The exception is the final question on years working, which appears to have a third peak (mode = 40, shown in italics), probably mainly consisting of students who have not yet spent much time in the workforce.

Table 1: Time spent living, studying and working in an English-speaking country

Just over 50% of the respondents had sat an IELTS test (65, with 60 having not sat one).

Table 2 shows the scores the students gave for the various parts of their IELTS tests.

The mode for overall IELTS score is 6.5, being the typical requirement for Australian universities.

Table 2: IELTS test scores

Modes are shown in italics and medians in bold

We asked students what year they sat their IELTS test (see Table 3).

Table 3: Year of IELTS test

Mode and median are the same.

Students stated (free-form text) where they sat their IELTS exam, which occurred in various countries including: Australia (19), China (10), Indonesia (5), Iran (4),

Malaysia (4), Bangladesh (3), Vietnam (3), Europe (3), South America (2), Sri Lanka (2), Korea (2), Pakistan (2), Chittagong (1), Saudi (1), Jordan (1), India (1), Philippines (1), and New Zealand (1).

In addition to IELTS, the Test of English as a Foreign Language (TOEFL) was a common test, with 21 respondents stating they had sat it previously. Of those who provided their TOEFL score, the range of results was 69-570. Other tests mentioned were Pearson (5), and Graduate Record Examinations (GRE) (3).

4.1.1.2 Writing skill as perceived by students

We received 116 responses to the four-level questions about writing proficiency on three writing tasks (application proposal, confirmation of candidature proposal, academic publications), and 115 for thesis writing. For each of the four writing tasks, the majority of students (66-73) perceived their English writing ability to be at least Proficient. When it came to writing for academic publications and thesis, however, the most common response shifted from Proficient to Adequate, and more participants selected Inadequate (See Table 4).

Figure 2 shows that the average perception of proficiency in writing of students who had previously completed an IELTS test was lower than for those who had never sat for an IELTS test. When the proficiency for publishing is examined in relation to past IELTS test scores, there is no clear linear relationship, with the median for IELTS 6.5 and 8 being Proficient, and medians for IELTS 6 and 7 being Adequate.

Figure 2: Average perception of proficiency: IELTS vs no IELTS

Table 4 shows the perception scores given for each specific writing task at each stage of candidature. On average, participants considered themselves Proficient for each writing task. Commencing students perceived themselves as more Proficient, on average, for writing an application proposal, compared to later stages. A higher proportion of completing students considered themselves Highly Proficient at writing theses and academic publications.

Table 4: Proficiency ratings for different candidature stages

Median in bold and light shading, quartiles in pink shading, mode in italics.

We asked students which aspects of English writing they had difficulty with. Table 5 shows their responses.

Table 5: Difficult aspects of English writing for 111 CS PhD students

4.1.2 Supervisor survey

Table 6 shows supervisors’ level of agreement with each statement about student writing, together with the average score for each statement (with ‘strongly disagree’ scoring 1 and ‘strongly agree’ scoring 5).

Table 6: Supervisors’ agreement level for each statement in question 18

Median of 42 responses shown in bold.

There was strong agreement amongst the supervisors that written communication skills are important (1), and that their students’ written communication skills improve during their PhD studies (8). However, there was also strong agreement that poor writing distracts focus from research (6), that insufficient writing skill have both impeded their students’ progress (3) and added to the supervisor’s workload (4), and that supervisors routinely edit their students’ writing (7).

There was also agreement (but to a lesser degree) that: students’ communication skills are appropriate for publishing papers by the end of their PhD; it was sometimes difficult to differentiate between poor writing skill and poor research skill; and supervisors would frequently refer students to the English writing support.

The most disparate support was for the acceptance of a PhD student with strong research skills but poor written communication skills, on which opinion was almost evenly divided, with a slight leaning towards disagreement. The strongest level of disagreement was with the statement that English language requirements are adequate, with significant but lesser levels of disagreement about the use of professional editors, whether for a thesis, a paper or for other writing tasks. There was also some disagreement with the statement that the English language services provided are adequate.

Question 19 was an open-ended question in which we asked: “Under what conditions would you accept a student with poor written communication skills?” Table 7 summarises the analysis for the question, and henceforth reported findings indicate the main identified themes in boldface italics. Forty-one respondents answered this question, with 10 stating they wouldn’t take on such a student, and two saying they wouldn’t anymore.

“They are too time-consuming to be worthwhile.”

“Following many bad experiences I will no longer accept such students.”

Another stated:

“If I didn’t have direct evidence of the poor written communication skills!”

One respondent indicated it depended on whether it was only “…minor issues - grammar, spelling, etc.”.

Twenty-four responses indicated that demonstrated prior competencies, such as research (9), logical thought (4), technical skills (3), academic results (4) mathematics (2), coding (1) or domain (1), would need to be present. Some mentioned previous publications. In some cases, demonstrating strong potential was sufficient, or if they were “…highly recommended”, had a “great personality”, in addition to other factors being present. Motivation was mentioned 11 times, and this either referred to general passion to succeed, or “willingness to improve” their English. Funding was mentioned twice, with specific reference to a scholarship or that funding needs “to be spent on a student immediately”.

Three respondents mentioned that the topic would need to be closely aligned with their research, and one stated that it should be a “Good research problem to work on”. Support was mentioned as a factor once.

“If the students is very promising and there are adequate support services for written and verbal communication skills.”

One respondent emphasised that:

“All students’ writing starts out poor (research writing is a skill many native English speaking students also take time to acquire).”

Another stated:

“When they bluff their way through the admissions process (with high IELTS scores and research proposals that turn out to have been edited by someone else).”

Table 7: Statistics related to qualitative analysis of supervisor survey question 19

The supervisors were asked to rank 10 different writing difficulties seen in their students’ work from 1 to 10, with 1 indicating most frequently occurring and 10 indicating least frequent. Table 8 shows that the difficulties of most concern were:

• clarity of meaning
• cohesion (flow).

Both received the two highest numbers of 1st or 2nd rankings (24 and 20 respectively), and two lowest in the Sum row, which is the sum of the weighted rankings, that is, the ranking multiplied by the number of supervisors who gave it this ranking.

Structure, Expression and Grammar were the next highest, both in terms of the number of 1st or 2nd rankings (14, 9 and 10 respectively) and on the sum measure, with structure ranking slightly ahead of Grammar, which in turn was slightly ahead of Expression. Vocabulary (technical or general) was ranked rather low with only 6 and 5 supervisors ranking these in the top four difficulties, and most (38 and 30) putting these in the bottom five. Spelling was ranked only marginally higher, with 29 supervisors ranking it outside the top four.

The other difficulties supervisors specified were incompleteness, lack of precision, lack of practice, plurals and articles, and synthesis. Lack of practice clearly is not in the same category as the other difficulties listed, which are about skills or elements of writing or language. One supervisor noted that “…this is a difficult question because difficulties vary greatly between students”.

Another supervisor said that:

“The above ranking exercise was quite difficult, in that I found myself involuntarily tending to rank in order of seriousness, and it required a conscientious effort to force myself to rank in order of frequency, and I’m still not confident I’ve achieved this! So I imagine there may be some distortion to the former ranking in the data you get.”

Table 8: Rankings of writing difficulty

Median rank in bold.

Question 16 was an open-ended question in which we asked which aspect of student English writing is the most difficult to manage. Forty-two respondents answered this question, with structural aspects being reported most frequently as the aspect most difficult to manage. (Table 9 shows the summary of answer codes for this question, as well as for question 17.) Most foundational aspects mentioned by supervisors were of a grammatical nature, with one person mentioning: “Taking proper care with spelling, punctuation, and typesetting”. Of the 18 structural comments, nine respondents specifically mentioned “Cohesion”, while four used the term “Flow”. Seven mentioned “Structure”. One participant stated a reason for the difficulty with structure (and expression) “as there is no single solution to offer them”. Another emphasised that:

“This is a problem for both native English and non-native English speakers”.

Two respondents referred to difficulties with structuring or building an argument.

Fourteen responses were coded as related to Expression. “Clarity” was explicitly mentioned nine times and “Expression” three times. One respondent stated:

“Clarity of meaning is the most difficult to manage as a supervisor because it’s difficult to correct. You need to ascertain what they are trying to say before you can suggest improvements.”

Another mentioned the difficulty where “…there is a need to explain or translate a conceptual or mathematical idea into a form that is more accessible by people who may not be exactly in the same area.” Yet another respondent referred to “story telling”. One respondent emphasised that cohesion, structure, and clarity “…are co-dependent. Students have difficulty managing these”. Another mentioned “precision” as a difficult aspect, in addition to clarity.

Some comments related specifically to research skills. Building an argument and “their scholarly thought processes” were the main two. Other comments included “student fudging”, “Getting them to write at all” and general proficiency “…in English writing of international students” being “below high school level, despite passing IELTS test [gaining level 6.5 and above in writing]”. Another respondent emphasised the “lack of professional writing skill, even native speaking students”.

Question 17 was another open question, asking which aspects of student English writing are the most important to improve. Nine respondents mentioned “Grammar” as the aspect most important to improve. One gave the reason “…so that feedback can instead focus on things such as research content!”, and similarly, another said:

“Avoiding distracting, basic errors which cause the reader to focus on trivia instead of the message”.

Ten respondents mentioned “Cohesion”, three “Flow”, and nine “Structure” for this question. One wrote:

“If people don’t understand the difference between highly cohesive writing and poorly cohesive writing, they can’t write cohesion into their work. Often providing examples doesn’t work because cohesion can be too nuanced for an un-seasoned reader”.

Another wrote:

“There is a need to identify all the important ideas and to place them in the right order”.

Seventeen respondents mentioned “Clarity” in their answers, and “Expression” was mentioned four times.

Three responses were related to considering the reader’s perspective, for example, “stopping them from assuming the reader can interpret what is in their head”. One respondent mentioned:

“The fact that content and facts alone are insufficient, and that the information has to be communicated effectively to the audience”.

The research-related main ideas expressed included “working to a plan”, and the logical presentation of concepts, reasoning and argumentation.

Finally, one respondent stated:

“Increase writing capability of international students up to high school level.”

Table 9: Number of responses and codes for supervisor survey questions 16 and 17

4.1.3 Standard setting

Thirty-two student participants completed the survey that incorporated the writing task online in a lab on campus. In this section, we report on the standard setting activity that used the writing samples collected from the student participants. The following section reports on a thematic analysis of panel members’ reasons for the scores they gave to writing samples.

In the standard setting exercise, 13 academics each rated 18 writing samples according to the scale from 1(unsatisfactory) to 7 (strong), resulting in 6, 7 or 13 judgements per piece of writing (four writing samples received ratings from all academics). Two judgements were made for each writing sample, one assuming it was the work of a commencing student, and the other a completing student. The scores of interest for standard setting are the “between” categories, which are used to determine the cut-off scores for the main categories. In the scale used here, scores 2, 4 and 6 represent between ‘not yet competent’ and ‘unsatisfactory’, between ‘competent’ and ‘not yet competent’, and between ‘strong’ and ‘competent’ respectively. The standard setting calculation is based on reference “fair scores”, which for our study are the IELTS overall writing band determined for the writing samples, by the IELTS examiner. The cut-off score for a particular between score is then calculated by averaging the IELTS fair score for the writing samples that received that between score.

Table 10 shows the result of the standard setting exercise. When academics assumed the writing was by commencing students, only two items received a score of 2, and the remaining “between” scores were distributed between 4 and 6. The score of 2 was associated with an average IELTS mark of 6, a score of 4 equating to approximately 6.5 and the score of 6 equating to approximately 7 on the IELTS bands. Expected writing standard for completing students was slightly higher, leading to more writing samples being given lower scores, as reflected in the lower mean of all standard setting scores, compared to commencing students.

On average, the writing samples were judged to be between ‘not competent’ and ‘competent’ for completing students, whereas for commencing students they were considered ‘competent’ on average (5.24, where a competent score is 5). Consequently, the standard setting technique produced cut-off scores for completing students that were higher, with the ‘not yet competent’ cut-off approaching 6.5, and the ‘competent’ cut-off approaching 7.

Table 10: Number of “between” panel scores given during the standard setting of 32 pieces of writing by 13 academics, and the resulting IELTS band scores from applying the writing task IELTS band score

Table 11 shows the correlations between the panel score means for each writing sample and the corresponding IELTS scores, including the components, Task Achievement/Response (TA/TR), Coherence and Cohesion (CC), Lexical Resource (LR), and Grammatical Range and Accuracy (GRA). The strongest correlation, shown in bold, was with the combined IELTS score. The strongest correlation with an IELTS component score occurred with lexical resource, and the weakest with task achievement/response (shown in italics).

Table 11: Pearson correlation between mean standard setting judgements at commencement and completion of a CS PhD respectively, and writing task IELTS scores

4.1.4 Panel qualitative analysis

The purpose of the panel members’ qualitative analysis was to discover what factors influence supervisors and EAP professionals when assessing CS doctoral student writing competency from the perspective of a commencing and completing student. Insights into what contributes to the change or lack of English writing skill development during a CS doctoral degree were also noted.

The qualitative analysis that follows is from the transcribed and anonymised discussion of the second phase of each of the three panel activities. Three major themes with multiple sub-categories emerged from the panel members’ text, namely: research and writing skill (69 comments); language characteristics (43 comments); and competence (32 comments). A fourth category was identified (3 comments) - where a participant felt unsure, undecided, and unable to articulate the reason for their assessment.

4.1.4.1 Research and writing skill

Research and writing skill was the most frequently identified theme in the panel members’ data. The sub-categories within this theme highlight critical thinking, writing skill, research skills, and the combination of writing and research skills (see below for participants’ quotes for each of these themes). Literature that discusses what is required to complete a PhD will include items such as the ability to recognise research problems, review critically, and have a sound knowledge of research methods, along with the ability to work independently, manage time etc. Also included are ‘communication skills’ covering academic writing skill and oral presentation skills for academic and non-academic audiences. The panel members’ data highlights the tension between ‘writing’ ability and ‘research’ ability when discussing the writing samples for example, “For completing PhD it is not something I feel very exciting but still I find that the writing is quite good”. Different sub-categories that support this theme are noted below.

Critical thinking

“… I think they can write quite coherently I just think they can’t think.”

“You’d want to find out more about whether they’d spent all their time trying to sound good rather than be accurate. But you’d know that they’ve at least got basic intelligence there …”

“Because I think if you can’t think, if you can’t assess a problem and answer it rationally. If you don’t have that skill, that’s much more important than your level of writing when entering a PhD. If that person has no training in a scientific viewpoint, they haven’t really answered the question. I think they’re totally incompetent for a PhD probably, but in terms of writing skill, sure they can write.”

“… they were asked to answer this question. I don’t think they’ve answered it. If they’re at the level of entering a PhD or leaving a PhD in either case they should know how to read a question and answer it. I would not give them another chance …”

“Indeed, my supervisor encouraged me [panel member]. At the beginning I said my English is not good, she said, the brain is more important than your English - you need to create something in science, you know with the idea not in English. That’s why he said you can do a PhD.”

Writing skill

“To me it comes to the question of what does the question mean by writing skill. Number two clearly can write coherent English, but I agree I would not take on example two as a PhD student. Even though I’d rate their writing skill as competent, I would not want to take them on.”

“Well I think there’s this dichotomy here of what does writing skill mean. If it’s just the ability to write correct English sentences then yes you can give person two a seven. But is it the ability to write an answer to the question, then I’m not sure they’ve done that.”

Research skills

“When they understand how to express - what order they have to express their ideas, what claims they’re making, how they’re supporting those claims, then the quality of their English wasn’t a major problem. People are going to be reading their work for the ideas not to be impressed by the flowery language.”

Writing and research

“Well, for example one, I would have said that it’s okay for a commencing student, but it’s obvious that the student will need to do work both in improving their ability to say things in English and quite possibly in their ability to sort out the ideas in the best order.”

“And so, when I’ve had the experience of supervising students who are writing up and their first language is not English, it was something that I had to get through to them really quickly, is that they have to work out which ideas they want to put down, and then expressing it in English is a separate skill.”

4.1.4.2 Language characteristics

Language characteristics refer to participant comments referencing the “mechanics” of writing, with sub-categories including grammar, syntax, style, and communicability. This was the second most commonly identified theme. There is a mix of comments. Some participants specifically addressed one item, “… I couldn’t say competent because there were just too many grammatical errors”, while others noted a number of language items in the one piece of writing. For example, the following comments identify cohesion, grammar, style, paragraphs, and topic sentences.

All panel members noted obvious language errors in the student writing, however, the significance of these language flaws in relation to overall successful progression and completion were perceived differently.

“You cannot submit a PhD that looks like this kind of level of English; it won’t get past the examiners. It’s true if you think about…if the supervisor was doing a lot of corrections, a lot of writing, maybe this student with that level of English could get through. But in some sense they shouldn’t because essentially someone else is writing their thesis.”

In contrast to:

“… again the language could be improved. But I see that the mistakes are things that can be addressed. They’re not huge, like the sentences are generally structured okay. It’s not like you’re starting from scratch” and “I think because I could read this and, even though there were errors, I could understand it”.

Further examples of Language Characteristic sub-categories are noted below.

Grammar (syntax, sentences, paragraphs …)

“So…I thought really I couldn’t say competent because there were just too many grammatical errors.”

“… and I think I was focused on lots of little grammatical errors. Um…that I found throughout it. And I thought okay, I’d like a student to be at least grammatically correct when they started.”

“But the problem is that the sentence are quite ah…the student used all simple sentences and sometimes I feel it’s a bit tedious or redundant…”

“…You know, the grammar’s not too bad. As you say, it’s understandable and I’ve read work that’s not. Um…so from that perspective it’s not bad. Sentence structure is pretty good. It’s sort of common grammar issues that we always find with English.”

Style

“It tends to be descriptive rather than analytical.”

“So that’s why I think…yeah it’s perfect English for a storyteller, but as a PhD in science, not to mention CS, whatever is maths, whatever is [unclear] demand to bring the technical component.”

“But, you know I don’t like anybody saying that something’s interesting. I don’t like anybody telling me that something is classic, that’s just not academic writing. You know, ‘there is no technology yet that adds hours to the day’ - all that sort of informal stuff is not, for me, something that academic writing does. It’s perfectly fine in other places, but not in academic writing.”

“I think I rated it for completing student down, way down, mainly because of the lack of conciseness of the language. You’re right, the grammar mistakes can be fixed, and they’re just the standard grammar mistakes that we see with English language all the time. You know, prepositions etc. etc. There’s a good sentence structure, but it’s not concise.”

Communicability

“Um yeah, there’s a broad generally good structure: the introduction and two set paragraphs with discussion and conclusion. However, the grammar is poorly written…the goal of a PhD is to be able to communicate clearly and this hasn’t been communicated clearly.”

“And there’s a lot of, um, complex noun phrases dumped in. So, it’s not my field, but it becomes confusing with all of those noun phrases. You aren’t being able to communicate well.”

4.1.4.3 Competence

Competence, the third theme and the least frequently identified in the data, refers to participant comments as to what informs their ideas on what is proficient and adequate writing for commencing and completing CS doctoral students. The comments revealed a range of responses with sub-categories including: competence according to stage (i.e. writing competence varies for commencing to completing students); publishable and ready (i.e. writing competence for drafts and final thesis and publishing vary); and competent with a caveat (i.e. if the writing was perceived as being by a national/international student - English as a first/additional language).

Competent - commencing/completing

“I would be definitely very happy if I had a PhD student starting the course with such writing.”

“I think I’ve taken on PhD students with this level of English at the start and get them through.”

“So basically they’re competent. They kind of, yeah, but yes it’s a different thing from whether I’d take them on. I don’t think I’d want to take this student on, whereas I’d take the other student on without a problem.”

“I would think that this student could improve their ability to say things in English over the course of their candidature. For completing PhD student, it would be a bit of a worry, and you might almost have to say it’s a three in the sense that in order…”

“So, if this was a commencing student, I would expect the student to have improved immensely by the time he or she is a completing student. The rudiments are there, the basics are there and I think it would be relatively easy for the student to continue working on those issues. They’re not insurmountable.”

“I think that if I got that writing at the start then there are things that you can work with. I’m just trying to think why that actually makes a difference, why is it…I’d obviously want them to be better than that if it were towards the end.”

“I’ve also taken on students with this level of English [a low score]. But they can develop over a candidature and hopefully the student would improve English over that time.”

Publishable-ready

“Clearly it’s not of publishable quality - whatever that phrase actually means.

Um and you would expect that for a completing PhD student, you know they should have language that’s at least approaching publishable quality, it might need brushing up still before you submit for publication…”

“Yeah… if it’s publishable quality, is it? It’s tricky isn’t it.”

“I kind of assume students are either writing a thesis or a paper, and so it’s not good enough to be submitted anywhere I would say.”

“I mean I guess you know, um, I mean competent to me means someone who could probably write something that could be submitted to a good publication venue. So, I don’t think this is ready yet for that, for my view. But I mean that said, you know, I’ve had PhD students start who probably have written worse than that. And certainly if someone was writing final thesis like that, it’s not ready.”

National/international - English as a first/additional language

“…the English could be quite good in that it looks like it comes from an international student. I’m not too sure, I’m just guessing. However, if it’s written by an international student, it’s quite good. I have supervised so many international PhD students; sometimes maybe you come to the stage that maybe I wrote half of it [laughs].”

“Just because I have dealt with so many PhD students and my expectations…I mean international PhD students, and my expectation is more or less what’s aligned with the standard which I have dealt with.”

“…maybe international PhD student nothing [unclear], their English is maybe not good but very hard working understand the maths, the theory and some very strict academic things. So for PhD they have the potential.”

“So that’s something when I first started supervising HDR students, I had HDR students whose first language wasn’t English. I thought that it might be a major problem, and it has never been a major problem because the students understand the distinction between the content and the way it’s expressed.”

“Maybe because I’ve dealt with a lot of international students and I sort of think… well this is a better quality than the students I’ve seen so I would put them in this space [competent].”

“…example one, the person who wrote that is…probably someone whose first language isn’t English and needs to polish up the grammar and things of that nature. That’s not the case for the person writing example two.”

“But we’re also, I was not thinking international/national - just a PhD student, just a person. I mean [panel member] and I have had a PhD [international] student, a couple, that are really very good. I mean [student] learned English five years ago and you can’t…you know now he writes…When he first started he was…down here, but now he’s strong. You know, really strong with his English. There are very few corrections you need to make.”

In conclusion, what factors influence supervisors and EAP professionals’ assessments of CS doctoral students’ writing competence and what factors may or may not contribute to progress were complex and multi-variable. Three themes that emerge from this set of data are the role and interplay of research and writing skill, an individuals’ English language ability, and their overall competence depending upon context such as stage, audience, English as an additional or first language. It was evident, however, that none of these were stand-alone influences; rather they are inter-dependent and evident at each stage of student candidature.

4.2 Research question 2: Changes in writing skill

The second research question is restated below, followed by the findings from student and staff surveys, and the writing task.

2. How does writing skill change throughout the course of a CS research degree?

4.2.2 Student survey

For those who were at the midway or completing stages of their PhD, the majority (46) felt their English writing ability had improved slightly during their candidature (see Table 12).

Table 12: How English writing ability has changed during candidature

4.2.2 Supervisor survey

There was strong agreement amongst supervisors that their students’ written communication skills improve during their PhD studies, and slightly weaker agreement that their skills are appropriate for publishing papers.

Table 13: Supervisor Likert scale responses related to change in writing skills

4.2.3 Writing task

Eighteen writing task participants had provided the IELTS scores for their most recent IELTS test. Table 14 shows the correlation between the writing band from their IELTS test and the writing task IELTS score. The strongest correlation is between the grammar component of the writing task scores and the past IELTS writing band. The relative strength of relationship appears to be more marked when using Spearman correlation than with Pearson.

Table 14: Spearman (Pearson) correlation between past IELTS writing band score and writing task IELTS scores

Figure 3 shows the difference between the means of the past IELTS writing band score and the writing task IELTS score, with 95% confidence intervals. The mean writing task score is slightly higher than the original score, but the confidence intervals overlap substantially, suggesting that there is no significant difference. However, with an effect size of about 0.34 standard deviations, there appears to be a medium size effect. That is, as the range of IELTS scores is quite small (4-7.5 for the IELTS test and 5.5-8 for the writing task), the relatively small change in average score represents a substantial change for the sample of participants.

Figure 3: Comparing the means of the past IELTS writing score and the writing task score

4.3 Research question 3: Perceived reasons for changes in writing skills

The third research question and subquestion are restated below, followed by the findings from student and staff surveys.

3. What are the perceived reasons for variation in English writing skill during a CS PhD Degree?

3.1. What are the opinions and attitudes of CS PhD students and supervisors regarding existing services that support student writing?

4.3.1 Student survey

Ninety-four out of 125 (75%) of all students answered question 23, which was an open question asking: “What writing language support do you think would improve your written English skills?” The key themes identified were practice/feedback, specific parts of English writing, and genre-specific help. Table 15 shows how frequently the themes were noted. The themes are discussed below (and shown in bold).

Practice and feedback were frequently mentioned as useful, whether it be from regular activities (“Weekly writing task and assessment”), or from ongoing research activities (“support for revising my publications drafts”). Some just mentioned “practice” without including feedback in any form in their response.

Where specific aspects of English writing were identified, the majority highlighted the need for grammar support. Other aspects mentioned include punctuation, sentence structure, and vocabulary. Some individual responses were “English”, “connections”, “expression”, “semantics”, and “organisation”.

Fourteen students mentioned genre-specific support. This was expressed as “academic writing” or “technical writing”, or specifically about papers (“how to write academic papers”).

Regarding the source of help, 18 mentioned formal help in the form of classes or workshops, such as “thesis writing workshops”.

Various resources were suggested, including grammar/spell checking tools, thesaurus, and good books on writing. Others suggested experts, academics, mentors, proof readers or native speakers for help.

Table 15: Count of qualitative codes for question 23

Eighty-six out of 125 students answered open question 24: “What writing language support do you think could have improved your written English skills earlier in your candidature?” This question was analysed with the same thematic categories as question 23. Again, practice and feedback featured frequently in responses. Similarly, grammar was the main language component stated. Students also commonly referred to genre-specific needs (“academic writing”) or specific pieces of academic writing, such as “literature review” and “proposal”.

In terms of support sources (question 24), similar responses were found to question 23, except one participant, who stated, “mandatory undergraduate academic writing courses”. Two students mentioned “self-study”. The resources that were identified were similar to question 23, although one participant mentioned “writing blogs” and another mentioned “reading more theses”. The experts mentioned were also similar to question 23. A summary of codes is found in Table 16.

In the next sections, we look at different student respondents’ experience of different types of support. Table 17 summarises the thematic analysis for these survey questions.

Table 16: Count of qualitative codes for Question 24

4.3.1.1 Writing drop-in centre

A writing drop-in centre is a student service language support centre. We asked students about the availability of a writing drop-in centre at their university (question 25). Their replies were Yes (54), No (4), and Don’t know (58), while the remaining 10 left the question blank.

When asked an open question on how they found out about a writing drop-in centre (question 26), most replied from their supervisor (14), school (11), friends (9), student services, and PhD administration (6). Twenty-six had used the centre while 27 had not. Of those who had used it, seven found it very helpful, 15 found it helpful, and four found it unhelpful.

Twenty-five out of 36 students answered open question 29: “Why did you not use a writing drop-in centre?” The most prevalent answer was that the service was not needed, either at the time, or in general, with several students expressing confidence in their writing skill (“I believe I am already proficient”). A smaller number of students stated that time constraints were the factor that prevented them using the service.

A few students stated limitations of the service, such as “1-page limitation”, “They didn’t help on research papers”, and that it didn’t help with “…academic technical writing”.

Table 17: Count of qualitative codes for reasons not to use a writing support service

4.3.1.2 Writing circle

A writing circle is where a number of students and a facilitator meet to collaborate on improving student writing. When asked if a writing circle was available at their university (question 30), students said Yes (42), No (6), and Don’t know (66). Those aware of the service had found out about it from their supervisor (8), school (6), friends (6), email list (5), other students (4), and university PhD administration (5). Twenty-three students had used a writing circle. It was rated as very helpful (8), helpful (14) or unhelpful (1).

Similar reasons were stated for not using writing circles as for the writing drop-in centre (time constraints, not needed), but one person stated:

“I think technical writing needs someone from the same group, and the people I met are from other disciplines, and always we have conflicts in the style of writing so that why I do not use this”.

Another stated:

“I follow the strategies they posted online, but would rather write alone as I can focus better that way”.

4.3.1.3 Journal club

In a journal club, students and a facilitator meet to critically evaluate recent articles in the academic literature and collaborate on supporting student writing, such as literature reviews. Responses as to whether there was a journal club at their university (question 35), students chose Yes (10), No (10) or Don’t know (95). When asked who informed them, the primary source was supervisor (3), followed by student services (2) and one each for school, email, friends, other students, and university PhD administration. Only four had made use of a journal club with six responses for No. Two rated it very helpful and two rated it helpful.

Only six out of 17 students stated why they did not use a journal club. Lack of need for a journal club was the main reason expressed. One stated “Too time consuming” as their reason.

4.3.1.4 Thesis boot camp

A thesis boot camp is an intensive group writing program designed to provide late candidature research students with support in a focused writing environment, often for two to three days, the opportunity to progress their thesis. When asked if a thesis boot camp was available at their university (question 40), participants replied Yes (33), No (13), and Don’t know (68). The majority were informed by an email list (11), supervisor (7), friends (3), student services (3), school (2), other students (1), as well as the university website and word of mouth.

Only three students had made use of a thesis boot camp, while 30 said they had not. Two students found it very helpful and one found it helpful.

Thirty out of 41 students stated why they did not use a Thesis Boot Camp. The majority stated that they were not at the writing up stage yet. Others stated that it was either not available at a time they could attend, or there was too much demand for it. Another stated that it had a cost associated with it. One said:

“We formed a mini-boot camp in our department”.

Two indicated they might look into it in the future:

“Have not yet investigated the thesis boot camp”.

One mentioned they’d rather write alone “to better concentrate”.

4.3.1.5 Writing tutor

A writing tutor works with students at various stages of the writing process, not as a proof-reader or editor, rather to help students learn how to improve the organisation, flow, grammar etc. of their written communication. Many students were uncertain about the availability of a writing tutor at their university (question 45). Results to this question were: Yes (29), No (11), and Don’t know (75). They were informed by their school (7), supervisor (4), friends (4), an email list (4), the university PhD administration (4), student services (4), and other students (1).

Thirteen students had made use of a writing tutor, rating it as very helpful (8), helpful (4), and unhelpful (1).

Fifteen students out of 27 stated why they did not use a writing tutor. Again, lack of need and time constraints were mentioned, as well as cost (“needs to be paid”). Another stated “tutors come from different fields of study”. One participant stated they were “too shy, worry about my communication skills”.

4.3.1.6 Writing mentor

A writing mentor is a “critical” reader of students’ work, not judging what they have written, rather asking lots of questions so as to guide a student to clarify what they want to communicate and how best to do that. Thirteen respondents said a writing mentor was available at their university (question 50) while 12 students said No and 89 replied Don’t know.

They found out about a writing mentor from their supervisor (5), school (4), other students (2), PhD administration (1), and student services (1). Six had used one, while seven had not, with five finding it very helpful and one, helpful.

To those who gave reasons for not using a writing mentor, mentors were seen as either not needed or “Not convenient, the mentor was not student-oriented”. One stated “some are paid for”.

4.3.1.7 Other language services

Students were asked if there was another language service available at their university (question 55) and the responses were: Yes (9), No (8), Don’t know (97).

They were informed by their supervisor (1), school (2), friends (1), other students (1), student services (2), and library (1). The services mentioned were “library”, supervisor, Asian languages, ESL/English/International services, “new orientation”, and “Mandatory unit of study on writing academic documents”. One participant was “Not sure of the name”. Five respondents made use of it, with three finding it very helpful and two finding it helpful. Reasons for not using it were that it was not needed, although one person stated they were “not sure” why they didn’t use it. For the Asian languages, there was a lack of interest.

4.3.1.8 Factors contributing to change in writing skill

Seventy-one out of 90 students responded to open question 62, which asked what factors contributed to a change in their English writing ability (statistics related to this question are shown in Table 18). The majority emphasised practice and feedback as the main contributing factors. Within this category, six mentioned daily writing. Some referred to genre-specific writing, with 15 specifically mentioning writing papers, and a few others mentioning the other writing they were required to do in their PhD, such as proposals, and reports for their supervisor. Reading was also emphasised, with 16 of the 49 practice/feedback responses attributing the reading of papers to part of their improvement. Feedback and corrections from supervisors (expert help) was another source of improvement. One mentioned, “being open to learning and improvement”.

Specific English language elements were not mentioned often, but one mentioned “more vocabulary”, and another outlined the specific type of corrections from their supervisor: “Sentence structure and phrasing corrections”.

Various types of support were found in the student responses. Of the formal support mentioned, two attributed some of their improvement to writing circles, and others cited a writing workshop, and writing training. One person was taking “two subjects, in which I have to use English”. Informal support included writing in a team, and just being in an “English environment”. Resource use included “Following style-guides when writing papers”, “working by myself and being proactive of accessing resources”. Eleven students specifically mentioned their supervisors. A few felt their writing had not improved. One said: “It has remained the same but my confident has weaken [sic] as I was criticised by so many academics”. One stated: “I have improved in writing scientific papers, but this is offset by a general rustiness”. Another said: “Writing ability was already at a high level”.

Table 18: Count of qualitative codes for question 62

4.3.2 Supervisor survey

Overall, most supervisors stated that they did refer students to writing support services. However, question 20 asked: “If you don’t refer students to writing support services, please state why not”. Table 19 summarises the analysis of this question. Fifteen answers were excluded as respondents simply stated that they did refer students to writing support services. Three respondents were doubtful that the support services would help. Four emphasised the generic nature of the support services not being useful for technical disciplines. Three indicated that students were uninterested in using them, with one respondent stating that the:

“…effectiveness of these services relies on students recognising there is a problem and wanting to improve. I have trouble with some students thinking they do not need help with their written English and resisting my feedback”.

Two respondents indicated it was unnecessary because they only accepted students with a “reasonable level of communication skill”. Four mentioned a lack of awareness of what was available. A substantial number of respondents stated that they do refer students, one stating:

“I do refer them to the support services. Some do not use these as much as I would hope.”

Another emphasised that they did so “…no matter their English background”.

Yet another respondent said that:

“I do frequently and run the [service] myself for the faculty”.

Two other reasons were that academic writing could not be learnt for mathematics, and that courses already exist that some students choose to attend.

Table 19: Supervisor responses to question 20

Question 21 asked: “Which of the following writing services are available for students at your university?” Nearly half of the supervisors did not know whether a particular service existed at their university or not. Of those that knew, a drop-in centre was the most prevalent service in the knowledge of the supervisors (18), with a thesis boot camp the next most known (13), and a writing tutor the next (10). All services were known by some of the supervisors, but overall the level of knowledge of available English language services was limited.

The findings presented in this section are discussed in the following section. The discussion addresses writing skill requirements, changes in writing skill, and perceived reasons for these changes before concluding with reflections and lessons learned.

5 Discussion

This project allowed for a detailed picture of CS within Australian universities, with respect to doctoral writing. The 125 CS PhD students surveyed were spread fairly evenly across the three stages of candidature, and included a wide range of language backgrounds and ages. Thirty-two of the student respondents also completed writing tasks, which were assessed by an IELTS examiner to have overall bands ranging from 5.5 to 8.

The 44 supervisors surveyed had a range of ages and supervision experience, with a gender ratio of 25% female, almost double the typical proportion of female CS staff, and came from a variety of universities in Australia (with some bias towards the G8 and ATN groupings). Over half of them identified English as their first language, with the vast majority feeling most proficient in English for written tasks. This suggests that the supervisors as a group felt confident in their English writing skill for CS purposes, with a reasonable level of affinity with other languages enabling them to sympathise and understand the difficulties faced by non-native English speakers.

The sections below discuss the findings related to each of the research questions, and present our reflections and further recommendations. The results from the two surveys and the writing task analysis reveal how student and supervisor perceptions of student writing differ and the context for their perceptions.

5.1 Writing skill requirements

Research question 1 asked: What are the writing skill requirements for success in a CS PhD degree (as perceived by supervisors, students, English language assessors, and student services English language support specialists?

While various aspects of writing skill were identified as being relevant to decisions by supervisors, the majority of evidence collected tended to refer to a standard of writing skill, rather than specific elements. The writing standard was considered at two time points: commencing, and completing a CS PhD.

Students generally considered their skill to be proficient. In contrast, supervisors were almost unanimous in believing that English language entry requirements for PhD students were inadequate, with poor writing skill being an issue impacting both student progress and supervisor workload.

Evidence from the standard setting exercise suggests that academics’ intuitions about writing standards roughly matched the current entry requirements for Australian PhD programs, the average being just below 6.5. However, for many institutions the Writing band is only required to be 6.0, with a minimum overall IELTS score of 6.5.

It must be noted that official IELTS tests would probably lead to slightly lower scores than our writing task activity, and the magnitude of that difference is unknown. It is also unclear whether the IELTS entry score has influenced academics’ judgement about what is sufficient, or whether the IELTS band score reflects what academics would normally choose in the absence of formal English entry requirements. It is likely that there is a complex interaction of factors leading to the current expectations of academic supervisors, with the evidence of some of these factors revealed by the comments collected from supervisors, such as the amount of extra work required when supervising students with poor writing skill leading to supervisors refusing to accept such students.

There are a few limitations to be considered when interpreting these results. First, the IELTS-like cut-off scores cannot be equated with actual IELTS writing bands due to differences in how the writing task was executed, as well as its nature. Second, there were fewer than the recommended judgements for each specific writing task, to reduce the burden of participation for volunteer academics in the panel. Third, there were different numbers of judgements for some pieces of writing (four received 13 judgements, whereas others received six or seven), which may have an unknown effect on the final scores.

5.1.1. Main difficulties experienced

Research sub-question 1.1 asked:

What are the main difficulties with writing that CS PhD students experience?

The primary writing issues identified by students include cohesion, and writing for clarity of meaning, with the next factors being expression, structure, and technical vocabulary. More than half of the students surveyed considered cohesion to be an aspect with which they had difficulty.

The most frequent difficulties encountered by the supervisors were also clarity of meaning and cohesion, which were ranked more or less equally. Ranked lower but also close together were structure, grammar and expression, followed by punctuation. Technical and general vocabulary were ranked next, with spelling appearing to be of little concern. This ranking of clarity and cohesion above grammar and expression reflects student survey feedback and suggests that more problems arise from the logical flow of an argument than from the formal correctness of the English used, and a confounding factor could be cultural (Kachru, 1999). It is somewhat unsurprising to note that problems with vocabulary of either kind were relatively uncommon, although punctuation issues appear to be more frequent than may have been expected. Thus, problems experienced may be more to do with constructing a logical argument than with the intricacies of English grammar and expression, although these are more common than issues with punctuation, spelling, or vocabulary. Of course, in practice, these components of writing are likely to be linked.

It should be noted that there was no explanation given regarding the meaning of terms like “structure”, when presenting the questions to participants, so it is possible that the terms were interpreted in different ways. The only term that was described was cohesion, which was parenthetically defined as “flow”. Therefore, caution is required in interpreting these results.

The evidence from the writing task assessment by an IELTS examiner was that Task Achievement/Task Response was the weakest area, followed by Coherence and Cohesion. As with both student and supervisor survey responses, Grammar and Vocabulary appeared to be less of a problem.

The 15 statements about student writing were most revealing. As a group, the supervisors somewhat disagreed with the statement that PhD English entry requirements are adequate, yet they also fairly strongly agreed that their students’ written communication skills improved during their PhD studies to the point where the students’ skills were adequate for publishing papers. This apparently contradictory stance may be explained by the practice of supervisors routinely editing their students’ work, and the less common practice of referring students to the English writing services. There are other possible explanations, for example, supervisors’ initial perception of students’ entry level writing is inaccurate, supervisors are ‘teaching’ students how to improve their written communication rather than only editing their work, or students are accessing support outside of the university. Further investigation is needed to better understand this finding. It would also be interesting to discover more about incidental, as well as intentional improvement in writing during candidature.

Supervisors’ tendency to edit their students’ work comes at the cost of increasing the supervisors’ workload, impeding the students’ progress, and distracting the focus of both from research. This paints a picture of supervisors perceiving problems with their students written skills, and to a large extent, taking it on themselves to solve the problem, rather than relying on external services (with some exceptions). The opposition by the supervisors to the use of professional editors seems paradoxical, in that the use of editors would presumably lessen some of their load, but is perhaps due to supervisors feeling that such editors are not appropriate for technical writing.

The division of opinion on whether to accept a PhD student with strong research skills but poor written communication skills may reflect some bad past experiences on the part of some supervisors, where the supervision was too time-consuming as a result, or a conscious decision to prioritise research ability over the pragmatics of producing a PhD thesis. It seems clear that the supervisors perceive poor written communication skills to be a general problem, and that they are generally unaware of the English writing services provided by their university, or feel that the services are not appropriate for this group of PhD students.

The dilemma of distinguishing research skill from writing skill was discussed by panel members, and within the research team. While it is difficult to completely separate the two, it does seem likely that panel members did do so to a degree, which was reflected in the lower correlation between their judgements and the task achievement/resource component IELTS score. Perhaps, despite its goal being only to assess academic language and readiness for academic study, the academic IELTS test captures aspects of research skill via the task achievement/resource assessment. It may be dependent on the specific set writing tasks, however. The writing task we developed for the present study, while modelled on typical IELTS writing tasks, simulated the type of writing that a PhD student might be expected to do in a research paper, thesis chapter or paper review. It would be interesting in future work to compare supervisor judgements of research skill, based on writing samples in contrast to the writing skill.

5.2 Changes in writing skill

Research question 2 asked:

How does writing skill change throughout the course of a CS research degree?

Student feedback from our survey aligned with recent findings about writing skill of international students in Australia (Son & Park, 2014). Of the 125 students surveyed, most students (62%) felt they improved their writing skill since the commencement of their PhD career, but only 23% felt they had greatly improved.

Supervisors also generally agreed that their students’ written communication skills improved during their PhD studies to the point where the students’ skills were adequate for publishing papers.

The standard setting exercise provided both qualitative and quantitative evidence of supervisor expectations of CS doctoral writing. The panel’s scoring of student writing showed logical trends, in that expectations were consistently higher for completing students than for commencing ones and, on average, low-scored writing was associated with low IELTS writing scores.

When comparing past IELTS writing bands with writing task scores, a slight improvement was also found, but it is unclear whether this improvement is an artefact of the different methods of testing writing. However, it can be seen from our study that four different types of evidence agree that writing improves during candidature.

Writing task participants’ past IELTS test writing bands showed the strongest relationship with the writing task’s grammar score. Perhaps this suggests that grammatical skill is slower to change than other components of writing skill, particularly given that the previous IELTS test dates ranged from 2010 to 2017. Panel scores, in contrast, were most correlated to the overall past IELTS score, followed by lexical resource.

5.3 Perceived reasons for changes in writing skills

Research question 3 asked:

What are the perceived reasons for variation in English writing skill during a CS PhD Degree?

The evidence collected to answer this question was both direct and indirect. Students responded to an explicit survey question about the reasons for their change in writing skill, for which the majority attributed the change to practice and feedback, and many (11/71) citing the source of feedback as their supervisors. Indirect evidence was collected by asking about services that are typically available to support student writing. This is discussed in the subsection below.

5.3.1 Perceptions of existing services

Research sub-question 3.1 asked:

What are the opinions and attitudes of CS PhD students and supervisors regarding existing services that support student writing?

Despite almost all student respondents identifying aspects of writing that they found difficult, they tended to believe that they were proficient in writing. This self-belief, in addition to time constraints and limitations of the services available, were the main reasons for not using the support services they were aware of. A few students did access writing support, indicating a willingness to work on their writing, and they attributed their improvement to the use of the services.

While most universities offer services that can improve students’ writing skill, such as writing circles and journal clubs, the majority of students appear to be unaware of these services. Students who have participated in group-writing activities appear to have found out about them from a variety of sources including emails, friends, university administration, and supervisors. Based on the Australian PhD students surveyed, it appears the research training students derived from their supervisors may not be well supported by, or integrated with, many centrally organised writing support services.

The supervisor provided the primary source for recommendations to access the various writing support services, and this is of concern considering the lack of awareness of many supervisors of available support services. Student services and the school were the next most commonly used. In the majority of cases, the help provided by these services was rated as very helpful or helpful. Several students also mentioned they used tools for writing, such as the online grammar checker Grammarly, which are becoming more powerful for helping improve writing, spelling, and grammar.

5.4 Reflections and recommendations on methodology - lessons learned

We acknowledge that this study had some limitations, which indicate that conclusions should be interpreted with caution. However, many insights have arisen from the work, particularly the contrast in attitudes between students and supervisors regarding writing skill. In addition, the benefit of several different sources of evidence has provided corroboration in some instances and a more nuanced understanding in others.

Although we focused mainly on writing specific to CS, the type of writing skill needed may vary significantly between qualitative, experimental, and formal methods research fields within CS. We intend to organise, collect, and analyse data related to different areas separately in the future.

Some feedback that came from one of the standard setting panel members was that the table of information given in the writing task would normally have units, and be labelled more precisely, which may have affected the way some students wrote.

Other feedback from participants in the survey included the observations that seemingly simple questions, such as whether English was a participant’s first language, or whether the participant had studied, worked or lived in an English-speaking country, were more difficult to answer than it may seem. For example, one participant explicitly asked whether India was considered an English-speaking country, given the number of English-speakers who reside there. It was also noted that someone who was brought up to be bilingual would find it difficult to answer a question posed in such simplistic terms. Hence, it seems that the notions of “first language” and “English-speaking country” need to be carefully considered and explicitly defined, and any questions about such notions need to be crafted with these considerations in mind. In addition, it may be better to ask the participant to name the countries where they have lived, worked or studied, rather than asking them to classify the countries as either English-speaking or not. Further, for some countries where English is not an official language, the medium of instruction may be English, and possibly not consistently so throughout the country.

A further issue that arose from the supervisor survey was that the survey had asked them to rank the given writing difficulties in order of frequency. However, this supervisor had also considered ranking them in order of importance, which suggests that it would be an improvement to ask supervisors to provide two separate rankings, one for frequency and one for importance.

The conclusions to be drawn from the IELTS-like assessment of CS PhD writing are limited in that the method of answering the question differed - being online, the question itself was not a genuine IELTS test question, and the attitudes of the participants would be different to those sitting a genuine IELTS tests.

In this project, the constraints of time meant that we were unable to follow up issues that arose from the panels of supervisors. However, it would have been very helpful to be able to explore these issues more deeply with focus groups.

We trialled running a further online panel with academics, in which they assessed the first two writing examples in an online questionnaire, stated their reasons, read key points from the panel transcripts, and then assessed the remaining writing examples.

However, time constraints and recruitment issues meant we had insufficient participants in this project, although we can see this being used for testing scalable standard setting exercises in future.

The next section summarises the study, and makes recommendations and proposals for future research.

6 Conclusion

6.1 Summary

Our goal was to answer the following research questions:

1. What are the writing skill requirements for success in a CS PhD degree (as perceived by supervisors, students, and English language assessors)?
2. How does writing skill change throughout the course of a CS research degree?
3. What are the perceived reasons for variation in English writing skill during a CS PhD Degree?

We discuss each of these in the sections below.

6.1.1 Writing skill requirements

According to evidence from the standard setting exercise, the writing skill expectations of supervisors appeared to be slightly higher than the typical 6.0 writing band minimum requirement found in university entry requirements. However, our method of measurement was not an IELTS test, so it is not clear whether this was an artefact of the measuring instrument. Those who completed the student survey rated their skills between adequate and highly proficient. However, some supervisors commented in their survey that it was difficult to convince students to use services available to improve their writing. In contrast, one student found all the criticism from academics destroyed their confidence. The general impression is that academics would prefer greater writing skill from their students to make supervision easier, but most students are unaware of the need for greater writing skill.

6.1.2 Changes in writing skill

The standard-setting exercise showed an expectation from supervisors of improvement in writing (before completion) to a writing band approaching 7.0. Nearly all staff responding to the survey agreed that student writing improves during candidature. Most students also agreed that their writing improved. Writing task IELTS scores were also slightly higher than past IELTS writing bands on average. Further research would be required to determine whether students do in fact achieve the equivalent of a 7.0 writing band by the end of their candidature.

6.1.3 Reasons for variation in writing skill

Students mostly attributed change in writing skill to practice and feedback, with a significant number mentioning the source of feedback being their supervisors. While not many attributed their change in writing skill to formal support such as writing circles, and a relatively small proportion of participants used each type of formal support, nearly all who used them found them helpful.

6.2 Recommendations

Evidence from our study suggests that supervisors are more concerned about PhD student writing ability than students are, but that student writing skill does improve during candidature. Supervisors carry a significant load in developing their students’ academic writing skill, which could be better supported. Writing support services are helpful, though not always known about, and tend to be more appreciated when they specifically target technical writing. With increasing emphasis placed on publications, it may be necessary for academics to collaborate with EAP professionals to address specific needs for PhD students in CS, such as interpretation of data and paper organisation (Kayfetz & Almeroth, 2008; Wilmot, 2016).

Our recommendations are that more support for technical writing be made available, early on and continuously throughout the PhD, and that this be well communicated to both students and supervisors. A minimum IELTS writing band of 6.5 is recommended to match supervisor expectations, but the accuracy of this recommendation is uncertain. To obtain a more precise gauge of supervisors’ preferences, or for supervisors themselves to have a better understanding of the meaning of IELTS writing band levels when deciding on the acceptability of PhD candidates, samples of writing that are known to be of specific IELTS bands could be provided to them.

6.3 Future work

We have collected a substantial amount of data through this project, and there remain further questions that can be answered from it, such as the impact of supervisor seniority on attitudes and ratings of student writing, and the relationship between student variables such as confidence in writing and use of writing support services. It would also be interesting to discover if students do - or do not - reach the perceived and desired IELTS 7.0. While we had enough panel members for standard setting, it would be better to scale up the panel due to the need to divide the pool of writing assessments in half to limit the burden on panel members. Using a transcript from other panel members’ discussion as part of an online panel questionnaire was created and trialled, and we consider it a worthwhile project to pursue in the future.

Standard setting panel members mentioned various factors that influenced their decision regarding the standard of student writing samples, one of which was whether it was of “publishable” quality. While standard setting provided an IELTS score that represented CS PhD writing competence as judged by academics, it is left for future work to discover and define what publishable writing is.

Other areas that are promising to pursue include collecting data directly from student support services on the types of support accessed and specific writing issues that students wish to address. Data from student services will strengthen the evidence beyond that reported by students and supervisors by capturing actual activity as it happens. It would also be interesting to discover the type and extent of independent learning strategies students adopt to further build their research writing skill.

References

• Arkoudis, S., Hawthorne, L., Baik, C., O’Loughlin, K., Leach, D. & Bexley, E. (2009). The impact of English language proficiency and workplace readiness on the employment outcomes of tertiary international students. Canberra, ACT: Department of Employment, Education and Workplace Relations.
• Belcher, D. & Hirvela, A. (2005). Writing the qualitative dissertation: what motivates and sustains commitment to a fuzzy genre?. Journal of English for Academic Purposes, vol 4, pp 187-205.
• Burge, J.E., Gannod, G.C., Anderson, P.V., Rosine, K., Vouk, M.A. & Carter, M. (2012). Characterizing communication instruction in computer science and engineering programs: Methods and applications. Frontiers in Education Conference, 3-6 October 2012, IEEE, Seattle, WA, pp 1-6.
• Cargill, M. & O’Connor, P. (2006). Developing Chinese scientists’ skills for publishing in English: Evaluating collaborating-colleague workshops based on genre analysis. Journal of English for Academic Purposes, vol 5, pp 207-221.
• Carter, M., Vouk, M., Gannod, G.C., Burge, J.E., Anderson, P.V. & Hoffman, M.E. (2011). Communication genres: Integrating communication into the software engineering curriculum. 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T), 22-24 May 2011, IEEE, Honolulu, HI, pp 21-30.
• Casanave, C. & Hubbard, P. (1992). The writing assignments and writing problems of doctoral students: Faculty perceptions, pedagogical issues, and needed research. English for Specific Purposes, vol 11, pp 33-49.
• Cooper, A. & Bikowski, D. (2007). Writing at the graduate level: what tasks do professors actually require. Journal of English for Academic Purposes, vol 6, pp 206-221.
• Dugan, R.F. & Polanski, V.G. (2006). Writing for computer science: a taxonomy of writing tasks and general advice. Journal of Computing Sciences in Colleges, vol 21, pp 91-203.
• Graduate Careers Australia. (2014). The Report of the Graduate Outlook Survey: Employers’ Perspectives on Graduate Recruitment. Graduate Careers Australia.
• Group of Eight. (2013). The Changing PhD: Discussion paper. Canberra, ACT: Group of Eight.
• Gurel, N. (2010). An examination of linguistic and sociocultural variables in writing a dissertation among Turkish doctoral students. State University of New York, Buffalo, NY.
• Hellmann, K. (2013). What do I need to succeed: The case of Arab engineering graduate students’ self-perceptions of writing. University of Idaho, Moscow, ID.
• Huang, J.C. (2010). Publishing and learning writing for publication in English: Perspectives of NNES PhD students in science. Journal of English for Academic Purposes, vol 9, pp 33-44.
• Hyland, K. (2002). Genre: Language, context, and literacy. Annual Review of Applied Linguistics, vol 22, pp 113-135.
• Kachru, Y. (1999). Culture, context, and writing. In E. Hinkel (ed), Culture in second language teaching and learning (pp 75-89). Cambridge: Cambridge University Press.
• Kayfetz, J.L. & Almeroth, K.C. (2008). Creating Innovative Writing Instruction for Computer Science Graduate Students. ASEE/IEEE Frontiers in Education Conference, 22-25 October 2008, IEEE Saratoga Springs, NY, pp T4F1-T4F6.
• Liu, L. (2011). An international graduate student’s ESL learning experience beyond the classroom. TESL Canada Journal, vol 29, pp 77-92.
• Orr, T. (1999). Genre in the field of computer science and computer engineering. IEEE Transactions on Professional Communication, vol 42(1), pp 32-37.
• Pill, J. & McNamara, T. (2015). How much is enough? Involving occupational experts in setting standards on a specific-purpose language test for health professionals. Language Testing, vol 33(2), pp 217-234.
• Schreier, M. (2012). Qualitative content analysis in practice. Sage: London, UK.
• Son, J.B. & Park, S.S. (2014). Academic Experiences of International PhD Students in Australian Higher Education: From an EAP program to a PhD program. International Journal of Pedagogies and Learning, vol 9, pp 26-37.
• Swales, J.M. & Feak, C.B. (2000). English in Today’s Research World. Ann Arbor, MI: University of Michigan Press.
• Taffe, W.J. (1989). Writing in the Computer Science Curriculum. Writing Across the Curriculum, vol 1, pp 17-22.
• Taylor, H. & Paine, K. (1993). An interdisciplinary approach to the development of writing skills in computer science students. ACM SIGCSE Bulletin, vol 25(1), pp 274-278.
• Taylor, M., Martin, B. & Wilsdon, J. (2010). The scientific century: securing our future prosperity. The Royal Society: London, UK.
• Terui, S. (2011). Second language learners’ coping strategy in conversations with native speakers. Journal of International Students, vol 2, pp 168-183.
• Thomas, R.M. & Brubaker, D.L. (2000). Theses and dissertations: A guide to planning, research and writing. Westport, CT: Greenwood Publishing Group.
• Tircuit, A. (2012). Teaching Writing in Computer Science. Proceedings of the International Conference on Frontiers in Education: Computer Science and Computer Engineering (FECS), 16-19 July 2012, The Steering Committee of the World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp), Las Vegas, NV, pp 1-8.
• Wilmot, K. (2016). Designing writing groups to support postgraduate students’ academic writing: a case study from a South African university. Innovations in Education and Teaching International, DOI: 10.1080/14703297.2016.123877
• Wingate, U. & Tribble, C. (2012). The best of both worlds? Towards an English for Academic Purposes/Academic Literacies writing pedagogy. Studies in Higher Education, vol 37, pp 481-549.
• Zhu, W. (2004). Faculty views on the importance of writing, the nature of academic writing, and teaching and responding to writing in the disciplines. Journal of Second Language Writing, vol 13, pp 29-48.
• Zobel, J. (2004). Writing for Computer Science (2nd edn). London: Springer.

Appendix A: Student survey - questionnaire and summary

Due to the way that Qualtrics (the survey system) works, questions 1 and 2 were informing the students about the project. We received a total of 125 meaningful responses (some participants exited the survey without completing enough questions to be useful). The question and responses are given below.

Q3: What is your age?

Table A1: Student participant age range

Q4: What is your gender?

Table A2: Student participant gender

Q5: What is the field of your prior study? Please select all that apply.

• Computer Science (CS)
• Electronic/Computer Systems Engineering (E/CSE)
• Mathematics (M)
• Information Systems (IS)
• Linguistics (L)
• Social Science (S)
• Arts (A)
• Economics (E)
• Business (B)
• Other (please specify) ____

Table A3: Student participant field of prior study

Q6: What type of university are you currently attending for your PhD studies?

• Group of Eight: University of Adelaide, Australian National University, Monash, University of Melbourne, University of NSW, University of QLD, University of Sydney, University of WA (1)
• Australian Technology Network: Curtin, QUT, RMIT, University of SA, UTS (2)
• Innovative Research: Charles Darwin, Flinders, Griffith, James Cook, LaTrobe, Murdoch (3)
• Regional: CQU, Federation, SCU, UNE, USC, USQ (4)
• Other public: Australian Catholic University, University of Ballarat, University of Canberra, Charles Sturt University, Deakin, Edith Cowan, Macquarie University, Murdoch, University of Newcastle, University of Western Sydney, University of Wollongong, University of Tasmania, Victoria University (5)
• Other private: Bond University, University of Notre Dame, Torrens University, Carnegie Mellon University (6)

Table A4: Type of university currently attending

Q7: Approximately which stage of your PhD are you up to?

• Commencing
• Mid-candidature
• Completing

Table A5: Stage of PhD

Q8: What is your first language?

The responses to the question are shown in Table A6.

Table A6: Student first language responses

Q9: If you chose other, please specify your first language. None.

Q10: What languages do you speak other than English and your first language?

Answers are in Table A7.

Table A7: Responses to question 10, language other than English and first language

Q11: If you chose other, please specify the languages you speak other than English and your first language.

Twi, Ghanian Language, Visayan

Q12: What is the language in which you are most proficient for written tasks? Answers are shown in Table A8.

Table A8: Most proficient written language

Q13: If you chose other, please specify the language in which you are most proficient for written tasks.

Both answers were blank.

Q14: Time spent in an English-speaking country

1. How many years have you lived in an English-speaking country?
2. How many years have you studied in an English-speaking country?
3. How many years have you worked in an English-speaking country?

Table A9: Period of time spent in an English-speaking country

Q15: Have you ever sat an IELTS test?

Table A10: Number of participants having sat an IELTS test

Q16: When and where was your last IELTS test?

• Month
• Year
• Location

Table A11: Time and location of last IELTS test Month

Year

Location: N/A - 60, Australia - 19, China - 10, Bangladesh - 3, South America - 2, Indonesia - 5, Vietnam - 3, Chittagong -1, Sri Lanka - 2, Europe - 3, Iran - 4, Saudi - 1, Jordan - 1, India - 1, Korea - 2, Pakistan - 2, Malaysia - 4, Philippines - 1, NZ -1, Not specified -1.

Q17: In your last IELTS test what was your result for:

1. Writing

2. Reading

3. Speaking

4. Listening

5. Overall Score

Table A12: Score of last IELTS test

Q18: Have you ever sat an English language proficiency test apart from IELTS (for example TOEFL)?

Table A13: Other English language proficiency test

Q19: Date of recent test:

A broad range of years 2003-2016. But N/A = 92 and Not Specified = 5.

Q20: Name of test:

• TOEFL
• Cambridge
• Pearson
• Other (please specify) ____

Table A14: Name of other English language proficiency test

Q21: Score:

Results range: 69-570 and ‘can’t remember’. But N/A=92 and Not Specified = 5.

Q22: Please rate your perception of your English writing ability for the following tasks.

Table A15: Perception of English writing ability

Q23: What writing language support do you think would improve your written English skills? Not Specified (32)

Logic orginasation

Guidance on improving sentence structures

Personal feedback on writing tasks

“Course about good writing, clear assignment and clear feedbacks, evaluating writing skill of publications and discussion”

One-to-one guidance or group guidance with similar area of study

“I think I need to get more practise, It has been quite a time that I am into writing. I think I am not fluent in writing. So more writing tasks and assessment/feedback on them would benefit me.”

“I do tend to use simple language in my writing. I believe a support that helps in choosing better words is helpful.”

Academic

Academic

Sitting one by one with language expert to check our writing regularly

Advanced coaching

Grammar, structure and expression in sentences

Mentors who can help to review the work and give feedback

Continuous feedback on my academic writings

Editing services / proof reading / grammar check tools

PhD Thesis-writing Workshops

Workshops

Online trial pad with (near) real-time feedback for analyzing writing skill

A thesaurus and spell check

Weekly writing task and assessment

Analysis of writing of papers at conferences etc

Vocaulary, grammar and writing style support

Semantics and complex use of punctuation marks. Technical report writing techniques

Grammar classes

Weekly writing tutoring that check my writing and tell me the mistake

More feedbacks

Learn more academic words - have homework essay writing

Some advice on how to write academic papers

Writing communication seminar

Q24: What writing language support do you think could have improved your written English skills earlier in your candidature? Not Specified (38)

Essays

Grammar usage and writhing structure

Personal feedback on writing tasks

“Academic writing (rule of thumb e.g. active voice/passive voice, tenses, linking

phrase and others) and literature review skill”

More of a technical writing in research rather than just into coding and proof reading.

I would say examples of reports is helpful.

Academic

Academic

Sitting one by one with language expert to check our writing.

Intermediate coaching

Support in grammar and language expression for structural writing.

Training for proposal and publication writing

Same as above, continuous feedback.

None

Academic writing workshops

Workshops

A thesaurus and spell check, reading more theses to compare academic lingua

Weekly writing task and assessment

Vocabulary and grammar support

Complex use of sentences

Grammar and appropriate preposition

Weekly writing tutoring that check my writing and tell me the mistake

More feedbacks

Discuss my writing with proficinal and let here/him face to face tell me the right things

Some support on English grammar.

A course called research method

Q25: Is a writing drop-in centre available at your university?

Table A16: Writing drop-in centre availability

Q26: How did you find out about the writing drop-in centre? Please select all that apply:

1. Supervisor

2. Other students

3. Library

4. School

5. University PhD administration

6. Other (please specify)

7. Friends

8. Scholarship adviser/sponsor

9. Email list

10. Student services

Table A17: Method of finding out about writing drop-in centre

10. Other: flyer

Q27: Do you make use of the writing drop-in centre?

Table A18: Drop-in centre use

Q28: Please rate how helpful the writing drop-in centre was

- Very helpful

- Helpful

- Unhelpful

• Very unhelpful

Table A19: Usefulness of drop-in centre

Q29: Why did you not use a writing drop-in centre?

I did not feel urgency to do so

Not useful for academic technical writing

No pressing need at the moment

I am comfortable in English

I just started my candidature, didn’t feel the need to, quite yet

I always forget it

Q30: Is a writing circle available at your university?

Table A20: Awareness of writing circle

Q31: How did you find out about the writing circle? Please select all that apply

1. Supervisor

2. Other students

3. Library

4. School

5. University PhD administration

6. Other (please specify)

7. Friends

8. Scholarship adviser/sponsor

9. Email list

10. Student services

Table A21: Method of awareness of writing circle

Other: Attending master research writing classes / The web

Q32: Do you make use of a writing centre?

Table A22: Use of writing centre

Display this question if yes is selected:

Q33: Please rate how helpful the writing circle was.

• Very helpful
• Helpful
• Unhelpful
• Very unhelpful

Table A23: Helpfulness of writing circle

Q34: Why did you not use a writing circle? (NA = 96, Not Specified =14)

Did not feel urgency and did not have much time

Clashing time

I investigated but did not perceive ongoing value

I follow the strategies they posted online, but would rather write alone as I can focus

better that way.

No pressing need at this stage of my candidature.

Q35: Is a journal club available at your university?

Table A24: Awareness of journal club

If yes is not selected, then skip to Question 40: Is a thesis boot camp available at your university?

Q36: How did you find out about the journal club? Please select all that apply

1. Supervisor

2. Other students

3. Library

4. School

5. University PhD administration

6. Other (please specify)

7. Friends

8. Scholarship adviser/sponsor

9. Email list

10. Student services

Table A25: Method of awareness of journal club

Q37: Do you make use of a journal club?

Table A26: Use of journal club

Q38: Please rate how helpful the journal club was

• Very helpful
• Helpful
• Unhelpful
• Very unhelpful

Table A27: Helpfulness of journal club

Q39: Why did you not use a journal club?

I think they will not help much. I have written a thesis of 140 pages during my master program and a couple of papers. So, writing is not an issue by itself for me.

I don’t find any use for it.

No need to use the journal club at this stage of my PhD candidature.

Q40: Is a thesis boot camp available at your university?

Table A28: Awareness of thesis boot camp

Q41: How did you find out about the thesis boot camp? Please select all that apply.

1. Supervisor

2. Other students

3. Library

4. School

5. University PhD administration

6. Other (please specify)

7. Friends

8. Scholarship adviser/sponsor

9. Email list

10. Student services

Table A29: Method of awareness of thesis boot camp

Other: University website / Word of mouth

Q42: Do you make use of a thesis boot camp?

Table A30: Use of thesis boot camp

Q43: Please rate how helpful the thesis boot camp was:

• Very helpful
• Helpful
• Unhelpful
• Very unhelpful

Table A31: Helpfulness of thesis boot camp

Q44: Why did you not use a thesis boot camp?

Still did not complete my mid-candidature

I do not have much time these days.

Didn’t get the chance due to high volume of graduating students

I plan to apply next year.

Have not yet investigated the thesis bootcamp.

“I was not available on the thesis bootcamp date, due to writing for journal/ conference paper deadlines.”

I also just follow the strategies online, but would rather write alone to better concentrate.

I am still within the first six months of my PhD candidature.

Planning to do it next time since I just commenced my PhD

We formed a mini-bootcamp in our department

Still too early for me

I just started my PhD and things are not defined yet

I am in early stage of my PhD

In the last thesis bootcamp, I just started my third year of PhD (not applicable to register)

I think i am not at right position to join the group, may be at the last year

There are not that many. When I submit my application, there are no positions available.

Q45: Is a writing tutor available at your university?

Table A32: Awareness of writing tutor

Q46: How did you find out about the writing tutor? Please select all that apply.

1. Supervisor

2. Other students

3. Library

4. School

5. University PhD administration

6. Other (please specify)

7. Friends

8. Scholarship adviser/sponsor

9. Email list

10. Student services

Table A33: Method of awareness of writing tutor

Q47: Do you make use of a writing tutor?

Table A34: Use of writing tutor

Q48: Please rate how helpful the writing tutor was.

• Very helpful
• Helpful
• Unhelpful
• Very unhelpful

Table A35: Helpfulness of writing tutor

Q49: Why did you not use a writing tutor?

Too shy, worry about my communication skills

I don’t think I need it at the moment.

Didn’t feel the need to

Just started my PhD and also some needs to be paid

I am busy with my work, and i hear about then know

Q50: Is a writing mentor available at your university?

Table A36: Awareness of writing mentor

If yes is not selected, then skip to Question 55: Is there another language service available at your university?

Q51: How did you find out about the writing mentor? Please select all that apply.

1. Supervisor

2. Other students

3. Library

4. School

5. University PhD administration

6. Other (please specify)

7. Friends

8. Scholarship adviser/sponsor

9. Email list

10. Student services

Table A37: Method of awareness of writing mentor

Q52: Do you make use of a writing mentor?

Table A38: Use of writing mentor

Q48: Please rate how helpful the writing mentor was:

• Very helpful
• Helpful
• Unhelpful
• Very unhelpful

Table A39: Helpfulness of writing mentor

Q54: Why did you not use a writing mentor?

I did not feel it was necessary

Didn’t feel the need to

Just started my PhD and some are paid for

Q55: Is there another language service available at your university?

If yes, please specify:

Table A40: Awareness of language service

Q56: How did you find out about it? Please select all that apply.

1. Supervisor

2. Other students

3. Library

4. School

5. University PhD administration

6. Other (please specify)

7. Friends

8. Scholarship adviser/sponsor

9. Email list

10. Student services

Table A41: Method of awareness of language service

Other: Saw it myself within the University

Q57: Do you make use of it?

Table A42: Use of language service

Q58: Please rate how helpful it was:

• Very helpful
• Helpful
• Unhelpful
• Very unhelpful

Table A43: Helpfulness of language service

Q59: Why did you not use it?

I do not need it yet at this time

Not sure

Lack of interest in the language

Q60: Which aspects of English writing do you find difficult?

1. Clarity of meaning 4. Grammar 7. Structure
2. Cohesion (flow) 5. Punctuation 8. General English vocabulary
3. Expression 6. Spelling 9. Technical vocabulary
4. Other (please specify) ____

Table A44: Difficult aspects of English writing

Other reasons specified are listed below.

To put it in lay terms, making my paper sound more academic/smarter

Logic of western professors

My (first draft) writing is often a bit too descriptive for technical publications

Writing more concise sentences.

My English writing is very good

Q61: How do you think your English writing ability has changed during your candidature?

• Improved greatly
• Improved slightly
• Remained the same
• Worsened

Table A45: Perceived changes in English writing

Q62: What do you think has contributed the most to this change?

Journal Papers

Practice and write every day

Daily practice of writing

All the reading and writing I have done throughout the years, mainly.

Writing practice, reading other academic article (comparing the way they write for each section)

Writing academic papers.

I wrote and published academic papers. I read other academic papers.

“It has not been my focus to improve my writing ability. I have only sat down to write as needed. I do not feel my writing ability has improved.”

Reading and writing research papers.

Better understanding of the content has made it easier to form arguments

Research

“Reading academic papers in my field - especially particular scholars who write really really well.”

“Sentence structure and phrasing corrections that my supervisor makes on my publication drafts.”

Writing weekly report

“I practice most of my writing when providing updates to my supervisor. Besides, engaging with writing up academic publication drafts helps with my writing.”

“I think my English writing ability is improved because I often write English documents during my PhD, such as: proposals, research papers, ethics applications, etc.”

I think if their specific tutor for each school for different area.

“1. I take two subjects, in which I have to use English to write the assignment and take exams.”

2. I write some materials every week to my supervisor, and wrote a conference paper and confirmation report. My supervisor, who is a local, gave me a lot of advice.

Practice makes perfect.”

More reading and more practice

Q66: Would you like to enter the draw to win one of five $50 Coles/Myer vouchers? If you select yes you will be redirected to a separate survey to enter your contact details.

Table A46: Interest in entering competition

Appendix B: Staff survey - questionnaire and summary

Due to the way that Qualtrics (the survey system) works, Questions 1 and 2 were informing the supervisors about the project. We received a total of 44 responses. The question and responses are given below.

Q3: What is your age?

Table B1: Age of supervisor participants

Q4: What is your gender?

11 Female, 33 Male

Q5: Where are you currently employed?

• Group of Eight: University of Adelaide, Australian National University, Monash, University of Melbourne, University of NSW, University of QLD, University of Sydney, University of WA (1)
• Australian Technology Network: Curtin, QUT, RMIT, University of SA, UTS (2)
• Innovative Research: Charles Darwin, Flinders, Griffith, James Cook, LaTrobe, Murdoch (3)
• Regional: CQU, Federation, SCU, UNE, USC, USQ (4)
• Other public: Australian Catholic University, University of Ballarat, University of Canberra, Charles Sturt University, Deakin, Edith Cowan, Macquarie University, Murdoch, University of Newcastle, University of Western Sydney, University of Wollongong, University of Tasmania, Victoria University (5)
• Other private: Bond University, University of Notre Dame, Torrens University, Carnegie Mellon University (6)

Table B2: Place of employment of supervisor participants

Q6: What is your first language?

The responses to the question are shown in Table B3.

Table B3: First language of supervisor participants

Q7 If you chose other, please specify your first language.

There were no such responses.

Q8: What languages do you speak other than English?

Table B4: Other languages of supervisor participants

Totals

Q9: If you chose other, please specify what languages you speak other than English and your first language.

These responses have been included in the above totals.

Q10: What is the language in which you are most proficient for written tasks?

The same list as in Question 6 was presented. The responses are summarised below.

Table B5: Language most used for written tasks

Q11: If you chose other, please specify the language in which you are most proficient for written tasks.

There were no such responses.

Q12: Please rate your perception of your English writing ability.

Table B6: Perceptions of proficiency in written English

Q13: Please answer the following questions about time spent in an English-speaking country.

13.1: How many years have you lived in an English-speaking country? 13.2: How many years have you studied in an English-speaking country? 13.3: How many years have you worked in an English-speaking country?

Table B7: Length of time in an English-speaking country

Q14: Supervision and research

14.1: How many PhD students have you supervised to completion as first supervisor? 14.2: How many PhD theses have you examined? 14.3: How many ARC grants have you held? 14.4: How many other grants have you held?

Table B8: Number of PhD students/grants

Q15: Please rank the student English writing difficulties that occur, in order from the most to the least frequent.

1. Clarity of meaning 4. Grammar 7. Structure
2. Cohesion (flow) 5. Punctuation 8. General English vocabulary
3. Expression 6. Spelling 9. Technical vocabulary
4. Other (please specify) ____

Table B9: Ranking of English writing difficulties

Other (please specify):

Incompleteness

Lack of precision

Lack of practice

Plurals and articles synthesis

This is a difficult question because difficulties vary greatly between students.

[The above ranking exercise was quite difficult, in that I found myself involuntarily tending to rank in order of *seriousness*, and it required a conscientious effort to force myself to rank in order of *frequency*, and I’m still not confident I’ve achieved this! So I imagine there may be some distortion to the former ranking in the data you get.]

Q16: Which aspect of student English writing is the most difficult to manage?

“Story telling”/narrative generation

Clarity x 2

Clarity of expression

Clarity of meaning x 2

Clarity of meaning is the most difficult to manage as a supervisor because it’s difficult to correct. You need to ascertain what they are trying to say before you can suggest improvements.

Clear explanations

Cohesion x 6

Cohesion (flow)

Cohesion, Structure and Clarity and three most import aspects. They are codependent. Student have difficulty managing these.

Even students with English as foreign language try to miss communicate to the supervisor using English as a language. I would say student fudging is most difficult to handle and manage.

Expression and flow, as there is no single solution to offer them

Expression

Flow

General proficiency in English writing of international students is below high school level, despite passing IELTS test [gaining level 6.5 and above in writing].

Getting them to write at all. They think they are writing for the benefit of their supervisors and their supervisors know it all anyway.

Grammar x 2

Grammar - adapting to inconsistencies and exceptions.

How to structure an argument in a clear way.

Lack of clarity and precision

Lack of professional writing skill, even native speaking students.

Learning how to build an argument and create a cohesive narrative. This is a problem for both native English and non-native English speakers.

Low-level grammatical errors

Maintaining the student’s voice.

Overall structure and flow, compounded by their invariable use of sentences and paragraphs that are way too long

Placement of articles: “the” “a” and more generally grammar.

Their scholarly thought processes

Q17: Which aspects of student English writing are the most important to improve?

“Story telling”/narrative generation

As above, Cohesion, Structure and Clarity and three most import aspects. The overall goal is clarity, so this is probably the most difficult to improve.

Avoiding distracting, basic errors which cause the reader to focus on trivia instead of the message. Clear argumentation.

Clarity x 3

Clarity and cohesion

Clarity and cohesion (assuming that it is grammatically comprehensible)

Clarity of meaning x 3

Clarity of meaning and flow are very high priorities (though they require adequate spelling and grammar). See previous response.

Clarity of meaning, cohesion, expression

Clarity of thinking and presentation. To repeat in other words: the problem is to factually communicate ones thought.

Clarity, but this encompasses many other aspects (grammar, expression, structure, etc.)

Clarity, cohesion

Clarity, expression

Cohesion x 3

Cohesion. If people don’t understand the difference between highly cohesive writing and poorly cohesive writing, they can’t write cohesion into their work. Often providing examples doesn’t work because cohesion can be too nuanced for an un-seasoned reader.

Cohesive writing

Flow

Flow of argument, writing for the reader

Grammar x 2

Grammar, so that feedback can instead focus on things such as research content!

How to structure an argument in a clear way.

Increase writing capability of international students up to high school level.

Logical presentation of concepts, explanation of reasoning

Narrative

Precise and clear meaning; logical grammatical structure (even if grammar doesn’t follow the rules exactly, it should give a logically sound and clear structure).

Reviewing their own work from an outsider’s perspective.

See above. Grammatical issues are problematic but are easier to resolve than writing that just doesn’t make any sense. The most important improvement is to encourage students to be able to look at their writing from the readers’ perspective, so they can see when things might not make sense, i.e., when there are flaws in the argument, inconsistencies, or insufficient explanations, etc.

Stopping them from assuming the reader can interpret what is in their head

Structure x 3

Structure, Cohesion and Clarity of meaning

Structuring an argument in a cogent fashion.

The fact that content and facts alone are insufficient, and that the information has to be communicated effectively to the audience

There is a need to identify all the important ideas and to place them in the right order.

Working to a plan

Q18: Please answer the following questions about written communication skills on a 5 point scale between strongly agree and strongly disagree.

1. Written communication skills are important for PhD students.
2. The English language entry requirements for PhD students are adequate.
3. Insufficient skill in written communication has impeded the progress of some of my PhD students.
4. Students with insufficient written communication skills have significantly added to my workload.
5. The English language support services provided by the university for PhD students are sufficient.
6. Poor writing distracts my focus from the student’s research issues.
7. I routinely edit my student’s writing.
8. My students’ written communication skills improve during their candidature.
9. By the end of the PhD, my students’ written communication skills are appropriate for publishing research papers.
10. For some students, I find it difficult to distinguish between poor written communication skills and poor research skills.
11. Students should use professional editors for writing their thesis.
12. Students should use professional editors for writing papers.
13. Students should use professional editors for other writing tasks.
14. I would accept a PhD student with strong research skills but poor written communication skills.
15. I frequently refer students to the English writing support.

Table B10: Supervisor responses to statements around written English issues for PhD students

Q19: Under what conditions would you accept a student with poor written communication skills?

After an interview or other interaction convincing me of the student’s ability to develop, and motivation to do so.

All students’ writing starts out poor (research writing is a skill many native English speaking students also take time to acquire).

Cant thing of any.

Excellent academic results in the mathematical/technical subjects.

Excellent mathematical skills

Excellent research skill

Excellent research skills and clear willingness to improve

Exceptional coding abilities, great personality, demonstrated abilities to get papers published.

Funding (scholarship), written evidence produced demonstrating potential for research.

Good academic transcripts, several years work or research experience and good English in other skills (reading and speaking) and willingness to improve writing skill.

Good motivation for research and good problem solving skills. Good research problem to work on.

I don’t accept them.

I look for the level of motivation in a postgrad. It should be because the student wants to learn how to make a research contribution. A poor motivation is that the student wants a PhD as a certificate, a box to be ticked on a job application or on promotion criteria. Another thing I look for is the ability and desire to see the big picture and to find what is important as opposed to what is co-incidental. I would accept a student with poor written communication skills provided the student recognised the need to improve their skills, was motivated to work on improving their skills and was able to show that they had made progress after they had applied themselves to improving their skills.

I won’t do it any more. They are too time-consuming to be worthwhile.

I would consider doing this if the student had excellent technical skills and I was convinced that they were motivated and showed potential to improve their English

I would need evidence that (a) they have excellent logical thinking skills; (b) they recognise the need to improve; (c) they are receptive to advice; (d) the logical structure of their writing is clear, even though there may be many errors.

I’d have to be convinced that their research skills are excellent and the topic is completely aligned with my interests.

If a candidate successfully demonstrate his/her logical thinking capability.

If a student is highly recommended, passionate about the research (has the drive to succeed) and is enthusiastic about a research topic closely aligned with my research.

If having very strong academic record.

If I didn’t have direct evidence of the poor written communication skills!

If the student demonstrates good research skills and the ability to interpret results.

If the students is very promising and there are adequate support services for written and verbal communication skills.

If their written communication seems to only have minor issues - grammar, spelling, etc. then I am happy to accept them. I also think that the written communication style is something all students have to learn, so do not expect full proficiency at the start.

However, if the written communication skills are too poor, I would not accept the student regardless of how good they were on other tasks. But sometimes it is not that clear initially, particularly with regard to the more important skills of clarity, cohesion and structure.

If they are already taking steps to improve their skills, e.g., are undertaking a writing course, or have joined a language school. They must also have excellent research skills.

If they have a plan to improve their written communication skills.

If they have published previously and have a topic exactly in my research area.

Never.

Never. I can’t imagine ever accepting a PhD student who clearly has poor written communication skills.

No.

None.

None. Following many bad experiences I will no longer accept such students.

Only acceptable if the student has an incredibly strong academic background to compensate for the poor writing.

Only if it is a very exceptional student, who I know will have no problem in acquiring what is needed to publish papers.

They have specific domain expertise or technical skill that is required for the project. (Very rarely!)

They must have an orderly way of thinking.

Very strong research skills and a clear commitment to work on communication skills

When funding has to be spent on a student immediately.

When the student has exceptional research skills.

When they bluff their way through the admissions process (with high IELTS scores and research proposals that turn out to have been edited by someone else).

Q20: If you don’t refer students to writing support services, please state why not.

Often referred but many students do not take advantage of these. The skills provided are often generic and not always relevant to technical writing skill appropriate to a particular (sub)discipline.

I don’t think it will help.

No point referring them over and over. Some of the international students don’t seem to want to improve, as the supervisor will do the editing for them anyway eventually.

I refer them, but the services are very limited and generally inadequate.

I do frequently and run them myself for the faculty.

This is something that cannot be learnt for mathematics at least.

Improving English does not usually improve the quality of scientific information and knowledge being communicated.

I always refer students to writing support services, no matter their English background.

I do refer students to writing support services.

Unaware that my university provides such a service for free.

Lack of awareness of what’s available and skepticism that they can assist in technical disciplines.

I am not fully aware of what services are available or how they work, though I am vaguely aware that the services exist. Some students have taken the initiative to seek out these services and it seems to help with their confidence - but the effectiveness of these services relies on students recognising there is a problem and wanting to improve. I have trouble with some students thinking they do not need help with their written English and resisting my feedback. I sometimes wonder if my standards are too high.

Existing courses already available, some students choose to attend.

I don’t have to because I don’t accept students with poor written communication skills.

“Because I’m not familiar with the services offered myself, and because I suspect they’re too generic to be helpful for our particular style of technical writing. (And because they encourage the use of Microsoft Word instead of LaTeX.)

[And, yes, I know all the sentences in this box are grammatically incorrect, but this isn’t a formal document.]

Not sure how much they can do.

I routinely refer most students early on and encourage them to use this service.

However the student who has had biggest improvement is one who independently found online writing and english courses/help.

I do refer them to the support services. Some do not use these as much as I would hope.

I have not found it necessary. The students I have taken on have a reasonable level of communication skill. As CS/IT students they will have developed the ability to structure programs and structuring written documents requires similar skills. As CS/IT students they will have developed the ability to pick up detailed, technical knowledge and improving basic expression skills also requires similar skills.

Q21: Which of the following writing services are available for students at your university? Please select all that apply.

Table B11: Awareness of writing support systems

Other (please specify):

Academic writing course

Faculty writing groups

I suspect the top 5 writing services are available but I do not know.

Some of these services are available in some areas but not in others. Some are only done informally and ad hoc by conscientious individuals.

Appendix C: The writing task

Q2 This is the main writing task we would like you to complete.

This task will take approximately 30-40 minutes at most.

Writing task

The table below shows the space and time utilisation of three compression algorithms*. Some researchers state that Algorithm A is superior for all applications. Others argue that it depends on the application. For example, for mobile phones space is critical, whereas for interactive streaming applications time is more critical.

Discuss both of these views in relation to the table of results and give your own opinion. Please structure your answer to include an introduction, discussion, and conclusion.

Please take your time and try to write between 200-300 words.

*Compression algorithms encode information in a smaller amount of space than the original data

Word count: 0

Appendix D: Standard setting score sheet for Computer Science PhD student English writing skill

For PhD student ____ (participant number), rate the English writing skill by marking one of the seven boxes for commencing PhD student and another box for completing PhD student.

I rate the English writing skill as:

Author biodata

Alexandra L. Uitdenbogerd

Dr Alexandra Uitdenbogerd has been with RMIT Computer Science and Information Technology since 2001. She has a Graduate Diploma in Education and has taught computer-related skills for nearly 30 years. She is internationally known for her pioneering work in Music Information Retrieval. Since 2003, she has also worked in the field of Computer Assisted Language Learning (CALL). Her goal is to determine the optimal extensive reading strategy and associated resources for additional language acquisition. In 2012, she obtained technology funding from the Victorian Government for automated optical inspection of circuit boards. In 2014, she received $43,000 of category 1 seed funding from the Office for Learning and Teaching (OLT) to better understand vocabulary acquisition from reading in English as an Additional Language. She grew up Dutch-English bilingual, and has attained CEFR level B1 in French. Alexandra is the grant project leader.

Kath Lynch

Dr Kath Lynch has worked for over 20 years in the higher education sector specialising in migration, international education and teaching English to second and foreign language learners. She has expertise as an IELTS examiner, teacher, and IELTS resource developer. Kath co-wrote the tender for, and was special content editor of, the IELTS textbook, IELTS to Success: Preparation Tips and Practice Tests (Tucker & Van Bemmel, 2002). She has collaborated on Australian university-funded professional language grants, for, e.g. Curtin University People’s Republic of China Teacher Exchange, and the University of Melbourne Language School Lao Teachers PD Program. Her research focuses on the role language, culture, and intercultural communication plays in higher education. Her Master’s research focused on the academic adjustment of Japanese students to Western learning environments and her PhD examined how Australian universities prepare and support academics who teach transnationally.

James Harland

Associate Professor James Harland has over 20 years’ experience in research and teaching. He is known internationally for his work on intelligent agent systems, automated reasoning, logic programming and computer science education research. Together with colleagues from RMIT and others from UTS, QUT, Monash and Newcastle, he was a key contributor to the BABELnot project, funded by a grant from the OLT, from 2011 to 2013, which developed an epistemology of competency in computer programming. In 2007, James received a Carrick (now OLT) Citation for Outstanding Contributions to Student Learning for his work on teaching Computing Theory, which many students find conceptually difficult. His experience in supervising PhD students from a variety of non-English-speaking backgrounds (including Vietnam, Serbia, Bangladesh, Saudi Arabia and Mexico), as well as the assessment of PhD theses and selection of students for PhD study, is particularly relevant to this project.

Charles Thevathayan

Dr Charles Thevathayan has over 30 years’ teaching experience both in Singapore and Australia, receiving many awards for instructional design, teaching techniques and course coordination. Since moving to RMIT, he has completed a PhD and has published several papers in security, trust and education. Charles has designed and taught several industry relevant courses which improve the chances of students securing permanent employment in industry. He promotes problem-based learning in the School of Computer Science and Information Technology. Charles has supervised several industry projects involving international students and is aware of some common language problems they face. Charles has been promoting closer links with overseas institutions by creating special pathways taking into account the students’ background and educational needs.

Margaret Hamilton

Associate Professor Margaret Hamilton researches in Computer Science education and human computer interaction, where she works with new technologies to research areas around people, mobility and sustainability. She has effected several OLT grants: Developing graduate employability through partnerships with industry and professional associations; Web 2.0 Authoring Tools in Higher Education Learning and Teaching: New Directions for Assessment and Academic Integrity; and A shared applied epistemology for competency in computer programming. Margaret has published over 50 peer-reviewed papers in Computer Science education and technology journals and conferences, and has over 30 years’ experience in teaching programming to tertiary students at TAFE and university. For this project, she was particularly interested in how the assessment of written English skill were made by the IELTS tests, PhD students and their supervisors, and brought experience in the design of surveys, interviews, focus groups and statistical analyses of the qualitative and quantitative data.

Daryl D’Souza

Dr Daryl D’Souza has taught for more than 30 years within the discipline of Computer Science and Information Technology at RMIT, with excellent teaching scores and recognition for good teaching at all levels. He has pursued computing education research since 2006, and is also interested in automatic text classification and data analytics for health and teaching and learning. He has led two successful internal RMIT Learning and Teaching grants, which established a sustainable peer mentoring service and which has operated since 2007. Daryl has chaired two national computing education conferences; published in computing education research for the last five years; served in program leadership roles, in which he established an important pathway for non-IT, mature-age students to enable transition into IT employment. He brings to the project his expertise in developing support services that enable a diverse range of students to succeed in Computer Science and Information Technology courses.

Sarah Zydervelt

Sarah Zydervelt worked from 2012 to 2016 as a Research Fellow at the Centre for Investigative Interviewing at Deakin University in Australia and as an Assistant Research Fellow and Research Assistant at the University of Otago in New Zealand. She has a diverse set of research skills from conducting literature reviews, data collection and analysis for both quantitative and qualitative studies, and has prepared a report for the Australian Royal Commission in one of her studies. As a barrister and solicitor admitted to the High Court of New Zealand, she is also eligible for admission to the Supreme Court of Victoria, Australia. Sarah has worked (both professionally and in a voluntary capacity) as a helpline counsellor and mentor for Youthline, with the Innocence Project and at the Dunedin Community Law Centre, in New Zealand. In this IELTS project, she was involved in a wide range of tasks including recruiting and managing the student writing tasks and assessment panels of academics, and contributing to the analysis of qualitative data.

Additional staff

Additional staff included Sarah Zydervelt as the research assistant (see above), an IELTS examiner to assess the writing of the PhD participants writing tasks, and an editor to assist with the preparation of the final IELTS report.