Comparability of IELTS Academic Writing Tasks and graduate-level multimodal writing tasks at universities in Türkiye, UK, and USA

Abstract

Multimodal writing has become increasingly important in today’s academic contexts. It is important that standardised writing tests reflect and model the features of writing tasks required in university settings since students’ performance in the tests is assumed to predict the extent to which they will successfully meet the requirements of the written assignments in the real academic context (Chapelle et al., 2008; Cumming et al., 2000). However, the correspondence between IELTS Academic Writing Tasks and multimodal writing tasks completed by students in university classes is unknown.

To better understand the use of multimodal writing tasks in university graduate programs and how IELTS Academic Writing Tasks compare to graduate-level multimodal writing tasks, this project examines the similarities and differences between the writing tasks across the two domains (i.e., test and academic) in terms of task features, students’ linguistic performance, and perceptions of students and academic staff members.

Over three million test-takers sit the IELTS exam globally every year, with the majority of them taking the exam for educational purposes such as admission into undergraduate and graduate programs at university. In this project, we only address graduate students for convenience reasons, taking into account our professional contexts.

We specifically examine:

1. the extent to which IELTS Academic Writing Tasks are comparable to graduate-level multimodal writing tasks required at university in terms of genre, topic and modality
2. the extent to which graduate-level students’ performance on IELTS Academic Writing Tasks are comparable to their performance on graduate-level multimodal writing tasks in terms of lexical sophistication and phrasal complexity
3. the extent to which graduate-level students and academic staff perceive IELTS Academic Writing Tasks to be comparable to graduate-level multimodal writing tasks.

Data include multimodal writing assignment sheets from graduate courses, students’ responses to the IELTS Academic Writing Tasks, students’ and academic staff participants’ responses to online questionnaires, and interviews with students and academic staff.

Textual analysis of IELTS writing task prompts and assignment sheets was conducted to examine the similarities and differences of writing tasks in those two domains in terms of their task features. Quantitative analyses of the students’ performance data were used to identify how the test tasks compared to the graduate assignments in terms of eliciting different linguistic features in student writing. Analysis of the perception data, including questionnaire responses and interviews, was used to better understand the similarities and differences identified in the textual and quantitative analyses.

To broaden perspectives on the research questions, we present three studies conducted in three different cultural contexts:

1. Türkiye
2. UK
3. USA.

We present country-specific findings with relevant discussion and conclusions in an effort to provide more context-relevant information to help the readers interpret the findings.

Based on trends identified from a synthesis of the findings, we discuss implications for designing and using multimodal writing tasks in tests, acknowledge our limitations, and conclude by offering future research directions.

1 Introduction

A 21^st-century understanding of writing centres around a socially-constructed multimodal perspective, in which writing goes beyond the traditional textual mode and includes other modes of expression such as colour, sound, images or videos (Hafiner & Ho, 2020). The emergence of diverse communication choices over the last 20 years is also changing academic writing practices in university settings (Belcher, 2017). Writing tasks in today’s higher education classes are commonly designed around multimodality (i.e., requiring students to integrate different modes in their written outputs) to promote effective disciplinary practice (Belcher, 2017; Lim & Polio, 2020). However, the extent to which high-stakes academic writing tests reflect the changes in university academic writing tasks remains unknown. L2 English-speaking students are admitted to universities based on their scores from IELTS or TOEFL, and students’ performance in these tests is assumed to predict how they will perform in real academic contexts (Chapelle, Enright, & Jamieson, 2008). Results from the writing tests inform institutions about whether or not students will be able to meet the requirements of the university written assignments. Therefore, it is important to examine the assumption that the high-stakes writing tasks model the writing tasks students encounter at university (Cumming, Kantor, Powers, Santos, & Taylor, 2000) against the background of heightened importance of multimodality.

An extensive empirical literature has investigated the comparability of tasks across high-stakes assessment and classroom-based assessment at university and students’ performance across these two domains (e.g., Cooper, 2013; Llosa & Malone, 2019; Moore & Morton, 1999; Riazi, 2016; Weigle, 2010; Weigle & Friginal, 2015). However, the number of studies focusing on IELTS is much smaller than the number of studies on TOEFL. In order to contribute to the empirical investigations of IELTS and in response to the growing importance of multimodality in today’s academic writing, we proposed to examine the authenticity of IELTS Academic Writing Tasks.

In this project, we aimed to compare IELTS Academic Writing Tasks to multimodal writing tasks that graduate-level students are required to complete in their disciplines as a part of their degree programs. While a large number of undergraduate students all around the world also take IELTS, the target group of test-takers for our study is graduate students for reasons of availability and convenience within our professional contexts. We specifically aimed to examine:

1. the correspondence between IELTS Academic Writing Tasks and graduate-level multimodal writing tasks in terms of task features such as genre, topic, rhetorical function, and modality
2. the similarities and differences between graduate-level students’ performance on IELTS Academic Writing Tasks and their performance on multimodal writing tasks in terms of linguistic features
3. graduate-level students’ and academic staff participants’ perceptions regarding the comparability between writing tasks across the domains of test versus target language use.

2 Literature review

2.1 Theoretical framework

The argument-based approach to validity (Kane, 2006, 2013; Messick, 1989) focuses on the development of two types of interconnected argument: 1) an interpretive argument where a detailed list of potential interpretations and uses of language test outcomes are explicitly outlined, and 2) a validity argument where evidence is collected, and a critical analysis of the evidence is conducted to support the inferences of the interpretive argument. Chapelle et al.’s (2008) work on validating the TOEFL test expanded this framework into a chain of six types of inferences: domain definition, evaluation, generalisation, explanation, extrapolation, and utilisation. Each inference focuses on a different stage of the assessment and thus different types of evidence needed. Thus, the argument-based approach to validation provides a systematic way to link chains of reasoning together based on empirical data or observations in order to construct a coherent validity argument. This approach has been widely used for validating language assessments in different contexts (e.g., Chapelle et al., 2008; Li, 2015; Yan & Staples, 2020). In this study, we aimed to seek evidence for the domain definition inference by examining the authenticity and representativeness of the IELTS Academic Writing Tasks in comparison to graduate-level multimodal writing tasks in English-speaking educational contexts in three countries.

For the IELTS Academic Writing Tasks, the target domain is typically university contexts in English-speaking countries. The scope is quite broad; therefore, we focus on graduate-level multimodal writing tasks at universities. The domain definition inference links the knowledge, skills, and abilities required in the target domain to the test-takers’ performance on a test, which is expected to consist of assessment tasks representative of the target domain. The claims, warrants, assumptions, and sources of backing for this inference are summarised in the table below.

Table 1: Domain definition claims, warrants, assumptions, and evidence sought

2.2 Multimodality

Language is not the only resource that individuals use to express or interpret meaning. Within social contexts, communication is realised through the use of several different resources that co-construct and co-represent meaning. Different resources or forms of representation that are used for making meaning are defined as “mode” and are socially and culturally shaped (Bezemer & Kress, 2008). Modes exist in several different forms such as images, comics, videos, music, gestures, posture, symbols, tables, graphs, and language (Jewitt & Kress, 2003). Based on this dimension of human communication, multimodality is defined as the use or integration of two or more meaning-making resources for communicative purposes (Kress & van Leeuwen, 2001).

Because multimodality seems to have gained prominence as a result of the developments in digital technologies, it is often considered to be a feature of digital communication. However, multimodality is neither a new phenomenon nor does it have to be digital (Early, Kendrick, & Potts, 2015). “[T]here has always been a plurality of modalities, in the same way that the world has always been populated by a multiplicity of languages, cultures, and literacies” (Darvin, 2015, p. 591). The co-existence of textual and visual modes has long been an integral part of real-life communication and academic communication. Multimodality has not emerged as a result of digitisation, but digitisation has certainly made it easier to bring together different modes via the same device (Gourlay, 2016). The prominence of multimodal communication is also noticed in the higher education landscape (Lim & Polio, 2020). Multimodal assignments are becoming increasingly common in classrooms, requiring students to engage with, and draw upon, different semiotic resources.

Multimodality can best be understood by looking at how it is implemented for pedagogical purposes at different educational levels. The majority of existing studies on multimodal pedagogical implementations are from high school contexts while the number of studies from higher education contexts is smaller. In one of the examples from a high school context, 10^th graders in an English Language Arts class in a high school in the United States completed three multimodal tasks: a hypertext poetry analysis, a persuasive podcast, and a video literary analysis (Smith, Amgott, & Malova, 2022). In the hypertext poetry analysis, students created a PowerPoint home slide where they presented their poems and created hyperlinks from the home slide to other slides which explored in detail some keywords, phrases, literary devices, or personal reactions. In the persuasive podcast, students created podcasts on a controversial topic using different rhetorical elements and remixing a variety of sounds such as music, sound effects, speech effects, and media clips. In the video literary analysis, students created a video analysing a literary theme from Kurt Vonnegut’s short story Harrison Bergeron (1961) in different video genres such as animation, documentary, or narrative.

In another implementation, graduate-level students in a General Linguistics class at a university in the United States completed a Mid-Term Creative Multimodal Project, and graduate students in a TESOL Methods I class completed a Multimodal Instructional Materials Project (Li, 2020). In the mid-term creative review/reflection project, students individually demonstrated understanding of a topic or a combination of topics such as phonetics, phonology, or morphology using a technology tool such as PowerPoint, Prezi, Storybird or WeVideo. In the Multimodal Instructional Materials Project, students designed multimodal instructional materials using a technology tool such as Storybird, Glogster, Vyond, Powtoon or Prezi, and utilising one or more teaching methods. In addition, students submitted written narratives introducing their technology tools and specific teaching methods.

Figure 1 presents sample multimodal products by the students in these studies.

Figure 1: Sample multimodal student works from Smith et al. (2022) and Li (2020)

2.3 Research on multimodal writing

According to a review of digital multimodal composing in post-secondary L2 settings done by Zhang et al. (2023), qualitative research is dominant in digital multimodal composition research, especially case study design. A large percentage of research studies on multimodality, particularly recent ones, focus on digital multimodality. Researchers most commonly collect data including semi-structured or focus group interviews, multimodal products, surveys, written reflections or journals, and observation notes, which according to Zhang et al. (2023), suggests an interpretive and naturalistic trend toward data collection and analysis in multimodal composition research.

Eliciting student and teacher perspectives has been an important theme in multimodality research. Yi and Choi (2015) explored in-service teachers’ views of using multimodal practices to teach English in K-12 classrooms through qualitative questionnaires that gathered information about teachers’ perceptions of, and prior experiences, with multimodal practices, online posts on weekly reading assignments on issues of multimodality, and reflection and response posts on multimodal projects. The results showed the potential of multimodal practices as demonstrated in the positive view of the teachers regarding the motivating and engaging aspects of multimodal practices, especially when teachers taught in a multimodal way. Multimodal practices provided noticeable opportunities for students with lower-level language skills to better express themselves through different meaning-making and communication possibilities. The results also yielded negative views held by the teachers, which were mainly related to the limited amount of time for teacher preparation and the limited use of academic language or literacy by English language learners. Tan and Matsuda (2020) examined instructors’ beliefs and practices of integrating multimodal writing into first-year composition and how internal and external factors influenced their beliefs and practices. They found that the instructors’ beliefs about modal complexity, critical analysis, subject-matter knowledge, and rhetorical awareness were reflected in their multimodal teaching practices.

Regarding the influential factors related to students, the results revealed students’ dismissive attitude toward multimodal assignments, increased anxiety, confusion, resistance, and lack of interest.

In an undergraduate science course in English, where students conducted a simple scientific experiment and reported their findings as a multimodal scientific video documentary and as a written lab report, Hafiner (2013) explored the challenges that students perceived in this task and students’ strategies in response. More than half of the students in this course reported little or no experience in creating multimedia projects in general, and scientific documentaries in particular. The analysis revealed two challenges as perceived by the students: audience attention and multimodal orchestration. In another study, Smith, Amgott and Malova (2021) examined the perspectives of 10^th-grade students on multimodal composing for academic purposes in an English Language Arts class, with a particular focus on the affordances and constraints of multimodal meaning-making. Students completed three multimodal projects (i.e., a hypertext poetry analysis, a persuasive podcast, and a video literary analysis) and submitted a written reflection after each project. Their results revealed six main perceived affordances of multimodal composing: to learn and think in a different way through multimodal composing processes; to make meaning in an innovative way; to express bilingual and bicultural identities; to feel the emotions of the content; to collaborate and learn from peers; and to make multilevel connections to other works. The constraints included technical difficulties, the uncertainty of expectations and assessment, lack of enough time to complete the projects, and coherence issues.

2.4 Research on comparability of writing tasks

Several researchers have conducted comparability studies between large-scale writing assessment and classroom-based writing assessment at university. Earlier studies have commonly compared the academic writing tasks across test and target language use domains for scores and linguistic/textual features, as well as student and teacher perceptions (e.g., Cooper, 2013; Llosa & Malone, 2017; 2019; Moore & Morton, 1999; Riazi, 2016; Weigle, 2010; Weigle & Friginal, 2015).

To date, a fairly limited number of studies has addressed the comparability between IELTS Academic Writing Tasks and university tasks. In one study, Moore and Morton (1999) compared the tasks from IELTS Academic Writing Task 2 with university assignment tasks based on four dimensions: genre, information source, rhetorical function, and object of enquiry. The two domains were found to be similar in genre, with “essay” also being the predominant genre of university writing tasks accounting for around 60% of the tasks. However, there were differences in the other dimensions. For example, the university tasks required using primary and/or secondary information sources. Regarding the rhetorical function, evaluation was common in both domains; summarisation and description were common among the university tasks but not in IELTS; and comparison, explanation and recommendation were more common at university.

In another study, Cooper (2013) compared students’ responses to IELTS Task 2 with their academic essays to find out whether the language structures that the IELTS tests elicit predict students’ linguistic performance in academic writing at university by investigating lexical bundles. The findings showed considerable differences, with IELTS Task 2 eliciting lexical bundles that are typical of spoken discourse while the academic essay writing tasks elicited lexical bundles typical of written discourse. Cooper (2013) concluded that IELTS Task 2 “does not serve as a predictor of students’ abilities to produce academic writing” (p. 78).

While very important for the literature, these studies were conducted a long time ago. Because writing practices have been changing rapidly, writing tasks in today’s higher education classes involve more use of technology and different meaning-making resources. Thus, it is important to revisit the issue of comparability between IELTS Academic Writing Tasks and university writing tasks. This exploration is important as many universities around the world accept L2 English-speaking students based on their IELTS scores, which serve as indicators of how well students will perform in the real academic context.

To contribute to efforts in this regard, this study aimed to compare IELTS Academic Writing Tasks to graduate-level multimodal writing tasks. It specifically aimed to examine: (a) the extent to which IELTS Academic Writing Tasks correspond to graduate-level multimodal writing tasks required at university classes in terms of task features such as genre, topic, rhetorical function, and modality; (b) the similarities and differences between graduate-level students’ performance on IELTS Academic Writing Tasks and their performance on multimodal writing tasks in terms of linguistic features, and; (c) graduate students’ and academic staff participants’ perceptions of the comparability between writing tasks across the domains of test (IELTS) versus target language use (graduate-level multimodal writing). It is important to note that IELTS Academic Writing Tasks elicit only textual language from the test-takers. Test-takers are not expected to produce outputs in modes other than writing. Writing Task 1 is multimodal at the receptive level as test-takers are presented with visual input (i.e., charts or graphs) while the input in Writing Task 2 is a written statement. Test-takers respond to both tasks in writing only. However, while the nature of writing has been changing in higher-education contexts as a result of digitalisation and becoming more multimodal, the question of to what extent high-stakes language proficiency tests provide representative coverage of the knowledge, skills, and processes needed and assessed in the target domain (i.e., university classrooms) has not been examined. It is necessary and timely to re-evaluate the authenticity of writing tasks in high-stakes tests from multimodality perspectives. It is also important to acknowledge that our research design is limited to only a specific group of IELTS test-takers (i.e., graduate-level students).

3 Research questions

This exploratory research project aimed to address the following three research questions.

RQ1: To what extent are IELTS Academic Writing Tasks comparable to graduate-level multimodal writing tasks required at university in terms of task features such as genre, topic, and modality?

RQ2: To what extent is graduate-level students’ performance on IELTS Academic Writing Tasks comparable to their performance on multimodal writing tasks required at university in terms of linguistic features?

RQ3: To what extent do stakeholders (graduate-level students and academic staff members) perceive IELTS Academic Writing Tasks to be comparable to graduate-level multimodal writing tasks required at university?

4 Methodology

4.1 Research design

This project employed an explanatory sequential mixed-methods design to address the research questions. A sequential mixed-methods design, which is underutilised in language assessment research (Lim, 2019), provides explanations for quantitative results with qualitative data and a more in-depth understanding of the findings (Creswell, 2014). It should be noted that this research project was conducted separately by each researcher in three different countries. Each author was solely responsible for data collection and analysis in their country of employment (Aysel in Türkiye; Duygu in the UK; and Jinrong in the USA). However, we attempted to synthesise our results in the Findings section after data analysis was completed.

4.2 Participants

This study involves two groups of participants: graduate-level students and academic staff members. In addition to reasons of availability and convenience, our focus on graduate-level students is motivated by the growth in international student recruitment at the graduate level (Choudaha, 2017) in several countries and the requirement of English proficiency test scores for admission into graduate programs. From 2022 to 2023, for example, the increase in the number of international students pursuing graduate degrees in the USA was about 21% while it was 1% for students pursuing undergraduate degrees (Institute of International Education, n.d.).

Three non-probability sampling strategies were used to recruit student participants: purposive sampling, convenience sampling, and snowball sampling. The inclusion criteria for student participants were international graduate-level students studying at a university in Türkiye with English as a medium of instruction (Türkiye), international graduate-level students studying at a UK university (UK), and international graduate-level students studying at a US university (USA). Two non-probability sampling strategies were used to recruit academic staff participants: purposive sampling and convenience sampling. The inclusion criteria for academic staff participants were teaching at a graduate-level university program (UK & USA) and teaching at a graduate-level university program with English as a medium of instruction (Türkiye).

The participants involved 108 graduate-level students and 216 academic staff members from three research sites (i.e., Türkiye, the UK, and the USA). Among the student participants, 59.3% were female, 39.8% were male, and 0.9% were non-binary, ranging from 21 to 42 years old. They represented a wide range of L1 backgrounds and academic disciplines. Table 2 provides an overview of the demographic information of the student participants.

Table 2: Selected demographics of student participants

In Türkiye, 40 graduate-level students (19 female and 21 male) studying in English-medium instruction universities completed the student questionnaire and submitted multimodal assignment sheets. Out of these participants, 36 completed the IELTS Academic Writing Tasks, and 17 also joined the semi-structured student interviews. The age of the student participants ranged from 25 to 34 years old (M=29.60, SD=3.65), and they were from 15 different L1 backgrounds (Table 3). They reported having different years of experience in learning English: since primary school (n=9), since high school (n=2), less than 10 years (n=4), 10-15 years (n=11), 16-20 years (n=5), 21-25 years (n=5), and more than 20 years (n=5).

For their most recent English proficiency test scores, 10 participants reported their TOEFL scores, which ranged from 92 to 114 (M=102, SD=7.69); 11 participants reported their IELTS scores, which ranged from 6.5 to 8 (M=7.45, SD=0.47); nine participants reported their YDS (Yabancı Dil Sınavı - Foreign Language Exam) scores, The Foreign Language Knowledge Level Determination Exam administered by Türkiye’s Centre for Assessment, Selection and Placement (ÖSYM), which ranged from 88.75 to 98 (out of 100) (M=92.39, SD=4.41); two participants reported their institutional English proficiency exam scores, which were 74 and 90 out of 100; and two participants reported their Erasmus English exam scores, which were 76 and 93 out of 100. Six participants did not report their exam scores.

More than half of the student participants were MA students (n=23, 57.5%), while the rest were PhD students (n=17, 42.5%). Participants were from several different fields of study: arts and humanities (n=13, 32.5%), social sciences (n=11, 27.5%), and sciences and engineering (n=11, 27.5%). Five participants (12.5%) did not report their majors. Students were in different years within their graduate programs. The majority were first-year students (n=14, 35.0%), followed by third-year students (n=11, 27.5%), second-year students (n=9, 22.5%), and fifth-year (n=3, 7.5%) and fourth-year (n=3, 7.5%) students.

In the UK, 40 postgraduate students studying for a master’s degree completed the student questionnaire and the IELTS Academic Writing Tasks and submitted assessed discipline-specific multimodal assignments. Sixteen of these participants also participated in the semi-structured student interviews. The age of the student participants ranged from 21 to 42 years old (M=25.9, SD=3.89), and they were from 12 different L1 backgrounds (Table 3). They reported having different years of experience in learning English (M=15.79, SD=5.93). The participants also reported their most recent English proficiency test scores, which were IELTS or Duolingo. The participants’ overall IELTS scores ranged from 6.0 to 8.5. The most frequently reported (mode) overall IELTS score was 7.0. The participants studied a range of different master’s programs (Table 3), with social sciences (n=16, 40.0%) and sciences and engineering (n=16, 40.0%) more heavily represented than arts and humanities (n=8, 20.0%).

In the USA, 28 graduate-level students completed the student survey (19 female, 8 male, and 1 non-binary), ranging from the age of 25 to 38 years old (M=29, SD=3). The students were from nine L1 backgrounds (Table 3). All student participants reported substantial English learning experience: 10 students (36%) had over 20 years of English learning experience; eight (29%) reported 16-20 years of English learning; nine (32%) had 10-15 years of English learning; and only one (4%) reported less than 10 years of English learning. For the most recent English proficiency test scores, 17 students reported their TOEFL scores (M=97, SD=10) and 11 students reported IELTS scores. Among the students who reported IELTS scores, two received a score of 7.5, eight received a score of 7.0, and one received a score of 6.5.

The student participants were enrolled in a fairly diverse range of graduate programs in terms of educational levels and disciplines. Sixteen students were enrolled in PhD programs and 12 were enrolled in master’s degree programs. The majority of these students (n=20, 71%) were either in their first or second year of their program. The students represented 19 fields of study that can be broadly categorised as sciences and engineering (n=10, 36%), social sciences (n=12, 43%), and arts and humanities (n=6, 21%). Out of the 28 student participants, seven participated in the writing test and follow-up interviews.

Among the academic staff participants, 56.9% were female, 42.1% were male, and 0.9% were non-binary or non-reporting, ranging from 21 to 73 years old. They represented a fairly diverse group of academic disciplines and teaching experience. Table 3 provides an overview of the demographic information of the academic staff participants.

Table 3: Selected demographics of academic staff participants

In Türkiye, 84 academic staff members teaching at the graduate level in English-medium instruction universities completed the academic staff questionnaire, and 17 of them also joined the semi-structured interviews. Their ages ranged from 31 to 73 years old (M=44.14, SD=8.28). Academic staff members were teaching in several different disciplines, as can be seen in Table 3. One participant just indicated that they were teaching in an MA program without a specific major. Academic staff members had different years of experience in teaching: less than 10 years (n=18, 21.5%), 10-15 years (n=20, 24.0%), 16-20 years (n=18, 21.5%), 21-25 years (n=16, 19.0%), and more than 20 years (n=12, 14.0%).

In the UK, 70 academic staff members teaching at the postgraduate level completed the academic staff questionnaire, 11 of whom also participated in the semi-structured interviews. Academic staff members had a wide range of teaching experience (from one year to 35 years) and their self-reported disciplinary backgrounds and gender can be seen in Table 3. The age of the academic staff members participating in this study ranged from 28 to 69 years old (M=40.16, SD=10.24).

In the USA, 62 academic staff members completed the survey (35 female, 25 male, one non-binary, and one not reporting). The age of the faculty participants ranged from 27 to 71 years old (M=47, SD=10), while three participants did not report their age. These participants varied greatly in terms of their length of teaching experience: from two years to 39 years (M=16, SD=10). The academic staff participants were from a fairly diverse range of disciplines, as shown in Table 3. Of the 62 academic staff participants, 39 submitted an assignment sheet, and 13 participated in the follow-up interviews.

4.3 Data collection materials

Ethical approval was obtained for the study separately in the three research sites: the Social Sciences University of Ankara (ID: 54118), the University of Southampton in the UK (ID: 80464), and Georgia Southern University in the USA (ID: H22201). Data were collected through four main materials: questionnaire, assignment sheets, writing tasks, and semi-structured interviews.

4.3.1 Questionnaires

The purpose of the student questionnaire (Appendix A) was to gather information about the student participants’ demographics, the types of assessments in their graduate classes and their frequencies, the types of multimodal resources used in their graduate class assignments and their frequencies, the evaluation of their multimodal assignments, their interest in working in multimodal assignments, their self-assessment of their multimodal skills, and their perceptions regarding the similarities between IELTS Writing Tasks and multimodal assignments in their graduate classes in terms of the required skills. It consisted of 19 items, with eight demographic questions, nine closed-ended, Likert-scale questions, and two check-box questions. A total of 108 student participants (40 in Türkiye, 40 in the UK, and 28 in the USA) completed the online questionnaire.

The purpose of the academic staff questionnaire (Appendix C) was to gather information about academic staff members’ demographics, how often they assign multimodal assignments in the graduate classes they teach, what types of multimodal resources they require in their graduate class assignments and how often and what types of assessment they do, how they evaluate multimodal assignments, their perceptions regarding their students’ interest in working in multimodal assignments, their perceptions regarding their students’ multimodal skills, and their perceptions regarding the similarities between IELTS writing tasks and multimodal assignments in their graduate classes in terms of the required skills. It consisted of 15 items, with four demographic questions, nine closed-ended, Likert-scale questions, and two check-box questions. A total of 216 academic staff participants (84 in Türkiye, 70 in the UK, and 62 in the USA) completed the online questionnaire.

4.3.2 Assignment samples

The purpose of collecting multimodal assignment samples from student participants was to provide more insights into the expectations regarding the use of multimodal resources in graduate-level writing assignments. Similarly, to better understand the expectations regarding the use of multimodal resources in graduate-level writing assignments, the academic staff participants were asked to share a multimodal writing assignment. Explanations of multimodal writing defined in the study were provided to the participants prior to their submitting the assignments. Each student participant submitted one self-selected multimodal assignment, and the researchers reviewed and verified that all submitted assignments met the definition of multimodal writing used in the study. A total of 87 student participants (40 in Türkiye, 40 in the UK, and 7 in the USA) submitted multimodal writing assignments. Multimodal writing assignments were only collected from the academic staff members in the USA (n=39).

4.3.3 IELTS writing tasks

Two retired IELTS Academic Writing Tasks were selected from IELTS Academic 16, published by Cambridge University Press (2021). IELTS Academic 16 included four authentic practice tests. Because the vocabulary in Test 1 seemed to be less technical than that in the other tests, we decided to use the writing tasks in Test 1. Using the same content and format from the writing test, we created an online version of the test for data collection purposes. The writing test used in the study is provided in Appendix B.

A total of 83 student participants (36 in Türkiye, 40 in the UK, and 7 in the USA ¹ ) completed the IELTS Academic Writing Tasks.

4.3.4 Interviews

The purpose of the semi-structured interviews was to provide more insights regarding the participants’ perspectives on multimodality, graduate-level multimodal writing assignments, and their comparison with the IELTS Academic Writing Tasks. After completing the online questionnaires and the related tasks, the student and academic staff participants who indicated that they were interested participated in a follow-up interview. In total, 40 student participants (17 in Türkiye, 16 in the UK, and 7 in the USA) and 41 academic staff participants (17 ² Türkiye, 11 UK, and 13 USA) were interviewed individually online (via Zoom in Türkiye and the USA, and Microsoft Teams in the UK). The semi-structured interviews were conducted based on an interview protocol for students and an interview protocol for academic staff. The interview protocol for students (Appendix D) included 12 open-ended questions and the interview protocol for academic staff (Appendix E) included 16 open-ended questions.

4.4 Data collection procedures

To recruit student participants, the researchers shared project flyers with the target students on their campuses, sent a formal letter and/or a project flyer to institutional offices and/or local community contacts that regularly worked with the target student population (e.g., general secretary of ASBU, the Ankara Foreigners Community, and listservs for international students in the USA) and kindly asked the first participants to share the information about this project with their networks. To recruit academic staff participants, the researchers sent recruitment messages via emails including project introduction and/or project flyers to their personal contacts, other academic staff members who were teaching graduate-level courses at different universities, and professional listservs. After receiving responses from volunteers, the researchers followed up with more information about the project, the consent form, an online questionnaire, and other relevant documents (e.g., the participation information sheet for the participants in the UK).

Data collection was completed in two main stages (i.e., Questionnaires, Assignment sheets, and IELTS Academic Writing Tasks in Stage 1 and Semi-structured interviews in Stage 2) from both student and academic staff participants in the three research sites. Data collection followed a similar procedure using the same set of instruments at the three research sites. Each student participant was provided with instructions via email and/or project flyers to complete an online questionnaire, to submit a multimodal assignment, and to complete two retired IELTS Academic Writing Tasks. The participants first signed their consent form and completed the questionnaire on Qualtrics and were then asked to submit a multimodal written assignment. In Türkiye, the participants signed the consent form, completed the student questionnaire, and submitted the multimodal assignment during their meeting with the researcher on Zoom. In the UK and the USA, the participants signed the consent form, completed the questionnaire, and submitted their multimodal assignments online prior to meeting with the researcher to complete the writing tasks.

The writing test was administered online via Zoom in Türkiye. Each participant was provided with a unique link to a Google doc to work on the writing tasks and was asked to share their screen so that the researchers could monitor their writing processes. In the UK, the student participants completed the writing tasks under the same conditions as the test. The writing tasks were completed within the prescribed time limits by IELTS: 20 minutes for Writing Task 1 and 40 minutes for Writing Task 2.

1. Given the small number of student participants in the USA who had submitted multimodal writing assignments and completed the IELTS Academic Writing Tasks (N=7), the student-submitted multimodal writing assignments and their performance data on the IELTS Academic Writing Tasks were not included in the analysis for this report.

2. We aimed to conduct 10 interviews per country as per our proposal, but more interviews were done to provide perspectives from academic staff from different disciplinary backgrounds.

For each task, participants were given “five minutes left” oral reminders. Participants were not explicitly told that the tasks were IELTS Academic Writing Tasks, but many of them stated that the tasks were the IELTS writing tasks once they finished the tasks.

Due to the slight variation in the data collection procedure, the data collection sessions in Türkiye lasted between 80 and 90 minutes each. Data collection sessions in the UK and the USA took 60 minutes. All the student participants who completed the three tasks (questionnaire, assignment submission, and IELTS Academic Writing Tasks) received Amazon gift cards as compensation. Two student participants in Türkiye received partial compensation as they did not complete the writing tasks.

At the beginning of each interview session, the participant was first asked to review and sign the consent form via Qualtrics. Each interview was approximately 30-60 minutes, with the average length of the student interviews slightly shorter than that of the academic staff interviews. The interviews were audio-recorded on Zoom (Türkiye and the USA) and on Microsoft Teams (the UK), and the automatically generated transcriptions were obtained and reviewed and corrected as necessary. All the participants who completed the interviews received Amazon gift cards as compensation.

4.5 Data analysis

The first research question was addressed by coding and analysing the 119 multimodal writing assignments submitted by the participants (40 multimodal writing assignments in Türkiye, 40 multimodal writing assignments in the UK, and 39 multimodal assignment sheets in the USA). These assignments were classified into genre families that were put forward by Gardner and Nesi (2013) and by examining the topics and multimodal resources in the assignments collected in each research site. Based on the coding of the assignment instructions, the assignments were first qualitatively coded for three main task features: genre, topic, and modality. Gardner and Nesi’s (2013) classification system, which includes 13 genre families with different genres in each (Table 4), was used to guide the coding of the assignment instructions, with several new genres added based on the data collected in the study. The assignment instructions were also coded for topic in two categories as general and discipline-specific and for modality in two categories as main mode (e.g., text) and sub-modes (e.g., table or chart). Sub-modes were coded for both types of mode and the frequencies. The coded data were analysed using descriptive statistics, in particular frequency and percentage distributions. Although a second coder was not involved, the researchers were able to discuss and resolve any uncertainties or confusions in the coding process and thus ensure an adequate level of trustworthiness of the process (Saldaña, 2021).

Table 4: Genre families and example genres adopted from Gardner and Nesi (2013)

The second research question was addressed by comparing the multimodal writing samples submitted by 71 student participants (31 in Türkiye, 40 in the UK) and their performance on the two IELTS writing tasks for lexical sophistication and syntactic complexity. In terms of lexical sophistication, register-specific indices were selected; hence, the academic section of the Corpus of Contemporary American English (COCA) (Davies, 2009) was utilised as the reference for the frequencies of single words, bigrams and trigrams. The tool for the automatic analysis of lexical sophistication (TAALES) version 2.8.1 (Kyle et al., 2018) was used to analyse lexical sophistication, taking into account the previous validation and empirical studies (e.g., Crossley et al., 2019; Kyle et al., 2018). The analysis focused on eight measures of lexical sophistication: content word range, content word frequency, function word range, function word frequency, bigram (i.e., 2-word sequences) range, bigram frequency, trigram (i.e., 3-word sequences) frequency, and trigram range. In contrast to frequency, word range evaluates the extent to which a particular word appears across texts within a corpus (Kyle & Crossley, 2015). The frequency indices of content and function words were examined separately rather than examining all words since content and function words provide “distinct information from each other” (Durrant et al., 2019, p. 47). The frequencies were log-transformed to facilitate comparisons and statistical analysis. All these measures were normalised per 100 words per text to ensure reliable comparisons.

For syntactic complexity, the following three phrasal sophistication indices in the L2 Syntactic Complexity Analyser (L2SCA) (Lu, 2010) were used: Complex nominals (i.e., nouns plus adjective, possessive, prepositional phrase, adjective clause, participle, or appositive) per clause (CN/C); Complex nominals per T-Unit (CN/T); and Verb phrases per T-unit (VP/T). Phrasal complexity measures were utilised instead of clausal measures because phrasal complexity is predictive of higher proficiency and writing quality in academic writing rather than clausal complexity (Biber et al., 2020; Lu, 2017).

A non-parametric test for paired samples, the Wilcoxon-signed rank test, was selected due to the non-normal distribution of the data. A Bonferroni correction was applied to avoid spurious positives while making simultaneous multiple comparisons. An open-source programming language, R version 4.2.1, was used to perform the Wilcoxon-signed rank test, and effect sizes (r) were reported (R Core Team, 2022).

The third research question was addressed primarily by qualitatively analysing the interview data from 40 student participants (17 in Türkiye, 16 in the UK, and 7 in the USA) and 41 academic staff participants (17 in Türkiye, 11 in the UK, and 13 in the USA). Responses to relevant questions on the student and the academic staff questionnaires were included and analysed quantitatively to provide a more general picture of the perspectives from the stakeholders.

The interview data were analysed to help better understand the perspectives. After reviewing and verifying the interview transcriptions provided by automatic transcription software to ensure their accuracy, the researchers followed the thematic analysis process and analysed the interview data inductively (see Braun & Clarke, 2013). We created four themes, and these are listed below in the order of prevalence (as determined by the proportion of coded transcription segments) in the interview data:

1. Theme 1: Text as the primary mode in both the IELTS Academic Writing Tasks and postgraduate multimodal writing assignments
2. Theme 2: Perceived similarities and differences between the IELTS Academic Writing Tasks and multimodal writing assignments at university in terms of multimodality
3. Theme 3: Perceived similarities and differences between the IELTS Academic Writing Tasks and multimodal writing assignments at university in terms of language skills/features
4. Theme 4: Perceived possibility of IELTS Academic Writing Tasks to predict postgraduate multimodal writing performance.

5 Findings and discussion

5.1 To what extent are IELTS Academic Writing Tasks comparable to graduate-level multimodal writing tasks required at university in terms of task features such as genre, topic, and modality?

5.1.1 Comparability of genres

The analysis of the graduate-level multimodal writing assignments collected in Türkiye shows that students completed multimodal assignments in a variety of genre families in their higher education classes. Out of the 40 graduate-level multimodal writing tasks collected, the majority were of the research report genre family (n=10, 25%), followed by the critique genre family (n=9, 22.5%). Table 5 presents the distribution of genre families in the graduate-level multimodal writing assignments collected in Türkiye.

Table 5: Genre families of the multimodal writing assignments in Türkiye

The lower frequency of the essay genre family among the graduate-level multimodal writing assignments collected in Türkiye is notable and important regarding its comparability to the essay genre targeted by IELTS Academic Writing Task 2.

It is apparent that expressing opinion on a topic is not a common writing assignment for graduate students in Türkiye. On the other hand, the four more common genre families of research report, critique, methodology recount, and explanation are more comparable to IELTS as most assignments in these genre families are also reading-to-write tasks or integrated writing tasks, with either textual information or visual information presented in a chart/graph, as in IELTS Academic Writing Task 1.

Regarding the postgraduate writing assignments collected in the UK, there was a great diversity in the genre families represented in the multimodal writing assignments at a UK university. Gardner and Nesi’s (2013) 11 different genre families and one emerging genre family (blog post) were represented in this sample. Out of the 40 multimodal writing tasks, the most common genre family was case study (see Table 6 for the distribution of the genre families of the multimodal writing assignments in the UK).

Table 6: Genre families of the multimodal writing assignments in the UK

It may seem surprising that the case study genre family was the most common; however, it can be expected given that 16 out of 40 participants had social sciences backgrounds and were asked to submit business genres, such as company reports and reports on a single issue for their module assignments. Out of these genres, the exercise, problem question, and research report genre families may show some similarities with the IELTS Writing Task 1 since students are expected to conduct data analysis and summarise their analysis in these genre families and the IELTS Writing Task 1. The explanation, essay, and critique genre families represented in the assignments that students were asked to produce are likely to be similar to the IELTS Writing Task 2 because students are expected to provide reasons and justifications for the ideas that they propose in these genre families and the IELTS Writing Task 2.

In the 39 assignment sheets submitted by the academic staff participants in the USA, in addition to the seven genre families (research report, methodology recount, explanation, case study, narrative recount, literature survey, and essay) identified by Gardner and Nesi (2013), there were also two oral presentation assignments and a debate assignment. The most commonly used genres were in the three genre families of research report, methodology recount, and explanation (Table 7). This result suggests that essay is not a common genre in the graduate-level assignments collected in the USA dataset; instead, research writing, both research report and methodology recount, appears to be more frequently used. Therefore, in comparison to IELTS Writing Task 2, IELTS Writing Task 1 may share more similarities with the graduate-level writing assignments in the USA in terms of genre characteristics.

Table 7: Genre families of the multimodal writing assignments in the US

5.1.2 Comparability of topics

All of the assignments that graduate student participants in Türkiye submitted as sample graduate-level multimodal writing tasks were on discipline specific topics. Some example topics from the assignments are as follows:

• Homeland security of the USA upon pandemic
• Drones born in Switzerland (see Figure 2)
• The non-profit organisation Fortify Rights
• Exploring the meaning of buildings at Aşıklı Höyük and Çatalhöyük
• A free educational policy for rural population and disadvantaged groups in Ghana
• Tea consumption and cognitive functioning
• Türkiye’s migration policies and implementing education migration as soft power
• Life cycle assessment and comparison study of clean coal power generation technologies in China.

Drone No. 2: ATLANTIKSOLAR. Towards low-altitude multi-day endurance

Originally built for the very first Atlantic crossing of an unmanned solar-powered aircraft, AtlantikSolar has demonstrated its long endurance capabilities: Its non-stop flight of 82 hours in summer 2015 set a new world record in flight endurance for UAVs of its size. By taking high-resolution images of remote glaciers in the arctic region in summer 2017, AtlantikSolar contributed to the research of glaciologists at ETH Zurich.

Year of birth: 2013

Place of birth: Zurich

Measurements: 560 x 200 x 45 cm

Weight: 6.7 kg

Functions: Multi-day sensing, mapping

atlantiksolar.ethz.ch

Link to video

Figure 2: An excerpt from the multimodal assignment ‘Drones born in Switzerland’

Similarly, all the multimodal assignments collected in the UK were written on discipline-specific topics rather than general topics, unlike the IELTS Academic Writing tasks. The students, for instance, were asked to write a company report on a brand for a case study genre family. The students were asked to design a product or luxury brand for a specific audience for a design specification genre family. For such professional genres, the students had some flexibility to choose their own topic, specifically, a company or a product to focus on in their assignments. Except for the exercise genre family, most of the other genre families represented in the sample allowed students some flexibility to focus on an exact topic that they determined as long as it was relevant to the assignment brief or in line with the topic options provided in the assignment briefs. The exercise genre family topics were pre-determined in that students were asked to analyse the data and show their analysis together with justifications provided by their lecturers. Figure 3 exemplifies a design specification assignment brief in which students are expected to combine both text and visual resources. The requirement to write an article that includes images is explicitly stated (please see the highlighted text). In addition to that, students are expected to write a blog that may include other multimodal resources within the same assignment.

Figure 3: An excerpt from the instructions of a multimodal writing assignment

The graduate assignments collected in the USA were also shown to be different from the IELTS Writing tasks in terms of the specificity of the topic. Whereas the IELTS Writing tasks tended to have topics that were more general, all the graduate assignments were on discipline-specific topics and required frequent use of lexicogrammar specific to the disciplines when completing the assignments. For example, in a finance assignment (assignment05_USA), the students were required “to complete a research survey of a behavioral tendency in financial decision-making” and to analyse and report their findings. In a food science assignment (assignment09_USA), the students were required to “explain the importance of batter viscosity to bubble retention and final cake volume” using both text and visuals and the answer “must include viscosity, stability, coalescence, disproportionation and expansion”. In a literature assignment (assignment14_USA), the students were required to complete a research project on Howard Finster’s Paradise Garden and southern outsider art through field trips and collection and analysis of “photographs, notes, internet sources, articles, reproductions of sources, transcribed interviews…”. It is interesting to note that the discipline-specific nature of the graduate-level assignments is evidenced by both the topic and the methods and types of sources frequently used in the respective disciplines. Figure 4 exemplifies an assignment in which students were required by the faculty member to use a table, and there is also a mention of figures.

Figure 4: An excerpt from the instructions of an assignment with a table requirement

To sum up, most of the graduate assignments in our sample differed from the IELTS Academic Writing tasks in two ways. First, the graduate assignments focused on discipline-specific topics as opposed to general topics. Second, the graduate assignments almost always required students to obtain and use primary, secondary, or a combination of both types of sources, as shown in the examples above.

5.1.3 Comparability of modalities

Regarding modality, the main mode of all the assignments collected from the participants in Türkiye was text. However, different types of meaning-making resources were used in them. Assignments generally demonstrated the use of two types of modes (n=19, 48%) or one type of mode (n=14, 25%) while a few assignments involved three (n=6, 15%) or more types of modes (n=1, 3%). Findings show that the main meaning-making resource in students’ course assignments is text while they also commonly involve other semiotic resources such as images, tables, graphs, or charts.

Similarly, the primary output mode of all the assignments collected from the participants at a UK university was text. This feature shows similarity with the expected output of the IELTS Writing tasks. The main difference between the multimodal assignments and the IELTS Writing tasks seems to lie in the nature of multimodality in that the input/prompt of the IELTS Writing Task 1 includes multimodality whereas the output of the university assignments in the UK involves multimodality for all the genre families, except for the exercise genre family in which students are provided with data or graphs. In the multimodal assignments, the students also made use of different types of meaning-making resources, including tables, images, and graphs. Apart from the text, tables (n=23, 58%) and images (n=23, 58%) were used, followed by graphs (n=18, 45%) in 40 multimodal assignments analysed. These percentages suggest that there were also occasions in which these multimodal features co-occurred in the multimodal written assignments. Although all the assignments collected in the UK were text-based, they all included at least one multimodal resource.

The graduate assignments used by the academic staff participants in the USA were also largely text-based assignments, although all of them required the use of other modes of communication such as audio, video, graphs (n=12, 30.7%), tables (n=14, 35.8%), charts (n=7, 17.9%), and images. Three assignments, two oral presentations and a debate, showed a clear shift from heavily text-based written assignments to multimodal assignments relying on audio and visuals. In all the three assignments, the written component was limited to written texts on PowerPoint slides or other visual aids that the students chose to use.

The analysis of the use of multimodal resources in the assignments collected in the USA further showed that only four assignments did not focus strongly on the use of written texts. These included one of the oral presentations, the debate assignment, an infographic assignment, and an assignment requiring students to design and create a balanced scorecard for a company. The rest of the assignments, however, had explicit and extensive requirements regarding the use of written texts. In addition to texts, all the assignments required the use of other multimodal resources. The three most frequently required multimodal resources were tables, graphs, and charts. This requirement of the use of visuals is similar to IELTS Writing Task 1 in the sense that students would be required to understand how visuals such as tables and charts present information and to summarise, report, and discuss such information. However, all the graduate assignments also required students to produce tables, charts, and graphs to help present data and illustrate patterns in the data. Although the results indicated a strong preference for the use of tables and charts, it is important to note that most of the assignments (over 60%) were from sciences and social sciences, and thus this result may also be a reflection of the disciplinary practices in these fields.

In several assignments from the humanities, for example, multimodal resources such as videos, films, and photographs were often required or suggested. The analysis of the assignment sheets also revealed two other patterns regarding the use of multimodal resources in written communication. First, some assignment requirements suggested that there was more flexibility in choices of communication modes. For example, assignment #2 (public health) is a research project that requires students to conduct literature reviews and interviews; however, in addition to the traditional format of a written report, students are also allowed to choose to present their results using either documentary-style films or graphic ethnography. Second, there were assignments aiming to help students learn to connect writing or text-based communication to other modes of communication. For example, assignment #4 (criminal justice) requires students to create an infographic for a research article to translate research evidence and findings into more engaging messages that are easier to understand for the general public.

In summary, regarding the main mode of writing, our findings show that IELTS Writing tasks are comparable to graduate-level course assignments as written text is the main mode in both at the production level. However, regarding multimodality at the production level, IELTS Writing tasks are different from the graduate-level course assignment samples that we collected in our contexts as test-takers produce only written texts in their responses. At the receptive level, Writing Task 1 is multimodal, thus can be considered more similar to graduate-level course assignments in modality than Writing Task 2.

5.2 To what extent is students’ performance on IELTS Academic Writing Tasks comparable to their performance on graduate-level multimodal writing tasks required at university in terms of linguistic features?

This section reports the findings based on the student data collected from Türkiye and the UK because there was not enough data to report descriptive or inferential statistics in the USA. Some students’ written assignment samples collected in Türkiye were product developments (e.g., coursebook unit) with a lot of visuals or were mostly numerical (e.g., statistical analysis) and had a limited amount of text. These assignment samples did not involve enough written text generated by the students and would not allow to analyse linguistic features to make meaningful comparisons; hence, their assignments were removed from the linguistic analysis. In total, 31 student multimodal assignments collected in Türkiye were included for the comparative analysis of linguistic features.

5.2.1 Lexical sophistication

Table 8 presents the descriptive statistics for six lexical sophistication measures across three tasks collected in both Türkiye and the UK. The mean lexical sophistication measures were similar across the different tasks in both Türkiye and the UK in that all the mean values were more similar between the IELTS Academic Writing Task 1 and the multimodal assignments in comparison to those between the IELTS Academic Writing Task 2 and the multimodal assignments. This suggests that lexical sophistication elicited by multimodal assignments and the IELTS Academic Writing Task has some similarities in terms of frequency and range of single words, bigrams, and trigrams in these two countries at the postgraduate level.

In Türkiye, for the range of content words, the means are observed to be closer between IELTS Academic Writing Task 1 and multimodal assignments than between IELTS Academic Writing Task 2 and multimodal assignments. The words that the students used in Writing Task 2 have a broader range than the words they used in Writing Task 1 and in their multimodal assignments. Given that “words that are used in fewer contexts are judged to be more sophisticated than those that are widely used” (Kyle & Crossley, 2015, p. 772), the descriptive results suggest that graduate-level course assignments elicit lexically more sophisticated writing than IELTS Writing Task 2 while Writing Task 1 elicits similar lexical sophistication to the course assignments. For the frequency of content words, and function words a similar trend is observed, with IELTS Writing Task 1 and multimodal course assignments being more similar in eliciting less frequent content and function words than IELTS Writing Task 2. More similar range and frequency means are observed for bigrams and trigrams across the domains.

Table 8: Descriptive statistics for lexical sophistication across domains in Türkiye and the UK

Overall, similar trends were observed for lexical sophistication in the data collected in the UK. First, the IELTS Academic Writing Task 1 was similar to the multimodal written assignments in terms of the frequency and range of single content words as well as the range and frequency of bigrams and the range of trigrams with reference to the COCA (Davies, 2009). This means that both the IELTS Writing Task 1 and multimodal writing assignments elicited similarly frequent and widely used single content words and bigrams, as well as similarly widely used trigrams by postgraduate students at a UK university (see Kyle & Crossley, 2015). When it comes to differences, the function words in the multimodal assignments were more widely used in the COCA than those in the IELTS Academic Writing Task 1, indicating a higher degree of lexical sophistication for the function words in the IELTS Academic Writing Task 1. This finding was also observed in the frequency of single function words. These two differences may be traced back to the genre families represented in the multimodal assignments. These genre families, such as case study and design specification, may have involved the use of function words,

such as articles, pronouns, conjunctions and auxiliary verbs that are more widely found in the COCA. The IELTS Academic Writing Task 1, on the other hand, elicited less widely used function words, including subordinating conjunctions, prepositions and particles, which are less likely to be featured in the COCA but are required to answer the prompt in the IELTS. This may also be due to the participants’ preference for varying their functional language use while summarising the charts and making comparisons between them.

The statistical comparisons of lexical sophistication measures suggest several similar trends across the three writing tasks collected in both Türkiye and the UK (see Tables 9 and 10). First, no statistically significant difference was observed in terms of the lexical sophistication measures (range and frequency of single content words and bigrams as well as the frequency of trigrams) between the IELTS Academic Writing Task 1 and the multimodal assignments in both Türkiye and the UK. Second, statistically significant differences were found between the IELTS Academic Writing Task 2 and the multimodal assignments in both Türkiye and the UK with regard to the range and frequency of single content words. However, it should be noted that there were more statically significant differences in terms of lexical sophistication measures in the UK context across the tasks, suggesting that the L2 writers in the UK tended to use more sophisticated vocabulary (single words) in their multimodal assignments than their counterparts in Türkiye. This may be due to their language proficiency and higher levels of exposure to English in the UK.

Table 9 presents the Wilcoxon signed-rank test results for lexical sophistication across the three types of writing collected in Türkiye. The results illustrate statistically significant differences across the domains for three measures with large effect sizes: academic range content words, academic frequency content words, and academic frequency function words. The range of the content words used by the graduate-level students in their responses to IELTS Writing Task 2 was significantly different from the range of the content words they used in their multimodal course assignments. The frequencies of both the content words and the function words in IELTS Writing Task 2 responses were significantly different from the frequencies of the content and function words in the course assignments. There were no significant differences across the three types of writing in bigram and trigram measures. These findings confirm that IELTS Writing Task 1 is more similar to graduate-level multimodal course assignments in eliciting similar lexical sophistication features, in particular both range and frequency of single content words and frequency of function words.

Table 9: Wilcoxon signed-rank test results for lexical sophistication across domains in Türkiye

Alpha level = 0.003. * Statistically significant

Table 10: Wilcoxon signed-rank test results for lexical sophistication across domains in the UK

Alpha level = 0.003. * Statistically significant

Table 10 presents the results of the Wilcoxon signed-rank test results for lexical sophistication measures across the three tasks in the UK. One interesting trend is that the similarities in terms of lexical sophistication were much fewer when the IELTS Academic Writing Task 2 and multimodal assignments were compared. The only non-significant differences were observed in terms of trigram frequency and range between the IELTS Academic Writing Task 2 and multimodal assignments, suggesting that both tasks elicited similarly frequent and widely used trigrams with reference to the COCA. It seems unsurprising to observe more differences with regard to lexical sophistication between the IELTS Academic Writing Task 2 and multimodal assignments at a UK university because the IELTS Academic Writing Task 2 is not multimodal in nature.

The general finding was that the IELTS Academic Writing Task 2 involved the use of less sophisticated single content and function words as well as bigrams than the multimodal assignments because such words and bigrams were more frequent and widely used in the COCA. It is worth noting that effect sizes were large for these differences, as can be seen in Table 10. These findings may be expected because the IELTS Academic Writing Task 2 relies on personal knowledge and experiences while the (multimodal) writing assignments at the postgraduate level at a UK university require support from the academic literature in building arguments and explanations (see Nesi & Gardner, 2012).

5.2.2 Phrasal complexity

Table 11 displays the descriptive statistics for three phrasal complexity measures (i.e., verb phrases per T-unit: VP/T), complex nominals per T-unit: CN/T, and complex nominals per clause: CN/C) across the three tasks collected in Türkiye and the UK. Unlike the indices of lexical sophistication, phrasal complexity indices showed strikingly different trends across the tasks in Türkiye and the UK. The L2 writers in Türkiye, for example, used phrasal complexity indices to a greater extent than their counterparts in the UK. Also, in Türkiye, the mean values of all the phrasal complexity indices showed substantial differences across the three tasks.

In contrast, the L2 writers in the UK used the phrasal complexity features in similar frequencies across the IELTS Academic Writing Task 1 and the multimodal assignments.

For all phrasal complexity measures in Türkiye, the means were highest for IELTS Academic Writing Task 1, followed by IELTS Academic Writing Task 2, and the lowest means were observed in the multimodal assignments. Students used more verb phrases per T-Unit, more complex nominals per T-unit, and more complex nominals per clauses in Writing Task 1 than in the other tasks. This means that IELTS Writing Task 1 elicited the highest phrasal complexity in graduate-level students’ writing. In the three measures, the mean differences were smaller between IELTS Writing Task 2 and the multimodal assignments than between IELTS Writing Task 1 and the multimodal assignments. This means that IELTS Writing Task 2 is more similar to graduate-level multimodal course assignments than Writing Task 1 in eliciting similar phrasal complexity in student writing.

Table 11: Descriptive results for phrasal complexity across domains in Türkiye and the UK

Unlike the trends in Türkiye, the opposite trends were observed in the tasks collected in the UK. Consistent with the findings of lexical sophistication, the IELTS Academic Writing Task 1 and multimodal writing assignments at a UK university had more similarities in phrasal complexity than the IELTS Academic Writing Task 2 and multimodal assignments, as shown in Table 11. For instance, the mean frequencies of complex nominals per T-unit and complex nominals per clause were highest in the IELTS Academic Writing Task 1 out of all the three tasks, but these values were very close to those of the multimodal writing assignments at a UK university. Interestingly, verbal phrases per T-unit were more frequent in the multimodal written assignments than in the IELTS Academic Writing Task 1; hence, meaning-making in the multimodal written assignments required the use of complex verb phrases which would involve a greater frequency of modals, finite and non-finite verbs.

Tables 12 and 13 present the Wilcoxon signed-rank test results for phrasal complexity features across the three types of tasks collected in Türkiye and the UK, respectively. In Türkiye, there were statistically significant differences in the three phrasal complexity measures across all tasks. In the UK, there was no statistically significant difference between the IELTS Academic Writing Task 1 and the multimodal writing assignments in terms of the frequency of complex nominals per T-unit and complex nominals per clause, suggesting that L2 writers at a UK university employed complex nominals similarly in both tasks. Inconsistent findings in the two contexts may be traced back to a number of factors, such as genre and disciplinary differences, and differences in the nature of multimodal writing assignments. As we were unable to model the data, we can only speculate about the reasons behind these inconsistent findings.

For each phrasal complexity measure, the results show statistically significant differences between student writing in IELTS Academic Writing Task 1 or IELTS Academic Writing Task 2 and multimodal assignments, all with large effect sizes in Türkiye. This means that students used the highest numbers of verb phrases per T-Unit, complex nominals per T-unit, and complex nominals per clauses in IELTS Writing Task 1 than in IELTS Writing Task 2 and in course assignments. These results confirm that IELTS writing tasks elicit higher phrasal complexity than multimodal course assignments, with Writing Task 1 eliciting higher phrasal complexity than Writing Task 2 in the writing of graduate-level students.

Table 12: Wilcoxon signed-rank test results for phrasal complexity across domains in Türkiye

Alpha level = 0.003. * Statistically significant

Table 13 shows that all three phrasal complexity features exhibited statistically significant differences between the IELTS Academic Writing Task 2 and multimodal writing assignments in the UK. There was a higher frequency of complex nominals per T-unit and clause in the multimodal writing assignments than in the IELTS Academic Writing Task 2.

This finding is unsurprising in that previous studies consistently found that discipline-specific postgraduate-level academic writing involved the use of nouns, nominalisations and other phrasal complexity features, such as attributive adjectives to express informational discourse (e.g., Nesi & Gardner, 2012; Staples et al., 2016). This suggests that a higher frequency of nominals is required to make meaning in discipline-specific multimodal writing assignments at UK universities (see Nesi & Gardner, 2012) than in the IELTS Academic Writing Task 2. There was a different trend for the frequency of verb phrases per T-unit in that the frequency of this phrasal complexity feature showed statistically significant trends across the three tasks. The IELTS Academic Writing Task 2, for example, included significantly more verb phrases per T-unit than the multimodal writing assignments. This may be due to the demands of the IELTS Writing Task 2, which requires students to write reasons behind their arguments from their own knowledge and experiences. Writing from knowledge and experiences may necessitate the frequent use of modal verbs, finite and non-finite verbs in comparison to discipline-specific multimodal assignments. There was no statistically significant difference in the frequency of complex nominals per T-unit and complex nominals per clause between the IELTS Academic Writing Task 1 and multimodal writing assignments. This is probably because both are multimodal tasks although this may not be the only reason for the similarities. In addition to being multimodal, both tasks require students to summarise information and compare and contrast data.

Table 13: Wilcoxon signed-rank test results for phrasal complexity across domains in the UK

Alpha level = 0.008.* Statistically significant

In summary, our analysis identified more differences between the IELTS Academic Writing Task 2 and the multimodal assignments at the postgraduate level in terms of both lexical sophistication and phrasal complexity indices in both Türkiye and the UK than similarities. However, our findings revealed some similarities between the IELTS Academic Writing Task 1 and multimodal writing assignments at a UK university and a Turkish university in terms of lexical sophistication features used in the present study, suggesting both tasks elicited similarly sophisticated vocabulary. On the other hand, the similarities between the IELTS Academic Writing 1 and multimodal writing assignments in terms of phrasal complexity indices were only observed in the student data in the UK. The strikingly different findings with regard to phrasal complexity indices between the data in Türkiye and in the UK suggest caution and further research in this area.

5.3 To what extent do students and academic staff members perceive IELTS Academic Writing Tasks to be comparable to graduate-level multimodal writing tasks required at university?

Several relevant questions on the questionnaires and the interview data were analysed to address this research question. Using thematic analysis, we created four themes:

1. text as the primary mode in both the IELTS Academic Writing Tasks and postgraduate multimodal writing assignments
2. perceived similarities and differences between IELTS Academic Writing Tasks and multimodal writing assignments at university in terms of multimodality
3. perceived similarities and difference between IELTS Academic Writing Tasks and multimodal writing assignments at university in terms of skills/linguistic feature
4. perceived possibility of IELTS Academic Writing Tasks to predict postgraduate multimodal writing performance.

Each of the themes is presented below, with discussion and interpretation of representative quotes from student participants and academic staff participants.

5.3.1 Text as the primary mode in both the IELTS Academic Writing Tasks and postgraduate multimodal writing assignments

This theme first emerged from the analysis of the participants’ responses to the questions about the assessment types used in graduate-level classes on the questionnaires and during the interviews. The responses from both the student participants and the academic staff across the three research sites indicated that the most frequently encountered assessment types included more traditional types of assignments such as essays, reports, summaries, reviews, oral presentations, and written reflections, which relied heavily on text as the primary mode of communication.

In Türkiye (Table 14), the student participants identified oral presentations, essays, reports, written reflections, reviews, and summaries as the most common assessment types. Tests/exams, posters, portfolios, product developments, exhibitions, or interviews, on the other hand, were rare.

Table 14: Frequencies of assessment types reported by student participants in Türkiye

Note: Minor discrepancies in the percentages in these tables are due to the rounding of decimal points.

In the UK (Table 15), 78% of the postgraduate students always or often wrote essays whose primary mode of communication is often text while assessment types that include multimodal elements, including videos and oral presentations, had much lower percentages in the ‘often’ or ‘always’ categories.

Table 15: Frequencies of assessment types reported by student participants in the UK

In the USA (Table 16), similarly, the student participants reported more frequent encountering with essays, reports, summaries, reviews, and other traditional text-based assessment types. Technology-mediated assignments or assignments incorporating different modalities seemed to be less frequently used and those included videos, posters, interviews, exhibitions, product development, and performance. Additionally, one student reported that empirical research papers and synthesis papers had always been used in their graduate classes; and another student reported peer review writing had often been used in their graduate classes.

Table 16: Frequencies of assessment types reported by student participants in the USA

The responses from the academic staff participants largely echoed those from the student participants across the three research sites. In Türkiye (Table 17), the academic staff participants indicated that the most frequently used assessment types were essays, oral presentations, reports, written reflections, reviews, and summaries; and assignment types such as posters, videos, product developments, or interviews were rare. Similarly, except for the oral presentations, text was the main mode in graduate-level writing assignments given by academic staff members in Türkiye.

Table 17: Frequencies of assessment types reported by academic staff participants in Türkiye

In the UK (Table 18), the academic staff participants also expressed more pronounced preference for essays over other assessment, with 49% of the participants indicated that essays were always used as an assessment type in their classes.

Table 18: Frequencies of assessment types reported by academic staff participants in the UK

These UK participants also highlighted the dominance of essay- or report-based assignments in their modules and emphasised that the IELTS Academic Writing Task 2 was monomodal traditional writing. In fact, two student interviewees wondered about the multimodal aspect of this research project during the interviews since they explicitly stated that there was no multimodal production in the IELTS writing tasks. Although all the interviewees noted the prevalence of essay- or report-based assignments in their disciplines, there was a great diversity of genre families represented in the multimodal assignments collected at a UK university; hence, students and subject lecturers may not have a great awareness of genres in UK higher education. One academic staff interviewee, Ralph, stated the following when asked about what kind of assignments he provided for his students:

“On the whole, it is … written sort of essay style assessment regardless of whether it is single or multimodal. In general it is a written [text] because that’s how students are assessed in the programs.”

Both students and academic staff interviewees of social sciences and humanities backgrounds shared similar sentiments that a written text would come first and that multimodality was not their primary concern when preparing assignments at UK universities. In a similar vein, a social sciences student, Helen, stated: “Most of the time I only include text in my assignments”. However, several students added that they would love to do more multimodal assignments at UK universities because they would enjoy doing them more than writing only texts.

In the USA (Table 19), 65% of the academic staff participants either often or always used essays as assessments. Technology-mediated assignments such as videos, posters, and portfolios were reported to be less frequently used. Additionally, several academic staff members reported using interdisciplinary art, different forms of qualitative research, annotations, and competition as assessments.

Table 19: Frequencies of assessment types reported by academic staff participants in the USA

Additionally, tests and exams were reported to be infrequently used by student participants in the USA while they were reported to be fairly commonly used by the academic staff. This inconsistency is probably because most of the student participants in this research sites were enrolled in doctoral programs but most of the academic staff participants taught both master’s and doctoral students. The results also showed that certain types of assignments appeared to be closely related to the disciplines. For example, product development assignments were mostly used in information technology, and interviews were more often used in social sciences. Thus, the relatively low frequencies of some of the assignment types, such as exhibitions and performance, might be associated with the small number of academic staff participants from some disciplines in the data set from the USA.

5.3.2 Perceived similarities and differences between IELTS Academic Writing Tasks and multimodal writing assignments at university in terms of multimodality

This theme was identified in the analysis of the interview data regarding the participants’ perspectives comparing IELTS Academic Writing Tasks and graduate-level multimodal writing assignments in terms of multimodality. Across the three research sites, most of the student participants and the academic staff participants considered IELTS Writing Task 1 to be more similar to their university multimodal writing assignments than IELTS Writing Task 2 in terms of multimodality, although some evidence suggests that this observation may be affected by the participants’ disciplinary backgrounds.

Regarding IELTS Writing Task 1, the perceived similarities focused primarily on the shared requirement for students to process and interpret visual information and to describe or summarise this information in subsequent writing.

Thus, in Türkiye, most participants considered IELTS Writing Task 1 to be quite similar to a multimodal writing assignment, as exemplified by the comments from a student participant, Thomas, below:

I think in the first task, actually, we are expected to interpret the numbers or the statistical information given on the charts. So basically, we have some visual support for meaning-making. The use of these charts serve as a facilitator for us, so it becomes easier for us to put our ideas on the paper by just looking at these charts. But of course we should have the ability to interpret correctly the information on the chart. But when we look at the second one, this is not the case, so there are just some basic descriptions. There is just a written question.

In the UK, several participants from physical and life sciences noted that IELTS Writing Task 1 was a sound foundation for the multimodal writing assignments at the graduate level. The student interviewees stated that they were asked to describe and summarise their data in their lab reports and dissertations, and the academic staff members of the abovementioned disciplinary backgrounds shared that their modules included data analysis or exercise assignments that had similarities with IELTS Writing Task 1. Alessia, an academic staff member expert in the life sciences discipline, for example, pointed this out in the quote below.

The first task is similar to our assignments because it is visual. The analysis of quantitative data would work in my field. Describing the patterns in the data would be in line with [practices in] our field.

The perceptions of Alessia were shared by the other academic staff and student interviewees who had life sciences or physical sciences backgrounds. One student, Suzy, had similar opinions.

The first task I think it’s related to analysing the data and giving…like making a comparison which is a lot more of what we do some in some of the modules. It includes a bit of numbers and some analysis. So we have to do some analysis. We have to give out some conclusions of what we think of when we look at the graphs. So, I think, yeah, it is similar in certain ways.

In the USA, most participants also saw more similarities between IELTS Writing Task 1 and the graduate-level writing assignments in terms of requiring students to process and understand information presented with both text and visuals. Most students perceived this similarity to be obvious and one of the students stated that she was “always required to interpret and summarise information presented in charts and tables” (Stella, PhD student, student interview). The majority of the academic staff participants interviewed echoed this observation, pointing out specifically that the requirement to read and understand visuals, such as charts and tables, aligned well with the goals in their courses to teach students to understand information presented in different manners. For example, Taylor, a professor in grain science, offered the following observation.

This first one with the graphs, this is absolutely what we do all the time, in my area. You see the information. You need to summarise it. You need to explain it. You either build this or you explain it. So, this is absolutely the case.

There seemed to be some differences between different disciplines, with the participants from sciences and social sciences generally perceiving IELTS Writing Task 1 as similar to multimodal writing assignments. In the UK, for example, while the interviewees of physical and life sciences backgrounds noted similarities between the IELTS Writing Task 1 and their assignments, these perceptions were not shared by most of the participants of arts and humanities and social sciences backgrounds. These participants stated that they would not use or interpret graphs because of the nature of their disciplines. However, it is worth noting that at the time of the interviews, those students had recently started their dissertations. Their perceptions about the use of graphs might have changed while they were writing their findings in their dissertations.

Most participants from Türkiye and the USA highlighted the similarities between IELTS Writing Task 1 and graduate-level multimodal writing assignments, but some participants in the UK also noted their differences. The main reason for this perception was the nature of modality in the task. The students stated that they would need to create multimodal resources themselves in their assignments rather than use multimodal resources created by the lecturers or included in the assignment prompts.

Hence, there seems to be a perceived difference between multimodal skills required in IELTS Writing Task 1 (receptive) and multimodal skills required in the multimodal discipline-specific assignments (productive). When asked about the differences between the IELTS writing tasks and multimodal writing assignments, one student interviewee, Sophie, studying marketing (social sciences) shared the comments below.

We have to constantly collect data like company performance, read their financial report and marketing metrics. I use Excel to create a graph first and also use Google Data Studio which is a dashboard building tool. We have to collect, comprehend and interpret data for my assignments, but these (the IELTS writing tasks) are more simple.

Sophie’s statements were representative of the UK participants’ perceptions regarding the differences between the IELTS Academic Writing Tasks and multimodal assignments. In addition to the differences in receptive multimodal skills required in the IELTS Writing Task 1 and productive multimodal skills required in the multimodal writing assignments, the participants also pointed out that the complexity level of the multimodal writing assignments was much higher than that of the IELTS tasks because collecting data, selecting data and deciding on how to report data required more time and thinking, based on the opinions of the interviewees.

In comparison, IELTS Writing Task 2 was perceived to be less similar to graduate-level multimodal writing assignments by most of the participants in all three research sites; and the perceived differences focused on the presence or absence of multimodal elements in the task prompt and the level of analysis and synthesis needed to complete the writing task. When asked to comment on the differences between the IELTS Writing Task 2 and multimodal writing assignments, many students and academic staff would first notice and describe the absence of the visual elements in the task prompt, as illustrated in the following quote from Aaron, an academic staff interviewee with an engineering background.

In my experience, we tend not to ask students to write an essay like this [IELTS Writing Task 2]. In engineering, 90% of the things are done by figures, equations and other things. In reports, students need to discuss things based on data not on personal thoughts.

This view was shared by many other participants, especially those from physical sciences and life sciences backgrounds. While university assignments asked students to use figures and tables to present their data or results, this component was missing in IELTS Writing Task 2. This observation seems to suggest, again, that there are disciplinary variations in how multimodality is enacted in student assignments, and this possible disciplinary variation is also supported by interview data from Türkiye and the USA.

Moreover, many participants considered IELTS Writing Task 2 to be more comparable to undergraduate essay-writing tasks, as illustrated by the following quote from Tristan, a student from Türkiye.

I think the first one is more similar to what we do because we are using some statistical programs and we generate some charts and statistics. And we are required to make some meaningful interpretations based on these charts. You have to, of course, report important information from those statistical analyses. So the first one, in that sense, prepares you better for that. But the second one, I think, this is more like undergraduate level kind of question. So, we are not asked this kind of questions any more in the graduate level.

Tristan considered the second writing task to be more similar to undergraduate-level essay questions, and this view was shared and further explained by quite a few academic staff participants. For example, in an interview with Timothy, an academic staff participant in Türkiye, he pointed out that IELTS Writing Task 2 reflects writing done in first-year composition classes or the types of questions asked in an “exam”. Another academic staff participant in Türkiye stated: “Well, it looks more like questions I prefer to ask in exams. But in the classes, it’s more like giving them a template and some instructions rather than giving them prompts. So yeah, this looks different than what I usually do.” Thus, it seems that, in addition to the absence of the requirement of using visual elements, the lack of the requirement to research, analyse, and synthesise external sources might be another reason why IELTS Writing Task 2 was considered to be different.

Overall, students’ and academic staff members’ perceptions of IELTS Academic Writing Tasks provide support for previous research findings that independent essay writing task alone is less representative of writing done in the university while tasks that integrate different skills elicit writing that is more similar to successful academic writing in the university (Llosa & Malone, 2019, Llosa et al., 2020; Riazi, 2016; Weigle, 2010).

5.3.3 Perceived similarities and differences between IELTS Academic Writing Tasks and multimodal writing assignments at university in terms of skills/linguistic features

This theme was identified in the analysis of the responses to the relevant questionnaire items and the interview data regarding the participants’ perspectives comparing IELTS Academic Writing Tasks and graduate-level multimodal writing assignments in terms of the necessary skills and/or linguistic features for completing these tasks and assignments. Across the three research sites, the participants identified both similarities and differences, and there are some variations in the findings from the three research sites.

In Türkiye, participants found IELTS Writing Task 1 to be more similar to graduate-level multimodal writing assignments. When comparing the skills required, an academic staff member, Anders, made the following comment referring to Writing Task 1: “Students need to evaluate, make some assessment of what is available, make some analysis of what they are presented with, and then produce their own version, so in that way I guess they are similar.” Gavin mentioned: “It’s a graphic, which means, in order to understand how to translate this into words, it’s not an easy work. It needs more skill actually to translate this graphics into the real sentences. It’s quite similar because most of the time our job is to translate the graphics and the images.” Ahmadi and Mansoordehghan (2014) refer to this as “graphical literacy” (p. 14), which according to them reflects academic writing better than a traditional independent essay writing task.

While both groups of participants mentioned more differences in the skills required by the IELTS writing tasks and graduate-level multimodal writing assignments, they mentioned more similarities in the necessary linguistic features. Tristan, for example, mentioned that features such as “syntactic structures, academic language, academic vocabulary, a regular introduction, body, conclusion” are used in both IELTS writing tasks and their graduate course assignments. “Coherence” was mentioned by another student participant as another similar linguistic feature.

The comments from the student and academic staff participants, as exemplified in the above paragraphs, indicate perceived similarity between IELTS Writing Task 1 and graduate-level writing assignments in three features: type of the input, types of the skills required, and features of the written product. It is clear that the multimodal input in Writing Task 1 (i.e., the chart/graph) is similar to the types of input that graduate level students encounter in their writing assignments.

It is also noticed that due to the visual input, both Writing Task 1 and graduate-level writing assignments appear to stimulate similar cognitive processes that are involved in analysing, interpreting, and synthesising as students or test-takers transform the visual content into written descriptions or summaries. Finally, both IELTS writing tasks and graduate-level writing assignments were considered to elicit written products with similar linguistic features as both require certain linguistic, rhetorical and stylistic conventions such as grammatical accuracy and complexity, lexical complexity and range, and formal language use.

In the UK, most student interviewees perceived that generic language skills/features, including grammar, organisation, vocabulary and “good English” required in the IELTS Academic Writing Tasks were similar to those of the multimodal writing assignments at the university. Nicole, a student of social sciences background, summarised this as follows:

When it comes to the language skills, they are quite similar, I guess. Because when we are writing an essay, we need to have a structure to share ideas.

I mean, [language] skills for these writing tasks are important to write an essay.

I mean, these are basic skills, but when we are writing our essay, we use them.

They require very basic things like…can you analyse something? Can you tell the differences…?

In addition to perceived similarities in surface-level language skills between the IELTS Academic Writing Tasks and multimodal writing assignments, several students also noted that language skills at the discourse level in the IELTS writing tasks were similar to those in the multimodal writing assignments, as Sophie notes:

I think they involve similar quite similar language skills, so interpreting data in this task is quite similar in my assignments. And framing before making arguments - that is also similar to my assignments a lot of time.

Some academic staff members offered the same observation. However, fewer academic staff members than students were in agreement that the IELTS writing tasks and multimodal writing assignments would require similar language skills. They stated that although clear English language use would be required to make meaning in both the IELTS writing tasks and multimodal writing assignments, discipline-specific language use, which is not required in the IELTS writing tasks, would be necessary in the multimodal discipline-specific writing assignments. The importance of discipline-specific vocabulary and meaning-making practices required for the multimodal writing assignments was highlighted by both the student and academic staff interviewees. The following quote from a student interviewee is an example.

I think there are a lot of intricacies in a lot of our assignments. Um, that are very specific to the industry. There’s quite a lot of jargon and quite a lot of very specific vocabulary that it’s from the business world And I think that can make it quite challenging and that’s something that I think neither [the IELTS] Writing Task 1 or 2 really have. They’re quite kind of…for anyone. And I found it there were quite a lot of new words that I’ve never heard of that I’ve just suddenly been expected to use in my course.

Moreover, several students and more than half of the academic staff interviewees observed the differences in the task requirements regarding reading academic sources, critical thinking, and using support from academic literature. Mary, an academic staff member, notes:

In postgraduate assessment, most of the assessment is academic. So these tasks don’t require any citation of evidences or resources. So this [the IELTS Writing Task 2] would be about personal opinions. In the first task, there is very little thinking skills assessed in this and most of it is just, you know, transforming the visual information to text.

In the USA, the questionnaire results showed that over half of the student participants (n=17, 61%) considered the skills needed to complete IELTS Writing Task 1 to be comparable to those needed to complete the graduate multimodal assignments. Specifically, 7% of the students (n=2) indicated that the required skills were very similar, 36% (n=10) similar, and 21% (n=6) slightly similar. The majority of these participants were in sciences, social sciences, or business. Only one student participant from science believed the skills required were different. Among the academic staff participants, about 70% suggested that the skills required to complete IELTS Writing Task 1 were comparable to those needed to complete the graduate-level multimodal assignments, with 10% (n=6) indicating very similar, 31% (n=19) similar, and 29% (n=18) slightly similar. Most of these respondents taught in sciences and social sciences and they suggested that being able to interpret and summarise information from charts, tables, and graphs was an important skill in their fields. The academic staff participants from education and humanities, on the other hand, showed some disagreements. During the follow-up interviews, it was shown that their disagreements seemed to focus on the way the information was presented in the prompt. Tori, a professor in education, provided the following observation.

Um, there’re some differences with the graph. I do have students look at information and work to make sense of it. But it’s typically not information presented like this. It’s usually, like I’m describing a scenario, perhaps using short video clips, or something like that. And I’m asking my students to make sense of the information using the ideas from what we have learned in class.

The quote above seemed to suggest two differences in terms of the skills needed to complete IELTS Writing Task 1 and graduate multimodal assignments in some disciplines. The first one was the students’ abilities to process and understand information from static visuals as opposed to moving visuals, and in the assignments described by Tori, verbal skills appeared to be more prominent. The second one was the additional requirement in the graduate multimodal assignments to make connections between the information presented in the prompt and that from previous classes. Thus, both the students’ and the academic staff participants’ responses seemed to highlight some disciplinary differences in their writing practices.

More than two thirds of the students indicated that IELTS Writing Task 2 was comparable to the multimodal assignments in their graduate classes in terms of the skills needed to complete the tasks, with 7% (n=2) indicating very similar, 32% (n=9) similar, and 43% (n=12) slightly similar. When asked to comment on the similarities and differences in skills and linguistic features during the interviews, most students shared that they saw IELTS Writing Task 2 as primarily testing them to develop support for a point of view; and that they believed their abilities to select and use examples and to put forward different reasons in order to support a main argument or point of view were important for them to complete the graduate multimodal assignments successfully.

A slightly lower percentage of the academic staff participants (n=40, 65%) thought that there were similarities between IELTS Writing Task 2 and graduate-level multimodal assignments in terms of the skills needed to complete the tasks, with 8% (n=5) indicating very similar, 26% (n=16) similar, and 32% (n=20) slightly similar. In the follow-up interviews, some academic staff participants compared IELTS Writing Task 2 to various reflective writing or discussion-based assignments in their classes, emphasising the role of writing in facilitating students’ learning of different concepts and development of other important skills in the class. The following quote from Trisha who taught English academic writing for graduate students is an example.

For the second one, I think this would be something that I would use to develop students’ persuasive skills. Because, because this just looks like the kind of short prompts that they would do as short homework assignments, um, or, if we are brainstorming ideas in class, or maybe a discussion board type of questions. Yeah, I can definitely see this as a discussion board assignment.

Regarding the differences between the skills required to complete IELTS Writing Task 2 and those needed for the graduate multimodal assignments, some academic staff participants suggested that the graduate multimodal assignments typically required the ability to describe and synthesise information from data rather than presenting personal opinions. Taylor, the grain science professor, commented on the difference below.

The second one is more argumentative, or if you think something is more positive or negative. This is not what we do in my area. You need to use the data to tell you if it works or not. There’s not much space for what your belief is regarding one thing or another.

In this quote, Taylor noted the differences between the underlying data-driven approach in research writing and the use of beliefs or opinions in persuasive writing. There also seemed to be some more emphasis on connecting writing with the students’ experiences in the graduate multimodal assignments. Tammy, an anthropology professor, shared the following observation.

And the second question, um, I said it was not similar just because it was not the type of questions that I might ask students, like, I tend to ask them to do a lot of reflective assignments, which I think are actually helpful for students who may be having issues with writing. Otherwise, if it’s just telling stories about themselves and their families, like, their writing comes out really well. So, in my classes, I have assignments like that. For example, in my health and cultural class, I asked them to write about how health and illness in the body were understood when they were growing up. You know, like, if they got sick, what would their family do, you know, would they go to a doctor, or would they have home remedies, or teas, or something like that. You know, so they were able to think. It’s a little easier, I think, to do that rather than commenting on something that’s more historical.

In this quote, Tammy explained that the main reason why she thought IELTS Writing Task 2 was different from her graduate multimodal assignments was that there was more emphasis on building upon students’ experiences in the graduate assignments whereas IELTS Writing Task 2 seemed to lack that component. Another academic staff member in humanities echoed this perspective, emphasising the importance of the experiential component in multimodal writing assignments that seemed to be missing from IELTS Writing Task 2.

Overall, while most student participants across the three research sites considered IELTS Writing Task 1 to be similar to graduate-level multimodal writing assignments in requiring students to analyse and summarise, some academic participants from the UK regarded this task as having “very little thinking skills” involved. It is possible that some disciplinary differences may have led to the different interpretations of the language skills/features needed to complete these tasks. Although there was evidence for the possible disciplinary differences in the data across the three research sites, the sample sizes were not large enough for further analysis. The interpretations of the comparison between IELTS Writing Task 2 and multimodal writing assignments seem to suggest that IELTS Writing Task 2 is comparable to multimodal writing assignments aiming to develop students’ basic writing skills but is less representative of other components such as discipline-specific vocabulary, evaluation, analysis, and synthesis of external sources that are typically required in graduate multimodal writing assignments.

5.3.4 Perceived possibility of IELTS Academic Writing Tasks to predict postgraduate multimodal writing performance

Across the three research sites, it seemed that the academic staff participants were more cautious than the student participants when commenting on the possibility of the IELTS Academic Writing Tasks to predict postgraduate multimodal writing performance.

Specifically, most student participants in the three research sites thought their scores on the IELTS Academic Writing Tasks could predict their performance in graduate-level multimodal writing assignments. This is particularly true for IELTS Writing Task 1. For example, a student participant from Türkiye shared the following observation.

I think it is kind of an indication that I can process, I have those thinking abilities, such as integrating different sorts of information, and summarising the main features of these different sources of evidence, and reporting main details. I think this also gives kind of evidence that I am cognitively ready for this type of assignments.

Similar observations were shared by some academic staff participants; however, many of them expressed reservations and/or discussed constraints regarding the predictability of the IELTS Academic Writing Tasks. For example, when commenting on the ability to read charts, an academic staff participant in Türkiye believed that IELTS Writing Task 1 could only partially predict students’ critical thinking ability, as illustrated in this quote.

In a limited way, yes, but not fully. Critical thinking, you know, they see something and they need to turn it into their own words. They need to turn it into something different. They need to infer the ideas when they see when they look at the chart. So being able to read the graph and then turn it into a text. That skill, I think.

This perspective that the IELTS Academic Writing Tasks could help predict certain aspects of students’ performance on multimodal writing assignments was shared by several academic staff participants from the UK (i.e., contrasting opinions, critical thinking, and processing visual information in life and physical sciences) and many from the USA (i.e., processing visual information, critical thinking, and developing arguments).

In considering the possibility of IELTS Academic Writing Tasks to predict postgraduate multimodal writing performance, the participants seemed to focus on the thinking processes and cognitive skills involved in input-based writing rather than a focus on the written product. Participants obviously believe that if test-takers can successfully complete a writing task that requires comprehension or interpretation and analysis of some visual input and its transformation to a written text, they will also be successful in completing graduate-level writing tasks. In this sense, as the input-based task of the IELTS writing exam that requires the mentioned skills, Writing Task 1 seems to be perceived as a better predictor of writing performance at the graduate level than Writing Task 2.

However, it is important to note that most academic staff participants in the UK stated that empirical research would be needed to better understand if and how the IELTS Academic Writing Tasks could predict students’ performance on graduate-level multimodal writing assignments. Several academic members pointed out that IELTS Academic Writing Tasks could not predict their students’ multimodal writing performance since they noted that critical thinking/writing and disciplinary knowledge were required to be successful in postgraduate classes in addition to multimodal literacies and that they perceived that IELTS Academic Writing Tasks would not require much criticality, disciplinary knowledge, or multimodal literacies.

6 Discussion and conclusions

This study set out to investigate the extent to which IELTS Academic Writing Tasks were comparable to graduate-level multimodal writing assignments through both performance and perception data. Due to the differences in the participant demographics of the three research sites and separate data analyses, no definitive conclusions and comparisons can be drawn in this project. However, we can suggest some general trends for the differences and similarities in graduate-level multimodal writing assignments and IELTS Academic Writing Tasks in all three countries.

The findings of the first research question showed some similarities between the IELTS Academic Writing Task 1 and multimodal writing assignments given in graduate-level courses in Türkiye, UK, and USA in terms of both genre characteristics (analysing, summarising, interpreting visual information, comparing and contrasting, discussing, explaining) and modality with the use of text as the main mode of meaning-making in both domains and with the use of visuals (i.e., graphs/charts) at the reception level again in both domains. The findings showed more differences between the IELTS Academic Writing Task 2 and graduate-level multimodal writing assignments in terms of genre, topic, and modality. In genre characteristics and modality, IELTS Writing Task 1 resembles the graduate-level assignments in its use of visuals at the reception level and the requirement to understand and interpret data when completing the writing task.

The nature of meaning-making practices was found to be different, with personal opinions, typically expressed in an essay required in the IELTS Academic Writing Task 2 versus arguments based on reading in university writing assignments, typically expressed in more research-oriented genre families. This is in contrast with Moore and Morton’s (1999) findings, who found essay to be the predominant genre of university writing tasks. The difference might be due to two reasons. First, it has been quite a long time since Moore and Morton’s (1999) study, thus, our findings may be a reflection of the changing nature of modality in today’s writing assignments. Second, Moore and Morton’s (1999) study included assignment tasks from both undergraduate and graduate levels, with the majority being undergraduate-level samples. The essay genre might be a more common genre in the undergraduate level than in the graduate level.

According to the findings of the first research question, graduate-level writing tasks were based on more specific topics from the disciplinary fields compared to IELTS Academic Writing Tasks. Topic specificity has long been a matter of discussion within the field of language assessment as it is not easy to use discipline-specific prompts in large-scale L2 wetting tests. IELTS (ELTS back then) was originally designed in a more discipline-specific way including three discipline-specific subtests (i.e., Academic Modules A, B and C) in the broad disciplines of Physical Sciences and Technology, Life and Medical Sciences, and Arts and Social Sciences (Taylor, 2012). While this version of IELTS might be considered more appropriate for the academic context and for the measurement of writing for academic purposes, it was not possible and sustainable to develop relevant and comparable tests across different disciplines. Apart from practicality reasons, design discipline-specific tasks were proved to affect the fairness of the test since every task topic would advantage students from a certain disciplinary (Clapham, 1996; Read, 2022). Therefore, IELTS replaced its discipline-specific modules with general academic modules in 1995. The graduate-level multimodal assignments were more research-focused writing tasks, engaging students in more research-oriented genre families such as critique, methodology recount, or research report and requiring them to write on discipline-specific topics by integrating external sources.

Regarding modality, across the three research sites, the main difference between the IELTS Academic Writing Tasks and graduate-level writing assignments was the use of multimodal resources as receptive or productive skills. Our findings support the changes in writing tasks in higher education classes and the role of the use of various semiotic resources in disciplinary academic writing practices (Belcher, 2017; Lim & Polio, 2020). While the participants of the three research sites were primarily expected to use multimodal composing skills productively in graduate-level multimodal writing assignments, only receptive multimodal skills were needed to complete the IELTS Academic Writing Task 1 successfully. However, this does not mean that receptive multimodal skills were not required for graduate-level multimodal writing assignments. Indeed, graduate students needed to utilise both productive and receptive multimodal skills in their multimodal writing assignments; however, our perception data suggested that productive multimodal skills were more important than receptive ones to accomplish graduate-level multimodal writing assignments that necessitated research, data compilation and analysis. The way multimodality was demanded in the graduate level classes in the three countries, as exemplified in Figures 2 to 4, shows similarities to the other graduate-level multimodal demands as reported in the literature (e.g., Li, 2020) as well as to the demands in high school contexts (e.g., Howell et al., 2017; Smith et al., 2022). This suggests a common or similar understanding of the demands of multimodal writing at various educational levels.

Our findings draw attention to differences in construct representation in the two domains. As a result of the rapid technological developments and the relevant changes in writing practices in the digital age, the construct of L2 writing ability has been re-defined in a way that includes the multiliterate, multimodal, social, and collaborative aspects of writing (Belcher, 2017; Jiang, 2018; Li & Li, 2022). In this regard, the writing tasks given at tertiary level education in the three contexts that we explored reflect the re-conceptualised construct of L2 writing ability. However, IELTS Academic Writing Task 2 still reflects the assumption that the construct of L2 writing for academic purposes is an independent skill, “wholly separate from reading or listening abilities” (Cumming, 2013, p. 4), but does not acknowledge the new construct definitions of academic literacy or multiliteracies (Cope & Kalantazis, 2000).

The findings of the third research question from the three research sites strongly support the observation of text as the primary mode of communication in both the IELTS Academic Writing Tasks and postgraduate multimodal writing assignments. The participants’ perceptions regarding the comparability of the IELTS Academic Writing Tasks and graduate-level multimodal assignments in terms of multimodality and language skills/features required differed between the research sites and possibly between different disciplines. IELTS Writing Task 1 was perceived to be similar to multimodal writing assignments in terms of multimodality primarily because of its requirement for test-takers to process, interpret, and summarise visual information in writing; it was perceived to be different mostly because graduate-level multimodal writing assignments would typically require students to produce visuals or to use multimodal resources productively. IELTS Writing Task 2 was perceived to be less similar to multimodal writing assignments in terms of multimodality because there was no requirement regarding the use of any multimodal resources either receptively or productively. In terms of the language skills/features, however, the participants generally focused more on the linguistic features expected and the underlying thinking process, and thus, the IELTS Academic Writing Tasks were perceived to be more similar to graduate-level multimodal writing assignments in this aspect. One noticeable difference between IELTS Writing Task 2 and multimodal writing assignments in this aspect, however, is its reliance on personal opinions as opposed to synthesis of external sources required in multimodal assignments.

Regarding the possibility of IELTS Academic Writing Tasks to predict students’ performance on multimodal writing assignments, the academic staff participants expressed more caution than the student participants, noting the need for more empirical evidence and the aspects of student performance that might not be accurately predicted. It is important to note that depending on disciplinary background and the types of multimodal resources that are often used, the participants might offer different perspectives on the comparability of these tasks.

With regard to any differences and similarities in linguistic features of the IELTS Academic Writing Tasks and graduate-level multimodal writing assignments, our findings in the three research sites were inconclusive. While there were quantitative similarities in terms of lexical sophistication and phrasal complexity between the IELTS Academic Writing Task 1 and multimodal assignments in the UK context, this similarity was not observed in Türkiye. The finding that students’ writing in their course assignments included less phrasal complexity than their writing in IELTS Academic Writing tasks in Türkiye is surprising given that we would expect disciplinary writing to be more complex as “students need to use increasingly complex and sophisticated language in order to convey precise and specialised meanings within disciplinary writing” (Staples et al., 2016, p. 154). However, this finding might be related to the features of multimodal tasks as expressing meaning using multiple modes might lead to less focus on the linguistic features of writing. Due to the multimodal nature of the graduate-level course assignments, students might pay more attention to meaning-making or content rather than the syntactic features of their writing. Thus, different modes of expression may reduce “demands of formulation and leave more attention for the processes of selecting, organising and preparing ideas” (Vandommele et al., 2017, p. 24). Compared to the multimodal assignments, students completed the IELTS writing tasks in test conditions where their writings were timed and where they expressed meaning only through text without using any other meaning-making resources. Thus, they might have paid more attention to the linguistic aspects of writing, which may have resulted in higher complexity. Similarly, we would expect the independent argumentative writing task (Writing Task 2) to elicit more phrasal complexity features in student writing due to increased reasoning demand, as hypothesised by Robinson’s (2001) Cognition Hypothesis. However, in Writing Task 1, students produced more phrasal complexity, which might be due to the prompt effect. There is evidence in the L2 writing literature that the linguistic features of writing prompts affect the linguistic features of the responses students produce (e.g., Mostafa & Crossley, 2020). In our study, the prompt of Writing Task 1 included more phrasal complexity features than the prompt of Writing Task 2. Thus, it is not surprising that this highly complex prompt resulted in more phrasal complexity in student responses than the prompt in the second writing task.

The disparate findings regarding the linguistic features of the graduate-level students’ responses to the IELTS Academic Writing Tasks and their multimodal writing assignments across Türkiye and the UK can be traced back to a number of factors, including differences in the genres of graduate-level multimodal writing assignments collected in the two countries, the English language proficiency levels of the participants, and local writing practices at universities. At this point, we can only speculate on the potential reasons for these differences. There were not enough data in the US context to do statistical analysis for this question.

Finally, our interview data indicated that both students and academic staff participants in the three research sites perceived some similarities between the IELTS Academic Writing Task 1 and graduate-level multimodal writing assignments in terms of the use of multimodality and writing skills (e.g. summarisation) required to complete both tasks. Having said that, some researchers may not consider the IELTS Academic Writing Task 1 as multimodal because it requires no multimodal composing.

However, in this study, we regard it as a multimodal writing task, which we define as a task that involves the use of semiotic resources at levels of reception and/or production to complete writing. At the same time, both groups perceived that the IELTS Academic Writing Task 2 was substantially different from the graduate-level multimodal writing assignments concerning multimodality, use of external resources and writing skills required to complete the tasks.

6.1 Limitations and future research

Several limitations of this project should be acknowledged. First, we were unable to model all the data or make cross-country comparisons in a systematic manner utilising statistical analysis due to the data-sharing agreement not being signed by the universities in Türkiye and the US.

Second, the three research sites had different demographic characteristics of participants. For instance, the case study in the UK only involved master’s students because most doctoral students take no courses and submit no assignments during their PhD studies in UK higher education. In Türkiye and the US, PhD students take courses and submit (multimodal) writing assignments. Such differences in higher education systems make it difficult to draw comparisons in this project.

Third, while IELTS is a language proficiency test for undergraduate and graduate admissions, our study participants were limited to graduate students. The real-life student assignments sampled in this report are unlikely to be representative of all international students. Undergraduate-level multimodal writing is an important area for further research since multimodal writing at this writing may be very different in the use of other semiotic resources. Moreover, the disciplinary backgrounds of the students and academic staff members were not controlled in this study. We tentatively argue for disciplinary differences in the degree of multimodality of writing assignments at the graduate level. The multimodal nature of writing assignments tended to be more pronounced in the disciplines of physical and life sciences than those of social sciences and art and humanities. However, further research is needed to investigate disciplinary differences in multimodal writing assignments at universities at different levels, including both undergraduate and postgraduate levels in a more systematic way.

Finally, multimodal assignments and the IELTS Academic Writing Task responses were collected from a small number of participants in the US context, which makes it hard to reach any meaningful conclusions about the differences and similarities of students’ performance in terms of linguistic features between the IELTS Academic Writing Tasks and multimodal writing assignments in the US context. We suggest controlling for participant demographic characteristics and using a large sample size in future studies. Future studies could also investigate the operationalisation of a multimodal writing task in different contexts.

6.2 Implications for language testing

The findings of this study suggest a number of implications for language testing. Given that most graduate students of this study submitted multimodal writing assignments produced based on reading from external resources or analysing data in all three countries, the current IELTS Academic Writing Task 2 may not represent the type of writing graduate students encounter at English-medium universities. Graduate-level writing assignments that we examined in this study often assessed students’ ability to construct an argument based on evidence from external resources rather than based on personal opinion. Addressing this misalignment, IELTS Academic Writing Task 2 could be reconsidered and designed to be an integrated writing task based on reading from external resources.

However, redesigning an independent writing task as a reading-into-writing task in a standardised like IELTS is challenging and will create several issues. First, the integrated task will lengthen the overall writing test time as test-takers will need more time for repeated reading when constructing their responses (Plakans, 2010). Second, a reading-into-writing task will impose validity constraints as the writing scores will depend on the ability of the test-takers to comprehend what they read (Plakans, 2009); thus, “compromising the validity of measurements of writing abilities for learners without adequate comprehension abilities” (Cumming, 2013, p. 4). Third, the source text will add complexity to the assessment process as human raters, if used in addition to automated scoring, will also need to understand the content in the source material. Raters will not only evaluate the quality of the written response but will also evaluate how accurately and effectively the source text is interpreted and integrated by the test-taker. The slower rating process and increased human costs may not be practical for a large-scale writing exam. There will also be more issues with reliability. It will be necessary to add new content-related criteria to the scoring rubric and to do more extensive rater training to ensure high inter-rater reliability, which still may not guarantee handling the diversity of responses that can be generated. As test-takers will engage with the source text in several different ways and will focus on the different parts of the text, it is very likely that the reading-into-writing task will elicit a great variety of responses, making the consistent use of a standardised scoring rubric difficult. It would be essential to strictly control the source material to be used by maybe limiting its length and its linguistic difficulty so that test-takers from low-level proficiency levels can also attempt at the task, which will create extra administrative burdens on test developers.

It is evident from our findings that graduate students write discipline-specific assignments (see also Nesi & Gardner, 2012) rather than general topics that the IELTS Academic Writing Tasks demand. Hence, test developers may identify common topics that students write at universities and refine the topics of the academic writing tasks accordingly, which is a controversial decision to make. While the practicality concerns and the issue that some test-takers may be discriminated for or against in discipline-specific writing tests (Clapham, 1996; Read, 2022; Taylor, 2012) persist today, artificial intelligence (AI) techniques available may help to address some of the challenges that have historically made the implementation of discipline-specific L2 writing tasks difficult in large-scale assessment. For example, by analysing huge amounts of academic texts from different disciplines, AI can identify a wide range of discipline-specific topics and common themes across disciplines so that cross-disciplinary writing tasks that are relevant to multiple disciplines and that address a broader range of test-takers can be developed.

Although we found in our study that text was the primary mode of graduate-level writing assignments, multimodal composing and orchestration of multimodal resources in assignments were common practices, in line with some earlier classroom-based studies (e.g., Fjørtoft, 2020; Lim & Polio, 2020). It seems that classroom assessment allows for the use of multimodal resources in L2 writing tasks and for multimodal writing assessment, given smaller groups of students, the focus on formative assessment and the learning process, the flexibility to design tasks that address student needs, interests, and abilities as well as the absence of the need for standardisation. However, applying multimodal writing assessment to high-stakes testing is not easy and will create several challenges. For example, assessment of a multimodal product requires evaluating not only the linguistic performance but also the design features, as, considering the theoretical foundation of multimodality, the roles of different modes are equally important in written communication (Zhang et al., 2023). The literature on assessing multimodal writing includes a few examples of analytical and holistic rubrics (e.g., Dzekoe, 2017; Hung et al., 2013; VanKooten, 2013) that provide insights into how different modes are integrated to express meaning.

However, the field of multimodal writing assessment is still developing, and there is a need for continued research on the reliability and validity of assessment tools that effectively assess the increased range of skills involved in multimodal writing.

In conclusion, going back to our claim and related assumptions regarding the domain definition that the key writing skills involved in graduate-level multimodal writing tasks are reflected in the IELTS Academic Writing Tasks and that IELTS Academic Writing Tasks are representative of graduate-level multimodal writing tasks, there are potential areas for improvement in the IELTS writing test, particularly in the Academic Writing Task 2. While changing the task type or design provides valuable opportunities in domain definition and construct representation, challenges inherent in changing established assessment practices cannot be ignored. There are several interrelated factors that will affect the process of task alterations (e.g., administration, scoring criteria, rater training, test materials, validity, reliability, fairness, accessibility, test-taker related factors, test preparation, stakeholders, etc.). Any changes in task type or design will require considerable amount of quantitative and qualitative validation research. A wide range of stakeholders will be affected by any change, from test developers and test-takers to educational institutions and employers. Thus, while there is potential for improvement, making task changes is a significant challenging task itself.

References

• Ahmadi, A., & Mansoordehghan, S. (2014). Task type and prompt effect on test performance: A focus on IELTS Academic Writing Tasks. The Journal of Teaching Language Skills, 6(3), 1-20. https://doi.org/10.22099/jtls. 2015.2897
• Belcher, D. D. (2017). On becoming facilitators of multimodal composing and digital design. Journal of Second Language Writing, 38, 80-85.
• https://doi.org/10.1016/j.jslw. 2017.10.004
• Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication, 25(2), 166-195.
• https://doi.org/10.1177/0741088307313177
• Biber, D., Gray, B., & Poonpon, K. (2011). Should we use characteristics of conversation to measure grammatical complexity in L2 writing development? TESOL Quarterly, 45(1), 5-35. https://doi.org/10.5054/tq. 2011.244483
• Biber, D., Gray, B., Staples, S., & Egbert, J. (2020). Investigating grammatical complexity in L2 English writing research: Linguistic description versus predictive measurement. Journal of English for Academic Purposes, 46, 100869.
• https://doi.org/10.1016/j.jeap. 2020.100869
• Braun, V., & Clarke, V. (2013). Successful qualitative research: A practical guide for beginners. Sage.
• Chapelle, C. A., Cotos, E., & Lee, J. (2015). Validity arguments for diagnostic assessment using automated writing evaluation. Language Testing, 32(3), 385-405.
• https://doi.org/10.1177/0265532214565386
• Chapelle, C. A., Enright, M. K., & Jamieson, J. M. (2008). Test score interpretation and use. In C. A. Chapelle, M. K. Enright, & J. M. Jamieson (Eds.), Building a validity argument for the Test of English as a Foreign Language (pp. 1-25). New York, NY: Routledge.
• Choudaha, R. (2017). Three waves of international student mobility (1999-2020). Studies in Higher Education, 42(5), 825-832. https://doi.org/10.1080/03075079.2017.1293872
• Cope, B., & Kalantazis, M. (Eds.). (2000). Multiliteracies: Literacy learning and the design of social futures. London, UK: Routledge.
• Cooper, T. (2013). Can IELTS writing scores predict university performance? Comparing the use of lexical bundles in IELTS writing tests and first-year academic writing. Stellenbosch Papers in Linguistics Plus, 42, 63-79.
• Creswell, J. W. (2014). Qualitative, quantitative and mixed-methods approaches ( 4^th ed.). Sage.
• Crossley, S. A., Skalicky, S., Kyle, K., & Monteiro, K. (2019). Absolute frequency effects in second language lexical acquisition. Studies in Second Language Acquisition, 41(4), 721-744. https://doi.org/10.1017/S0272263118000268
• Cumming, A. (2013). Assessing integrated writing tasks for academic purposes: Promises and perils. Language Assessment Quarterly, 10(1), 1-8. https://doi.org/10.1080/15434303.2011.622016
• Cumming, A., Kantor, R., Powers, D., Santos, T., & Taylor, C. (2000). TOEFL 2000 writing framework: A working paper. TOEFL Monograph Series 18. Princeton, NJ: Educational Testing Service.
• Darvin, R. (2015). Representing the margins: Multimodal performance as a tool for critical reflection and pedagogy. TESOL Quarterly, 49(3), 590-600. https://doi.org/10.1002/tesq. 235
• Davies, M. (2009). The 385+ million word Corpus of Contemporary American English (1990-2008+): Design, architecture, and linguistic insights. International Journal of Corpus Linguistics, 14(2), 159-190. https://doi.org/10.1075/ijcl. 14.2.02dav
• Durrant, P., Moxley, J., & McCallum, L. (2019). Vocabulary sophistication in first-year composition assignments. International Journal of Corpus Linguistics, 24(1), 33-66. https://doi.org/10.1075/ijcl. 17052.dur
• Dzekoe, R. (2017). Computer-based multimodal composing activities, self-revision, and L2 acquisition through writing. Language Learning & Technology, 21(2), 73-95. https://dx.doi.org/10125/44612
• Early, M., Kendrick, M., & Potts, D. (2015). Multimodality: Out from the margins of English language teaching. TESOL Quarterly, 49(3), 447-460. https://doi.org/10.1002/tesq. 246
• Elola, I., & Oskoz, A. (2016). Supporting second language writing using multimodal feedback. Foreign Language Annals, 49(1), 58-74. https://doi.org/10.1111/flan. 12183
• Fjørtoft, H. (2020). Multimodal digital classroom assessments. Computers & Education, 152, 103892. https://doi.org/10.1016/j.compedu. 2020.103892
• Gardner, S., & Nesi, H. (2013). A classification of genre families in university student writing. Applied Linguistics, 34(1), 25-52. https://doi.org/10.1093/applin/ams024
• Gourlay, L. (2016). Multimodality, argument and the persistence of written text. In R. Fidalgo & T. Olive (Series Eds.) & A. Archer, & E.O. Breuer (Vol. Eds.), Studies in Writing: Vol. 33, Multimodality in Higher Education, (pp. 79-90). Leiden: Brill.
• Hafiner, C. A., & Miller, L. (2011). Fostering learner autonomy in English for science: A collaborative digital video project in a technological learning environment. Language Learning and Technology, 15(3), 68-86. http://dx.doi.org/10125/44263
• Hafiner, C. A., & Ho, W. Y. J. (2020). Assessing digital multimodal composing in second language writing: Towards a process-based model. Journal of Second Language Writing, 47, 100710. https://doi.org/10.1016/j.jslw. 2020.100710
• Halliday, M.A.K. (1975). Learning How to Mean: Explorations in the Development of Language. London: Edwald Arnold.
• Halliday, M.A.K. (2003). On Language and Linguistics. Edited by Webster, J. London: continuum.
• Howell, E., Butler, T., & Reinking, D. (2017). Integrating multimodal arguments into high school writing instruction. Journal of Literacy Research, 49(2), 181-209. https://doi.org/10.1177/1086296X17700456
• Hung, H.-T., Chiu, Y.-C. J., & Yeh, H.-C. (2013). Multimodal assessment of and for learning: A theory-driven design rubric. British Journal of Educational Technology, 44(3), 400-409. https://doi.org/10.1111/j. 1467-8535.2012.01337.x
• Jewitt, C., & Kress, G. (Eds.). (2003). Multimodal literacy. New York, NY: Peter Lang
• Jiang, L. (2018). Digital multimodal composing and investment change in learners’ writing in English as a foreign language. Journal of Second Language Writing, 40, 60-72. doi:10.1016/j.jslw. 2018.03.002
• IELTS. (2018). IELTS score guide. Retrieved from https://ielts.kz/wp-content/uploads/2019/01/ielts_score-guide_a4_2018_web.pdf
• Institute of International Education. (2023). United States hosts over 1 million international students, fastest enrollment growth rate in more than 40 years. Open Doors® 2023 Report on International Educational Exchange. Retrieved from https://www.iie.org/news/us-hosts-over-1-million-fastest-growth-for-40-years/#:\~:text=For%20the%20first%20time%20since.year-over-year
• Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational Measurement (pp. 17-64). Westport, CT: American Council on Education and Praeger Publishers.
• Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1-73. https://doi.org/10.1111/jedm. 12000
• Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London, England: Edward Arnold.
• Kress, G., & van Leeuwen. T. (2006). Reading images: The grammar of visual design ( 2^nd ed.). New York, NY: Routledge.
• Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly, 49, 757-786. https://doi.org/10.1002/tesq. 194
• Kyle, K., Crossley, S., & Berger, C. (2018). The tool for the automatic analysis of lexical sophistication (TAALES): Version 2.0. Behavior Research Methods, 50, 1030-1046. https://doi.org/10.3758/s13428-017-0924-4.
• Li, M. (2020). Multimodal pedagogy in TESOL teacher education: Students’ perspectives. System, 94, 102337. https://doi.org/10.1016/j.system. 2020.102337.
• Li, Z. (2015). An argument-based validation study of the English Placement Test (EPT) - Focusing on the inferences of extrapolation and ramification. Doctoral dissertation. Ames, IA: Iowa State University.
• Li., J., & Li, M. (2022). Assessing L2 writing in the digital age: Opportunities and challenges. Journal of Second Language Writing, 57, 100913. https://doi.org/10.1016/j.jslw. 2022.100913
• Lim, J. (2019). An investigation of the text features of discrepantly-scored ESL essays: A mixed methods study. Assessing Writing, 39, 1-13. https://doi.org/10.1016/j.asw. 2018.10.003
• Lim, J., & Polio, C. (2020). Multimodal assignments in higher education: Implications multimodal writing tasks for L2 writers. Journal of Second Language Writing, 47. https://doi.org/10.1016/j.jslw. 2020.100713
• Llosa, L., & Malone, M. E. (2017). Student and instructor perceptions of writing tasks and performance on TOEFL iBT versus university writing courses. Assessing Writing, 34, 88-99. https://doi.org/10.1016/j.asw. 2017.09.004
• Llosa, L., & Malone, M. E. (2019). Comparability of students’ writing performance on TOEFL iBT and in required university writing courses. Language Testing, 36(2) 235-263.
• New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60-92.
• New London Group. (2000). A pedagogy of multiliteracies. In B. Cope & M. Kalatzis (Eds.), Multiliteracies. South Yarra, Australia: Macmillan.
• Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13-104). New York, NY: MacMillan.
• Moore, T., & Morton, J. (1999). Authenticity in the IELTS Academic Module Writing Test: A comparative study of Task 2 items and university assignments. In R. Tulloh (Ed.), IELTS Research Reports, Volume 2. IELTS Australia Pty Limited, Canberra
• Mostafa, T., & Crossley, S. A. (2020). Verb argument construction complexity indices and L2 writing quality: Effects of writing tasks and prompts. Journal of Second Language Writing, 49, 100730. https://doi.org/10.1016/j.jslw. 2020.100730
• Nesi, H., & Gardner, S. (2012). Genres across the disciplines: Student writing in higher education. Cambridge University Press.
• Nesi, H., & Gardner, S. (2018). The BAWE corpus and genre families classification of assessed student writing. Assessing Writing, 38(1), 51-55. https://doi.org/10.1016/j.asw. 2018.06.005
• Plakans, L. (2010). Independent vs. integrated writing tasks: A comparison of task representation. TESOL Quarterly, 44(1), 185-194. https://doi.org/10.5054/tq. 2010.215251
• Qu, W. (2017). For L2 writers, it is always the problem of the language. Journal of Second Language Writing, 38, 92-93. https://doi.org/10.1016/j.jslw. 2017.10.007
• Plakans, L. (2009). The role of reading strategies in integrated L2 writing tasks. Journal of English for Academic Purposes, 8(4), 252-266. https://doi.org/10.1016/j.jeap. 2009.05.001
• R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www. R-project.org/
• Riazi, A. M. (2016). Comparing writing performance in TOEFL-iBT and academic assignments: An exploration of textual features. Assessing Writing, 28, 15-27. https://doi.org/10.1016/j.asw. 2016.02.001
• Robinson, P. (2011). Second language task complexity, the Cognition Hypothesis, language learning, and performance. In Robinson, P. (ed.), Second language task complexity: Researching the Cognition Hypothesis of language learning and performance (pp. 3-38). John Benjamins.
• Smith, B. E., Amgott, N., & Malova, I. (2022). “It Made Me Think in a Different Way”: Bilingual Students’ Perspectives on Multimodal Composing in the English Language Arts Classroom. TESOL Quarterly, 56(2), 525-551. https://doi.org/10.1002/tesq. 3064
• Staples, S., Egbert, J., Biber, D., & Gray, B. (2016). Academic writing development at the university level: Phrasal and clausal complexity across level of study, discipline, and genre. Written Communication, 33(2), 149-183.
• https://doi.org/10.1177/0741088316631527
• Stein, P. (2008). Multimodal pedagogies in diverse classrooms: Representation, rights and resources. London & New York: Routledge.
• Tan, X., & Matsuda, P. K. (2020). Teacher beliefs and pedagogical practices of integrating multimodality into first-year composition. Computers and Composition, 58, 102614. https://doi.org/10.1016/j.compcom. 2020.102614
• Taylor, L. (2011). Introduction. IELTS Research Reports, Volume 11. IDP: IELTS Australia and British Council. Available at https://ielts.org/researchers/our-research/research-reports
• Van Leeuwen, T. (2005). Introducing Social Semiotics. London: Routledge.
• Vandommele, G., Van den Branden, K., Van Gorp, K., & De Maeyer, S. (2017). In-school and out-of-school multimodal writing as an L2 writing resource for beginner learners of Dutch. Journal of Second Language Writing, 36, 23-36. https://doi.org/10.1016/j.jslw. 2017.05.010
• VanKooten, C. (2013). Toward a rhetorically sensitive assessment model for new media composition. In H. A. McKee, & D. N. DeVoss (Eds.). Digital writing assessment and evaluation. http://ccdigitalpress.org/dwae/09_vankooten.html
• Weigle, S. C. (2010). Validation of automated scores of TOEFL iBT tasks against non-test indicators of writing ability. Language Testing, 27(3), 335-353. https://doi.org/10.1177/0265532210364406
• Weigle, S. C., & Friginal, E. (2015). Linguistic dimensions of impromptu test essays compared with successful student disciplinary writing: Effects of language background, topic, and L2 proficiency. Journal of English for Academic Purposes, 18, 25-39. https://doi.org/10.1016/j.jeap. 2015.03.006
• Yan, X., & Staples, S. (2020). Fitting MD analysis in an argument-based validity framework for writing assessment: Explanation and generalisation inferences for the ECPE. Language Testing, 37(2), 189-214. https://doi.org/10.1177/0265532219876226
• Yi, Y., & Choi, J. (2015). Teachers’ views of multimodal practices in K-12 classrooms: Voices from teachers in the United States. TESOL Quarterly, 49(4), 838-847.
• Zhang, M., Akoto, M., & Li, M. (2023). Digital multimodal composing in post-secondary L2 settings: A review of the empirical landscape. Computer Assisted Language Learning, 36(4), 694-721. https://doi.org/10.1080/09588221.2021.1942068

Appendix A: Student questionnaire

Q1 What is your native (first) language? ____ Q2 What is your gender?

○ Male ○ Female ○ Non-binary / third gender

Q3 How old are you? ____ Q4 In what program do you currently study? ____ Q5 How long have you been learning English? ____ Q6 What is the score of your most recent English proficiency test (e.g., TOEFL, IELTS)?

Which test did you take? ____

Q7 What degree are you studying to get?

○ MA ○ PhD ○ Other (Please specify) ____

Q8 How long have you been in your current graduate program?

○ First year ○ Second year ○ Third year ○ Fourth year ○ Fifth year

Other (Please specify) ____

Q9 How often are the following used as assessments in your courses?

Q10 How often do you have individual, pair, or group work in your graduate class assignments?

Q11 How often are you given a multimodal assignment as part of your graduate classes? (A multimodal assignment is where you use different meaning-making resources such as images, sound, tables, figures, movement, gesture, video, etc. to express or interpret meaning.)

○ Never ○ Rarely ○ Sometimes ○ Often ○ Always

Q12 How often do you use the following multimodal resources in your graduate class assignments?

Q13 When you receive the graded assignments, do you receive feedback on the multimodal aspects or are those aspects graded?

○ Yes, I receive both a grade and feedback on the multimodal aspects of my work. ○ I receive a grade on the multimodal aspects of my work, but I don’t receive feedback on these aspects. ○ No, I don’t receive a grade or feedback on the multimodal aspects of my work. ○ I don’t receive a grade on the multimodal aspects of my work, but I receive feedback on these aspects.

Q14 When your multimodal assignments are evaluated, is the evaluation based on the final product/output or the process or both?

○ I only receive evaluation on the final multimodal product/output. ○ I only receive evaluation on the process of completing the multimodal assignment. ○ I receive evaluation on both the final multimodal product/output and the process.

Q15 How would you rate your interest in working on multimodal assignments?

○ Not interested at all ○ Slightly interested ○ Neutral ○ Moderately interested ○ Highly interested

Q16 How would you rate your multimodal skills?

○ None ○ Low ○ Moderate ○ High

Q17 How often do you have opportunities to learn to complete multimodal assignments in your classes?

○ Never ○ Rarely ○ Sometimes ○ Often ○ Always

Q18 To what extent do you think the skills needed to do this writing task are similar to those needed for the multimodal assignments in your graduate classes?

○ Very similar ○ Similar ○ Slightly similar ○ Slightly different

○ Different ○ Very different

WRITING TASK 1

You should spend about 20 minutes on this task.

The charts below show the changes in ownership of electrical appliances and amount of time spent doing housework in households in one country between 1920 and 2019.

Summarise the information by selecting and reporting the main features, and make comparisons where relevant.

Write at least 150 words.

Percentage of households with electrical appliances (1920-2019)

Q19 To what extent do you think the skills needed to do this writing task are similar to those needed for the multimodal assignments in your graduate classes?

○ Very similar ○ Similar ○ Slightly similar ○ Slightly different

○ Different ○ Very different

WRITING TASK 2

You should spend about 40 minutes on this task.

Write about the following topic:

In some countries, more and more people are becoming interested in finding out about the history of the house or building they live in.

What are the reasons for this?

How can people research this?

Give reasons for your answer and include any relevant examples from your own knowledge or experience.

Write at least 250 words.

Appendix B: Academic staff questionnaire

Q1 In what program do you teach? ____ Q2 How long have you been teaching? ____ Q3 What is your gender?

○ Male ○ Female ○ Non-binary / third gender

Q5 How often do you ask your students to complete multimodal assignments as a part of your classes? (A multimodal assignment is where you use different meaning-making resources such as images, sound, tables, figures, movement, gesture, video, etc. to express or interpret meaning.)

○ Never ○ Rarely ○ Sometimes ○ Often ○ Always

Q6 How often do you typically require the following multimodal resources in the writing assignments as part of your graduate classes?

Q7 How often do you use the following as part of the assessment in your graduate-level classes?

Q8 How often do you have individual, pair, or group work in your graduate class assignments?

Q9 When you evaluate your students’ multimodal assignments, do you give feedback on the multimodal aspects or do you grade these aspects?

○ Yes, I both grade and give feedback on the multimodal aspects of students’ work. ○ I grade the multimodal aspects of students’ work, but I don’t give feedback on these aspects. ○ No, I don’t grade or give feedback on the multimodal aspects of students’ work. ○ I don’t grade the multimodal aspects of students’ work, but I give feedback on these aspects.

Q10 When you evaluate your students’ multimodal assignments, do you grade the final product/output or the process or both?

○ I only grade the final multimodal product / output. ○ I only grade students’ processed of completing the multimodal assignment. ○ I grade both the final multimodal product/output and the processes of students’ work.

Q11 How would you rate your students’ interest in completing multimodal assignments?

○ Not interested at all ○ Slightly interested ○ Neutral ○ Moderately interested ○ Highly interested

Q12 How would you rate your students’ multimodal skills?

○ None ○ Low ○ Moderate ○ High

Q13 How often do you have institutional training opportunities for designing multimodal assignments?

○ Never ○ Rarely ○ Sometimes ○ Often ○ Always

Q14 To what extent do you think the skills needed to do this writing task are similar to those needed for the multimodal assignments in your graduate classes?

○ Very similar ○ Similar ○ Slightly similar ○ Slightly different ○ Different ○ Very different

WRITING TASK 1

You should spend about 20 minutes on this task.

The charts below show the changes in ownership of electrical appliances and amount of time spent doing housework in households in one country between 1920 and 2019.

Summarise the information by selecting and reporting the main features, and make comparisons where relevant.

Write at least 150 words.

Percentage of households with electrical appliances (1920-2019)

Q15 To what extent do you think the skills needed to do this writing task are similar to those needed for the multimodal assignments in your graduate classes?

○ Very similar ○ Similar ○ Slightly similar ○ Slightly different

○ Different ○ Very different

WRITING TASK 2

You should spend about 40 minutes on this task.

Write about the following topic:

In some countries, more and more people are becoming interested in finding out about the history of the house or building they live in.

What are the reasons for this?

How can people research this?

Give reasons for your answer and include any relevant examples from your own knowledge or experience.

Write at least 250 words.

Appendix C: IELTS writing test

WRITING TASK 1

You should spend about 20 minutes on this task.

The charts below show the changes in ownership of electrical appliances and amount of time spent doing housework in households in one country between 1920 and 2019.

Summarise the information by selecting and reporting the main features, and make comparisons where relevant.

Write at least 150 words

*housework= washing clothes, preparing meals, cleaning

WRITING TASK 2

You should spend about 40 minutes on this task.

Write about the following topic:

In some countries, more and more people are becoming interested in finding out about the history of the house or building they live in.

What are the reasons for this?

How can people research this?

Give reasons for your answer and include any relevant examples from your own knowledge or experience.

Write at least 250 words.

Appendix D: Student interview questions

1. Do you know what multimodality is? Or what do you understand from multimodality? How would you define it?

Thank you for sharing your understanding of multimodality. There are many interpretations, and it’s great to have your view. To have consistency in relation to our study, we define multimodality as the integration of different meaning-making resources to express or interpret meaning, such as image, sound, text, movement, gesture, video, etc. Please answer the questions according to this definition.

2. Could you describe the role that multimodality plays in your academic life? Is using a variety of modes important in your academic life? Or is one mode (textual typically) enough for your graduate studies and academic success? Please do not think of your graduate courses only. Please consider your academic life outside your university as well.
3. How often are you given multimodal assignments in your graduate classes? Could you describe what kind of assignments you do in your graduate classes? Do you generally do textual assignments in which you typically write or do you integrate different modes in your assignments?
4. Now, let’s specify the role of multimodality in your graduate classes. Could you describe the role that multimodality plays in your class assignments? How do you think using a single mode or different modes affects your success in your assignments?
5. Do you think different modes contribute to your assignments? If yes, please elaborate in what ways.
6. How would you describe the differences and/or similarities between a multimodal writing assignment and a traditional writing assignment?
7. What are the challenges that you encounter or would encounter when doing multimodal assignments?
8. Please look at these sample academic writing tasks (over the page). Do you think these academic writing tasks and multimodal writing assignments require similar (language) skills? How do you think these academic writing tasks are similar to the assignments you are given in your graduate classes?
9. Can you describe a typical multimodal assignment in your graduate classes?
10. If any, what kind of instructions or guidelines are you provided for your multimodal assignments?
11. Are your multimodal assignments evaluated based on some explicit criteria? If yes, what are these criteria?

WRITING TASK 1

You should spend about 20 minutes on this task.

The charts below show what UK graduate and postgraduate students who did not go into full-time work did after leaving college in 2008.

Summarise the information by selecting and reporting the main features, and make comparisons where relevant.

Write at least 150 words.

Destination of UK graduates (excluding full-time work) 2008

Destination of UK postgraduates (excluding full-time work) 2008

WRITING TASK 2

You should spend about 40 minutes on this task.

Write about the following topic:

Countries are becoming more and more similar because people are able to buy the same products anywhere in the world.

Do you think this is a positive or negative development?

Give reasons for your answer and include any relevant examples from your own knowledge or experience.

Write at least 250 words.

12. Do you think the results from the academic writing tasks you just saw would provide a good indication of your ability to complete and do well in the multimodal assignments given in your graduate classes? In other words, if you did well in Writing Tasks 1 and 2, do you think you would also do well in the multimodal assignments you are asked to do in your classes? Can you give reasons for your answer?

Appendix E: Academic staff interview questions

1. What kind of graduate classes are you teaching?
2. What types of assignments do you typically give in your graduate classes?

Could you describe what you typically ask your students to do in their assignments, such as to produce a written text, to create a video, or to make a presentation, etc.?

You can give examples from specific graduate level classes.

3. Do you know what multimodality is? Or what do you understand from multimodality? How would you define it?

Thank you for sharing your understanding of multimodality. There are many interpretations, and it’s great to have your view. To have consistency in relation to our study, we define multimodality as the integration of different meaning-making resources to express or interpret meaning, such as image, sound, text, movement, gesture, video, etc. Please answer the questions according to this definition.

4. Based on our definition, do you give your students multimodal assignments in your classes? If yes, could you please give examples of the multimodal assignments you give in your classes? For example, what are the common modes that you ask your students to integrate in their assignments?

5. Do you think different modes contribute to student assignments? If yes, please elaborate in what ways.

6. Could you describe the role that multimodality plays in your academic life? Is using a variety of modes important in your academic life? Why? Or is one mode (textual typically) enough for your graduate studies and academic success? Please do not think of the graduate classes you teach only. Please consider your academic life outside your institution as well.

7. Could you describe the role that multimodality plays in your class assignments? How do you think using a single mode or different modes affects your students’ success in your assignments?

8. How would you describe the differences and/or similarities between a multimodal assignment and a traditional assignment, essay writing, for example?

9. Can you describe a typical multimodal assignment that you give in your classes?

10. What kind of instructions or guidelines do you provide in your multimodal assignments?

11. Do you have explicit criteria for evaluating multimodal assignments? If yes, what are these criteria? If no, can you tell us why you do not prefer using explicit criteria and how you do your evaluation?

12. What are the challenges that you encounter when designing multimodal writing assignments?

13. What are the challenges that you encounter when evaluating multimodal writing assignments?

14. What are the challenges that you observe your students encounter when doing multimodal writing assignments?

15. Please look at these sample academic writing tasks. How do you think these tasks are similar to the assignments you give to your graduate students? Do you think the language skills and abilities needed to do well in Writing Tasks 1 and 2 are similar to the language skills and abilities needed to do well in the assignments you give your graduate students?

WRITING TASK 1

You should spend about 20 minutes on this task.

The charts below show what UK graduate and postgraduate students who did not go into full-time work did after leaving college in 2008.

Summarise the information by selecting and reporting the main features, and make comparisons where relevant.

Write at least 150 words.

Destination of UK graduates (excluding full-time work) 2008

Destination of UK postgraduates (excluding full-time work) 2008

WRITING TASK 2

You should spend about 40 minutes on this task.

Write about the following topic:

Countries are becoming more and more similar because people are able to buy the same products anywhere in the world.

Do you think this is a positive or negative development?

Give reasons for your answer and include any relevant examples from your own knowledge or experience.

Write at least 250 words.

16. Do you think students’ performance on these sample academic writing tasks can represent or predict their performance on the multimodal assignments that you assign in your classes? Please explain why or why not?

Author biodata

Dr Aysel Saricaoglu (PhD) works as an associate professor in the Department of English Language and Literature at Social Sciences University of Ankara, Türkiye. She investigates academic writing, syntactic complexity, automated assessment, technology-enhanced language learning, and teacher research literacy. She has disseminated her research in journals such as Assessing Writing, Studies in Second Language Acquisition, Journal of English for Academic Purposes, and ReCALL.

Dr Duygu Candarli (PhD) is currently Lecturer in TESOL at the University of Southampton, UK. She specialises in second language writing, writing assessment, corpus linguistics, and academic discourse. She has published in international journals such as Reading and Writing, Corpora, Critical Discourse Studies and Journal of English for Academic Purposes. She is the co-author of The Linguistic Challenge of the Transition to Secondary School: A Corpus Study of Academic Language (Routledge, 2023).

Dr Jinrong Li (PhD) is a professor in the Department of English at Georgia Southern University. Her research focuses on L2 writing assessment, language learning and technology, and information literacy. Her recent work has appeared in Journal of Second Language Writing, Assessing Writing, Language Learning & Technology, and Computers and Composition.

Acknowledgements

We would like to thank our research assistants who contributed to this project: Betül Dursun (Research Assistant, Social Sciences University of Ankara, Türkiye), Ouacila Ait Eldjoudi (Researcher, University of Southampton, UK), and Fazley Rabby (MA student, Georgia Southern University, USA). We also would like to express our gratitude to all the students and academic staff members who devoted their valuable time to participate in this project. We are also grateful to Ms Jenny Osborne (IDP: IELTS Australia) for her support and guidance throughout this project.