The road to understanding in lecture listening: Cognitive processes engaged in the integration of auditory and textual information

Abstract

Recent technological advances offer test developers the opportunity to develop new assessment tasks which closely replicate behaviour in real-world domains such as academic lectures, thus achieving greater ecological and cognitive validity. However, developing new task types that demand the integration of auditory and textual information relies on understanding the relationship between what is said and what is written on the slides, and how students integrate these two streams of information.

With a dual focus, this study first examined the relationship between the lecturer’s speech and the textual information on the lecture slides before investigating the processes students use to integrate these two streams of information to develop an understanding of the lecture content.

In Phase 1, five lecture recordings were collected from universities across the UK. Following Hallewell and Crook (2019), the lecturer’s speech was segmented and mapped to slide-text units to form discourse units. The discourse units were coded according to the role that the lecturer’s speech performed in relation to the slide text.

In Phase 2, the most obvious speech/slide variable-the extent to which the information delivered verbally also appears in the slide text-was manipulated to assess the effect on participants’ processes. Two lecture excerpts were selected from Phase 1 and two versions of the slides were created for each lecture clip: 1) topic headings plus a short text outlining the main teaching points, and 2) topic headings only. Using a counterbalanced presentation, eye-tracking and stimulated recall were used to investigate participants’ perceptual processes as they read the slides and listened to the lecture excerpts.

Phase 1 resulted in a taxonomy of discourse relations and Phase 2 reported the factors which impacted participants’ ability to develop an understanding of the lecture content. The implications for teaching and assessing academic listening are discussed.

1 Rationale

The field of language testing has witnessed a transition towards performance testing and using integrated test tasks that simulate real-life performance in the target language use (TLU) domains, for example, using reading-into-writing tasks to assess academic literacy (Cumming, 2013). This transition has been partly facilitated by transitions towards computer-delivered language tests which allow greater flexibility in task design. A case can be made for utilising the affordances of computer-based test delivery to develop new test tasks which more closely replicate listener behaviour in real-world domains such as academic lectures, thus achieving greater ecological and cognitive validity. This study aims to contribute to an important area not covered by current tests of academic listening and pave the way to a new task type in the IELTS Listening test.

2 Literature review

Currently, the listening component in most major international language tests used for university admissions (e.g. IELTS, TOEFL iBT, PTE Academic, CAE) involves the presentation of audio material as the primary/sole source of information for comprehension. Written text does appear in test items, but often as information to compare or reason against the information heard (e.g. in MCQs), rather than a source of information to integrate with the audio input so as to arrive at a coherent representation of content. While some tests require more complex synthesis in other parts of the test, e.g. TOEFL iBT uses text and audio inputs which need to be synthesised for the speaking component, the testing of listening seems to lag behind in this respect. Concerns have been raised about the cognitive validity of such conventions and the artificial nature of the test methods (see Field, 2019, pp.73-86 for a review). Recent eye-tracking research suggests that the multiple-choice format demands listening processes which are not representative of real-world listening processes - and may indeed be more cognitively demanding. For example, see Chan and Latimer (2020) which used eye-tracking to investigate test-taker processes in a multiple-choice listening test.

Information processing in academic lecture comprehension would appear to involve considerably different processes from those demanded by conventional listening tests. Academic lectures nowadays are mostly delivered with presentation (often PowerPoint) slides, where written text (e.g. bullet points, quotations) continues to account for most of the content (Hallewell & Crook, 2019; Roberts, 2018). Understanding of academic lecture content is likely to involve (near-)synchronous integration of auditory input from the lecturer and textual (and other visual) information on presentation slides, together with visual cues of the lecturer (where available), and any relevant background knowledge the student has on the subject. Mapping the spoken information to the written information is certainly part of the lecture-listening experience - but the way textual information is presented (and indeed the whole process) is arguably entirely different from a battery of questions both in terms of the type of written information presented and in terms of the attentional and cognitive demands elicited by the task.

There can be little doubt that digital technology has revolutionised academic practices and nowadays both teachers and students are expected to be competent in multimodal communication across a range of platforms (O’Halloran, 2016). The online presentation of slide materials alongside the lecturer’s speech is commonplace in academic settings and is likely to form part of this multimodal competence, and research suggests that second-language students in particular benefit from accessing recordings of academic lectures (Shaw and Molnar, 2011).

Campoy-Cubillo and Querol-Julián (2015) argue that the assessments of listening should consider including a range wide of semiotic resources (gesture, facial expression, body posture, gaze etc.) as well as other visual elements such as textual information and diagrams. Several researchers have studied the way that listening interacts with visual information in tests of listening. For example, Ockey (2007) studied how test-takers used semiotic information when shown videos of the speaker compared to still images. Wagner (2010) compared the scores of test-takers on lecture listening tasks with and without access to videos of the lecturer. Some videos also included images relevant to the lecture topic. Test-takers’ scores were significantly improved in the ‘with video’ condition. More recently, Suvorov (2015) used eye-tracking to study the viewing behaviour of test-takers on a Video-based Academic Listening Test and compared the effect of videos containing contextual information with videos containing content information, and concluded that visual information played an important role in the construct of academic listening.

There are arguments within the multimedia learning (MML) literature that the integration of lecturers’ speech and slide-text information could be cognitively demanding, depending on factors such as whether there is a slide-text overload and whether the lecturer tends to ‘read off the slides’ (Sweller, Ayres & Kalyuga, 2011; Yue, Bjork & Bjork, 2013). These in turn are subject to variations in terms of disciplinary norms, as well as lecturers’ styles, content familiarity and pedagogic beliefs (Hallewell & Crook, 2019). For those on the receiving end of lectures, language proficiency level may play a role in how difficult the integration of auditory and written information is. For example, where reading and listening processes are less automatised (Gathercole & Baddeley, 1993) among lower-proficiency learners, they may experience more difficulties managing the two simultaneously and integrating two channels of information. The listener’s proficiency is also likely to interact with their familiarity with operating in this discourse mode/genre. Nonetheless, the argument remains that a test of L2 academic listening ability should perhaps incorporate elements in the test - in this case, slide text - that mirror real-life academic listening as far as possible, with cognitive ¹ and ecological ² validity in mind. And, with technology playing an ever more important role in higher education, ecological validity means understanding the integration of lecturers’ speech and slide-text information in an online setting.

2.1 Discourse relations between the lecturer’s speech and slide text

A first step towards operationalising assessment of the ability to integrate auditory and textual information in online academic lectures is to determine the nature of relationships between the lecturer’s speech and the textual information on the lecture slides. This area is surprisingly under-researched (Hallewell & Crook, 2019; Zhao & van Leeuwen, 2014). To be sure, there are studies of the contribution to understanding made by PowerPoint slides. However, these have largely focused on slide content or the tendency of lecturers/presenters to read slide content aloud (e.g. Adams, 2006; Knight, 2015), and have tended to examine individual modalities (slide-visuals, slide text, lecturer’s speech) in isolation (Zhao & van Leeuwen, 2014). Importantly, the systematic study of the integration of the speech and visual elements of a lecture is generally lacking.

One important exception is a recently published study by Hallewell & Crook (2019) which focuses on the integration of the lecturer’s speech and PowerPoint slide text. Their discourse analysis identified two main styles of integration: (1) a ‘referent style’ whereby a lecturer refers to and assesses the content on the slide; and (2) a ‘scaffolding style’ where the slide text is interwoven into the lecturer’s speech, rather than being explicitly referred to and assessed.

Notably, however, the scope of the study was confined to parts of the lecturer’s speech which identifies the slide text - it excluded speech that ‘develops’, ‘explains’ or ‘expands on’ the slide text (ibid., p.5), and which would otherwise be relevant to students’ comprehension of the lecture content. Relatedly, there seems to be very limited (if any) research that focuses on integration at the receiving end of a lecture - students’ cognitive processes in integrating the lecturer’s speech and slide text to arrive at an understanding of the lecture content. Both of these aspects are crucial to a construct definition for a task assessing lecture comprehension and are explored in the current study.

Developing new task types that demand the integration of auditory and textual information relies on understanding both the relationship between what is said and what is written on the slides, and how these two streams of information are integrated by students. While some research has focused on analysing lecture speech/slide discourse relations (Hallewell & Crook, 2019), the research limited its scope to parts of the lecturer’s speech which identified the slide text - it excluded speech that developed, explained or expanded on the slide text. In addition, there is a lack of research investigating how students simultaneously process and integrate the two streams of information to arrive at an understanding of the lecture. Therefore, this study posed two research questions, as set out below.

3 Research questions

1. What discourse relations exist between the lecturer’s speech and the slide text in PowerPoint slide lectures?
2. How do students integrate auditory and textual information to develop a mental representation of the lecture content in lectures?

4 Research design

This research was undertaken in two stages; a pilot study followed by a main study. Both the pilot study and the main study included two investigative strands, one relating to RQ1 and the other relating to RQ2. A mixed-methods approach utilising qualitative discourse analytic (DA) analysis, eye-tracking and stimulated recall was used to answer the research questions (the methodology and results for RQ1 are reported in Sections 5 and 6 respectively, and the methodology and results for RQ2 are reported in Sections 7 and 8).

A mixture of online lectures and video recordings of live lectures were collected to analyse the discourse relations between the lecturer’s speech and slide text (RQ1). Some of the slides included visuals and some of the recordings included video of the lecturer speaking. The full range of lectures (Lectures A-E in Table 1) were used for both the pilot and main study to answer RQ1.

This first phase of the pilot study focused on understanding the discourse relations between lecturers’ speech and the text appearing on the slides (RQ1). Using Hallewell and Crook (2019) as a starting point, a taxonomy of such discourse relations was developed by analysing parts of five real-life academic lecture recordings.

Phase one produced an initial taxonomy of discourse relations which included verbalising, conducting attention, translating, linking to previous text, assessing slide text, elaborating, merging, and combining.

In addition, it was noted that some lecturers highlighted particularly important information, while some communicated additional teaching points verbally that were absent from the slides and could not be considered an elaboration of a point on the slide. The pilot study also detected evidence of some notable patterns in the lecturers’ style of lecture delivery.

The pilot study for RQ1 generated an initial taxonomy of discourse relations from a sample of the lectures. The main study for RQ1 then continued the discourse analysis of the lectures, reviewing and, if appropriate, refining the taxonomy and analytic descriptions of the discourse relations and lecturers’ exposition patterns in response to RQ1. No new categories of discourse relations emerged from the main study.

To investigate how students integrate auditory and textual information (RQ2), participants were eye-tracked as they listened to clips taken from the lectures and read the related slides. Stimulated recall was also used to aid in the interpretation of the eye-tracking data.

The pilot study produced initial findings in relation to integrating information from reading and listening, and the way the design of the slides impacted participants’ ability to understand the lecture. Four key themes emerged from the pilot study.

1. At times, reading and listening seemed to compete for the participants’ attention and this is perhaps why many participants reported finding large amounts of text on slides sometimes unhelpful.
2. In most situations, participants reported trying to ‘read along’ with the lecturer. In other words, participants tried to find the text on the slide relevant to what was being said at the time. This pattern of ‘following’ the lecturers’ progress through the slide seemed consistent whether the slides contained minimal information or larger chunks of text.
3. There seems to be a ‘sweet spot’ regarding how much text on slides is helpful. Paragraph-sized chunks of text were almost categorically seen as unhelpful, even when the lecturer went on to read out the text. Very brief points which included just one or two words per teaching point risked offering insufficient support if they included unfamiliar vocabulary.
4. Participants reported having keywords/terms highlighted as very useful. This enabled them to identify keywords and key points more effectively; participants suggested that it helped them identify the main points more effectively.

From the pilot study for RQ2, researchers concluded that there were too many variables contained within lectures A-E. Some included the lecturers’ images as they spoke, others did not and some included slides with images. In order to manage the number of variables for the main study for RQ2, it was decided to exclude images of the lecturer and use recordings from native-speaking lecturers.

One of the findings from the pilot study of RQ1 was that the lectures analysed included a wide range of lecture delivery styles and many different types of discourse relationships between the lecturer’s speech and the slide text. It was not possible to investigate all these speech/slide text relations in the main study. Therefore, the main study compared the most obvious speech/slide variable, the extent to which information delivered verbally appears in the slide text. This variable related to the most pertinent findings from the pilot - that at times, reading and listening seemed to compete for the participants’ attention and that participants reported attempting to ‘read along’ with the lecture.

It would also provide insight into the question of whether there is an optimal amount of text for slides.

Therefore, the second strand of the main study (addressing RQ2) compared two speech/slide conditions - (1) where the slide contains topic headings plus a summary of the teaching point which is elaborated verbally, and (2) where the slide contains only topic headings and the main teaching point is communicated verbally.

The methodology, data analysis and findings for each of the two strands of the main study will be reported separately before the implications are drawn together in the Discussion section. Therefore, Section 5 reports the methodology for RQ1, followed by the results for RQ1 in Section 6. Section 7 reports the methodology for RQ2, followed by the results for RQ2 in Section 8. Section 9 is the Discussion where the results of the two strands of study are drawn together. For a detailed account of the pilot study and its findings, please see the interim report.

5 Methodology for investigation of discourse relations (RQ1)

5.1 Introduction

Aiming at discovering and arriving at a taxonomy of discourse relations between the lecturer’s speech and the slide text, a qualitative discourse analytic (DA) approach is used (cf. Hallewell & Crook, 2019). This involves mapping segments of the lecturer’s speech with corresponding parts of the slide text and comparing the semantic contents of the two to determine their discourse relations (ibid.). Relations already identified by Hallewell and Crook (2019) included, for example, referring (verbalising the slide text), translating (paraphrasing), and assessing (agreeing with or questioning the slide text). Note that, however, Hallewell and Crook excluded from their study the more indirect functions where the lecturer explained or elaborated on the content of the slide text. The present study includes these aspects of the lecturer’s speech. As indicated in the pilot study, these aspects contribute to students’ comprehension of the lecture and may contain key teaching points for students to heed. Moreover, the analysis takes a combination of deductive and inductive coding processes, with the initial coding scheme drawing on categories from Hallewell and Crook (2019), but also coding categories emerging from the data.

5.2 Sampling and data collection

The following criteria were applied in sampling academic lectures used in this study:

• five (5) real-life academic lectures
• five (5) different lecturers with at least five years of teaching experience
• all the lectures relate to the same relatively non-technical subject area (e.g. second language studies) at an introductory level.

Recordings of five real-life academic lectures from five different universities in the UK were collected for this study. Each lecturer participant signed an informed consent form for the use of their lecture recording in both Phase 1 and Phase 2 of this study. Each lecture recording contains: a) the lecturer’s speech; and b) the PowerPoint slides, with the lecturer’s speech synchronised with the progression of the slides. Details of the five slide-audio lecture recordings collected are provided in the following table.

Table 1: Details of the lectures collected for the study

The online lectures (A, B, C) were pre-recorded and made available to students online, whereas the live lectures (D and E) were delivered to students in a lecture theatre.

The length of the lectures shown here represents the duration of the lecture slot per the design of the module, which includes student activities or tasks excluded from analysis in this study.

5.3 Data processing and analysis

The purpose of the analysis is to gain insights into the discourse relationship between the content of the PowerPoint slide text used by the lecturers and their exposition of the corresponding slides in speech; and by extension, the relative ease or difficulty for students when mapping the two channels of information. The term ‘mapping’ is used here to explain the process of following the lecture, e.g. matching what the lecturer is talking about to a particular bullet point on the slide.

Stage 1: The lecturer’s speech in each lecture collected was orthographically transcribed. For each lecture, the PowerPoint slides have been segmented into slide-text units following Hallewell and Crook (2019). Each heading, bullet point, or block of text (e.g. a paragraph, a quotation) was treated as one slide-text unit. Some of the slides included images for illustrative purposes, others did not. However, it was felt that including images would introduce a significant variable and reduce the study’s ability to draw conclusions regarding the speech/slide text relations and therefore, visuals were excluded from the analysis. Similarly, slides labelled with tasks/activities for students, and the corresponding lecturer-student dialogues in the live lectures, were also excluded.

Stage 2: The next stage involved establishing lecture discourse units as units of analysis, by segmenting the lecturer’s speech and mapping it with the slide-text units. This was done through repeated reviewing of the slide-audio recording, to determine which chunk of the lecturer’s speech corresponds to which slide-text unit (e.g. bullet point), using pauses and slide animations as cues. It should be noted that this mapping procedure treats the slide-text unit as the ‘base unit’, on which a unit of lecturer’s speech is mapped. However, there are, of course, lecturers’ speech units which do not correspond to particular slide-text units, and there are parts of the lecturer’s speech which are not necessarily guided by the progression of the slide-text. Nevertheless, it allowed for the subsequent coding of discourse relations to take a consistent perspective (same ‘starting point’): What is the lecturer’s speech doing in relation to the slide text? Mapped together, each slide-text unit and lecturer’s speech unit formed a lecture discourse unit, represented by each row on an MS Excel file.

Stage 3: Each lecture discourse unit was then coded manually, using MS Excel, for the discourse relation(s) between the lecturer’s speech and the slide text. The initial coding scheme was derived from categories of relations identified by Hallewell and Crook (2019). Emergent categories arising from the data were then noted, defined, and added to the coding scheme. The coding scheme with a description of each category is presented in Appendix 1. Note that each lecture discourse unit often received two codes (sometimes up to four). This reflects that the lecturer’s spoken exposition of one slide-text unit may include several actions (e.g. conducting attention, verbalising, elaborating). Part of the analysis was also to gain insights into more nuanced patterns within each discourse relation (e.g. types of verbalising, types of elaborating) as well as sequences of discursive actions (e.g. verbalising —> translating —> elaborating —> highlighting teaching point), which will be discussed in the Findings Section below.

Overall, the inductive coding process was aimed at analytic generalisation (arriving at a taxonomy of discourse relations) and data saturation (no new categories arising).

Table 2 summarises the number of lecture discourse units coded.

Table 2: Summary of lecture discourse units coded

Of the data (63 units), 10% were double-coded. Disagreements in coding categories were resolved through discussion before the rest of the data set was coded. The inter-coder agreement rate was 93.65%.

6 Results for investigation of discourse relations (RQ1)

6.1 Lecturer speech’s relationship to slide text: A taxonomy

The coding described in Section 5 arrived at a taxonomy of 10 discourse relations that identified the function or purpose of the lecturer’s speech in relation to the slide text. Table 3 lists the different types of discourse relations, and subsequent sections describe each of them in turn, including any relevant sub-types emerging in the data, and illustrate them with examples.

Table 3: Taxonomy of speech to slide-text relations

6.1.1 Verbalising

The lecturer may render the slide-text unit (e.g. a bullet point) in their speech. The speech may render the slide text verbatim (Example 1); or close to verbatim, changing the wording only slightly (illustrated in bold in Example 2 and Example 3).

Example 1

Slide text: - What language are you a native speaker of?

Speech: Well, what language are you a native speaker of.

(Lecture A)

Example 2

Slide text: - Exploring how this concept has been challenged

Speech: We’ll then move on to think a little about how this concept has been challenged.

(Lecture A)

Example 3

Slide text: Do you need to know a language from very early childhood to make a claim to native speaker ‘ownership’ of a language?

Speech: So do we need to know a language from a very very early childhood stage to be able to make a claim to having native speaker ownership of that language?

(Lecture A)

A third sub-type of verbalising: paraphrasing or mangling, involves expressing the idea(s) on the slide text but with different words. As Hallewell and Crook (2019) describe it:

Although the concepts are the same in speech and text, the speech can be so different in structure and terminology that they are two separate entities that provide the same semantic message. (p.5)

Examples 4 and 5 illustrate how specific terms in the slide-text units are paraphrased in the spoken rendering. Examples 6 and 7 show cases of jumbling up the order, the selection of words, or the formulation of the proposition in the slide text.

Example 4

Slide text: Collocations

Speech: And we know also the words that co-occur with it.

(Lecture D)

Example 5

Slide text: - 5. native speaker through long residence in the adopted country.

Speech: And finally, Davies says it could be the case that you could make a claim to being a native speaker through having lived in a country for a very long time.

(Lecture A)

Example 6

Slide text: - ‘Picking it up’ from exposure to examples - an implicit approach

Speech: Alternatively, maybe you give them lots of example texts, and they pick it up just from simple exposure.

(Lecture C)

Example 7

Slide text: The answer might give insight into how to teach them.

Speech: So this translates into the language classroom for the language teacher — what is their role in encouraging the students, educating students, to come out with these patterns?

(Lecture C)

6.1.2 Conducting attention

The lecturer may at times verbally direct students’ attention to particular slide-text elements ³, as in the examples below.

Example 8

Slide text: Consider two of my friends…

Speech: And as an example of this, I want you to think about two of my friends.

(Lecture A)

Example 9

Slide text: (note, original slide text contained Japanese script)

ラドクリフ、マラソン五輪代表に1万m出場にも含み

Radcliffe to compete in Olympic marathon and 10,000 metres

Speech: So when you put it all together, as in a typical piece of writing. So it’s taken from a newspaper headline, you’ll find at the bottom of that slide all four scripts working together.

(Lecture C)

6.1.3 Translating

Drawing on Hallewell and Crook’s (2019) category, this refers to occasions where the lecturer deciphers the slide text for the students, explaining it ‘in other words’. In the following examples, the parts of lecturer speech that translate the slide text are given in bold, following the verbalisation of a slide-text unit (boundaries marked with //).

Example 10

Slide text: Receptive/passive ----> word recognition (and understanding?) vs

Productive/active > word production

Speech: So this is one distinction, between receptive or passive knowledge and productive or active knowledge. // So this means basically the ability to recognise a word and the ability to use a word.

(Lecture D)

Example 11

Slide text: - Considering what this means for English teaching and learning

Speech: And finally we’ll consider what this all means for English teaching and learning. // So we’ll finish by thinking a little bit about some of the practical implications, some of the implications for classroom contexts.

(Lecture A)

Note, therefore, how this is different from verbalising: paraphrasing in that the action of translating is often linguistically marked through ‘so’, ‘I mean’, or ‘in other words’ etc., and it also typically follows the verbalising of the slide-text unit.

6.1.4. Linking slide text to previous text

Lecturers often verbally link a current slide-text unit to some previous text - either earlier in the same lecture, or parts of a previous lecture.

Example 12

Slide text: Variable types

Speech: So now we know what the variables are, // but when it comes to variables, we can make a distinction in terms of what their function is when it comes to research.

(Lecture E)

Example 13

Slide text: Variation in the way English is used [UK map as image background]

(Lecture A)

In Example 12, the lecturer refers to the immediately previous slide where she has introduced a definition of ‘variables’ and uses this as a springboard for introducing the contents of the current slide - different types of variables in relation to their function in research. In Example 13, the lecturer references the lecture in the previous week, the gist of which is represented in the slide text and the background image; and she uses this to signpost the topic of the current lecture, which concerns the variations in English across the globe.

6.1.5 Assessing the slide text

Lecturers often assess (i.e. evaluate or comment on) particular units of text on a slide. These evaluations can be positive, negative or neutral, as illustrated in the following three examples respectively.

Example 14

Slide text: Age of arrival

Speech: Age of arrival is an important one.

(Lecture E)

Example 15

Slide text: - Native speakers and non-native speakers

Speech: So if someone is not a native speaker, we would call them a NON-native speaker. // And some authors in this particular field, some of the thinkers have said, well this really isn’t a fair way to define individuals who fall in that category.

(Lecture A)

Example 16

Slide text: - 1. native speaker by birth (that is by early childhood exposure)

Speech: Maybe that’s the most common way of defining what we mean by native speaker, but there are perhaps other ways too. This is what Davies is encouraging us to think about.

(Lecture A)

As seen in the examples, the action of assessing again often comes after the verbalising of the relevant slide-text unit. Lecturers sometimes evaluate the idea in a slide-text unit positively (Example 14) or agree with it. Alternatively, they may challenge or question the idea in the slide text (Example 15). They may also make comments with a more neutral, or less overtly positive or negative, tenor (Example 16). In evaluating or commenting on the ideas in the slide text, the lecturer’s speech then adds a layer of stance or value on top of presenting propositions or information (in the slide text). Pedagogically, it can be argued that lecturers can and do use this type of discourse relation between slide text and speech to encourage the students to engage more critically with the lecture content.

6.1.6 Elaborating

Elaborating is arguably the richest type of speech-to-slide-text discourse relation with the most sub-types. Elaborating enriches the slide text by adding detail and/or depth to it. An obvious way in which the lecturer’s speech elaborates on the slide text is by providing additional ideas or details related to the slide text.

Example 17

Slide text: - 2. native speaker by virtue of being a native user

Speech: So he says that we could be a native speaker of a language by virtue of being a native user. // So someone who is so competent, so skilled in the language, that maybe they’re not a native speaker by birth, but they are able to use it in all the same ways and all the same domains that a native speaker - who we would traditionally conceptualise as a native speaker - is able to.

(Lecture A)

After verbalising the slide text, the lecturer goes on to explain the notion of ‘native user’ in more detail, specifying several features (given in bold) that would characterise such an individual. This is to be distinguished from translating, which mostly concerns the lecturer providing a short gloss or explanation ‘in other words’ for a term or concept. The following example, which does not involve explaining a specific concept, further illustrates this.

Example 18

Slide text: - Exploring how this concept has been challenged

Speech: We’ll then move on to think a little about how this concept has been challenged. // So we’ll think about some of the problems of the native speaker concept, and maybe alternative ways of conceptualising as well, take you through some of the key research that has been done in that particular field.

(Lecture A)

Here, the lecturer is not explaining what ‘challenge’ means, but elaborating on the slide text by outlining three specific ways in which she would explore this topic in the upcoming part of the lecture.

A second way in which the lecturer’s speech elaborates on the slide text is through providing a context in which a slide-text unit can be or is to be understood. In Example 19, the lecturer communicates to students the fields of study in which the concept of operationalisation is relevant or often applied.

Example 19

Slide text: Operationalisation

Speech: Another key term when it comes to not just experimental research but quantitative research in general is the term operationalisation.

(Lecture E)

Elaborating through providing context occurs notably when lecturers are introducing a quote from other authors/researchers’ works, as in the following example.

Example 20

Slide text: - Global Englishes

‘most speakers of English are non-native speakers, and all English varieties, native or non-native, are accepted in their own right rather than evaluated against a NSE benchmark’ (Jenkins et al., 2011: 283)

Speech: So I cite here some work by Jennifer Jenkins and colleagues. And Jennifer Jenkins, I think is recognised really as probably the most well-known, the standout figure in this particular field of research. We really associate her name with Global Englishes and English as a Lingua Franca. // So Jenkins et al. say Global Englishes is a paradigm which recognises that most speakers of English are non-native speakers…

(Lecture A)

In this excerpt, the lecturer provides students with information about the lead author of the quotation as a prominent figure in the field, before moving on to verbalise the quotation itself. It therefore provides a contextual frame which supports the selection and presentation of the quote in the slide text, as well as encourages students to accept the propositions in the quote.

A third means of elaborating is through exemplification - providing example(s) that illustrate the idea in the slide text (without the examples themselves being included in the slide text).

Example 21

Slide text: Grammatical functions

Speech: First of all, we may know some things about its grammar, // like if it’s a verb, if it is a transitive or intransitive verb, for example; if it takes specific prepositional phrases.

(Lecture C)

Example 22

Slide text: Variation in the way English is used

Speech: …there’s a huge amount of variation in the way that English is used across the UK. // So the way that someone from Inverness uses English is going to be very different to the way that someone from Plymouth uses English.

(Lecture A)

Examples are useful in illustrating or assisting the comprehension of a concept (‘grammatical functions’) or proposition (there is ‘variation in the way English is used’), but are arguably non-essential pieces of information in the lecture content. There are implications for both the presentation and comprehension dimensions of a lecture: the selective inclusion/exclusion of examples in the slide text has implications for a slide’s information density ⁴, while their inclusion in the lecturer’s speech can be differentially comprehended as key vs. peripheral information (to understanding) by different students as they follow the lecture.

Finally, lecturers may elaborate by stating the implications of the idea or a quote in the slide text, such as in the following examples.

Example 23

Slide text: - 5. native speaker through long residence in the adopted country.

Speech: And finally, Davies says it could be the case that you could make a claim to being a native speaker through having lived in a country for a very long time, // and again that has enabled you to develop a high level of linguistic skills.

(Lecture A)

Example 24

Slide text: Native language variable: using only Chinese, English, Greek, Hungarian, or Japanese speakers

Speech: So if you have a study like that, you might want to control for the first language of the participants. Okay? // Because otherwise if you find any differences, you won’t know whether it was first language that made a difference or the independent variable that you are actually interested in.

Okay?

(Lecture E)

The elaboration in speech therefore enriches the understanding of the idea in the slide text, and these implications elaborated on in the lecturer’s speech often relate to the gist or key message(s) of the lecture, such as defining native speakership through ability rather than early childhood exposure in Lecture A (Example 23), and controlling variables which potentially confound a study’s results in Lecture E (Example 24).

6.1.7 Merging

As identified by Hallewell and Crook (2019), lecturers on occasions discuss or incorporate two or more slide-text units (e.g. two bullet points) in their speech.

Example 25

Slide text: Who is a ‘native speaker’? // Davies (2012: 4)

Speech: So I’m starting by looking at some work by Alan Davies, who was a big name in this field of defining native speakerism - who is and who is not a native speaker.

(Lecture A)

In Example 25, the lecturer introduces the first topic in the lecture, as represented by the first slide-text unit, and also refers to the author whose defining criteria for ‘native speaker’ are cited on the same side, with the author-date citation appearing at the end of the slide.

Example 26

Slide text: HOW TO TEACH: TEXT STRUCTURES [unit 1: slide heading]

Genre analysis (e.g. Swales) has revealed several organisational patterns that recur across texts: [unit 2: first slide-text unit under the heading]

Speech: One of the aims of genre analysis, or discourse analysis more generally, is to identify recurring patterns across texts. Those recurring patterns, presumably are things which could enter into the language classroom as patterns to teach. So there are patterns in narratives, so there’s lots of work on narratives.

(Lecture C)

In Example 26, the lecturer’s speech starts by verbalising the slide-text unit under the heading about genre analysis (unit 2) but then refers back to the slide heading (unit 1). This segment of his spoken exposition, therefore, merges the two slide-text units.

6.1.8 Combining

Lecturers’ speech may also relate to slide-text units in less of a referential way (vs. verbalising, conducting attention, assessing) - they sometimes combine speech and slide-text into a single narrative or message (Hallewell & Crook, 2019)

Example 27

Slide text: Gender variable: using only men or women, assuming that gender may affect the results

Speech: If you have a study where you think gender could make a difference.

Right? There are some stereotypes, that women are better at learning languages and, you know, there’s not really - anyway, I’m not sure if it’s a substantiated statement, but if you’re looking at language achievement, if I believe that’s the case, maybe you want to control for gender as a variable. Okay?

(Lecture E)

In Example 27, the lecturer’s speech segment does not start by verbalising the content of the slide text. Notably, rather, it is framed as a narrative of a teaching point ‘If you have a study where you think gender could make a difference…’, and the rest of the segment (about stereotypes) can be seen to be expanding on that narrative rather than elaborating on the slide text per se. This illustrates how the idea(s) in the slide text (particular words or their paraphrases) may be integrated into the speech rather than being referred to in the speech (as in, for example, conducting attention, assessing).

Example 28

Slide text: The problem: exhaustive academic enquiry to tell us what we already knew as competent readers/writers in a language; so how come we need exhaustive enquiry to figure this out?

Speech: The problem with this idea of having patterns in our head, which we apply to in constructing our texts, is that it’s quite paradoxical that it took much effort, lots of research for academics to identify those patterns, and then tell us what patterns that we had, which we’re supposed to have known already.

So this is knowledge, which is beyond consciousness if there is such knowledge.

(Lecture C)

Example 28, again, illustrates a combining discourse relation between the lecturer’s speech and the slide text, where terms in the slide text are paraphrased in the speech, and it is difficult to identify a linear, sequential progression from referring to the slide text (e.g. verbalising) to assessing or expanding on it; or even a parallel discursive progression in how the main idea or message is communicated comparing the slide text and the speech segment. It is not unreasonable to speculate that this type of discourse relation between slide text and speech would present challenges to students when engaging in simultaneous reception and processing of the aural and the textual input of a lecture.

6.1.9 Highlighting teaching points

Three different ways in which lecturers emphasise or foreground key points for students to notice (in relation to the slide text) have been identified. The first way for lecturers to highlight teaching points is through linguistically and discursively marking them (e.g. ‘the key thing to take away…’, ‘the key point to bear in mind…’), as in Example 29.

Example 29

Slide text: Variation in the way English is used [world map as image background]

Speech: Last time was just looking at the UK context but this time we’re going to look at variation in the way that English is used much more globally, but the key thing to take away here is that there is variation in both these settings, so both in what we might call a native speaker country, but also variation in the way that

English is used around the world.

(Lecture A)

Lecturers also highlight teaching points or key messages within the lecture through reiterating them.

Example 30

Slide text: - “which English?” / “whose English” has important implications for language teaching and learning

Speech: We’ve seen that questions of which English, which variety of English, or whose English has really important implications for language teaching and learning, // and again, I think that’s maybe something interesting for you to reflect on in terms of any previous language learning or either language teaching experience that you might have had: which variety, whose variety of language is being taught and learned.

(Lecture A)

Example 31

Slide text: a) Aspects of word knowledge (Nation 1990, Nation 2001; Richards 1976)

Speech: [after going through points a), b) and c) on the slide] But as I said, the first understanding of word knowledge is the most common. So we’re going to talk about that. And you’re going to see that in many of your readings.

(Lecture D)

In Example 30, the lecturer highlights the importance of the slide text she has just verbalised and prompts students to reflect on that aspect in their own experiences. In Example 31, the lecturer reiterated that the idea represented by the slide-text unit (point a) is the most common understanding of vocabulary knowledge, and highlighted to students that they would encounter the same in the reading texts for the subject.

It is worth noting that reiteration of key points can be done in the lecturer’s speech without necessarily referring to an immediate slide-text unit (the idea may have appeared in a previous slide), as in the example below:

Example 32

Slide text: [N/A]

Speech: So when we scrutinise experimental design, experimental studies, one of the things that I’m always looking for, and then we’re designing studies, we always try to make sure that we really measure what we want to measure.

And we really try to control for all the variables that might interfere when it comes to the relationship between our independent variable and dependent variable.

Okay?

(Lecture E)

Finally, lecturers may summarise the key ideas in a cited text and suggest the text to students as extended reading:

Example 33

Slide text: - ‘bad’ English?

Speech: And Joseph talks about how that was not necessarily seen even inside Hong Kong is necessarily a good thing, because people who consider themselves to speak a more standard variety of English looked down upon Hong Kong English. // So really, a fascinating discussion in that particular chapter of language and identity, if that’s something you would like to read about in more depth.

(Lecture A)

6.1.10 Verbally communicating teaching points

This final category of action in lecturers’ speech does not - strictly speaking - present a discourse relation between the slide text and speech. It concerns lecturers communicating teaching points in their speech, while these points appear to be entirely absent in the slide text. It is included here both as an emergent category in the analysis and in light of its implications for students’ lecture comprehension. Both examples presented below are from Lecture C, and this category is particularly characteristic of Lecturer C’s lecture discourse. However, Example 32 from Lecture E is another instance.

Example 34

Slide text: Berman and Slobin claim that the difference between these excerpts:

Speech: The conclusions that Berman and Slobin arrived at from their research // were a variation on the theme of the Sapir-Whorf Hypothesis - the idea that the language you acquire in effect is a pair of spectacles through which you see the world. And different languages, they constitute different pairs of spectacles, so you’ll see the world in slightly different ways.

(Lecture C)

Here, the speech segment beginning with ‘…were a variation…of the Sapir-Whorf Hypothesis’ can be argued as elaborating on the slide text. The following part that expounds the Sapir-Whorf Hypothesis, however, appears to be a teaching point in itself and is related to the explanation for Berman and Slobin’s findings. There is another slide-text unit in prose form following the slide-text unit shown here to provide a possible explanation for the difference, but it makes no reference to the Sapir-Whorf Hypothesis. Arguably, then, a teaching point has been communicated verbally in the lecturer’s speech without being presented as slide text.

Example 35

Slide text: Japanese [slide heading]

Speech: One of the consequences of the difficulties of the English writing system — By the way, the technical term for the nature of the difficulties of the English writing system is that, the English writing system is ‘opaque’, or sometimes called ‘deep’, whereas a writing system, as in Italian, uses the Roman alphabet, but that’s more or less a intelligent and logical realisation of the alphabetic principle, so more or less one letter, one graphene corresponds to one phoneme. So Italian is called a ‘shallow’ orthography or a ‘transparent’ orthography. English is opaque or deep. And one of the consequences for English kids, when they’re learning to read, is that it is enormously more difficult. So it takes two years, more literacy training for an Italian — for an English kid, sorry — to come to a certain standard compared to an Italian kid. And another consequence of that is there is a far higher incidence of dyslexia amongst children learning to read in English.

(Lecture C)

In Example 35, readers would promptly discover that the speech segment ⁵ has nothing to do with the slide heading which reads ‘Japanese’ and is expounded in the remainder of the slide. Here, the lecturer follows on from the discussion in the preceding slide and actually introduces several terms/concepts (‘opaque’/‘deep’ and ‘shallow’/‘transparent’ orthography) that were nowhere to be found on the PowerPoint slides. He went on to communicate the key point that the opaque orthography of English has implications for learning - presenting challenges for literacy development and relating to a higher incidence of dyslexia.

All in all, the analysis identified 10 discourse relations between lecturers’ speech and the corresponding slide text (see no.1 - no.10 of the coding scheme in Appendix 1).

This section has provided examples of each discourse relation from the lecture dataset, as well as illustrating key variations (sub-types) within each discourse relation identified. In addition to the taxonomy of discourse relations, the analysis further identified some patterns of lecture content exposition characterising individual lecturer’s delivery. The following section briefly discusses these.

6.2 Notable patterns of lecture content exposition

6.2.1 A sequential pattern of exposition

One pattern emerging from the analysis and the above discussion of speech-to-slide-text relations is a sequential pattern of exposition, whereby the lecturer begins their exposition of a slide-text unit by verbalising the slide text, followed by translating and/or elaborating (see, for instance, Examples 10, 17, 18, 21 and 22). A most extended operation of this sequential exposition pattern is observed in Lecturer D’s speech, where the exposition involves four actions in the following sequence:

Verbalising —> (translating) —> elaborating —> (highlighting teaching point) ⁶

Example 36 below illustrates the full sequence.

This would appear to be an exposition pattern that makes it quite easy for students to follow. It starts with drawing students’ attention to a main idea, rendered in both audio and visual channels of input; then either the proposition or a term/concept explained ‘in other words’; followed by elaboration with additional details or with examples. Finally, the key message is reiterated or highlighted. This pattern characterising much of Lecturer D’s speech could be potentially explained in terms of her idiosyncratic lecturing style, or the fact that Lecture D was the very first session of a taught module. It is not difficult to conceive that not all lecturers within the current dataset use this pattern of exposition, and for those who do, not necessarily for every slide or slide-text unit.

6.2.2 A ‘topic-comment’ pattern of exposition

A second pattern of exposition noted is characterised by a ‘topic-comment’ structure. This often involves bullet-point slide-text units with minimal text, displaying a topic only. The lecturer’s speech then contains the main messages or teaching points about the topic, as in the examples below, where the lecturer is introducing the characteristics of written language in contrast with spoken language.

Example 37

Slide text: b) Standard English

Speech: The point here is that English on the page, is extraordinary uniform, so glosses over differences, differences just don’t make their appearance on the written page. Any differences in pronunciation, you don’t hear or read differences in grammar. It’s all uniform. But in the spoken world, English is enormously varied. (Lecture C)

Example 38

Slide text: d) Context-free

Speech: When you were talking to somebody, typically you know who you’re talking to, you know what information they know, and they know what information that you know. When you’re writing, you’re writing to somebody who is simply not there, or you may not even know who is going to be reading it. And in that context free situation, then you as a writer have to supply the context, and that is what produces much more textual density on the written page.

(Lecture C)

While these instances of the lecturer’s speech have been coded as ‘elaborating’ on the slide text in the analysis, it is noted that the slide text alone would be insufficient for grasping the lecture content. This contrasts with slide-text units (e.g. bullet points) which contain not only the topic under discussion but also the key points or main messages, as in Example 36 above (receptive vocabulary is bigger than productive vocabulary), or in Examples 5 and 6 (native speaker membership can be obtained through long residence in the adopted country) under ‘verbalising’.

6.2.3 Communicating teaching points in speech only

Finally, it is worth noting again that lecturers sometimes convey important teaching points only in speech but not in the slide-text. In such instances, the input for lecture content comprehension comes in the audio channel only, with zero or minimal input in the visual channel. This was the case in Examples 37 and 38 under the ‘topic-comment’ exposition pattern, and is further illustrated by Example 39 (also from Lecture C) below, where the key terms or concepts to learn as part of the lecture appear in the lecturer’s speech only.

Example 39

Slide text: i) How do writing systems differ?

Having reported the 10 discourse patterns, we will report the methods and findings in relation to RQ2 (students’ processing in the integration of auditory and textual information) in Section 7. Implications of the findings of RQ1 will be discussed together with findings related to RQ2 in Section 9.

7 Methodology for investigation of integration of auditory and textual information (RQ2)

7.1 Introduction

RQ2 aimed to understand how students integrate auditory and textual information during lectures and develop a mental representation of the lecture content. As the focus of the study was on discourse patterns between auditory and textual information, the variable of native/non-native lecturers was excluded in the main study. Therefore, this along with considerations mentioned in Section 4, narrowed down the lectures for the main study to Lectures A and C (see Table 1) which were delivered by native speakers.

As reported in Section 6, the lectures analysed contained a wide range of speech-to-slide-text relations. For reasons outlined in Section 4, RQ2 focused on the most obvious speech/slide variable - the extent to which the information delivered verbally also appears in the slide text.

In order to investigate this variable and to eliminate several confounding variables such as varied text/font size, the presence or lack of illustrations, and the presence or lack of highlighted text, new slides were produced. The new slides to accompany the lectures had no illustrations or diagrams and text size, font and colour were standardised.

The researchers created two versions of the slides for each lecture clip.

• One version with topic headings plus a short text outlining the main teaching points.
• One version with only topic headings

This enabled the researchers to counterbalance these two patterns of speech-to-slide-text relations for the main study. Therefore, the main study presented each participant with extracts from two of the lectures. For one lecture extract, the slide contained topic headings accompanied by a short text outlining the main teaching point. For the other extract, the slide contained only topic headings.

Therefore, RQ2 focused on exploring any potential differences in students’ cognitive processes in lecture comprehension based on two specific patterns of speech-to-slide-text relations. Two lecture clips, each with one of the two patterns of speech-to-slide-text relations, were presented to participants. The two patterns can be summarised as follows:

Speech/slide-text pattern 1

• Slide text: topic headings plus a summary of the teaching points (key messages of the lecture)
• Lecturer speech: verbalises bullet points and elaborates on the slide text

Speech/slide-text pattern 2

• Slide text: topic headings only
• Lecturer speech: verbalises bullet points and communicates teaching points verbally.

7.2 Participants

For the main study, four participants at CEFR B2 level and 4 participants at CEFR C1 level were recruited from British universities as shown in Table 4.

Table 4: Summary of participant background information

7.3 Materials

One clip from Lecture A and one clip from Lecture C (approx. 2 minutes each) were selected. Both lectures were delivered by experienced lecturers (5+ years of lecturing experience) who are native speakers of British English, and both lectures were pre-recorded lectures delivered online rather than live lectures delivered in person. Each original lecture clip contains a PowerPoint slide (written text only) and the audio of the lecturer’s speech (no video image). The original lecture slides were adapted to capture the two patterns of speech/slide-text relations described above (i.e., topic headings plus a summary of the teaching point and topic headings only), as well as to ensure effective capture of participants’ eye movements by the eye-tracker (e.g. large enough font size; enough spacing between lines). The original audio recording of the lecture was used, and transcriptions of the lecture clips are provided in Appendix 2.

The adaptation of the original lecture clips produced four different experimental clips for data collection.

Table 5: Conditions for data collection

Images of the slides used to accompany the lectures are provided in Appendix 2.

7.4 Procedure

Data was collected at a Psychology Department eye-tracking laboratory at a British university. This provides a series of quiet rooms, free from visual distractions with controlled light conditions. Upon arrival, participants were asked to read the research information sheet and complete a consent form and background questionnaire.

Eye-tracking data was collected using an EyeLink 1000 which uses binocular sampling (it tracks both the left and right eye) at 1000hz (it takes 1000 readings per second). Participants were seated at a desk, in front of a computer monitor and a chin rest was used to ensure participant’s heads remained stable during data collection (Figure 1 gives an indication of the eye-tracker set-up). After adjusting the chin rest to ensure the participant was in a comfortable position, a nine-point calibration was performed.

Figure 1: Example of arrangement of equipment, participant and computer monitor for eye-tracking data collection

A counterbalanced order of both lecturer and condition was used with participants as outlined in Table 6.

Table 6: Order of presentation and lecturer/slide text condition

To replicate authentic listening conditions for participants, before participants listened to the lecture extract, they were:

• provided with information to contextualise what they were about to hear
• given the opportunity to become familiar with the speaker’s voice.

This was achieved by presenting the lecture extract as the last slide in a set of four slides as follows:

Slide one explained that slide two was an opportunity for participants to get used to the lecturer’s voice and that they would not be asked about what they had learned from slide two. There was no audio.

Slide two contained a two-minute audio sample of Lecturer A or C speaking. It was taken from the same lecture as the lecture extract and was accompanied by the relevant slide text. This provided context for slide four and provided an opportunity for the participant to get used to the lecturer’s voice.

Slide three told participants what the topic of slide four would be and reminded them they would be asked about what they had learned from slide four. There was no audio.

Slide four contained the lecture extract (A or C) and text giving topic headings only or topic headings plus summary teaching points associated with the lecture excerpt.

Immediately after the lecture clip finished, a blank screen was displayed, and the participant was asked to tell the researcher what they had learnt from the lecture excerpt. If participants failed to mention one of the topics covered by the lecture, the researcher prompted the participant by asking ‘Did you learn anything about…’ This was to try and minimise the role that memory/recall played. The participant’s explanation of what they had learnt was recorded.

The eye-tracker calibration was checked before the procedure was repeated with the other lecture and condition.

After a short break, the stimulated recall session took place. The session was recorded using screen capture and audio capture. Participants were shown a replay of the first lecture and shown how to pause the recording. Participants were asked to stop the video whenever they remembered what they were thinking during the lecture.

After the stimulated recall of the first lecture, the participant was also asked:

• How much of the text did you read? Why?
• Did you try to read sections of the slide to correspond/match what you were listening to? Why? Why not?
• What did you think about the amount of text on the slides?
• What did you think about the amount of information on the slide(s)?

This procedure was repeated for the second lecture. Once this had finished, the participants were also asked:

• Which lecture did you find it easiest to understand? Why?
• Do you have any strategies that you use when you are in lectures to help you understand the lecture?
• Is listening and reading at the same time easy?
• What makes it easier?
• What makes it more difficult?

7.5 Data analysis

The data collection procedure resulted in three sets of data: the eye-tracking data; the participants’ accounts of what they had learnt from the lectures; and the stimulated recall recordings. The way each type of data was analysed and processed will be described in turn.

7.5.1 Eye-tracking data

The eye-tracking software (DataViewer by SR Research) allows interest areas to be defined on the image of the screen that was presented to participants. The researchers identified interest areas by drawing rectangles on areas of the screen image that are of particular interest. These rectangles were labelled, and the software then codes any fixation that occurs within the boundary of a particular interest area (IA) with the relevant IA label.

The slide images with Interest Areas outlined in orange are shown for Lecture A in each condition (Topic heading only followed by Topic heading + summary) followed by the slide images for Lecture C

Figure 2: Slide image for Lecture A, Topic Heading Only condition with Interest Areas outlined in orange

Figure 3: Slide image for Lecture A, Topic Heading + Summary condition with Interest Areas outlined in orange

Figure 4: Lecture C, Topic Headings Only condition with Interest Areas outlined in orange

Figure 5: Lecture C, Topic Heading + Summary condition with Interest Areas outlined in orange

When setting up Interest Areas, each area is given a name, and the eye-tracking software tags any fixation occurring within an Interest Area with the relevant code. Interest area codes were allocated to each of the lecture slides as illustrated in Figures 6 to 9. Fixations which occurred outside any of the Interest Areas drawn were tagged with the code OUTSIDE.

Figure 6: Interest Area coding for lecture A, Topic Headings Only

Figure 7: Interest Area coding for lecture A, Topic Headings + Summary

Figure 8: Interest Area coding for lecture C, Topic Headings Only

Figure 9: Interest Area coding for lecture C, Topic Headings + Summary

This resulted in the fixation data being tagged with a code which indicated which part of the slide was being fixated throughout the period of each lecture. A sample of the eye-tracking data from Excel can be seen in Table 7. The columns of data represent (left to right) the participant reference, the participant’s CEFR level, the order in which fixations occurred, the number of milliseconds that have elapsed since the slide was first displayed, how long the fixations lasted (also in milliseconds), the interest area coding, the slide condition (0=topic heading only, 1= topic heading + summary) and the lecture (A or C).

Table 7: Sample of eye-tracking data when imported into Excel

After this, an additional code was added to each fixation to indicate which part of the lecture audio was being played when the fixation took place. Because fixation data is recorded in milliseconds, it was also necessary to use milliseconds when tagging the eye-tracking fixation data with codes to indicate which part of the audio was being played at the time of each fixation. This was achieved by:

1. reviewing the content of the audio and making a precise record of the timings for each lecture as the audio progressed from one topic to the next (see Table 8)
2. comparing the timing of each fixation (Fix_start_time in Table 7) with Starting millisecond and Ending millisecond data from Table 8 and appending the relevant code from the ‘Audio relating to…’ column in Table 8 to the fixation.

Table 8: Audio coding applied to fixation

Therefore, after this procedure, the fixation data had an additional column which indicated which part of the lecture audio was being played when the fixation took place. A sample of data with Audio Coding can be seen in Table 9.

Table 9: Sample of fixation data with audio code added

Thus, the researchers were able to analyse and compare the data in terms of differences between the two conditions (topic headings only versus topic headings + summary) and consider the extent to which participants coordinated their reading of the slides with the lecture audio. Results for B2 and C1 participants could also be compared to consider whether differences between the two slide conditions were consistent for the two groups.

7.5.2 Assessment of learning from the lectures

In order to assess whether the two slide conditions impacted participants’ ability to learn from the lectures, the participants’ accounts of what they had learned were transcribed and scored by two researchers.

Transcriptions of the lectures were analysed by two of the researchers and the lectures were broken into segments which related to the main teaching points. Each lecture was considered to contain five teaching points. The raters discussed and agreed upon the information which was key to each teaching point. So, for example, Figure 10 below shows the transcription of one of the teaching points from Lecture C with the key information highlighted in red.

Figure 10: Transcription of teaching point 2, from Lecture C (words highlighted in red indicate key information)

The two researchers then independently reviewed the transcriptions of participants’ accounts of what they had learned and awarded a score for each of the 10 teaching points (five from Lecture A and five from Lecture C) using the descriptors in Table 10.

Table 10: Descriptors and scores used to rate participants’ learning from the lectures

In total, each researcher awarded 80 scores (8 participants x 10 teaching points). In 77 of the 80 scores, the researchers’ scores agreed. For each participant, the 10 scores awarded by Researcher 1 and the 10 scores awarded by Researcher 2 were totalled and divided by 2 thus giving each participant a score based on the average of the two researchers’ scores. The total score awarded to each participant was added to a summary of the eye-tracking data to enable the researchers to consider the eye-tracking data in conjunction with the score data.

7.5.3 Stimulated recall and interview

All the stimulated recall interviews were transcribed and analysed thematically using MAXQDA 2022. One of the researchers went through all the relevant, transcribed verbal report data and iteratively worked out the emerging themes and sub-themes. The interview questions (as presented in Section 7.4) served as a basis for developing the initial coding categories, and the detailed coding scheme was developed by adding new themes and subthemes as the coding progressed. When a participant reported on more than one process and/or strategy at the same time, the relevant parts of the recall data were coded for multiple categories. The emerging codes were reviewed, discussed and revised by the researcher and one other researcher. The data was reviewed again in light of the revised coding scheme and then, in order to warrant coding reliability, verbal reports on both lectures from one of the eight participants (i.e., 12.5% of data) were double-coded independently by the second researcher. The exact agreement reached 90.57% for the types and occurrences of codes, which was deemed satisfactory.

The coded data was used to gain more in-depth, potential explanations to complement the results of eye-tracking data, including any similarities and differences across the two proficiency levels, two lectures, and slide conditions.

8 Results for investigation of the integration of auditory and textual information (RQ2)

8.1 Eye-tracking data

During the lectures, participants made a total of 6,540 fixations on the lecture extract slides. Fixations on the information and familiarisation slides (slides 1,2 and 3 detailed in Section 7.4) were excluded from the analysis. Table 11 summarises the fixation data.

Table 11: Summary of fixation data for all participants

The mean fixation durations are broadly in line with those reported in the literature for second-language reading (Cop, Drieghe & Duyck, 2015, Rayner, Pollatsek, Ashby & Clifton, 2012); however, it should be noted that P10 has a higher mean fixation duration (343.80 ms) with a much larger SD (of 804.19 ms) compared with other C1 participants. P10 made a small number of very long fixations (10-12 seconds) during which the participant fixed at a point on the screen that contained no text. In the stimulated recall interview, P10 commented that she finds listening and reading the slides at the same time difficult. Her staring at the blank parts of the slides might indicate her strategy to alleviate the difficulty of concurrent lecture listening and slide reading.

As reported in Section 7.5.1, the fixation data was coded to indicate which part of the lecture audio was being played when the fixation took place. This enabled the research to assess the extent to which participants focused on the bullet point that the lecturer was talking about as each lecture progressed. The researcher uses the word ‘correspond’ to indicate that, for example, while Lecturer A introduced the lecture, the participant fixated on the interest area labelled as INTRO in Figure 7, when the lecturer moved on to talk about terminology, the participant fixated on the interest area labelled as TERM and so on. Thus, all the fixations occurring within the INTRO interest area while the lecturer introduced the lecture would be coded as ‘yes’. However, if the participant looked elsewhere on the screen (outside any defined interest area or at another interest area, such as the TERM interest area which came next, while the lecturer was still talking about the introduction, the fixation would be coded as ‘no’ in the Correspond column. (See Table 12 for a sample of the coded data). Once the lecturer progressed to talking about Terminology, only fixations occurring on the TERM interest area would be coded as ‘Yes’ in the Correspond column and so on.

Table 12: Sample of eye-tracking data coded to indicate whether fixations focused on bullet point relevant to audio at the time fixation took place

Overall, 41.85% of fixations fixated on the bullet point which was being talked about when the fixation was made. When the data is subdivided into participant levels, B2 participants had a slightly higher percentage of correspondence (42.87%) than the C1 participants (40.74%). With a small number of participants, it is not possible to say whether this difference is statistically significant.

If we look at Table 13, we can see that both the B2 and C1 participants had higher rates of correspondence for Lecture C than for Lecture A, albeit that the difference between Lecture A and C is less marked for C1 participants.

Table 13: Breakdown of fixation corresponding with audio by participant level and lecture

However, if we look at Table 14 which reports the breakdown by the slide condition, we can see that the patterns of correspondence differ between B2 and C1 participants. This suggests that the alteration in slide condition affected the behaviour of B2 and C1 participants somewhat differently.

Table 14: Breakdown of fixation corresponding with audio by participant level and slide condition

The differences in B2 and C1 participants’ correspondence of fixations with the audio are also reflected in the assessment of learning results reported in Section 8.2.

8.2 Assessment of learning

Immediately following the lecture, participants were asked to recall what they learned. Their response was recorded, transcribed and independently rated by two of the researchers. Please see Section 7.5.2 for a detailed account of how the scores were generated. A score of 20 was the maximum possible score for the lecture indicating full coverage of all the teaching points in the lecture. The results of the rating procedure are reported in Table 15.

Table 15: Scores awarded for accuracy and coverage of learning from lectures

When the data is summarised by participant proficiency level and lecture, it is clear that all the participants achieved a more accurate and complete coverage of learning from Lecture C than from Lecture A. On the whole, C1 participants scored more highly than B2 participants.

Table 16: Summary of scores awarded for accuracy and coverage of learning from lectures by lecture and participant language proficiency level

The eye-tracking results for fixations corresponding with the audio showed that for both B2 and C1, participants’ fixations corresponded with the audio more for Lecture C than for Lecture A.

Data from the stimulated recall suggests that Lecture C may have been easier to understand and for participants to identify and fixate on the corresponding bullet point for the following reasons.

• It had a clearer structure - there were four points - all equally relevant to the main theme of the lecture (examples of the differences between speech and writing). Lecture A had a slightly more complex structure with a main point (Terminology is important) which generated a sub-point (which term was preferred) which had two supporting points (two reasons why the term was preferred).
• Lecturer C left clear pauses in his speech before moving on to the next point, so it was easy for participants to recognise which bullet point the lecturer’s speech related to.

In addition, assessment of learning scores may have been higher for Lecture C because the subject matter was slightly less abstract or less challenging than Lecture A which related to the reasons the term ‘native user’ was preferred to the term ‘native speaker’.

This was despite several participants reporting that they found Lecturer C’s voice more difficult to understand because the speech was rather slow or less clear than Lecturer A’s.

If we consider the differences between B2 and C1 participants’ scores in terms of the slide condition (Table 17), we can see that while the Topic Heading only condition resulted in slightly poorer scores in Assessment of Learning for the C1 participants, B2 participants scored better in this condition.

Table 17: Summary of scores awarded for accuracy and coverage of learning from lectures by slide condition and participant language proficiency level

When this is related to the eye-tracking data, it is interesting to note that once again, the slide condition seems to have affected B2 participants differently than C1 participants. In the Topic Heading only condition, B2 participants’ fixations corresponded with the audio more than in the Topic Heading + Summary condition. C1 participants had greater correspondence in the Topic Heading + Summary condition.

The number of participants is small, but these results could indicate that ‘less is more’ in terms of slide text for students at B2 level. In other words, having more text on the slide did not lead to a greater level of understanding for B2 participants. The stimulated recall and interview data suggest that concurrent reading and listening is challenging and having too much text may limit B2 participants’ ability to focus on listening. The challenges of concurrent reading and listening are discussed in greater depth in the stimulated recall and interview data, which is reported next.

8.3 Stimulated recall and interview

As noted above, participants reviewed recordings of the lecture made during eye-tracking and were asked to pause the replay, or the researcher paused the replay and prompted participants to report their reading and listening processes when they were able. After that, they were prompted to compare the two lectures before finally being asked about strategies used in lecture listening.

Five themes emerged from the data:

1. Slide text and lecturer speech relations
2. Aspects relating to lecture delivery
3. Strategies used by students when listening to lectures
4. Comparisons made between the lectures
5. Miscellaneous other factors.

Each theme consisted of several subthemes. A tally of the number of comments in each subtheme is reported in Table 18. The tally shows the breakdown of comments according to which lecture/condition they related to or whether they were made when participants were asked to compare the lectures or asked about their general lecture listening behaviour.

Table 18: Tally of stimulated recall and interview data comments by category and subcategory

The most important subthemes that emerged from each theme will now be discussed in turn.

8.3.1 Speech slide relations

Comments in the speech slide relations category concerned the amount of text on the slide, the relationship between the words on the slide and the extent to which the lecturers’ speech overlapped. The participant ID, CEFR level, and the relevant line/position in the transcript (specified as ‘Pos. [number]’) are specified at the end of each comment excerpt.

The first subtheme related to the amount of text on the slides. Several comments suggested that what was ideal varied according to the listener’s familiarity with the content or topic.

Example 40

Because I a think that really depends on what the lecture is about, how I am familiar with the content. (P7, (C1)Pos. 91)

Example 41

From from my knowledge, it depends on the topic, because if there is…of course we cannot have everything…what what is being said. (P4, (B2) Pos. 66)

Seven comments in this category suggested that the amount of text on the slides was appropriate.

Example 42

Enough information on the slide to catch things. When she’s talking. I mean, I can just understand better because it’s enough information. (P9, (B2) Pos. 37)

However, under the sub-themes of too much text, five comments suggested there was too much text on the slides with Topic headings + summaries. Of these, two related to Lecture A and two related to Lecture C. One related to lecture slides in general. No comments suggested that the Topic heading only condition slides contained too much text.

Example 43

Definitely enough, I think. Or maybe more than enough. (P7,(C1) Pos. 261)

Example 44

I think it’s too much. Uhm, I think the amount of words threw me off a little bit because I thought this is going to be a very heavy lecture maybe. And then I realised it isn’t. And then I was thinking, why are there so many words on this slide for? Single topic, yes, there are…subtopics to it, to some points, but it’s the same topic. I. Yeah, I think just fewer fewer words. I think would be… (P10,(C1) Pos. 173)

Under the sub-theme of too little text (10 comments), seven comments suggested that Lecturer A in the Topic heading only condition had too little text and two comments suggested that Lecturer C in the Topic heading only condition had too little text. No comments suggest that the Topic heading + summary condition slides contained too little text.

Example 45

Because it was…a comparison between users and speakers so she could somehow put in the slide, on the slides…difference between users and the speakers and after that to what’s the difference because one is born, one is not yes so…making comparison between them, it would be useful to put into the slides, yeah. (P8,(B2) Pos. 63)

This suggests that more of the participants were comfortable with the amount of text in the Topic heading + summary condition than in the Topic heading only condition. However, it is interesting to note that in the results of the Assessment of Learning (Table 15), participants scored slightly better in the Topic heading only condition.

One comment was made relating to difficulties that ensued when the lecturer’s speech deviates from talking about the topics in the order they are presented on lecture slides.

The sub-theme that occurred the most was the overlap, or lack of overlap between the wording of the slides and the lecturers’ speech. Altogether 45 of the 239 coded comments related to this subject, the most in any sub-theme.

Some comments related to the lack of overlap in places between lecturer speech and slide text and the difficulties this caused.

Example 46

So the headings were a bit different from what I was expecting to listen to. I was expecting to speak about these great words like ‘informal’. (P8,(B2) Pos. 5)

Example 47

Yeah, there is totally a lack of…key words here on the slides because of what she was saying was not on the slide, totally like nothing about it. (P8,(B2) Pos. 55)

Example 48

I’m not sure if he mentioned it…he didn’t mention ‘standard English’ actually. (P7, (C1)Pos. 199)

Example 49

I think these are not what she said yeah, so that’s more difficult. I need more time to process that information. (P3 (B2), Pos. 137)

Example 50

It would have been confusing at first because I would think he still might be talking about standard English, so only when I would hear that he’s talking about words and and the the breaks between words and I think he still hasn’t mentioned the word ‘discreet’. (P10, (C1) Pos. 69).

This seems to suggest that participants struggled when lecturers failed to use the same key terms or phrases in their speech that they used on the slides. This may lead to increased cognitive demands in attempting to relate the words/headings on the slide to the subjects under discussion. Example 49 suggests that a failure to use the key terms from the slide can prevent students from readily following the structure of the lecture. When there was a clear overlap of key terms or phrases, comments suggest that this was helpful.

Example 51

This was, yeah, she used more of the words also used on the slide. So I could very easily pinpoint where we are at with the listening, if if I would switch off for a second or think of something else and come back to it. Just hearing her say a few words I could point to you on the slides where she is, where she got to. (P10,(C1) Pos. 177)

Example 51 seems to suggest that language overlap makes recognising the structure of the lecture easy. The same participant suggested that while it was helpful, perhaps as English language proficiency improved, that clear overlap of language was less critical.

Example 52

Maybe at a lower level it’s helpful. At a higher level, I think it’s irrelevant. Yeah, it’s kind of useless, yeah. If I’m the kind of person who wants to listen. I just want to look at the screen, if I’m the kind of person who needs the screen, I’ll look at the screen. The purpose of having slides is not…is that the slides don’t have the exact same information in the same words as the speaker is going to use. Otherwise, he could just give us a handout. We read the text and then we talk about. We wouldn’t need the teacher. We wouldn’t need his slides, so yeah. (P10 (C1), Pos. 97)

One participant seemed to suggest that the amount of overlap between slide text and speech may also be related to attempts to read and listen concurrently, which is dealt with in the next section.

Example 53

It felt like a race. Almost. I had to read quickly before he started speaking again so I can listen to him and not focus on the slides. (P10,(C1) Pos. 85)

Five comments were made regarding the way in which slide information helped participants understand and follow the structure of the lecture.

Example 54

Like these kind of structured information on the PowerPoint slides. Help a lot. P7 (C1), Pos. 281)

Example 55

But as I, as it is in order I could, I could understand that it he was speaking about the first one, second one. (P4 (B2), Pos. 108)

Example 56

If I know the overall structure of this part, I think I and that will help me to follow the speaker even if there are something, maybe I cannot understand and I I still know what what the main point is, yeah. (P7 (C1), Pos. 83)

This suggests that the way in which points are set out and structured on the slides may have an important role in helping students understand how points relate to each other and perhaps assist in distinguishing main points from supporting details.

8.3.2 Lecture delivery

This theme draws together comments relating to the process of delivering the lecture, for example, whether the lecturer pauses between points, whether participants are required to read and listen concurrently and aspects relating to the lecturer’s voice and lecturing style.

The theme most frequently commented upon was the need to read and listen concurrently with 25 comments. Many of the comments attested to the fact that participants attempted to reading and listen concurrently.

Example 57

And then as the tape was was going by, I was following (participant refers to slide on screen) as as the speaker. There was, say, a specific time because there was another pause between them, so I just took in. (P4 (B2), Pos. 46)

Example 58

The speaker speak a little bit fast and we are trying to read the words and and follow the speaker. We just get sometimes a little bit lost if it’s a the speaker which speaks slowly and it’s raised the full sentence fully and clearly, we can follow. (P4 (B2), Pos. 194)

Some comments suggested that attempting to read and listen concurrently is tricky and engaging in one process can distract attention from the other, even for more proficient students.

Example 59

As he was talking a, I quickly scanned through all the terms to get a general idea about what is going to be covered in this leg but I didn’t read the sentence below because I was afraid of missing any information by listening. (P2 (C1), Pos. 171-175)

Example 60

At the same time I am listening and I’m not focused on reading, because listening and reading at the same time is not that easy, even if it’s about the same subjects. (P10 (C1), Pos. 213)

However, there were occasions when reading at the same time seems to assist with listening comprehension.

Example 61

It’s it’s sometimes from what I capture, what I hear, even though I understand, for me sometimes, to put it in words, I do, sometimes struggle to…to write to spell.

So, as I am reading I also capture some words and the spelling of the words, so it’s it’s also good for me to read to have a little bit of a of summary for what it’s been spoken, so I can also acknowledge what I what I heard, what I hear.

(P4 (B2), Pos. 50)

It also seems that reading can help with listening comprehension in terms of priming or preparing the listener for what they are about to hear. There were also 18 comments which related to the way participants attempted to read ‘ahead’ of the lecturer’s speech.

Example 62

And while listening to his lecturing, I would focus on the point he’s he was talking about, and then also scanned the…a little bit the next point. So to get myself prepared for the incoming information. (P2 (C1), Pos. 187)

Example 63

I think first, just as she started, started talking, I tried to have a overall look at so to read…everything, it’s just to find out, like the structure of the part where she’s going. (P7 (C1), Pos. 79)

Example 64

OK, I’m trying to read what’s on the screen. I’m trying to make sense of what what this lecture is going to be about so I know what I’m listening to. (P10 (C1), Pos. 7)

And when participants were asked about instances when their reading fixations had preceded their listening they accounted for the behaviour as helpful.

Example 65

I think that is very helpful because I can expect. I can expect what I was going to listen and mentally prepared for that? Yeah. (P7 (C1), Pos. 301)

Example 66

OK. Yes, that’s give me some hints what he is talking about because actually my listening ability, I think I need to improve. (P3 (B2), Pos. 17)

However, even when participants ‘read ahead’, it did not always ensure comprehension followed.

Example 67

Yes, this is my strategy and then try to listen and to see…Ohh which one he is talking about or she is…talking about but here I cannot. I’m, I’m just lost.

(P3 (B2), Pos. 161)

When asked about general lecture listening behaviours and strategies some participants mentioned being given access to lecture slides ahead of lectures. However, when asked about this, they reported that this was not always helpful. It seems that giving out slides which are very information dense can be off-putting.

Example 68

Not always depends on the subject. I will repeat again, because sometimes it just gives you a lot of information and you’re seeing it and it scares you.

(P8 (B2), Pos. 149)

It also seems that if slides contain vast amounts of information, which the lecturer then reads out during the lecture, is seen as a waste of time.

Example 69

So if it’s very important information, but that really depends because I I noticed some speaker, they put a lot of information on the slide but it’s not so useful to read before because what they are talking about is just the same as the slide, so it it doesn’t matter if we read it before or not, yeah. (P7 (C1), Pos. 289)

There were 14 comments which related to pauses in the lecturer’s speech. It should be noted that Lecturer C left very clear pauses, for a few seconds at the end of each point before moving on to the next point. Lecturer A had no noticeable pauses in their lecture. All but one of the comments related to Lecture C and were evenly divided between positive and negative comments.

The positive comments regarding the pauses seem to suggest that pauses help establish the structure of the lecture.

Example 70

It was talking about the A and the B, and then the C. That’s, so that pause between, between them of course, made me also understand, help. Yeah.

Help you move on. (P4 (B2), Pos. 96-98)

The pauses may also allow time for listeners to process what they have heard (Example 71) and prepare to listen afresh (Example 72).

Example 71

So and then the, the little gap that it made pause between, between the points was also good for us to, for for me to, to how to say it? To understand a little bit. (P4 (B2), Pos. 120)

Example 72

Also, the gap is on purpose. Yeah. Oh, I think it’s helpful. Yes. The silence gives me a warning that this section is over and the next section is coming. So I have a few seconds or one second to sort of close off point in my mind and opened my mind to Point C. I’m not still thinking about point B. OK, I find it…very helpful. (P10 (C1), Pos. 61)

However, pausing was not seen as helpful by all participants. Some participants seemed distracted or unsettled by the length of the pause.

Example 73

I was thinking why? How come? How come he stopped? (P2 (C1), Pos. 106)

Example 74

Also, I think he had…big gap between each point and it felt like he’s reading it from somewhere and then stopped point A and then he had to turn the page and then gone. And I mean, he wasn’t fluent. (P6 (C1), Pos. 95)

This perhaps reflected the fact that participants did not have an image of the lecturer and so had no facial expression or body language to help them understand or account for the pause.

A range of comments concerned aspects related to the lecturer’s voice. Two participants considered Lecturer C’s voice to be rather slow.

Example 75

I was expecting him to speak a little bit faster for me to better understand all the points because. For me, personally, he’s speaking, slowly. (P2 (C1), Pos. 61-65)

Example 76

So yeah, he looks like he’s doing something else when he’s talking. (P9 (B2), Pos. 67)

One participant reported finding Lecturer C more difficult to understand than Lecturer A.

Example 77

Because I think the second speaker and his accent is, is not as easy as the first one for me to follow. (P7 (C1), Pos. 95)

However, as was reported in Table 16, participants scored higher for accuracy and coverage of learning from Lecture C than from Lecture A. Therefore, this suggests that the effects of Lecture C’s rate of speech and pausing behaviour were relatively limited.

8.3.3 Comparing lectures

As mentioned above in Table 16, participants scored higher for accuracy and coverage of learning from Lecture C than from Lecture A. However, that did not concur with their perception of which lecture was easier to understand. There were nine comments which related to Lecture A being easier to understand. Six of the comments explaining why Lecture A was easier related to accent and clarity of speech.

Example 78

OK, she’s more clear definitely than the 2nd. (P9 (B2), Pos. 21)

Example 79

Because her accent, I think it’s more easy to understand and she’s talking more fluently. (P9 (B2), Pos. 121)

The remaining comments related to there being less information to understand.

Example 80

The woman is easier to understand because she doesn’t mention as many points as many details. It’s not as, it’s not as, not as informative. (P10 (C1), Pos. 185)

When asked which lecture was preferred in terms of the amount of text on the slides, the Topic heading + summary condition was preferred and perceived as being easier, with nine comments reflecting this view.

Example 81

About the amount of information he was giving and the information on the slide, on the second slide, it was she was speaking so much but giving so little on the slides. (P8 (B2), Pos. 105)

Example 82

Because it’s quite obvious that this one has more words or text written on the screen compared with the second one. And I think this is proper length. It would guide me through the major points in the lecture, but not really distract me a lot. But compared with the second one, which has really very limited text actually. I…it might help me understand more with a little bit more. (P2 (C1), Pos. 227-231)

Only one participant reported preferring the Topic heading only condition.

Example 83

It’s simple and it’s a little bit. I think it’s points here and you can easily get the main points. (P3 (B2), Pos. 149)

8.3.4 Strategies used

The fourth theme that emerged from the stimulated recall data was the strategies that participants mentioned using to facilitate comprehension and learning from lectures.

The strategy mentioned most, with 13 instances, was relating lecture content to participants’ own experiences. Comments suggested that several participants actively tried to relate what they were learning to their own experiences.

Example 84

It’s good because…if you’re given an example, you can find yourself some way and relate it to yourself so you can make a comparison, give you understand better. (P8 (B2), Pos. 51)

Example 85

When he mentioned this point, I…What I was thinking, I think it’s like… it’s right.

Context is is so important and and I’m…My my own experience that the, the most difficult thing for you to write something or… (P7 (C1), Pos. 235)

The end of Example 84 suggests that linking to personal experience facilitates improved understanding and, as in Example 86 below, perhaps improves recall of information.

Example 86

Yeah. So maybe to remember it, because you link it to something else. (P6 (C1), Pos. 35)

However, there were times when the process of making the link to personal experience seemed to distract the participant’s attention away from the lecture.

Example 87

Yeah, you’re not sure and, and, and she suggests you can probably think of an example, but that suggests you, couldn’t you? Weren’t sure what example you’re expected to come up. (P7 (C1), Pos. 13)

Example 88

So I would say when, when, when she said this. When I think about my own experience, I, maybe in a way, stop listening to her. (P7 (C1), Pos. 33-34)

Other strategies mentioned related to attempts to pre-empt what comes next either by reading ahead of the lecturer’s speech on the slide or by predicting. Participants 10 and 7 both reported actively trying to read ahead of the lecturer’s speech.

Example 89

And then once I figure out what the first paragraph was about and what she was talking about, I moved on to the bullet point. Because there’s information on the slides, you might want to read it. I know she’s not there yet. I know she’s going to get there, but the information is there and I want to read it. So I was not struggling, but I was trying to choose whether to listen or read. At this point, yeah. OK. (P10 (C1), Pos. 135)

Example 90

I think when I when I got the general idea I I kind of like, not straight away, but I want to read what’s coming. (P7 (C1), Pos. 253)

Comments seem to suggest that when participants read ahead, they begin to formulate ideas about what the lecturer might say.

Example 91

Researcher: Yes, he was talking about speaking and you were expecting that.

P3: Yes, right. And typically you should be speech typically I think well, why only writing typically, and then when I listen and then I forgot about that and then I mixed the information. (P3 (B2), Pos. 69-71)

Example 92

Here I was expecting the lecture talking or mentioning reading and listening and writing, but I wasn’t really expecting her mentioning sign language.

(P2 (C1), Pos. 382)

It seems that the process of understanding the lecture can involve not only this predicting of content but also trying to work out how the current point relates to or is relevant to what has come before:

Example 93

Researcher: So it’s, it’s quite off topic?

P2: Yeah, to me. To me, it’s like this point was.

Researcher: Irrelevant or maybe redundant?

P2: Yeah, I was feeling that, but now I’m listening to it. I think it’s, it’s, it’s summarising previous points. (P2 (C1), Pos. 404-411)

Example 94

I think I, I, I quite understand what she is talking about when because that when, when do you have like this receptive, receptive one of language but what, what it seems like what is talked about already in the first point. (P7 (C1), Pos. 65)

Participants also reported guessing the meaning of words they fail to hear or recognise.

Example 95

Here for this sentence, I, I didn’t get every word but but I, I thought I, I, I can manage to understand. (P7 (C1), Pos. 133)

Other strategies that participants reported using included taking or not taking notes, asking questions, making links to sensations and going back to earlier points on the slide.

8.3.5 Other factors

There were some factors mentioned by participants that the researchers felt fell outside the themes discussed so far (Speech slide relations, lecture delivery, comparing lectures and strategies used). These remaining subthemes included personal preferences/learning style, a failure to understand and background knowledge/interest, and were grouped together under the heading of ‘Other’.

Eight comments were made about the role of background knowledge/interest.

One participant suggests that their familiarity with the general area of the topic made comprehension easier.

Example 96

But for this lecture, because I think the content, I, I’m quite familiar with this area of knowledge. So, it’s very much easy for me to understand what she’s talking about. (P7 (C1), Pos. 33)

The participant goes on to suggest that not having background knowledge may make understanding more challenging.

Example 97

Say, like for my experience or my educational background because I studied applied linguistics and English language teaching, I think that’s quite normal terminology or, and easy to understand terminology, but for some other people, maybe they are just new to this field. They may need to get these two terms, or at least. (P7 (C1), Pos. 51)

Another participant seems to confirm that a lack of familiarity with the subject or certain terms means they need to concentrate more when unfamiliar topics are introduced.

Example 98

Not very familiar because…I didn’t read that. I didn’t know that ‘native users’.

Here’s a new word, ‘native users’, but then here is just thinking, oh, she’s talking about native users. Why? And then because you will concentrate on that.

(P3 (B2), Pos. 107)

Another participant suggests that whether you are interested in a subject has an impact on your ability to follow the lecture.

Example 99

But I also think it’s much easier to follow something that you’re interested in.

(P6 (C1), Pos. 139)

These comments suggest that familiarity with a topic can make it easier to understand. Perhaps because it lessens the cognitive load of processing what you are hearing and some connections between the points made in the lecture are, in part, already established. Logically, a lack of familiarity will make additional demands on the listener, perhaps requiring them to concentrate more carefully. Example 99 also suggests that, inevitably, motivation also has a role to play.

There were several comments relating to personal preference or learning style.

The remarks in this subtheme included remarks about whether participants felt they were good or bad at certain skills or what they found easy or difficult.

Some participants seemed to suggest their listening skills were weak, and reading information from the slide supported their listening skills.

Example 100

I have to say, I’m not very good listener in lecture. The reason why? I don’t know because maybe I, my mind stray away a lot during a lecture because it’s long and a lot of information or…I don’t know. But when, when I try to concentrate or when I try to get…Because I know I had, I had to get something from this lecture and the one. That’s why I was there…I think the…like these kind of structured information on the PowerPoint slides help a lot. OK. And sometimes when I want to get back to the lecture, I will look at this and like, oh, this is the point. (P7 (C1), Pos. 281)

Other participants seem to suggest that they preferred to listen without the distraction of reading points on the slide.

Example 101

So if I wanted to, if I just listened to the recording without looking at anything, I can remember more than looking at the screen and trying to set these points in my mind of what I need to listen to…rather, just listen to it like a story. And this is personally how I always remember stuff better or even memorise things of paper if I read the like a story, I would remember the…story. (P10 (C1), Pos. 15)

Some participants seemed to express a preference for verbal learning.

Example 102

I get things faster when I’m listening, same again when I have a conversation, than when I’m reading it. (P9 (B2), Pos. 29)

As with learning in a first language, different students will have different preferred learning styles and preferences for how content is delivered.

The final subtheme related to participants’ explanations for when they failed to understand. There were four comments where participants attempted to explain a breakdown in understanding. The explanations related to the content being unfamiliar or words being used in an unfamiliar way.

Example 103

‘Discrete’ yes, I know that word. Yeah, but discrete words is yes…stand alone word. Stand alone… But I didn’t notice that. I didn’t realise because you know, you just grasp the main meaning. You don’t understand the word by word. So I didn’t catch that, that. (P3 (B2), Pos. 39-43)

It is interesting to note here, although the participant reports that during the lecture they did not understand the word ‘discrete’ in this context, they have, during stimulated recall, correctly inferred the meaning of the word in this context.

On other occasions, comments suggest perhaps the amount of information contained in the lecture clips presented a challenge.

Example 104

(Quoting the lecturer) ‘emphasis(e) the skills people have acquired’. Acquired rather than this reason, but I don’t quite understand ..so many…point. (P3 (B2), Pos. 83)

In summary, the stimulated recall data suggests there are a wide range of factors that impact upon the ability of students with English as a second language to understand the content of lectures. The data also suggests that a number of these factors interact with each other.

The most commented-upon factor was the extent to which the language on the slides overlaps with the lecturer’s speech. The comments on concurrent reading and listening suggest that for most participants, processing the two channels of information simultaneously was challenging and therefore, having a clear overlap, especially of key terms was important.

The data suggests that the amount of text on the slides and the way the points on the slide are structured do matter. The Topic heading plus summary condition was preferred although the results of the assessment of learning from the lecture did not suggest that this condition made learning easier. Data suggests that if the wording of the slides is appropriate, it can support listening. Too much text and attempts to read long passages are likely to distract from listening.

The amount of text and appropriateness of text on the slides is likely to be important as data suggests that participants often previewed points before the lecturer talked about them, as this helped them understand how points related to each other and helped them to “prepare” to listen and contextualise what they were about to hear. Background knowledge and topic familiarity were also pinpointed as factors likely to impact on ease of comprehension.

Pausing between points was also commented upon with one lecturer leaving very distinct pauses while the other did not. In some cases, the pauses were seen as useful, acting as a clear signal of the structure of the lecture or allowing participants time to process the previous point or read the forthcoming point. However, some participants found the pauses too long or disconcerting.

Participants expressed clear preferences about the lecturers’ voices or speaking style and Lecturer A’s voice was preferred to Lecturer C’s which was perceived as being less clear and the rate of speech as too slow.

Finally, several participants related what they learnt to their own experiences, sometimes because they were prompted by the lecturer, sometimes unprompted. This seems to have been effective as an aid to learning or comprehension for some participants, but it also had the potential to distract others.

We have reported the results of the eye-tracking, the assessment of learning and stimulated recall and interview data in response. In the next section, the results of the findings in relation to RQ1 and RQ2 will be discussed and the implications for language teaching and assessment will be outlined.

9 Discussion

This section draws together the results of RQ1 and RQ2. As noted in Section 3, the research questions were:

1. What discourse relations exist between the lecturer’s speech and the slide text in PowerPoint slide lectures?
2. How do students integrate auditory and textual information to develop a mental representation of the lecture content?

In answer to RQ1, this study has produced a taxonomy of the discourse relations which emerged from the analysis of five real-life lectures collected for the study. The taxonomy (see Section 6.1) revealed 10 discourse relations. The analysis suggests there are not only differences in terms of what the lecturer’s speech is doing in relation to the slide text (verbalising, conducting attention, elaborating etc.), but also distinct patterns of lecture delivery, such as sequences of lecture content exposition, or how teaching points are communicated (see Section 6.2).

In answer to RQ2, it seems that students use a variety of strategies to integrate auditory and textual information in lectures, and a range of factors impact the comprehensiveness and accuracy of the mental representation of the lecture content that students can build. A summary of the key findings are as follows.

• About $40\%$ of students’ fixations fixated on the bullet point which was being talked about when the fixation was made.
• B2 participants had a slightly higher percentage of correspondence between eye fixation and content of slides (42.87%) than the C1 participants (40.74%).
• Both the B2 and C1 participants had higher rates of correspondence for Lecture C than for Lecture A, albeit that the difference between Lecture A and C is less marked for C1 participants.
• All the participants achieved a more accurate and more complete coverage of learning from Lecture C than from Lecture A. On the whole, C1 participants scored more highly than B2 participants.

- The discourse patterns of Lecture C, including a clearer structure and more pauses between key points, helped with students’ processing of the auditory and textual input.

The key factors which emerged from RQ1 and RQ2, and the way they interact with each other, will now be discussed.

The main implications from the discourse analysis are that: in order to comprehend real-life academic lectures, students need to develop a high degree of adaptability to: (a) different forms of integrating aural and textual input (re: different discourse relations between lecturer’s speech and slide text); as well as to (b) different degrees of integration, for example, teaching points primarily in speech only (cf. ‘verbally communicating teaching points’) vs. teaching points in slide text, then elaborated in speech (cf. ‘translating’ and ‘elaborating’) vs. parallel discourse running through slide text and lecturer’s speech to be integrated and teaching points to be synthesised or inferred (cf. ‘combining’). It is noteworthy that such different degrees of integration are not just a matter of individual variation across lecturers, but could well exist within one single lecture.

The findings from RQ2 suggest that students also need to be prepared to cope with a range of voices, accents, and styles of delivery as this seemed to have the biggest impact on their ability to develop a complete and accurate mental model of a lecture. Nevertheless, students’ perceptions of which voices and styles of lectures are easier to understand may not concur with reality (cf. assessment of learning measures). Alternatively, it may be that students concentrate more when listening to a voice they find more challenging to understand resulting in better comprehension.

This issue of student perception and preference is also relevant for the amount of text included on lecture slides. Participants expressed a preference for, and perceived the elaborated condition (Topic heading + summary), as being easier than the Topic heading only condition. However, B2 level participants developed more complete and accurate mental models of the lecture when there was less text on the slide. This may be because attempting to read and listen concurrently seems to be challenging, particularly for B2 level students.

The findings that participants were able to generate a more complete and accurate account of Lecture C than of Lecture A might relate to the differences in patterns of lecture content exposition. Lecture C tended to follow quite a simple ‘topic-comment’ exposition pattern (see Section 6.2.2) while Lecture A tended to adopt a more complex sequential pattern of ‘verbalising/translating/elaborating/highlighting’ (see Section 6.2.1). It may be that the ‘topic-comment’ exposition pattern is less demanding when simultaneous processing of listening and reading information is taking place (see further discussion below). However, caution needs to be exercised in drawing any conclusions here, as Lectures A and C also differ in terms of their content and conceptual complexity, and the number of participants generating the results was very small.

The ‘topic-comment’ exposition pattern or communicating teaching points in speech only identified in the analysis (see Section 6.2) may have both positive and negative implications. Having just topic headings on the slide may encourage, if not demand, more attention from students to the lecturer’s oral exposition of the lecture, as relying on reading the PowerPoint slides would result in missing several key points or concepts from the lecture ⁷. The reduced amount of slide text to simultaneously decode while reading and listening might make it easier for students to focus their cognitive resources on information from one particular channel (listening) ⁸. However, in this Topic heading only condition, the overlap between slide text and speech becomes particularly important because, if some of the key terms (e.g., ‘characteristic’, ‘term’ from Lecture A

or ‘uniform’, ‘spaces’ from Lecture C) do not get decoded or recognised successfully in listening, it may result in a failure to comprehend the respective content (there is further discussion of this point below).

Proficiency in integrating listening and reading in lectures may also concern the ability to distinguish main points from supporting details or examples, not only within each of the audio and visual channels but across the two channels, and being able to selectively allocate attention or cognitive resources to a particular input channel when both audio and visual inputs are presented simultaneously. Having less slide text, along with it being clearly structured, may facilitate easier switching between channels or less competition for cognitive resources.

However, it is not just the quantity of text on the slide that appeared to have implications for lecture learning and comprehension. The extent to which the slide text and the lecturer’s speech overlap also appears to be an important factor. Participants reported finding it helpful and, at times, necessary for there to be an overlap of the key terms on the slide text and the wording used by the lecturer. It seems that for less proficient listeners, this overlap helps them ‘keep track’ of where they are in the lecture. A lack of overlap in slide text and the lecturer’s speech may be another factor which contributes to competition between processes as students try to establish the link between what they are hearing and what they are reading.

Another reason why the amount of text on slides and the pertinence of wording matters is that participants reported actively looking or scanning ahead to the next point or heading on the slide to help prime them for what they were about to hear leading to improved comprehension. Reviewing or scanning slide text structure can also help students to establish the relationships between points and this, in turn, may help them distinguish main points from supporting details.

An inability to understand and follow the structure of the lecture may lead to a less complete and accurate understanding. Pauses in the lecturer’s speech between teaching points can be useful for helping to indicate moves from one point on the slide to the next. In addition, pauses may offer a chance to parse or process one point and give the listener a chance to prepare to listen afresh. Pauses may also allow students time to read without the need to compromise their listening. However, in situations without the assurance of facial expression or body language, prolonged pauses that are unaccounted for could cause distraction.

Metacognitive strategies such as encouraging students to link lecture content to personal experience can be a useful strategy but also risks taking their concentration away from the lecture momentarily. Therefore, it may be necessary to give students a moment for that reflection before calling their attention back to the lecture when using this strategy.

Finally, topic familiarity and background knowledge inevitably have a role to play. Therefore, teaching of topics or points which include a lot of new terminology or new concepts are likely to be more challenging and may need a slower rate of delivery, more contextualisation, repetition or reinforcement.

9.1 The implications for teaching academic listening skills

Students need to be exposed to a range of accents and lecture styles in conjunction with slides with differing amounts of text. In pre-sessional or in-sessional EAP courses, reviewing a range of lectures and presentations and asking students to review and compare what the lecturer’s speech is doing in relation to the text (reiterating the same idea but using different words, giving more information, illustrating using an example, linking this point to another point etc.) may help students develop metacognitive strategies relevant to the lecture listening genre and build resilience. Lecture listening is not a passive activity, and preparing students to plan how they will engage with, and respond to, different lecture styles is likely to prove worthwhile.

While the findings of this study have suggested that ‘less is more’ in terms of the amount of text on lecture slides, certainly for students at B2 or lower, the likelihood is that, in real life, students will have to cope with varying amounts of slide text. Therefore, in EAP listening classes, it would be useful to discuss and practise techniques for coping when students are confronted with large amounts of text. One possible strategy would be resisting the temptation to try and read all the text, especially while listening. Instead, students could be encouraged to read only the headings while listening and make guesses or inferences about what the remaining text might say. Starting with minimal slide text and slowly increasing the amount of text to allow students to establish the point at which reading becomes counter-productive and compromises their ability to listen might prove a useful exercise, as the ‘sweet spot’ in the balance between reading and listening is likely to vary from one student to another.

Students could also practise identifying main points and supporting details to help them establish the structure of the lecture. Reviewing and discussing the format of the text on the slides may prove useful for this. Listening to identify signposting phrases and phrases referring to earlier slides/points may also be useful for establishing lecture structure.

Reviewing strategies that could be used to reduce the challenges posed by lecture listening is likely to prove useful. Strategies such as reading/scanning ahead to help make predictions or contextualise forthcoming content are important. Even a small amount of preparatory reading before a lecture can assist with contextualising lecture content and reducing the challenge posed by new concepts or terms. Helping students to recognise the benefits of preparatory reading could be valuable. Judging the relevance of one teaching point in relation to another, identifying key terms and concepts and guessing or inferring the meaning of unknown words may all assist with comprehension and boost listening confidence. Encouraging learners to link content to their own experience can also be useful but the risks of becoming distracted should also be emphasised.

Lecture listening, even for short periods, is likely to prove challenging for many students with English as a second language. Exposing students to the variety that represents real-life lecture listening and helping them develop a range of strategies to mitigate some of those challenges will be time-consuming but worthwhile.

For lecturers delivering content to students listening in a second language, the advice would be to try and limit the amount of text that appears on slides, and to ensure that the text includes key terms and concepts, especially when they are specialist or unfamiliar terms. It is also important that there is a clear overlap between the lecturer’s speech and the wording on the slides. It is sometimes tempting to avoid verbalising the text which appears on the slide and go straight to ‘translating’ (explaining the concept or point using alternative language) or ‘elaborating’ (adding more detail or information).

However, doing this may make following the lecture more difficult for some students, unless the lecturer takes the time to indicate very clearly which point is being referred to.

Short pauses between points or slides will offer students the opportunity to digest what they have heard and to read the upcoming point or slide. Pauses can also be helpful for signalling the progression from one point to the next within a slide. Signposting language should also be used to clarify progression through points on a slide, which can help students distinguish main points from supporting details.

Finally, a note on speech rate. Of course, a very rapid speech rate is unlikely to be helpful, but the findings suggest that an overly slow rate will not be perceived as helpful either. Field (2019: 63) suggests that the inclusion of more brief (half-second) pauses leads to greater compressibility for second-language listeners.

9.2 The implications for assessment of lecture listening

The results of this study suggest that the construct of lecture listening is many-faceted and that there is also wide variety within individual facets. The facets identified include the lecture content, the lecturer, the slide text and the discourse relations between the lecturer’s speech and the text on the slides.

The number of facets involved and their variability make developing a definition of the construct challenging. However, the study has given some insight into the way these facets may interact to affect comprehension difficulty.

The findings suggest that reading and listening simultaneously is challenging, even for students with well-developed language skills.

We argue that tests of lecture listening should present audio and slide text simultaneously as input, as real-life academic lectures would. The input would then need to cover a range of degrees of integration between slide text and the lecturer’s speech. So, for example, items which only have bullet point ‘topic headings’ on slides and deliver more detail verbally would need to be balanced with items providing the main teaching points textually which are then verbally elaborated. Where more information is provided in slide texts, expecting candidates to read while also listening is likely to increase item difficulty. Careful consideration would need to be given to the overlap between slide text and speech to ensure key terminology is covered, especially for specialist terms or infrequent vocabulary.

Tests of lecture listening should also cover a variety of discourse relations, and this report has provided a taxonomy of the types occurring in the sample of lectures collected by this study. It is beyond the remit of this study to establish whether the different types of discourse relations are all equally prevalent or whether some occur more frequently in academic lectures, but it would seem wise to include a range of the types listed in Appendix 1.

Distinguishing main points from subordinate ones and generally understanding the structure of the lecture is important and could form part of a test of lecture listening. Consideration would need to be given to the way moves from one point to the next are signalled and how links between main and subordinating points are signposted.

As with all language testing, background knowledge/topic familiarity has a role to play in task difficulty. Less familiar topics will increase the challenge, as will more abstract subjects or subjects with greater levels of conceptual complexity.

10 Limitations and further research

It must be acknowledged that this research made a detailed study of the context of a small sample of lectures and the processes of a small group of participants. There were also several factors manipulated in the delivery of the lectures (slide condition, lecturer and content) which means that findings must be considered as indicative. In any research there needs to be a compromise between the use of authentic materials, which by nature contain high variability, and the use of materials constructed for research, providing consistency, comparability and balance. In this study, it was felt that the use of authentic lecture extracts would provide valuable insights into the processes of students in academic lectures.

The use of eye-tracking offers detailed insight into the eye movements of participants; however, eye movements are not categoric evidence that reading is taking place. Eye-tracking is based on the eye-mind hypothesis (Just and Carpenter, 1980) which relies on two underlying assumptions: firstly, when we are fixating on a word (or looking at something), we are reading it (or thinking about it) and secondly that the amount of time we spend fixating on something reflects cognitive effort (more time fixating = more cognitive effort). Therefore, conclusions based on eye-tracking data need to be tentative, even when supported by stimulated recall data. This is because stimulated recall, like any form of self-reported data, may only represent a proportion of cognitive processes or strategies used by participants. Some processes are automated and beyond the percept of participants, and other thoughts and processes will defy attempts to recall them. This is a universal problem pertinent to any self-reported data.

However, complementary eye-tracking and stimulated recall data have, together, provided a rich source of data, which has enabled this study to draw indicative conclusions, shedding new insights into the actual behaviour of lecture listening and the use of strategies and processes that participants were aware of.

Following on from this study, further insight may be offered by isolating some of the facets involved in lecture listening and studying them in isolation. For example, studies to establish the frequency with which different discourse relations occur in different contexts would be useful. Controlling a limited number of different discourse relation patterns and assessing their impact on learning/understanding would offer valuable data for universities and test providers alike. Insight could also be achieved by manipulating the amount of slide text and assessing its impact on learning/understanding, especially for students with different levels of proficiency.

In conclusion, this study suggests that lecture listening is a complex process which can make arduous demands on students with English as a second language, as they attempt to juggle two streams of information. Acknowledging the range of factors which impact lecture comprehension is a meaningful first step in shaping measures to support such students.

References

• Adams, C. (2006). PowerPoint, habits of mind, and classroom culture. Journal of Curriculum Studies, 38, 389-411.
• Campoy-Cubillo, M.C. & Querol-Julián, M. (2015). Assessing multimodal listening. In B. Crawford and I. Fortanet-Gómez (Eds.) Multimodal Analysis in Academic Settings: From Research to Teaching (pp. 193-212). London/ New York: Routledge.
• Chan, S., & Latimer, N. (2020). The cognitive processes of B1, B2 and C1 students performing the Linguaskill extended listening and reading tasks: An eye-tracking investigation. Project report submitted to Cambridge Assessment English.
• Cop, U., Drieghe, D., & Duyck, W. (2015). Eye Movement Patterns in Natural Reading: A Comparison of Monolingual and Bilingual Reading of a Novel. PLoSONE, 10(8), e0134008. doi:10.1371/journal.pone.0134008
• Cumming, A. (2013). Assessing integrated writing tasks for academic purposes: Promises and perils. Language Assessment Quarterly, 10(1), 1-8.
• Field, J. (2019). Rethinking the Second Language Listening Test. Sheffield: Equinox.
• Field, J. (2020). Cyril Weir and Cognitive Validity. In L. Taylor and N. Saville (Eds), Studies in Language Testing 50: Lessons and Legacy: A Tribute to Professor Cyril. J. Weir (1950-2018) (pp. 54-82). CUP
• Gathercole, S. E., & Baddeley, A. (1993). Working Memory and Language. Hove: Erlbaum.
• Hallewell, M. J., & Crook, C. (2019). Performing PowerPoint lectures: Examining the extent of slide-text integration into lecturers’ spoken expositions. Journal of Further and Higher Education, 44(4), 467-482. DOI: 10.1080/0309877X.2019.1579895
• Just, M. A., & Carpenter, P.A., (1980). A theory of reading: from eye fixations to comprehension. Psychological Review, 87(4), 329.
• Knight, M. (2015). The ubiquitousness of PowerPoint. Business and Professional Communication Quarterly, 78, 271-272.
• Loewen, S., & Plonsky, L. (2017). An A-Z of Applied Linguistics Research Methods. Bloomsbury Publishing. http://refhub.elsevier.com/S0346-251X(23)00196-3/sref35
• Ockey, G. J. (2007). Construct implications of including still image or video in computer-based listening tests. Language Testing, 24(4), 517-537.
• O’Halloran, K. L., Tan, S., & Smith, B.A. (2016). Multimodal approaches to English for academic purposes. In K. Hyland and P. Shaw (Eds.) The Routledge Handbook of English for Academic Purposes (pp. 256-269). Routledge.
• Rayner, K., Pollatsek, A., Ashby, J., & Clifton, C. (2012). Psychology of Reading. 2nd edition. Hove: Psychology Press.
• Roberts, D. (2018). The engagement agenda, multimedia learning and the use of images in higher education lecturing: Or, how to end death by PowerPoint. Journal of Further and Higher Education, 42(7), 969-985.
• Shaw, G. P., & Molnar, D. (2011). Non-native English language speakers benefit most from the use of lecture capture in medical school. Biochemistry and Molecular Biology Education, 39(6), 416-420.
• Suvorov, R. (2015). The use of eye-tracking in research on video-based second language (L2) listening assessment: A comparison of context videos and content videos. Language Testing, 32(4), 463-483. https://doi.org/10.1177/0265532214562099
• Sweller, J., Ayers, P., & Kalyuga, S. (2011). Cognitive Load Theory. London: Springer.
• Wagner, E. (2010). The effect of the use of video texts on ESL listening test-taker performance. Language Testing, 27(4), 493-513.
• Yue, C., Bjork, E. L., & Bjork, R. A. (2013). Reducing verbal redundancy in multimedia learning: An undesired desirable difficulty? Journal of Educational Psychology, 105(2), 266-277.
• Zhao, S., & van Leeuwen, T. (2014). Understanding semiotic technology in university classrooms: A social semiotic approach to PowerPoint-assisted cultural studies lectures. Classroom Discourse, 5, 71-90.

Appendix 1: Coding scheme for discourse relations

Coding scheme for analysing discourse relations between lecturer’s speech and slide text

Appendix 2: Transcription of lecture clips

Lecture A Transcript

00:00:01 LECTURER A

So I think there are so many discussions happening nowadays about why terminology is important.

00:00:08 LECTURER A

We really have to think carefully about the choice of language that we use to describe individuals and describe some of the characteristics that they may embody or characteristics that are part of them.

00:00:21 LECTURER A

And I’m sure from discussions with peers and discussions that are being had in the media as well, you can think of some other examples.

00:00:29 LECTURER A

I think this is actually quite a hot topic. Is is questioning terminology that we use for different aspects of an individual.

00:00:37 LECTURER A

And the different characteristics that they might have.

00:00:39 LECTURER A

And this is.

00:00:39 LECTURER A

Also true in this area of applied linguistics.

00:00:43 LECTURER A

So some suggestions have been that there’s maybe more inherent fairness or appropriateness to use the term native speakers. Sorry, native users rather than native speakers or non-native speakers, and the reason that some people in this field like this idea of saying native users rather than native or non-native speakers.

00:01:05 LECTURER A

It’s first of all that it doesn’t just focus on speaking, of course, there are many different ways that you can be skilled or be proficient in a language, not just the spoken skills.

00:01:15 LECTURER A

So of course, people also read, hear, write, even sign language, and you, you might know people or maybe you know yourself in certain languages you have receptive, but perhaps not productive skills.

00:01:29 LECTURER A

So it may be the case that in some language or languages that you know, you feel quite confident being able to hear.

00:01:36 LECTURER A

And understand or read and understand that language, but not necessarily to be able to speak or to write that language.

00:01:42 LECTURER A

And we would call that receptive rather than a productive.

00:01:45 LECTURER A

Skills. So there are different ways that people.

00:01:47 LECTURER A

Can use and, and understand a language.

00:01:50 LECTURER A

And the term native user also emphasises the skills themselves that people have acquired, rather than the point at which they acquired them.

00:01:59 LECTURER A

So we saw at the beginning of the lecture that a common way that we might think of someone being a native speaker of a language as if they have been exposed to it from very, very early childhood.

00:02:10 LECTURER A

The term native user encourages people to not just think about that, but think about the actual language skills that someone has, rather than whether they’ve been using that language or exposed to that language since birth and early childhood.

00:02:23 LECTURER A

That becomes slightly less relevant with this term native user, and that is seen as an advantage of that particular term.

Lecture C Transcript

00:00:01 LECTURER C

What we have here are a list of the dimensions whereby speech and writing differ, so have a quick think about each one and think how.

00:00:13 LECTURER C

Speech and writing differ in these respects and I’ll add my comments after each one.

00:00:22 LECTURER C

The first one is quite simple and simply people do not speak in sentences. So when you transcribe.

00:00:31 LECTURER C

A conversation or somebody talking. Perhaps if you transcribe what I’m saying at the moment, put it down on paper, and then trying to find the sentence, you’ll find it enormously difficult. The sentence is simply not a unit of speech.

00:00:51 LECTURER C

The point here is that English on the page is extraordinarily uniform, so it glosses over differences and differences. Just don’t make their appearance on the written page. You don’t hear differences in pronunciation, you don’t hear or read differences in. In grammar, it’s all uniform.

00:01:08 LECTURER C

But in the spoken world, English is enormously varied.

00:01:17 LECTURER C

Written English has a little technological device called spaces. The spaces between words so they become discrete on the page.

00:01:26 LECTURER C

Nice clear boundaries. But if you analyse anybody’s spoken language, words just flow into each other and they’re not discrete, they’re all, they all come in one.

00:01:42 LECTURER C

When you’re talking to somebody, typically you know who you’re talking to. You know what information that they know and they know what information that you know when you’re writing, you’re writing to somebody who is simply not there, or you may not even know who, who is going to be reading it. And in that context, free situation. Then you.

00:02:02 LECTURER C

As a writer, have to supply the context and that is what produces much more textual density on the written page.

Slide versions used to accompany Lecture A

Figure 11: Lecture A Topic Heading Only slide

Figure 12: Lecture A Topic Heading + Summary slide

Slide versions used to accompany Lecture C

Figure 13: Lecture C Topic Heading only slide

Figure 14: Lecture C Topic Heading + Summary slide

Author biodata

Dr Nicola Latimer is a research fellow at The Centre for Research in English Language Learning and Assessment (CRELLA) at the University of Bedfordshire. She has 20 years of experience in teaching and assessing English as a second language. Nicola completed her PhD at CRELLA in 2019 and since then has worked on a wide range of research projects. Her research interests include the assessment of reading, writing and listening, and assessments that integrate these skills. Many of Nicola’s research projects use technological methods such as eye-tracking and keystroke logging to offer insight into test-takers’ processes. Nicola’s work has been published in Studies in Language Testing and The International Journal of TESOL. Nicola is currently co-editing a special issue of The International Journal of TESOL (with Sathena Chan) discussing how eye-tracking technology is being used to inform second language teaching and assessment.

Dr Chihiro Inoue is an Associate Professor of Language Assessment at the University of Bedfordshire. She specialises in the assessment of speaking and listening, with keen interest in the relationship between test-taker characteristics, task design, and learner language, as well as test accommodations which bring the three together. In addition to leading a number of funded projects in the fields of language testing and assessment, she regularly gives invited talks for policymakers, in-service teachers, graduate students and testing organisations. Her publications have appeared in peer-reviewed journals such as Language Assessment Quarterly, Assessment in Education, Language Learning Journal, and Assessing Writing.

Dr Sathena Chan is an Associate Professor in Language Assessment. She specialises in large-scale language assessment and L2 cognitive processing. She has won and conducted over 30 externally funded projects as PI on language assessment. She is the founder and editor-in-chief of Research Synthesis in Applied Linguistics (T&F) and on the editorial board of Innovation of Language Learning and Teaching. She is the Special Interest Group (SIG) Co-ordinator of the British Association of Applied Linguistics (BAAL) and Chair of the Nomination and Membership Committee of the UK Association for Language Testing and Assessment. She is the immediate past Chair of the BAAL Testing, Evaluation and Assessment Special Interest Group. Her books include Defining Integrated Reading-into-Writing Constructs and Research and Practice in Assessing Academic English (both Cambridge University Press). She has written book chapters on topics such as Differences in L2 Reading and Listening (Routledge) and Construct Definition at the Heart of Language Assessment (Cambridge University Press).

Daniel MK Lam is Lecturer in TESOL at University of Glasgow. His research interests include interactive listening and interactional competence, the relationship between assessment, feedback and learning, and language assessment literacy in university admission contexts. Daniel’s work can be found in journals such as Applied Linguistics, ELT Journal, and Language Testing, as well as more recently, in the Routledge Handbook of Second Language Acquisition and Listening. In his former post at CRELLA, University of Bedfordshire, Daniel collaborated with colleagues on various research projects on Aptis, Cambridge English Qualifications, IELTS, and TOEFL iBT, the research reports of which can be located via his staff website: www.gla.ac.uk/schools/education/staff/daniellam/

Footnotes

1. Cognitive validity refers to the extent to which a test task elicits the same cognitive processes that would be demanded of the test-taker in life beyond the test (Field, 2020). ↩

2. Ecological validity refers to the degree of similarity between the environment in which a test task takes place and the authentic context in which the test user would be expected to operate in life beyond the test (Loewen & Plonsky, 2017). ↩

3. In lecture discourse, the lecturer may, of course, draw students’ attention to different elements of the lecture, such as a particular concept or idea (see Highlighting teaching point below), or a next part/stage of the lecture. Here, our focus is primarily on the lecturer drawing students’ attention to slide-text elements. ↩

4. which has been reported by students as having an effect on comprehension (see findings from stimulated recall in Phase 2 pilot study) ↩

5. as an audio clip placed directly next to the slide heading ‘Japanese’ in the lecture PowerPoint presentation. ↩

6. the parenthesis indicates an optional action that occurs in some lecture discourse units ↩

7. In our Phase 2 pilot study, some participants reported treating the lecturer’s speech as expounding on main points already on the slide-text only. ↩

8. Findings from students’ stimulated recall in the pilot study corroborate this. ↩