Multimodality study in English Language Education.pptx

Multimodality

1. Multimodality

2. Multimodality in communication and education refers to the use of multiple modes—such as text, images, gestures, sound, and digital media—to facilitate learning and meaning-making. This approach aligns with socio-semiotic perspectives, which suggest that meaning is constructed through diverse semiotic resources rather than just linguistic symbols.

3. Multimodality became a central topic in Literacy Studies from the 1990s, shaped, firstly, by Gunther Kress's work. As Gee (2001) described it, distinct sites of social practice or “semiotic domains,” as he named them, including first person-shooter-video games, advertising, cellular biology, midwifery and surgery incorporated multiple modalities (e.g., oral or written languages, images, equations, symbols, sounds, gestures, graphs, artifacts, and so forth) to communicate distinctive types of messages. Multimodal, trans-semiotic literacy and language studies have grown exponentially since

4. Multimodality is frequently linked to multiliteracies (Mills, 2009), new literacies (Albers and Harste , 2007), and New Literacy Studies (NLS) ( Rowsell et al., 2019). Regardless of the theoretical frameworks used, multimodality considers the ways in which people draw upon all modes to make meanings (e.g., combining images with print, using a gesture, using a gesture with a word and an image and so on).

5. M ultimodality is the study of interrelationships and interdependence between different communicative modes, no matter whether they are written or oral, visual or auditory. It is also a way to transcribe the meaning of discourses composed of different semiotic modes. Multimodality does not designate a pre-given entity or text-type .

6. In their original publication, the NLG (1996) posited that “One of the key ideas informing the notion of multiliteracies is the increasing complexity and inter-relationship of different modes of meaning” (p. 78). They indicated modes pertaining to linguistic, visual, audio, gestural, spatial, and multimodal design (see Fig. 1). The NLG differentiated multimodal design from other modes noting: “Multimodal Design is of a different order to the other five modes of meaning; it represents the patterns of interconnection among the other modes” (NLG, 1996, p. 78).

7. Theoretical Frameworks of Multimodality Social Semiotics (Kress & Van Leeuwen , 2001) : Explores how meaning is created through different modes and their cultural/social contexts. Multimodal Discourse Analysis : Analyzes how various modes interact in communication (e.g., analyzing a news broadcast's verbal and visual elements). Cognitive Theories : Examines how people process multimodal information and how different modes influence comprehension and memory.

8. Multimodality refers to the interplay between different representational modes, for instance, between images and written/spoken word. Multimodal representations mediate the sociocultural ways in which these modes are combined in the communication process (Kress & Van Leeuwen 2001, p. 20).

9. Multimodality in Communication Expanding Meaning-Making Beyond Text Multimodal communication emphasizes that meaning is created not only through language but also through visual, auditory, spatial, and gestural elements. Studies highlight that incorporating images, videos, and interactive digital tools enhances comprehension and engagement in various contexts ( Bezemer & Kress, 2016). Digital and Technological Influence The rise of digital media has transformed communication into a dynamic, multimodal experience where users combine video, sound, and text. Research suggests that in online learning environments, the use of multimodal strategies improves learner engagement and understanding ( Jewitt , 2009).

10. Multimodality in Education Enhancing Student Learning and Engagement Studies have shown that multimodal teaching strategies—such as integrating images, gestures, and sound—support students in constructing knowledge more effectively. For example, when teachers use visual aids alongside verbal explanations, comprehension increases (Archer & Breuer, 2016). Support for Diverse Learners Multimodal approaches cater to different learning styles, making education more inclusive. Students with disabilities, such as dyslexia, benefit from multimodal texts that provide auditory or visual alternatives to traditional reading (Cope & Kalantzis , 2009). Role in Digital Learning Environments Online learning platforms increasingly incorporate multimodal features, such as interactive videos, gamification, and virtual reality, to enhance engagement. Research suggests that such tools facilitate deeper learning and retention (Gee, 2004).

11. More recent sociocognitive approaches to literacy have increased the purview of texts with multimodality and with New Literacy Studies. Picturebooks , textbooks, websites, screen-based apps, digital texts, museum exhibits, television programs, and billboards are just a few examples of the wide variety of multimodal texts young readers experience in contemporary society.

12. Kress (2015) uses the term multimodal ensemble to refer to “a designed complex of different modes” (p. 57) and emphasizes how “the meaning of the whole arises out of the contribution of each part in its interaction with all other parts” (p. 57 ). In general, multimodality and social semiotics are concerned with how people generate meanings in transaction with multimodal texts, including written language, visual images, and design features, from a variety of perspectives to meet the requirements of particular social contexts ( Serafini , 2014).