Multimodal representations used in this entry are “the depiction or

communication of an idea or ideas using more than a single expressive mode,

either in synchrony or separately.” Because education has traditionally been so

strongly mediated by written and spoken language, an important aspect of

Multimodal projects are simply projects that have multiple “modes” of communicating a message. For example, while traditional papers typically

only have one mode (text), a multimodal project would include a combination of

text, images, motion, or audio. Multimodality is the application of

multiple literacies within one medium. For example, understanding a televised weather

forecast (medium) involves understanding spoken language, written language,

specific language (such as temperature scales), geography, and symbols (clouds,

sun, rain, etc.).

Multimodal learning suggests that when a number of

our senses – visual, auditory, and kinaesthetic – are engaged during

learning, we understand and remember more. By combining these modes, learners experience

learning in a variety of ways to create a diverse learning style.

Four core concepts are common across multimodal research: mode, semiotic resource, modal affordance and inter-semiotic relations.

Overview of multimodal literacy

videos with a speech in one language and subtitles in another.
bilingual picture books with text in multiple languages.
instruction manuals with the information presented in pictures and translated into multiple languages.

Multimodal refers to the integration of multiple modes of communication and expression, which can be perceived by senses such as sight, hearing, and touch. Using multiple modes of communication helps convey information to your audience. Multimodal learning in education means teaching concepts using multiple modes. Modes are channels of information or anything that communicates meaning in some way, including Pictures. Illustrations. Audio.

Knowledge representation learning (KRL) encodes enormous structured information with entities and relations into a continuous low-dimensional semantic space. Most conventional methods solely focus on learning knowledge representation from a single modality, yet neglect the complementary information from others. The more and more rich available multi-modal data on the Internet also drive us to explore a novel approach for KRL in a multi-modal way and overcome the limitations of previous single-modal based methods. This paper proposes a novel multi-modal knowledge representation learning (MM-KRL) framework which attempts to handle knowledge from both textual and visual modal web data. It consists of two stages, i.e., Weebly-supervised multi-modal relationship mining, and bi-enhanced cross-modal knowledge representation learning. Compared with existing knowledge representation methods, our framework has several advantages: (1) It can effectively mine multi-modal knowledge with structured textual and visual relationships from the web automatically. (2) It can learn a common knowledge space which is independent of both task and modality by the proposed Bi-enhanced Cross-modal Deep Neural Network (BC-DNN). (3) It can represent unseen multi-modal relationships by transferring the learned knowledge with isolated seen entities and relations into unseen relationships. We build a large-scale multi-modal relationship dataset (MMR-D) and the experimental results show that our framework achieves excellent performance in zero-shot multi-modal retrieval and visual relationship recognition.