Multimodal Deep Learning

62 papers with code • 1 benchmarks • 15 datasets

Multimodal deep learning is a type of deep learning that combines information from multiple modalities, such as text, image, audio, and video, to make more accurate and comprehensive predictions. It involves training deep neural networks on data that includes multiple types of information and using the network to make predictions based on this combined data.

One of the key challenges in multimodal deep learning is how to effectively combine information from multiple modalities. This can be done using a variety of techniques, such as fusing the features extracted from each modality, or using attention mechanisms to weight the contribution of each modality based on its importance for the task at hand.

Multimodal deep learning has many applications, including image captioning, speech recognition, natural language processing, and autonomous vehicles. By combining information from multiple modalities, multimodal deep learning can improve the accuracy and robustness of models, enabling them to perform better in real-world scenarios where multiple types of information are present.

Most implemented papers

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

opengvlab/llama-adapter 28 Mar 2023

We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model.

ShapeWorld - A new test methodology for multimodal language understanding

AlexKuhnle/ShapeWorld 14 Apr 2017

We introduce a novel framework for evaluating multimodal deep learning models with respect to their language understanding and generalization abilities.

Multimodal deep networks for text and image-based document classification

Quicksign/ocrized-text-dataset 15 Jul 2019

Classification of document images is a critical step for archival of old manuscripts, online subscription and administrative procedures.

Are These Birds Similar: Learning Branched Networks for Fine-grained Representations

nicolalandro/ntsnet-cub200 16 Jan 2020

In recent years, natural language descriptions are used to obtain information on discriminative parts of the object.

aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception

aimotive/aimotive_dataset 17 Nov 2022

The dataset consists of 176 scenes with synchronized and calibrated LiDAR, camera, and radar sensors covering a 360-degree field of view.

Multimodal Deep Learning for Robust RGB-D Object Recognition

isrugeek/robotics_recognition 24 Jul 2015

Robust object recognition is a crucial ingredient of many, if not all, real-world robotics applications.

Supervised Video Summarization via Multiple Feature Sets with Parallel Attention

TIBHannover/MSVA 23 Apr 2021

The proposed architecture utilizes an attention mechanism before fusing motion features and features representing the (static) visual content, i. e., derived from an image classification model.

A multimodal deep learning framework for scalable content based visual media retrieval

ambareeshravi/media_retrieval 18 May 2021

We propose a novel, efficient, modular and scalable framework for content based visual media retrieval systems by leveraging the power of Deep Learning which is flexible to work both for images and videos conjointly and we also introduce an efficient comparison and filtering metric for retrieval.

Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

declare-lab/multimodal-deep-learning 28 Jul 2021

Multimodal sentiment analysis aims to extract and integrate semantic information collected from multiple modalities to recognize the expressed emotions and sentiment in multimodal data.

XFlow: Cross-modal Deep Neural Networks for Audiovisual Classification

catalina17/XFlow 2 Sep 2017

Our work improves on existing multimodal deep learning algorithms in two essential ways: (1) it presents a novel method for performing cross-modality (before features are learned from individual modalities) and (2) extends the previously proposed cross-connections which only transfer information between streams that process compatible data.