Search Results for author:
Found 35 papers, 18 papers with code
Tutorial on Multimodal Machine Learning
Multimodal machine learning involves integrating and modeling information from multiple heterogeneous sources of data.
PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference
To address this, we propose PromptDistill, a novel, training-free method that improves inference efficiency while preserving generation quality.
Hi5: 2D Hand Pose Estimation with Zero Human Annotation
We propose a new large synthetic hand pose estimation dataset, Hi5, and a novel inexpensive method for collecting high-quality synthetic data that requires no human annotation or validation.
Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding
To mitigate this issue, parameter-efficient transfer learning algorithms, such as adapters and prefix tuning, have been proposed as a way to introduce a few trainable parameters that can be plugged into large pre-trained language models such as BERT, and HuBERT.
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
With the recent interest in video understanding, embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, multimodal machine learning has brought unique computational and theoretical challenges to the machine learning community given the heterogeneity of data sources and the interconnections often found between modalities.
Face-to-Face Contrastive Learning for Social Intelligence Question-Answering
Creating artificial social intelligence - algorithms that can understand the nuances of multi-person interactions - is an exciting and emerging challenge in processing facial expressions and gestures from multimodal videos.
Relay Variational Inference: A Method for Accelerated Encoderless VI
We find RVI to be a unique tool, often superior in both performance and convergence speed to previously proposed encoderless as well as amortized VI models (e. g. VAE).
M2H2: A Multimodal Multiparty Hindi Dataset For Humor Recognition in Conversations
We propose several strong multimodal baselines and show the importance of contextual and multimodal information for humor recognition in conversations.
Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Multimodal sentiment analysis aims to extract and integrate semantic information collected from multiple modalities to recognize the expressed emotions and sentiment in multimodal data.
StarNet: Gradient-free Training of Deep Generative Models using Determined System of Linear Equations
In this paper we present an approach for training deep generative models solely based on solving determined systems of linear equations.
MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences
Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed.
Multimodal Research in Vision and Language: A Review of Current and Emerging Trends
Deep Learning and its applications have cascaded impactful research and development with a diverse range of modalities present in the real-world data.
What Gives the Answer Away? Question Answering Bias Analysis on Video QA Datasets
In this paper, we analyze QA biases in popular video question answering datasets and discover pretrained language models can answer 37-48% questions correctly without using any multimodal context information, far exceeding the 20% random guess baseline for 5-choose-1 multiple-choice questions.
Improving Aspect-Level Sentiment Analysis with Aspect Extraction
Aspect-based sentiment analysis (ABSA), a popular research area in NLP has two distinct parts -- aspect extraction (AE) and labeling the aspects with sentiment polarity (ALSA).
Pseudo-Encoded Stochastic Variational Inference
Posterior inference in directed graphical models is commonly done using a probabilistic encoder (a. k. a inference model) conditioned on the input.
Factorized Multimodal Transformer for Multimodal Sequential Learning
As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental research area.
WildMix Dataset and Spectro-Temporal Transformer Model for Monoaural Audio Source Separation
To this end, we introduce a novel trasnformer-based model called Spectro-Temporal Transformer (STT).
Integrating Multimodal Information in Large Pretrained Transformers
It does so by generating a shift to internal representation of BERT and XLNet; a shift that is conditioned on the visual and acoustic modalities.
Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence
Human language offers a unique unconstrained approach to probe through questions and reason through answers about social situations.
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
The dataset and accompanying studies, present a framework in multimodal humor detection for the natural language processing community.
Variational Auto-Decoder: A Method for Neural Generative Modeling from Incomplete Data
We study a specific implementation of the Auto-Encoding Variational Bayes (AEVB) algorithm, named in this paper as a Variational Auto-Decoder (VAD).
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Humans convey their intentions through the usage of both verbal and nonverbal behaviors during face-to-face communication.
Multimodal Language Analysis with Recurrent Multistage Fusion
In this paper, we propose the Recurrent Multistage Fusion Network (RMFN) which decomposes the fusion problem into multiple stages, each of them focused on a subset of multimodal signals for specialized, effective fusion.
Learning Factorized Multimodal Representations
Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction.
Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos
Emotion recognition in conversations is crucial for the development of empathetic machines.
Ranked #57 on
Emotion Recognition in Conversation
on IEMOCAP
Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
Previous research in this field has exploited the expressiveness of tensors for multimodal representation.
Memory Fusion Network for Multi-view Sequential Learning
In this paper, we present a new neural architecture for multi-view sequential learning called the Memory Fusion Network (MFN) that explicitly accounts for both interactions in a neural architecture and continuously models them through time.
Multi-attention Recurrent Network for Human Communication Comprehension
AI must understand each modality and the interactions between them that shape human communication.
Ranked #10 on
Multimodal Sentiment Analysis
on MOSI
Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning
In this paper, we propose the Gated Multimodal Embedding LSTM with Temporal Attention (GME-LSTM(A)) model that is composed of 2 modules.
Tensor Fusion Network for Multimodal Sentiment Analysis
Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language.
Context-Dependent Sentiment Analysis in User-Generated Videos
Multimodal sentiment analysis is a developing area of research, which involves the identification of sentiments in videos.
Ranked #3 on
Emotion Recognition in Conversation
on CPED
Emotion Recognition in Conversation
General Classification
+4
Combating Human Trafficking with Multimodal Deep Models
Human trafficking is a global epidemic affecting millions of people across the planet.
Combating Human Trafficking with Deep Multimodal Models
Human trafficking is a global epidemic affecting millions of people across the planet.
Convolutional Experts Constrained Local Model for Facial Landmark Detection
In our work, we present a novel local detector -- Convolutional Experts Network (CEN) -- that brings together the advantages of neural architectures and mixtures of experts in an end-to-end framework.
MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
This paper introduces to the scientific community the first opinion-level annotated corpus of sentiment and subjectivity analysis in online videos called Multimodal Opinion-level Sentiment Intensity dataset (MOSI).
