Search Results for author:
Found 136 papers, 74 papers with code
Convolutional Experts Constrained Local Model for Facial Landmark Detection
In our work, we present a novel local detector -- Convolutional Experts Network (CEN) -- that brings together the advantages of neural architectures and mixtures of experts in an end-to-end framework.
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos
Emotion recognition in conversations is crucial for the development of empathetic machines.
Ranked #51 on
Emotion Recognition in Conversation
on IEMOCAP
Think Locally, Act Globally: Federated Learning with Local and Global Representations
To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors.
Ranked #5 on
Multimodal Sentiment Analysis
on MOSI
Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Multimodal sentiment analysis aims to extract and integrate semantic information collected from multiple modalities to recognize the expressed emotions and sentiment in multimodal data.
MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning
Learning multimodal representations involves integrating information from multiple heterogeneous sources of data.
MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
This paper introduces to the scientific community the first opinion-level annotated corpus of sentiment and subjectivity analysis in online videos called Multimodal Opinion-level Sentiment Intensity dataset (MOSI).
Context-Dependent Sentiment Analysis in User-Generated Videos
Multimodal sentiment analysis is a developing area of research, which involves the identification of sentiments in videos.
Ranked #3 on
Emotion Recognition in Conversation
on CPED
Emotion Recognition in Conversation
General Classification
+4
Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
Previous research in this field has exploited the expressiveness of tensors for multimodal representation.
Integrating Multimodal Information in Large Pretrained Transformers
It does so by generating a shift to internal representation of BERT and XLNet; a shift that is conditioned on the visual and acoustic modalities.
Tensor Fusion Network for Multimodal Sentiment Analysis
Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language.
Video Relationship Reasoning using Gated Spatio-Temporal Energy Graph
Visual relationship reasoning is a crucial yet challenging task for understanding rich interactions across visual concepts.
Speaker-Follower Models for Vision-and-Language Navigation
We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction.
Memory Fusion Network for Multi-view Sequential Learning
In this paper, we present a new neural architecture for multi-view sequential learning called the Memory Fusion Network (MFN) that explicitly accounts for both interactions in a neural architecture and continuously models them through time.
Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning
In this paper, we propose the Gated Multimodal Embedding LSTM with Temporal Attention (GME-LSTM(A)) model that is composed of 2 modules.
Multi-attention Recurrent Network for Human Communication Comprehension
AI must understand each modality and the interactions between them that shape human communication.
Ranked #9 on
Multimodal Sentiment Analysis
on MOSI
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Our method is based on the key insight that translation from a source to a target modality provides a method of learning joint representations using only the source modality as input.
High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on robots.
MultiViz: Towards Visualizing and Understanding Multimodal Models
How can we visualize the internal modeling of multimodal interactions in these models?
Diverse and Admissible Trajectory Forecasting through Multimodal Context Understanding
Multi-agent trajectory forecasting in autonomous driving requires an agent to accurately anticipate the behaviors of the surrounding vehicles and pedestrians, for safe and reliable decision-making.
Diverse and Admissible Trajectory Prediction through Multimodal Context Understanding
Multi-agent trajectory forecasting in autonomous driving requires an agent to accurately anticipate the behaviors of the surrounding vehicles and pedestrians, for safe and reliable decision-making.
A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
In this report, we relate the algorithmic design of Barlow Twins' method to the Hilbert-Schmidt Independence Criterion (HSIC), thus establishing it as a contrastive learning approach that is free of negative samples.
Learning Factorized Multimodal Representations
Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction.
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Humans convey their intentions through the usage of both verbal and nonverbal behaviors during face-to-face communication.
Language2Pose: Natural Language Grounded Pose Forecasting
In this paper, we address this multimodal problem by introducing a neural architecture called Joint Language to Pose (or JL2P), which learns a joint embedding of language and pose.
StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer
Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes (e. g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence.
Towards Debiasing Sentence Representations
As natural language processing methods are increasingly deployed in real-world scenarios such as healthcare, legal systems, and social science, it becomes necessary to recognize the role they potentially play in shaping social biases and stereotypes.
Towards Understanding and Mitigating Social Biases in Language Models
As machine learning methods are deployed in real-world settings such as healthcare, legal systems, and social science, it is crucial to recognize how they shape social biases and stereotypes in these sensitive decision-making processes.
Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel
This new formulation gives us a better way to understand individual components of the Transformer's attention, such as the better way to integrate the positional embedding.
Factorized Contrastive Learning: Going Beyond Multi-view Redundancy
How can we learn self-supervised multimodal representations to capture both shared and unique information relevant to downstream tasks?
MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences
Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed.
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and evaluate their social intelligence.
Self-supervised Learning from a Multi-view Perspective
In particular, we propose a composite objective that bridges the gap between prior contrastive and predictive learning objectives, and introduce an additional objective term to discard task-irrelevant information.
Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling
We evaluate this family on new LRU models on computational convergence rates and statistical efficiency.
Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment
To tackle this problem, we show that relation alignment can be enforced by encouraging the directed language attention from 'mug' to 'grass' (capturing the semantic relation 'in') to match the directed visual attention from the mug to the grass.
Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications
We propose two lower bounds based on the amount of shared information between modalities and the disagreement between separately trained unimodal classifiers, and derive an upper bound through connections to approximate algorithms for min-entropy couplings.
Multimodal Fusion Interactions: A Study of Human and Automatic Quantification
In order to perform multimodal fusion of heterogeneous signals, we need to understand their interactions: how each modality individually provides information useful for a task and how this information changes in the presence of other modalities.
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities.
To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations
In this paper, we introduce a neural architecture named Dyadic Residual-Attention Model (DRAM), which integrates intrapersonal (monadic) and interpersonal (dyadic) dynamics using selective attention to generate sequences of body pose conditioned on audio and body pose of the interlocutor and audio of the human operating the avatar.
Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach
A key challenge, called gesture style transfer, is to learn a model that generates these gestures for a speaking agent 'A' in the gesturing style of a target speaker 'B'.
Deep Gamblers: Learning to Abstain with Portfolio Theory
We deal with the \textit{selective classification} problem (supervised-learning problem with a rejection option), where we want to achieve the best performance at a certain level of coverage of the data.
Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides
As a step toward developing AI to aid in student learning as intelligent teacher assistants, we introduce the Multimodal Lecture Presentations dataset as a large-scale benchmark testing the capabilities of machine learning models in multimodal understanding of educational content.
Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis
In this paper, we propose a Select-Additive Learning (SAL) procedure that improves the generalizability of trained neural networks for multimodal sentiment analysis.
Temporal Attention-Gated Model for Robust Sequence Classification
An important advantage of our approach is interpretability since the temporal attention weights provide a meaningful value for the salience of each time step in the sequence.
No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures
We study relationships between spoken language and co-speech gestures in context of two key challenges.
Neural Methods for Point-wise Dependency Estimation
Since its inception, the neural estimation of mutual information (MI) has demonstrated the empirical success of modeling expected dependency between high-dimensional random variables.
Self-supervised Representation Learning with Relative Predictive Coding
This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance.
Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis
The human language can be expressed through multiple sources of information known as modalities, including tones of voice, facial gestures, and spoken language.
M2H2: A Multimodal Multiparty Hindi Dataset For Humor Recognition in Conversations
We propose several strong multimodal baselines and show the importance of contextual and multimodal information for humor recognition in conversations.
Learning Weakly-Supervised Contrastive Representations
The first stage is to cluster data according to its auxiliary information.
Using Syntax to Ground Referring Expressions in Natural Images
We introduce GroundNet, a neural network for referring expression recognition -- the task of localizing (or grounding) in an image the object referred to by a natural language expression.
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis
In particular, there are various considerations behind the pipeline: (1) the choice and (2) the size of PLM, (3) the choice of uncertainty quantifier, (4) the choice of fine-tuning loss, and many more.
Refer360$^\circ$: A Referring Expression Recognition Dataset in 360$^\circ$ Images
We propose a novel large-scale referring expression recognition dataset, Refer360{\mbox{$^\circ$}}, consisting of 17, 137 instruction sequences and ground-truth actions for completing these instructions in 360{\mbox{$^\circ$}} scenes.
Strong and Simple Baselines for Multimodal Utterance Embeddings
Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations.
PACS: A Dataset for Physical Audiovisual CommonSense Reasoning
Our paper takes a step towards real-world physical commonsense reasoning by contributing PACS: the first audiovisual benchmark annotated for physical commonsense attributes.
Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities
Human interactions are deeply rooted in the interplay of thoughts, beliefs, and desires made possible by Theory of Mind (ToM): our cognitive ability to understand the mental states of ourselves and others.
Crossmodal clustered contrastive learning: Grounding of spoken language to gesture
Crossmodal grounding is a key challenge for the task of generating relevant and well-timed gestures from just spoken language as an input.
Conditional Contrastive Learning with Kernel
Conditional contrastive learning frameworks consider the conditional sampling procedure that constructs positive or negative data pairs conditioned on specific variables.
DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations
The ability for a human to understand an Artificial Intelligence (AI) model's decision-making process is critical in enabling stakeholders to visualize model behavior, perform model debugging, promote trust in AI models, and assist in collaborative human-AI decision-making.
Understanding Masked Autoencoders via Hierarchical Latent Variable Models
In this work, we formally characterize and justify existing empirical insights and provide theoretical guarantees of MAE.
Visual Referring Expression Recognition: What Do Systems Actually Learn?
We present an empirical analysis of the state-of-the-art systems for referring expression recognition -- the task of identifying the object in an image referred to by a natural language expression -- with the goal of gaining insight into how these systems reason about language and vision.
Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
In this work, we propose algorithms for cross-modal generalization: a learning paradigm to train a model that can (1) quickly perform new tasks in a target modality (i. e. meta-learning) and (2) doing so while being trained on a different source modality.
On Emergent Communication in Competitive Multi-Agent Teams
Several recent works have found the emergence of grounded compositional language in the communication protocols developed by mostly cooperative multi-agent systems when learned end-to-end to maximize performance on a downstream task.
Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
As users increasingly rely on cloud-based computing services, it is important to ensure that uploaded speech data remains private.
Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
Existing techniques for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions.
Counterfactual Augmentation for Multimodal Learning Under Presentation Bias
In real-world machine learning systems, labels are often derived from user behaviors that the system wishes to encourage.
Text-Transport: Toward Learning Causal Effects of Natural Language
To address this issue, we leverage the notion of distribution shift to describe an estimator that transports causal effects between domains, bypassing the need for strong assumptions in the target domain.
SenteCon: Leveraging Lexicons to Learn Human-Interpretable Language Representations
Moreover, we find that SenteCon outperforms existing interpretable language representations with respect to both its downstream performance and its agreement with human characterizations of the text.
Multimodal Language Analysis with Recurrent Multistage Fusion
In this paper, we propose the Recurrent Multistage Fusion Network (RMFN) which decomposes the fusion problem into multiple stages, each of them focused on a subset of multimodal signals for specialized, effective fusion.
Difference-Masking: Choosing What to Mask in Continued Pretraining
The self-supervised objective of masking-and-predicting has led to promising performance gains on a variety of downstream tasks.
Neural Mixed Effects for Nonlinear Personalized Predictions
Empirically, we observe that NME improves performance across six unimodal and multimodal datasets, including a smartphone dataset to predict daily mood and a mother-adolescent dataset to predict affective state sequences where half the mothers experience at least moderate symptoms of depression.
Comparative Knowledge Distillation
We observe that prevalent KD techniques and state of the art data augmentation strategies fall short in this constrained setting.
Hand2Face: Automatic Synthesis and Recognition of Hand Over Face Occlusions
We then propose a model for facial occlusion type recognition to differentiate between hand over face occlusions and other types of occlusions such as scarves, hair, glasses and objects.
Multimodal Machine Learning: A Survey and Taxonomy
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors.
Preserving Intermediate Objectives: One Simple Trick to Improve Learning for Hierarchical Models
In this work, we focus on improving learning for such hierarchical models and demonstrate our method on the task of speaker trait prediction.
Combating Human Trafficking with Deep Multimodal Models
Human trafficking is a global epidemic affecting millions of people across the planet.
GazeDirector: Fully Articulated Eye Gaze Redirection in Video
We present GazeDirector, a new approach for eye gaze redirection that uses model-fitting.
Affect-LM: A Neural Language Model for Customizable Affective Text Generation
Human verbal communication includes affective messages which are conveyed through use of emotionally colored words.
Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks
Curriculum Learning emphasizes the order of training instances in a computational learning setup.
Learning Representations of Affect from Speech
Experiments on a well-established real-life speech dataset (IEMOCAP) show that the learnt representations are comparable to state of the art feature extractors (such as voice quality features and MFCCs) and are competitive with state-of-the-art approaches at emotion and dimensional affect recognition.
Relative Facial Action Unit Detection
This paper presents a subject-independent facial action unit (AU) detection method by introducing the concept of relative AU detection, for scenarios where the neutral face is not provided.
Multimodal Polynomial Fusion for Detecting Driver Distraction
Distracted driving is deadly, claiming 3, 477 lives in the U. S. in 2015 alone.
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Analyzing human multimodal language is an emerging area of research in NLP.
Ranked #11 on
Multimodal Sentiment Analysis
on CMU-MOSEI
(using extra training data)
Combating Human Trafficking with Multimodal Deep Models
Human trafficking is a global epidemic affecting millions of people across the planet.
Multimodal Machine Learning: Integrating Language, Vision and Speech
Multimodal machine learning is a vibrant multi-disciplinary research field which addresses some of the original goals of artificial intelligence by integrating and modeling multiple communicative modalities, including linguistic, acoustic and visual messages.
Audio-Visual Speech Recognition
BIG-bench Machine Learning
+9
Action Recognition by Hierarchical Sequence Summarization
We develop an efficient learning method to train our model and show that its complexity grows sublinearly with the size of the hierarchy.
The Distress Analysis Interview Corpus of human and computer interviews
The Distress Analysis Interview Corpus (DAIC) contains clinical interviews designed to support the diagnosis of psychological distress conditions such as anxiety, depression, and post traumatic stress disorder.
Variational Auto-Decoder: A Method for Neural Generative Modeling from Incomplete Data
We study a specific implementation of the Auto-Encoding Variational Bayes (AEVB) algorithm, named in this paper as a Variational Auto-Decoder (VAD).
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
The dataset and accompanying studies, present a framework in multimodal humor detection for the natural language processing community.
A Multimodal Corpus for the Assessment of Public Speaking Ability and Anxiety
As such, tools enabling the improvement of public speaking performance and the assessment and mitigation of anxiety related to public speaking would be very useful.
Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Our method is based on the observation that high-dimensional multimodal time series data often exhibit correlations across time and modalities which leads to low-rank tensor representations.
Transformer Dissection: An Unified Understanding for Transformer's Attention via the Lens of Kernel
This new formulation gives us a better way to understand individual components of the Transformer{'}s attention, such as the better way to integrate the positional embedding.
Factorized Multimodal Transformer for Multimodal Sequential Learning
As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental research area.
WildMix Dataset and Spectro-Temporal Transformer Model for Monoaural Audio Source Separation
To this end, we introduce a novel trasnformer-based model called Spectro-Temporal Transformer (STT).
Context-Dependent Models for Predicting and Characterizing Facial Expressiveness
In recent years, extensive research has emerged in affective computing on topics like automatic emotion recognition and determining the signals that characterize individual emotions.
Learning Not to Learn in the Presence of Noisy Labels
Learning in the presence of label noise is a challenging yet important task: it is crucial to design models that are robust in the presence of mislabeled datasets.
Improving Aspect-Level Sentiment Analysis with Aspect Extraction
Aspect-based sentiment analysis (ABSA), a popular research area in NLP has two distinct parts -- aspect extraction (AE) and labeling the aspects with sentiment polarity (ALSA).
Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions
In this paper, we propose \textbf{N3} (\textbf{N}eural \textbf{N}etworks from \textbf{N}atural Language) - a new paradigm of synthesizing task-specific neural networks from language descriptions and a generic pre-trained model.
What Gives the Answer Away? Question Answering Bias Analysis on Video QA Datasets
In this paper, we analyze QA biases in popular video question answering datasets and discover pretrained language models can answer 37-48% questions correctly without using any multimodal context information, far exceeding the 20% random guess baseline for 5-choose-1 multiple-choice questions.
Importance-based Multimodal Autoencoder
In this paper we propose the IMA (Importance-based Multimodal Autoencoder) model, a scalable model that learns modality importances and robust multimodal representations through a novel cross-covariance based loss function.
Unsupervised Domain Adaptation for Visual Navigation
We further show that our method can be used to transfer the navigation policies learned in simulation to the real world.
CMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French
It covers a diverse set topics and speakers, and carries supervision of 20 labels including sentiment (and subjectivity), emotions, and attributes.
Multimodal Privacy-preserving Mood Prediction from Mobile Data: A Preliminary Study
Mental health conditions remain under-diagnosed even in countries with common access to advanced medical care.
StarNet: Gradient-free Training of Deep Generative Models using Determined System of Linear Equations
In this paper we present an approach for training deep generative models solely based on solving determined systems of linear equations.
Integrating Auxiliary Information in Self-supervised Learning
Our approach contributes as follows: 1) Comparing to conventional self-supervised representations, the auxiliary-information-infused self-supervised representations bring the performance closer to the supervised representations; 2) The presented Cl-InfoNCE can also work with unsupervised constructed clusters (e. g., k-means clusters) and outperform strong clustering-based self-supervised learning approaches, such as the Prototypical Contrastive Learning (PCL) method; 3) We show that Cl-InfoNCE may be a better approach to leverage the data clustering information, by comparing it to the baseline approach - learning to predict the clustering assignments with cross-entropy loss.
Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning
In this paper, we propose a Conditional Contrastive Learning (CCL) approach to improve the fairness of contrastive SSL methods.
Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data
Using computational models, we find that language and multimodal representations of mobile typed text (spanning typed characters, words, keystroke timings, and app usage) are predictive of daily mood.
Learning Visual-Linguistic Adequacy, Fidelity, and Fluency for Novel Object Captioning
Novel object captioning (NOC) learns image captioning models for describing objects or visual concepts which are unseen (i. e., novel) in the training captions.
Relay Variational Inference: A Method for Accelerated Encoderless VI
We find RVI to be a unique tool, often superior in both performance and convergence speed to previously proposed encoderless as well as amortized VI models (e. g. VAE).
A Simple Approach to the Noisy Label Problem Through the Gambler's Loss
Learning in the presence of label noise is a challenging yet important task.
HOLM: Hallucinating Objects with Language Models for Referring Expression Recognition in Partially-Observed Scenes
AI systems embodied in the physical world face a fundamental challenge of partial observability; operating with only a limited view and knowledge of the environment.
Low-Resource Adaptation for Personalized Co-Speech Gesture Generation
Personalizing an avatar for co-speech gesture generation from spoken language requires learning the idiosyncrasies of a person's gesture style from a small amount of data.
Tutorial on Multimodal Machine Learning
Multimodal machine learning involves integrating and modeling information from multiple heterogeneous sources of data.
Face-to-Face Contrastive Learning for Social Intelligence Question-Answering
Creating artificial social intelligence - algorithms that can understand the nuances of multi-person interactions - is an exciting and emerging challenge in processing facial expressions and gestures from multimodal videos.
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
With the recent interest in video understanding, embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, multimodal machine learning has brought unique computational and theoretical challenges to the machine learning community given the heterogeneity of data sources and the interconnections often found between modalities.
Paraphrasing Is All You Need for Novel Object Captioning
Since no ground truth captions are available for novel object images during training, our P2C leverages cross-modality (image-text) association modules to ensure the above caption characteristics can be properly preserved.
SeedBERT: Recovering Annotator Rating Distributions from an Aggregated Label
However, machine learning datasets commonly have just one "ground truth" label for each sample, so models trained on these labels may not perform well on tasks that are subjective in nature.
Expanding the Role of Affective Phenomena in Multimodal Interaction Research
We find that this body of research has primarily focused on enabling machines to recognize and express affect and emotion.
Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition
Fusing multiple modalities has proven effective for multimodal information processing.
Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions
Since large-scale retraining of these models from scratch is both time and compute-expensive, a variety of approaches have been previously proposed that de-bias a pre-trained model.
Lecture Presentations Multimodal Dataset: Towards Understanding Multimodality in Educational Videos
We introduce three research tasks, (1) figure-to-text retrieval, (2) text-to-figure retrieval, and (3) generation of slide explanations, which are grounded in multimedia learning and psychology principles to test a vision-language model's understanding of multimodal content.
Continual Learning for Personalized Co-speech Gesture Generation
However in practical scenarios, speaker data comes sequentially and in small amounts as the agent personalizes with more speakers, akin to a continual learning paradigm.
MultiIoT: Towards Large-scale Multisensory Learning for the Internet of Things
The Internet of Things (IoT), the network integrating billions of smart physical devices embedded with sensors, software, and communication technologies for the purpose of connecting and exchanging data with other devices and systems, is a critical and rapidly expanding component of our modern world.
MMOE: Mixture of Multimodal Interaction Experts
Multimodal machine learning, which studies the information and interactions across various input modalities, has made significant advancements in understanding the relationship between images and descriptive text.
Does Structural Attention Improve Compositional Representations in Vision-Language Models?
Although scaling self-supervised approaches has gained widespread success in Vision-Language pre-training, a number of works providing structural knowledge of visually-grounded semantics have recently shown incremental performance gains.
Ranked #27 on
Visual Reasoning
on Winoground
Optimizing Language Models for Human Preferences is a Causal Inference Problem
In this paper, we present an initial exploration of language model optimization for human preferences from direct outcome datasets, where each sample consists of a text and an associated numerical outcome measuring the reader's response.
Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback
We describe an approach for aligning an LLM-based dialogue agent based on global (i. e., dialogue-level) rewards, while also taking into account naturally-occurring multimodal signals.
