Search Results for author:
Found 22 papers, 16 papers with code
PlanT: Explainable Planning Transformers via Object-Level Representations
Planning an optimal route in a complex environment requires efficient reasoning about the surrounding scene.
Ranked #6 on
CARLA longest6
on CARLA
Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models
However, existing benchmarks predate the popularization of large multi-modal models, such as CLIP and CLAP.
Self-supervised learning of a facial attribute embedding from video
We propose a self-supervised framework for learning facial attributes by simply watching videos of a human face speaking, laughing, and moving over time.
Ranked #2 on
Unsupervised Facial Landmark Detection
on 300W
Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
Having access to multi-modal cues (e. g. vision and audio) empowers some cognitive tasks to be done faster compared to learning from a single modality.
Waffling around for Performance: Visual Classification with Random Words and Broad Concepts
The visual classification performance of vision-language models such as CLIP has been shown to benefit from additional semantic knowledge from large language models (LLMs) such as GPT-3.
Audio Retrieval with Natural Language Queries: A Benchmark Study
Additionally, we introduce the SoundDescs benchmark, which consists of paired audio and natural language descriptions for a diverse collection of sounds that are complementary to those found in AudioCaps and Clotho.
Ranked #1 on
Audio to Text Retrieval
on SoundDescs
Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language
Focusing on the relatively underexplored task of audio-visual zero-shot learning, we propose to learn multi-modal representations from audio-visual data using cross-modal attention and exploit textual label embeddings for transferring knowledge from seen classes to unseen classes.
Ranked #1 on
ZSL Video Classification
on UCF-GZSL (cls)
Audio Retrieval with Natural Language Queries
We consider the task of retrieving audio using free-form natural language queries.
Ranked #1 on
Audio/Video to Text Retrieval
on AudioCaps
Temporal and cross-modal attention for audio-visual zero-shot learning
We show that our proposed framework that ingests temporal features yields state-of-the-art performance on the \ucf, \vgg, and \activity benchmarks for (generalised) zero-shot learning.
Ranked #2 on
GZSL Video Classification
on UCF-GZSL(main)
CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
We present baseline results for generating natural language explanations in the context of VQA using two state-of-the-art frameworks on the CLEVR-X dataset.
Ranked #1 on
Explanation Generation
on CLEVR-X
Image-free Classifier Injection for Zero-Shot Classification
We achieve this with our proposed Image-free Classifier Injection with Semantics (ICIS) that injects classifiers for new, unseen classes into pre-trained classification models in a post-hoc fashion without relying on image data.
Zero-shot audio captioning with audio-language model guidance and audio context keywords
In particular, our framework exploits a pre-trained large language model (LLM) for generating the text which is guided by a pre-trained audio-language model to produce captions that describe the audio content.
Ranked #1 on
Zero-shot Audio Captioning
on Clotho
Text-to-feature diffusion for audio-visual few-shot learning
Training deep learning models for video classification from audio-visual data commonly requires immense amounts of labeled training data collected via a costly process.
Video-adverb retrieval with compositional adverb-action embeddings
We propose a framework for video-to-adverb retrieval (and vice versa) that aligns video embeddings with their matching compositional adverb-action text embedding in a joint embedding space.
Addressing caveats of neural persistence with deep graph persistence
Whilst this captures useful information for linear classifiers, we find that no relevant spatial structure is present in later layers of deep neural networks, making neural persistence roughly equivalent to the variance of weights.
Zero-shot Translation of Attention Patterns in VQA Models to Natural Language
Converting a model's internals to text can yield human-understandable insights about the model.
X2Face: A network for controlling face generation by using images, audio, and pose codes
The objective of this paper is a neural network model that controls the pose and expression of a given face, using another face or modality (e. g. audio).
X2Face: A network for controlling face generation using images, audio, and pose codes
The objective of this paper is a neural network model that controls the pose and expression of a given face, using another face or modality (e. g. audio).
Self-supervised learning of class embeddings from video
This work explores how to use self-supervised learning on videos to learn a class-specific image embedding that encodes pose and shape information.
Where and When: Space-Time Attention for Audio-Visual Explanations
Recent advances in XAI provide explanations for models trained on still images.
Explainable artificial intelligence
Explainable Artificial Intelligence (XAI)
+1
Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval
In this work, we introduce ODmAP@k, an object decorrelation metric that measures a model's robustness to spurious correlations in the training data.
A SOUND APPROACH: Using Large Language Models to generate audio descriptions for egocentric text-audio retrieval
Furthermore, we show that using the same prompts, we can successfully employ LLMs to improve the retrieval on EpicSounds, compared to using the original audio class labels of the dataset.
