Search Results for author:
Found 31 papers, 11 papers with code
Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models
Audiovisual segmentation (AVS) aims to identify visual regions corresponding to sound sources, playing a vital role in video understanding, surveillance, and human-computer interaction.
DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimization
Language-queried Audio Source Separation (LASS) enables open-vocabulary sound separation via natural language queries.
Random Conditioning with Distillation for Data-Efficient Diffusion Model Compression
In this work, we propose Random Conditioning, a novel approach that pairs noised images with randomly selected text conditions to enable efficient, image-free knowledge distillation.
Spectral-Adaptive Modulation Networks for Visual Perception
Recent studies have shown that 2D convolution and self-attention exhibit distinct spectral behaviors, and optimizing their spectral properties can enhance vision model performance.
LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs
While large language models (LLMs) excel in generating coherent and contextually rich outputs, their capacity to efficiently handle long-form contexts is limited by fixed-length position embeddings.
Random Conditioning for Diffusion Model Compression with Distillation
In this work, we propose Random Conditioning, a novel approach that pairs noised images with randomly selected text conditions to enable efficient, image-free knowledge distillation.
Multi-Granularity Video Object Segmentation
Thus, developing a new video segmentation dataset aimed at tracking multi-granularity segmentation target in the video scene is necessary.
Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels
Large-scale vision-language models like CLIP have demonstrated impressive open-vocabulary capabilities for image-level tasks, excelling in recognizing what objects are present.
Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentation
To address this challenge, we propose two-fold strategies that generate distinctive captions: 1) 'distinctive caption sampling', a new decoding method for the captioning model, to generate multiple expression candidates with detailed words focusing on the target.
Learning Correlation Structures for Vision Transformers
We introduce a new attention mechanism, dubbed structural self-attention (StructSA), that leverages rich correlation patterns naturally emerging in key-query interactions of attention.
Ranked #5 on
Action Recognition
on Diving-48
Zero-shot Referring Image Segmentation with Global-Local Context Features
To overcome this issue, we propose a simple yet effective zero-shot referring image segmentation method by leveraging the pre-trained cross-modal knowledge from CLIP.
AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR
(ii) We also introduce a simple curriculum scheme during training which we show is crucial to enable the model to jointly process audio and visual information effectively; and finally (iii) we show that our model achieves state of the art zero-shot results on three different AV-ASR benchmarks (How2, VisSpeech and Ego4D), while also crucially preserving decent performance on traditional audio-only speech recognition benchmarks (LibriSpeech).
IFSeg: Image-free Semantic Segmentation via Vision-Language Model
In this paper, we introduce a novel image-free segmentation task where the goal is to perform semantic segmentation given only a set of the target semantic categories, but without any task-specific images and annotations.
CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation
Open-vocabulary semantic segmentation presents the challenge of labeling each pixel within an image based on a wide range of text descriptions.
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale.
Ranked #1 on
Dense Video Captioning
on ActivityNet Captions
(using extra training data)
AVATAR submission to the Ego4D AV Transcription Challenge
In this report, we describe our submission to the Ego4D AudioVisual (AV) Speech Transcription Challenge 2022.
AVATAR: Unconstrained Audiovisual Speech Recognition
Audio-visual automatic speech recognition (AV-ASR) is an extension of ASR that incorporates visual cues, often from the movements of a speaker's mouth.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Learning Audio-Video Modalities from Image Captions
To close this gap we propose a new video mining pipeline which involves transferring captions from image captioning datasets to video clips with no additional manual effort.
Ranked #6 on
Zero-shot Text to Audio Retrieval
on AudioCaps
End-to-end Generative Pretraining for Multimodal Video Captioning
Recent video and language pretraining frameworks lack the ability to generate sentences.
Ranked #15 on
Video Captioning
on MSR-VTT
(using extra training data)
Look Before you Speak: Visually Contextualized Utterances
Leveraging recent advances in multimodal learning, our model consists of a novel co-attentional multimodal video transformer, and when trained on both textual and visual context, outperforms baselines that use textual inputs alone.
Combinatorial Inference against Label Noise
Label noise is one of the critical sources that degrade generalization performance of deep neural networks significantly.
Reinforcing an Image Caption Generator Using Off-Line Human Feedback
Human ratings are currently the most accurate way to assess the quality of an image captioning model, yet most often the only used outcome of an expensive human rating evaluation is a few overall statistics over the evaluation dataset.
Regularizing Neural Networks via Stochastic Branch Layers
We introduce a novel stochastic regularization technique for deep neural networks, which decomposes a layer into multiple branches with different parameters and merges stochastically sampled combinations of the outputs from the branches during training.
Learning for Single-Shot Confidence Calibration in Deep Neural Networks through Stochastic Inferences
The proposed loss function enables us to learn deep neural networks that predict confidence calibrated scores using a single inference.
CPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of Maps
Image geolocalization is the task of identifying the location depicted in a photo based only on its visual information.
Ranked #1 on
Photo geolocation estimation
on Im2GPS
(Reference images metric)
Attentive Semantic Alignment with Offset-Aware Correlation Kernels
Semantic correspondence is the problem of establishing correspondences across images depicting different instances of the same object or scene class.
Visual Reference Resolution using Attention Memory for Visual Dialog
From this memory, the model retrieves the previous attention, taking into account recency, which is most relevant for the current question, in order to resolve potentially ambiguous references.
Ranked #13 on
Visual Dialog
on VisDial v0.9 val
(R@1 metric)
MarioQA: Answering Questions by Watching Gameplay Videos
To address this objective, we automatically generate a customized synthetic VideoQA dataset using {\em Super Mario Bros.} gameplay videos so that it contains events with different levels of reasoning complexity.
Progressive Attention Networks for Visual Attribute Prediction
We propose a novel attention model that can accurately attends to target objects of various scales and shapes in images.
Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
We tackle image question answering (ImageQA) problem by learning a convolutional neural network (CNN) with a dynamic parameter layer whose weights are determined adaptively based on questions.
Image Retrieval with Multi-Modal Query
Parameter Prediction
+3
