Search Results for author:
Found 26 papers, 16 papers with code
DINOv2: Learning Robust Visual Features without Supervision
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision.
Ranked #1 on
Image Classification
on CIFAR-10
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks.
Ranked #1 on
Temporal Action Localization
on CrossTask
(using extra training data)
Action Segmentation
Long Video Retrieval (Background Removed)
+4
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
We present a simplified, task-agnostic multi-modal pre-training approach that can accept either video or text input, or both for a variety of end tasks.
Ranked #2 on
Temporal Action Localization
on CrossTask
(using extra training data)
VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
We introduce VoiceCraft, a token infilling neural codec language model, that achieves state-of-the-art performance on both speech editing and zero-shot text-to-speech (TTS) on audiobooks, internet videos, and podcasts.
Masked Autoencoders that Listen
Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers.
Ranked #2 on
Speaker Identification
on VoxCeleb1
(using extra training data)
Demystifying CLIP Data
We believe that the main ingredient to the success of CLIP is its data and not the model architecture or pre-training objective.
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles
Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance.
Ranked #1 on
Image Classification
on iNaturalist 2019
(using extra training data)
Argus: Efficient Activity Detection System for Extended Video Analysis
We propose an Efficient Activity Detection System, Argus, for Extended Video Analysis in the surveillance scenario.
CiT: Curation in Training for Effective Vision-Language Data
Large vision-language models are generally applicable to many downstream tasks, but come at an exorbitant training cost that only large institutions can afford.
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextualized multilingual multimodal embeddings.
RWR-GAE: Random Walk Regularization for Graph Auto Encoders
Node embeddings have become an ubiquitous technique for representing graph data in a low dimensional space.
Ranked #2 on
Graph Clustering
on Pubmed
AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information.
MAViL: Masked Audio-Video Learners
We present Masked Audio-Video Learners (MAViL) to train audio-visual representations.
STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition
The model uses a hierarchical transformer with intra-frame off-set attention and inter-frame self-attention.
Audio-Visual Event Recognition through the lens of Adversary
This work aims to study several key questions related to multimodal learning through the lens of adversarial noises: 1) The trade-off between early/middle/late fusion affecting its robustness and accuracy 2) How do different frequency/time domain features contribute to the robustness?
Video Representation Learning and Latent Concept Mining for Large-scale Multi-label Video Classification
We report on CMU Informedia Lab's system used in Google's YouTube 8 Million Video Understanding Challenge.
RCAA: Relational Context-Aware Agents for Person Search
In this paper, we address this problem by training relational context-aware agents which learn the actions to localize the target person from the gallery of whole scene images.
Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations
With the aim of promoting and understanding the multilingual version of image search, we leverage visual object detection and propose a model with diverse multi-head attention to learn grounded multilingual multimodal representations.
Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
In this paper, we investigate how to utilize visual content for disambiguation and promoting latent space alignment in unsupervised MMT.
A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
Neural Architecture Search (NAS) is just such a revolutionary algorithm, and the related research work is complicated and rich.
Support-set bottlenecks for video-text representation learning
The dominant paradigm for learning video-text representations -- noise contrastive learning -- increases the similarity of the representations of pairs of samples that are known to be related, such as text and video from the same sample, and pushes away the representations of all other pairs.
Diffusion Models as Masked Autoencoders
There has been a longstanding belief that generation can facilitate a true understanding of visual data.
FLAP: Fast Language-Audio Pre-training
We propose Fast Language-Audio Pre-training (FLAP), a self-supervised approach that efficiently and effectively learns aligned audio and language representations through masking, contrastive learning and reconstruction.
Adversarially Masked Video Consistency for Unsupervised Domain Adaptation
The second is a Masked Consistency Learning module to learn class-discriminative representations.
