Search Results for author:
Found 44 papers, 18 papers with code
PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
In this paper, we study building a Perception Language Model (PLM) in a fully open and reproducible framework for transparent research in image and video understanding.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone
Video-language pre-training (VLP) has become increasingly important due to its ability to generalize to various vision and language tasks.
Egocentric Video Task Translation @ Ego4D Challenge 2022
With no modification to the baseline architectures, our proposed approach achieves competitive performance on two Ego4D challenges, ranking the 1st in the talking to me challenge and the 3rd in the PNR keyframe localization challenge.
Egocentric Video Task Translation
Different video understanding tasks are typically treated in isolation, and even with distinct types of curated data (e. g., classifying sports in one dataset, tracking animals in another).
PatchBlender: A Motion Prior for Video Transformers
We show that our method is successful at enabling vision transformers to encode the temporal component of video data.
Video Summarization Overview
With the broad growth of video capturing devices and applications on the web, it is more demanding to provide desired video content for users efficiently.
Neural-Sim: Learning to Generate Training Data with NeRF
However, existing approaches either require human experts to manually tune each scene property or use automatic methods that provide little to no control; this requires rendering large amounts of random data variations, which is slow and is often suboptimal for the target domain.
Scaling Novel Object Detection with Weakly Supervised Detection Transformers
A critical object detection task is finetuning an existing model to detect novel objects, but the standard workflow requires bounding box annotations which are time-consuming and expensive to collect.
Visual Attention Emerges from Recurrent Sparse Reconstruction
We present VARS, Visual Attention from Recurrent Sparse reconstruction, a new attention formulation built on two prominent features of the human visual attention mechanism: recurrency and sparsity.
One Network Doesn't Rule Them All: Moving Beyond Handcrafted Architectures in Self-Supervised Learning
We conduct a large-scale study with over 100 variants of ResNet and MobileNet architectures and evaluate them across 11 downstream scenarios in the SSL setting.
Robust Contrastive Learning against Noisy Views
Contrastive learning relies on an assumption that positive pairs contain related views, e. g., patches of an image or co-occurring multimodal signals of a video, that share certain underlying information about an instance.
Contrastive Learning of Global and Local Video Representations
In this work, we propose to learn video representations that generalize to both the tasks which require global semantic information (e. g., classification) and the tasks that require local fine-grained spatio-temporal information (e. g., localization).
CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning
The ability to perform causal and counterfactual reasoning are central properties of human intelligence.
Contrastive Learning of Global-Local Video Representations
In this work, we propose to learn video representations that generalize to both the tasks which require global semantic information (e. g., classification) and the tasks that require local fine-grained spatio-temporal information (e. g., localization).
DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents
Creating presentation materials requires complex multimodal reasoning skills to summarize key concepts and arrange them in a logical and visually pleasing manner.
ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning
We demonstrate that our approach finds videos with high audio-visual correspondence and show that self-supervised models trained on our data achieve competitive performances compared to models trained on existing manually curated datasets.
Contrastive Self-Supervised Learning of Global-Local Audio-Visual Representations
Contrastive self-supervised learning has delivered impressive results in many audio-visual recognition tasks.
Self-Supervised Learning of Compressed Video Representations
We show that our approach achieves competitive performance on self-supervised learning of video representations with a considerable improvement in speed compared to the traditional methods.
Parameter Efficient Multimodal Transformers for Video Representation Learning
The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model.
Learning to Transfer Visual Effects from Videos to Images
We study the problem of animating images by transferring spatio-temporal visual effects (such as melting) from a collection of videos.
Active Contrastive Learning of Audio-Visual Video Representations
Contrastive learning has been shown to produce generalizable representations of audio and visual data by maximizing the lower bound on the mutual information (MI) between different views of an instance.
Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistency
We use this method to transfer emotion from a dataset containing four emotions to a dataset with only a single emotion.
Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck
We propose a multimodal information bottleneck approach that learns the correspondence between modalities from unpaired data (image and speech) by leveraging the shared modality (text).
Image to Video Domain Adaptation Using Web Supervision
Training deep neural networks typically requires large amounts of labeled data which may be scarce or expensive to obtain for a particular target domain.
M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention
Generative adversarial networks have led to significant advances in cross-modal/domain translation.
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
In this work, we introduce Polysemous Instance Embedding Networks (PIE-Nets) that compute multiple and diverse representations of an instance by combining global context with locally-guided features via multi-head self-attention and residual learning.
Ranked #34 on
Cross-Modal Retrieval
on COCO 2014
Characterizing Bias in Classifiers using Generative Models
Models that are learned from real-world data are often biased because the data used to train them is biased.
Neural TTS Stylization with Adversarial and Collaborative Games
The synthesized audio waveform is expected to contain the verbal content of x_txt and the auditory style of x_aud.
Video Prediction with Appearance and Motion Conditions
Video prediction aims to generate realistic future frames by learning dynamic visual patterns.
Cross-Modal Retrieval with Implicit Concept Association
Traditional cross-modal retrieval assumes explicit association of concepts across modalities, where there is no ambiguity in how the concepts are linked to each other, e. g., when we do the image search with a query "dogs", we expect to see dog images.
Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks
Given a still photograph, one can imagine how dynamic objects might move against a static background.
ElasticPlay: Interactive Video Summarization with Dynamic Time Budgets
Video consumption is being shifted from sit-and-watch to selective skimming.
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
In this paper, we focus on extending VQA to the video domain and contribute to the literature in three important ways.
Ranked #33 on
Visual Question Answering (VQA)
on MSRVTT-QA
Improving Pairwise Ranking for Multi-label Image Classification
Pairwise ranking, in particular, has been successful in multi-label image classification, achieving state-of-the-art results on various benchmarks.
Learning from Noisy Labels with Distillation
The ability of learning from noisy labels is very useful in many visual recognition tasks, as a vast amount of data with noisy labels are relatively easy to obtain.
Real-Time Video Highlights for Yahoo Esports
We present a technique for detecting highlights from live streaming videos of esports game matches.
To Click or Not To Click: Automatic Selection of Beautiful Thumbnails from Videos
Our system selects attractive thumbnails by analyzing various visual quality and aesthetic metrics of video frames, and performs a clustering analysis to determine the relevance to video content, thus making the resulting thumbnails more representative of the video.
Multimedia
Video2GIF: Automatic Generation of Animated GIFs from Video
We introduce the novel problem of automatically generating animated GIFs from video.
Balancing Appearance and Context in Sketch Interpretation
We describe a sketch interpretation system that detects and classifies clock numerals created by subjects taking the Clock Drawing Test, a clinical tool widely used to screen for cognitive impairments (e. g., dementia).
TGIF: A New Dataset and Benchmark on Animated GIF Description
The motivation for this work is to develop a testbed for image sequence description systems, where the task is to generate natural language descriptions for animated GIFs or video clips.
TVSum: Summarizing Web Videos Using Titles
We observe that a video title is often carefully chosen to be maximally descriptive of its main topic, and hence images related to the title can serve as a proxy for important visual concepts of the main topic.
Video Co-Summarization: Video Summarization by Visual Co-Occurrence
We present video co-summarization, a novel perspective to video summarization that exploits visual co-occurrence across multiple videos.
Action Recognition by Hierarchical Sequence Summarization
We develop an efficient learning method to train our model and show that its complexity grows sublinearly with the size of the hierarchy.
