Search Results for author:
Found 33 papers, 16 papers with code
Rotationally-Temporally Consistent Novel View Synthesis of Human Performance Video
Second, we introduce a novel loss to explicitly enforce consistency across generated views both in space and in time.
VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models
We introduce VideoComp, a benchmark and learning framework for advancing video-text compositionality understanding, aimed at improving vision-language models (VLMs) in fine-grained temporal alignment.
Whats in a Video: Factorized Autoregressive Decoding for Online Dense Video Captioning
Instead, we propose an efficient, online approach which outputs frequent, detailed and temporally aligned captions, without access to future frames.
Learning Visual Grounding from Generative Vision and Language Model
In this work, we explore whether generative VLMs predominantly trained on image-text data could be leveraged to scale up the text annotation of visual grounding data.
Uni-DVPS: Unified Model for Depth-Aware Video Panoptic Segmentation
We present Uni-DVPS, a unified model for Depth-aware Video Panoptic Segmentation (DVPS) that jointly tackles distinct vision tasks, i. e., video panoptic segmentation, monocular depth estimation, and object tracking.
OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All
Specifically, we propose Cross-modal Alignment Distillation (CAD) to address the unequal-scale problem between student and teacher modalities and effectively align student modalities into the teacher modalities' representation space in stage one.
Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities
We propose a multimodal model, called Mirasol3B, consisting of an autoregressive component for the time-synchronized modalities (audio and video), and an autoregressive component for the context modalities which are not necessarily aligned in time but are still sequential.
Ranked #1 on
Audio Classification
on VGGSound
Region-centric Image-Language Pretraining for Open-Vocabulary Detection
We present a new open-vocabulary detection approach based on region-centric image-language pretraining to bridge the gap between image-level pretraining and open-vocabulary object detection.
Ranked #2 on
Open Vocabulary Object Detection
on LVIS v1.0
Contrastive Feature Masking Open-Vocabulary Vision Transformer
We present Contrastive Feature Masking Vision Transformer (CFM-ViT) - an image-text pretraining methodology that achieves simultaneous learning of image- and region-level representation for open-vocabulary object detection (OVD).
Ranked #8 on
Open Vocabulary Object Detection
on LVIS v1.0
Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation
By setting both the inputs and outputs of our learning problem as speech units, we propose to train an encoder-decoder model in a many-to-many spoken language translation setting, namely Unit-to-Unit Translation (UTUT).
Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers
We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a contrastive image-text pretraining recipe to bridge the gap between image-level pretraining and open-vocabulary object detection.
Ranked #6 on
Zero-Shot Cross-Modal Retrieval
on Flickr30k
RECLIP: Resource-efficient CLIP by Training with Small Images
We present RECLIP (Resource-efficient CLIP), a simple method that minimizes computational resource footprint for CLIP (Contrastive Language Image Pretraining).
Neural Image-based Avatars: Generalizable Radiance Fields for Human Avatar Modeling
We present a method that enables synthesizing novel views and novel poses of arbitrary human performers from sparse multi-view images.
Video-kMaX: A Simple Unified Approach for Online and Near-Online Video Panoptic Segmentation
The meta architecture of the proposed Video-kMaX consists of two components: within clip segmenter (for clip-level segmentation) and cross-clip associater (for association beyond clips).
MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks
We propose a novel paradigm of training with a decoder-only model for multimodal tasks, which is surprisingly effective in jointly learning of these disparate vision-language tasks.
Ranked #1 on
Video Captioning
on MSVD
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation
We propose Clustering Mask Transformer (CMT-DeepLab), a transformer-based framework for panoptic segmentation designed around clustering.
Ranked #6 on
Panoptic Segmentation
on COCO test-dev
TubeFormer-DeepLab: Video Mask Transformer
We present TubeFormer-DeepLab, the first attempt to tackle multiple core video segmentation tasks in a unified manner.
Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering
To tackle this, we propose Neural Human Performer, a novel approach that learns generalizable neural radiance fields based on a parametric human body model for robust performance capture.
Ranked #3 on
Generalizable Novel View Synthesis
on ZJU-MoCap
Learning Open-World Object Proposals without Learning to Classify
In this paper, we identify that the problem is that the binary classifiers in existing proposal methods tend to overfit to the training categories.
Ranked #2 on
Open World Object Detection
on COCO VOC to non-VOC
DeepLab2: A TensorFlow Library for Deep Labeling
DeepLab2 is a TensorFlow library for deep labeling, aiming to provide a state-of-the-art and easy-to-use TensorFlow codebase for general dense pixel prediction problems in computer vision.
Learning to Associate Every Segment for Video Panoptic Segmentation
Temporal correspondence - linking pixels or objects across frames - is a fundamental supervisory signal for the video models.
Ranked #6 on
Video Panoptic Segmentation
on Cityscapes-VPS
(using extra training data)
The Devil is in the Boundary: Exploiting Boundary Representation for Basis-based Instance Segmentation
In this work, we propose Boundary Basis based Instance Segmentation(B2Inst) to learn a global boundary representation that can complement existing global-mask-based methods that are often lacking high-frequency details.
Video Panoptic Segmentation
In this paper, we propose and explore a new video extension of this task, called video panoptic segmentation.
Ranked #7 on
Video Panoptic Segmentation
on Cityscapes-VPS
(using extra training data)
Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling
In this paper, we propose to explicitly learn to imagine a storyline that bridges the visual gap.
Ranked #7 on
Visual Storytelling
on VIST
Preserving Semantic and Temporal Consistency for Unpaired Video-to-Video Translation
In this paper, we investigate the problem of unpaired video-to-video translation.
Align-and-Attend Network for Globally and Locally Coherent Video Inpainting
Our video inpainting network consists of two stages.
Deep Blind Video Decaptioning by Temporal Aggregation and Recurrence
Blind video decaptioning is a problem of automatically removing text overlays and inpainting the occluded parts in videos without any input masks.
Deep Video Inpainting
Video inpainting aims to fill spatio-temporal holes with plausible content in a video.
Ranked #7 on
Video Inpainting
on DAVIS
Discriminative Feature Learning for Unsupervised Video Summarization
The proposed variance loss allows a network to predict output scores for each frame with high discrepancy which enables effective feature learning and significantly improves model performance.
Ranked #4 on
Unsupervised Video Summarization
on SumMe
Supervised Video Summarization
Unsupervised Video Summarization
Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles
Self-supervised tasks such as colorization, inpainting and zigsaw puzzle have been utilized for visual representation learning for still images, when the number of labeled images is limited or absent at all.
Ranked #42 on
Self-Supervised Action Recognition
on HMDB51
LinkNet: Relational Embedding for Scene Graph
In this paper, we present a method that improves scene graph generation by explicitly modeling inter-dependency among the entire object instances.
Learning Image Representations by Completing Damaged Jigsaw Puzzles
The recovery of the aforementioned damage pushes the network to obtain robust and general-purpose representations.
Two-Phase Learning for Weakly Supervised Object Localization
Weakly supervised semantic segmentation and localiza- tion have a problem of focusing only on the most important parts of an image since they use only image-level annota- tions.
