Search Results for author:
Found 33 papers, 17 papers with code
PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation
Various visual foundation models have distinct strengths and weaknesses, both of which can be improved through heterogeneous multi-teacher knowledge distillation without labels, termed "agglomerative models."
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies.
Wolf: Captioning Everything with a World Summarization Framework
Finally, we establish a benchmark for video captioning and introduce a leaderboard, aiming to accelerate advancements in video understanding, captioning, and data alignment.
X-VILA: Cross-Modality Alignment for Large Language Model
We introduce X-VILA, an omni-modality model designed to extend the capabilities of large language models (LLMs) by incorporating image, video, and audio modalities.
VILA: On Pre-training for Visual Language Models
Visual language models (VLMs) rapidly progressed with the recent success of large language models.
Ranked #3 on
Zero-Shot Video Question Answer
on MSVD-QA
FasterViT: Fast Vision Transformers with Hierarchical Attention
At a high level, global self-attentions enable the efficient cross-window communication at lower costs.
Progressive Learning of 3D Reconstruction Network from 2D GAN Data
In this work, to overcome these limitations of generated datasets, we have two main contributions which lead us to achieve state-of-the-art results on challenging objects: 1) A robust multi-stage learning scheme that gradually relies more on the models own predictions when calculating losses, 2) A novel adversarial learning pipeline with online pseudo-ground truth generations to achieve fine details.
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models
Despite tremendous progress in generating high-quality images using diffusion models, synthesizing a sequence of animated frames that are both photorealistic and temporally coherent is still in its infancy.
Ranked #8 on
Text-to-Video Generation
on UCF-101
Fine Detailed Texture Learning for 3D Meshes with Generative Models
The reconstruction is posed as an adaptation problem and is done progressively where in the first stage, we focus on learning accurate geometry, whereas in the second stage, we focus on learning the texture with a generative adversarial network.
Leveraging Bitstream Metadata for Fast, Accurate, Generalized Compressed Video Quality Enhancement
Video compression is a central feature of the modern internet powering technologies from social media to video conferencing.
Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers
AFNO is based on a principled foundation of operator learning which allows us to frame token mixing as a continuous global convolution without any dependence on the input resolution.
Efficient Token Mixing for Transformers via Adaptive Fourier Neural Operators
AFNO is based on a principled foundation of operator learning which allows us to frame token mixing as a continuous global convolution without any dependence on the input resolution.
View Generalization for Single Image Textured 3D Models
We describe a cycle consistency loss that encourages model textures to be aligned, so as to encourage sharing.
Dual Contrastive Loss and Attention for GANs
Lastly, we study different attention architectures in the discriminator, and propose a reference attention mechanism.
Neural FFTs for Universal Texture Image Synthesis
The conventional CNNs, recently adopted for synthesis, require to train and test on the same set of images and fail to generalize to unseen images.
Transposer: Universal Texture Synthesis Using Feature Maps as Transposed Convolution Filter
Specifically, we directly treat the whole encoded feature map of the input texture as transposed convolution filters and the features' self-similarity map, which captures the auto-correlation information, as input to the transposed convolution.
Hierarchical Multi-Scale Attention for Semantic Segmentation
Multi-scale inference is commonly used to improve the results of semantic segmentation.
Ranked #6 on
Semantic Segmentation
on Cityscapes val
(using extra training data)
Panoptic-based Image Synthesis
Conditional image synthesis for generating photorealistic images serves various applications for content editing to content generation.
Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos
However, the reconstruction task of the entire image forces the model to allocate landmarks to model the background.
Neural ODEs for Image Segmentation with Level Sets
We propose a novel approach for image segmentation that combines Neural Ordinary Differential Equations (NODEs) and the Level Set method.
Few-shot Video-to-Video Synthesis
To address the limitations, we propose a few-shot vid2vid framework, which learns to synthesize videos of previously unseen subjects or scenes by leveraging few example images of the target at test time.
Ranked #1 on
Video-to-Video Synthesis
on YouTube Dancing
Video Interpolation and Prediction with Unsupervised Landmarks
Prediction and interpolation for long-range video data involves the complex task of modeling motion trajectories for each visible object, occlusions and dis-occlusions, as well as appearance changes due to viewpoint and lighting.
Unsupervised Video Interpolation Using Cycle Consistency
We further introduce a pseudo supervised loss term that enforces the interpolated frames to be consistent with predictions of a pre-trained interpolation model.
Ranked #1 on
Video Frame Interpolation
on UCF101
(PSNR (sRGB) metric)
Graphical Contrastive Losses for Scene Graph Parsing
The first, Entity Instance Confusion, occurs when the model confuses multiple instances of the same type of entity (e. g. multiple cups).
Improving Semantic Segmentation via Video Propagation and Label Relaxation
In this paper, we present a video prediction-based methodology to scale up training sets by synthesizing new training samples in order to improve the accuracy of semantic segmentation networks.
Ranked #2 on
Semantic Segmentation
on KITTI Semantic Segmentation
(using extra training data)
Partial Convolution based Padding
In this paper, we present a simple yet effective padding scheme that can be used as a drop-in module for existing convolutional neural networks.
An Interpretable Model for Scene Graph Generation
We propose an efficient and interpretable scene graph generator.
SDCNet: Video Prediction Using Spatially-Displaced Convolution
We present an approach for high-resolution video frame prediction by conditioning on both past frames and past optical flows.
Introduction to the 1st Place Winning Model of OpenImages Relationship Detection Challenge
This article describes the model we built that achieved 1st place in the OpenImage Visual Relationship Detection Challenge on Kaggle.
SDC-Net: Video prediction using spatially-displaced convolution
We present an approach for high-resolution video frame prediction by conditioning on both past frames and past optical flows.
Ranked #1 on
Video Prediction
on YouTube-8M
Video-to-Video Synthesis
We study the problem of video-to-video synthesis, whose goal is to learn a mapping function from an input source video (e. g., a sequence of semantic segmentation masks) to an output photorealistic video that precisely depicts the content of the source video.
Image Inpainting for Irregular Holes Using Partial Convolutions
Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid pixels as well as the substitute values in the masked holes (typically the mean value).
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
We present a new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs).
Ranked #2 on
Sketch-to-Image Translation
on COCO-Stuff
Conditional Image Generation
Fundus to Angiography Generation
+5
