Search Results for author:
Found 48 papers, 21 papers with code
Movie Gen: A Cast of Media Foundation Models
Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation.
Pixel-Space Post-Training of Latent Diffusion Models
Latent diffusion models (LDMs) have made significant advancements in the field of image generation in recent years.
Imagine Flash: Accelerating Emu Diffusion Models with Backward Distillation
Diffusion models are a powerful generative framework, but come with expensive inference.
Animated Stickers: Bringing Stickers to Life with Video Diffusion
Our model is built on top of the state-of-the-art Emu text-to-image model, with the addition of temporal layers to model motion.
FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis
This enables our model for video synthesis by editing the first frame with any prevalent I2I models and then propagating edits to successive frames.
Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis
In this paper, we introduce Fairy, a minimalist yet robust adaptation of image-editing diffusion models, enhancing them for video editing applications.
MixRT: Mixed Neural Representations For Real-Time NeRF Rendering
Neural Radiance Field (NeRF) has emerged as a leading technique for novel view synthesis, owing to its impressive photorealistic reconstruction and rendering capability.
ControlRoom3D: Room Generation using Semantic Proxy Rooms
Central to our approach is a user-defined 3D semantic proxy room that outlines a rough room layout based on semantic bounding boxes and a textual description of the overall room style.
AVID: Any-Length Video Inpainting with Diffusion Model
Given a video, a masked region at its initial frame, and an editing prompt, it requires a model to do infilling at each frame following the editing guidance while keeping the out-of-mask region intact.
Cache Me if You Can: Accelerating Diffusion Models through Block Caching
However, one of the major drawbacks of diffusion models is that the image generation process is costly.
Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack
Training text-to-image models with web scale image-text pairs enables the generation of a wide range of visual concepts from text.
NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection
We present NeRF-Det, a novel method for indoor 3D detection with posed RGB images as input.
3D-CLFusion: Fast Text-to-3D Rendering with Contrastive Latent Diffusion
We note that the novelty of our model lies in that we introduce contrastive learning during training the diffusion prior which enables the generation of the valid view-invariant latent code.
Pruning Compact ConvNets for Efficient Inference
The resulting family of pruned models can consistently obtain better performance than existing FBNetV3 models at the same level of computation, and thus provide state-of-the-art results when trading off between computational complexity and generalization performance on the ImageNet benchmark.
INGeo: Accelerating Instant Neural Scene Reconstruction with Noisy Geometry Priors
We present a method that accelerates reconstruction of 3D scenes and objects, aiming to enable instant reconstruction on edge devices such as mobile phones and AR/VR headsets.
A Practical Stereo Depth System for Smart Glasses
We present the design of a productionized end-to-end stereo depth sensing system that does pre-processing, online stereo rectification, and stereo depth estimation with a fallback to monocular depth estimation when rectification is unreliable.
Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention at Vision Transformer Inference
Vision Transformers (ViTs) have shown impressive performance but still require a high computation cost as compared to convolutional neural networks (CNNs), one reason is that ViTs' attention measures global similarities and thus has a quadratic complexity with the number of input tokens.
XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse
We hope that our work will stimulate research and lead to the development of a new generation of ML systems for XR use cases.
3D-Aware Encoding for Style-based Neural Radiance Fields
In the first stage, we introduce a base encoder that converts the input image to a latent code.
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
To address this, we propose to finetune CLIP on a collection of masked image regions and their corresponding text descriptions.
Ranked #4 on
Semantic Segmentation
on Replica
Open-Set Semi-Supervised Object Detection
To address this problem, we consider online and offline OOD detection modules, which are integrated with SSOD methods.
Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation
Traditional computer vision models are trained to predict a fixed set of predefined categories.
Cross-Domain Adaptive Teacher for Object Detection
To mitigate this problem, we propose a teacher-student framework named Adaptive Teacher (AT) which leverages domain adversarial learning and weak-strong data augmentation to address the domain gap.
FBNetV5: Neural Architecture Search for Multiple Tasks in One Run
To tackle these challenges, we propose FBNetV5, a NAS framework that can search for neural architectures for a variety of vision tasks with much reduced computational cost and human effort.
Ranked #7 on
Neural Architecture Search
on ImageNet
Adaptive Unbiased Teacher for Cross-Domain Object Detection
This enables the student model to capture domain-invariant features.
An Investigation on Hardware-Aware Vision Transformer Scaling
In particular, when transferred to PiT, our scaling strategies lead to a boosted ImageNet top-1 accuracy of from $74. 6\%$ to $76. 7\%$ ($\uparrow2. 1\%$) under the same 0. 7G FLOPs; and when transferred to the COCO object detection task, the average precision is boosted by $\uparrow0. 7\%$ under a similar throughput on a V100 GPU.
Tackling the Ill-Posedness of Super-Resolution Through Adaptive Target Generation
By the one-to-many nature of the super-resolution (SR) problem, a single low-resolution (LR) image can be mapped to many high-resolution (HR) images.
Ranked #3 on
Blind Super-Resolution
on DIV2KRK - 4x upscaling
Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models
We discover that we can indeed use the same architecture and pretrained weights of a neural net model to understand both images and point-clouds.
Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation
Our model transfers knowledge from pretrained image and sentence encoders and achieves strong performance with only 3M image text pairs, 133x smaller than CLIP.
Unbiased Teacher for Semi-Supervised Object Detection
To address this, we introduce Unbiased Teacher, a simple yet effective approach that jointly trains a student and a gradually progressing teacher in a mutually-beneficial manner.
Visual Transformers: Where Do Transformers Really Belong in Vision Models?
A recent trend in computer vision is to replace convolutions with transformers.
FBWave: Efficient and Scalable Neural Vocoders for Streaming Text-To-Speech on the Edge
More efficient variants of FBWave can achieve up to 109x fewer MACs while still delivering acceptable audio quality.
One Shot 3D Photography
3D photos are static in time, like traditional photos, but are displayed with interactive parallax on mobile or desktop screens, as well as on Virtual Reality devices, where viewing it also includes stereo.
Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild
We present a novel 3D pose refinement approach based on differentiable rendering for objects of arbitrary categories in the wild.
Visual Transformers: Token-based Image Representation and Processing for Computer Vision
In this work, we challenge this paradigm by (a) representing images as semantic visual tokens and (b) running transformers to densely model token relationships.
FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
To address this, we present Neural Architecture-Recipe Search (NARS) to search both (a) architectures and (b) their corresponding training recipes, simultaneously.
Ranked #5 on
Neural Architecture Search
on ImageNet
FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
We propose a masking mechanism for feature map reuse, so that memory and computational costs stay nearly constant as the search space expands.
Ranked #68 on
Neural Architecture Search
on ImageNet
Deep Space-Time Video Upsampling Networks
Video super-resolution (VSR) and frame interpolation (FI) are traditional computer vision problems, and the performance have been improving by incorporating deep learning recently.
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation
Using standard convolutions to process such LiDAR images is problematic, as convolution filters pick up local features that are only active in specific regions in the image.
Ranked #24 on
3D Semantic Segmentation
on SemanticKITTI
Learning the Loss Functions in a Discriminative Space for Video Restoration
Our framework is similar to GANs in that we iteratively train two networks - a generator and a loss network.
Efficient Segmentation: Learning Downsampling Near Semantic Boundaries
Many automated processes such as auto-piloting rely on a good semantic segmentation as a critical component.
Learning to Generate Grounded Visual Captions without Localization Supervision
When automatically generating a sentence description for an image or video, it often remains unclear how well the generated caption is grounded, that is whether the model uses the correct image regions to output particular words, or if the model is hallucinating based on priors in the dataset and/or the language model.
Precision Highway for Ultra Low-Precision Quantization
We also report that the proposed method significantly outperforms the existing method in the 2-bit quantization of an LSTM for language modeling.
ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
We formulate platform-aware NN architecture search in an optimization framework and propose a novel algorithm to search for optimal architectures aided by efficient accuracy and resource (latency and/or energy) predictors.
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
Due to this, previous neural architecture search (NAS) methods are computationally expensive.
Ranked #946 on
Image Classification
on ImageNet
Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
Recent work in network quantization has substantially reduced the time and space complexity of neural network inference, enabling their deployment on embedded and mobile devices with limited computational and memory resources.
Value-aware Quantization for Training and Inference of Neural Networks
We propose a novel value-aware quantization which applies aggressively reduced precision to the majority of data while separately handling a small amount of large data in high precision, which reduces total quantization errors under very low precision.
DSD: Dense-Sparse-Dense Training for Deep Neural Networks
We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance.
