Search Results for author:
Found 76 papers, 32 papers with code
Motion Prompting: Controlling Video Generation with Motion Trajectories
Motion control is crucial for generating expressive and compelling video content; however, most existing video generation models rely mainly on text prompts for control, which struggle to capture the nuances of dynamic actions and temporal compositions.
HoliSDiP: Image Super-Resolution via Holistic Semantics and Diffusion Prior
Text-to-image diffusion models have emerged as powerful priors for real-world image super-resolution (Real-ISR).
Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens
Models based on continuous tokens achieve significantly better visual quality than those using discrete tokens.
High-Resolution Frame Interpolation with Patch-based Cascaded Diffusion
However, contrary to prior work on cascaded diffusion models which perform diffusion on increasingly large resolutions, we use a single model that always performs diffusion at the same resolution and upsamples by processing patches of the inputs and the prior solution.
Ranked #1 on
Video Frame Interpolation
on X4K1000FPS
A Simple Approach to Unifying Diffusion-based Conditional Generation
Recent progress in image generation has sparked research into controlling these models through condition signals, with various methods addressing specific challenges in conditional generation.
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion
Estimating geometry from dynamic scenes, where objects move and deform over time, remains a core challenge in computer vision.
SMooDi: Stylized Motion Diffusion Model
We introduce a novel Stylized Motion Diffusion model, dubbed SMooDi, to generate stylized motion driven by content texts and style motion sequences.
Probing the 3D Awareness of Visual Foundation Models
Given that such models can classify, delineate, and localize objects in 2D, we ask whether they also represent their 3D structure?
GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting
We present GALA3D, generative 3D GAussians with LAyout-guided control, for effective compositional text-to-3D generation.
Lumiere: A Space-Time Diffusion Model for Video Generation
We introduce Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis.
Ranked #5 on
Text-to-Video Generation
on UCF-101
SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild
We present SHINOBI, an end-to-end framework for the reconstruction of shape, material, and illumination from object images captured with varying lighting, pose, and background.
Efficient Hybrid Zoom using Camera Fusion on Mobile Phones
DSLR cameras can achieve multiple zoom levels via shifting lens distances or swapping lens types.
Boundary Attention: Learning curves, corners, junctions and grouping
We present a lightweight network that infers grouping and boundaries, including curves, corners and junctions.
Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Model
In contrast, we advocate a generic, task-agnostic diffusion model, with several advancements such as log-scale depth parameterization to enable joint modeling of indoor and outdoor scenes, conditioning on the field-of-view (FOV) to handle scale ambiguity and synthetically augmenting FOV during training to generalize beyond the limited camera intrinsics in training datasets.
Ranked #19 on
Monocular Depth Estimation
on NYU-Depth V2
(using extra training data)
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes
We present DrivingGaussian, an efficient and effective framework for surrounding dynamic autonomous driving scenes.
HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models
We also develop an affordance prediction diffusion model (APDM) to predict the contacting area between the human and object during the interactions driven by the textual prompt.
WonderJourney: Going from Anywhere to Everywhere
We introduce WonderJourney, a modularized framework for perpetual 3D scene generation.
Fine-grained Controllable Video Generation via Object Appearance and Context
To achieve detailed control, we propose a unified framework to jointly inject control signals into the existing text-to-video model.
One-Shot Open Affordance Learning with Foundation Models
We introduce One-shot Open Affordance Learning (OOAL), where a model is trained with just one example per base object category, but is expected to identify novel objects and affordances.
Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence
This paper identifies the importance of being geometry-aware for semantic correspondence and reveals a limitation of the features of current foundation models under simple post-processing.
Ranked #1 on
Semantic correspondence
on SPair-71k
ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image
Further, we observe that Score Distillation Sampling (SDS) tends to truncate the distribution of complex backgrounds during distillation of 360-degree scenes, and propose "SDS anchoring" to improve the diversity of synthesized novel views.
OmniControl: Control Any Joint at Any Time for Human Motion Generation
We present a novel approach named OmniControl for incorporating flexible spatial control signals into a text-conditioned human motion generation model based on the diffusion process.
Structure-Preserving Instance Segmentation via Skeleton-Aware Distance Transform
Objects with complex structures pose significant challenges to existing instance segmentation methods that rely on boundary or affinity maps, which are vulnerable to small errors around contacting pixels that cause noticeable connectivity change.
SAMPLING: Scene-adaptive Hierarchical Multiplane Images Representation for Novel View Synthesis from a Single Image
Recent novel view synthesis methods obtain promising results for relatively small scenes, e. g., indoor environments and scenes with a few objects, but tend to fail for unbounded outdoor scenes with a single image as input.
Substance or Style: What Does Your Image Embedding Know?
We find that image-text models (CLIP and ALIGN) are better at recognizing new examples of style transfer than masking-based models (CAN and MAE).
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
Text-to-image diffusion models have made significant advances in generating and editing high-quality images.
Ranked #1 on
Dense Pixel Correspondence Estimation
on TSS
Dense Pixel Correspondence Estimation
Representation Learning
+2
ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimation
We propose a new dataset and a novel approach to learning hand-object interaction priors for hand and articulated object pose estimation.
LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Grounding
A key step to acquire this skill is to identify what part of the object affords each action, which is called affordance grounding.
VQ3D: Learning a 3D-Aware Generative Model on ImageNet
Recent work has shown the possibility of training generative models of 3D content from 2D image collections on small datasets corresponding to a single object class, such as human faces, animal faces, or cars.
Accidental Light Probes
Recovering lighting in a scene from a single image is a fundamental problem in computer vision.
Self-supervised AutoFlow
Recently, AutoFlow has shown promising results on learning a training set for optical flow, but requires ground truth labels in the target domain to compute its search metric.
Face Deblurring using Dual Camera Fusion on Mobile Phones
To the best of our knowledge, our work is the first mobile solution for face motion deblurring that works reliably and robustly over thousands of images in diverse motion and lighting conditions.
SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections
Our method works on in-the-wild online image collections of an object and produces relightable 3D assets for several use-cases such as AR/VR.
An Extendable, Efficient and Effective Transformer-based Object Detector
Transformers have been widely used in numerous vision problems especially for visual recognition and detection.
Adaptive Transformers for Robust Few-shot Cross-domain Face Anti-spoofing
While recent face anti-spoofing methods perform well under the intra-domain setups, an effective approach needs to account for much larger appearance variations of images acquired in complex scenes with different sensors for robust performance.
Disentangling Architecture and Training for Optical Flow
Our newly trained RAFT achieves an Fl-all score of 4. 31% on KITTI 2015, more accurate than all published optical flow methods at the time of writing.
Kubric: A scalable dataset generator
Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance of a system than architecture and training details.
FILM: Frame Interpolation for Large Motion
Recent methods use multiple networks to estimate optical flow or depth and a separate network dedicated to frame synthesis.
Ranked #2 on
Video Frame Interpolation
on Middlebury
(SSIM metric)
Revisiting RCAN: Improved Training for Image Super-Resolution
Image super-resolution (SR) is a fast-moving field with novel architectures attracting the spotlight.
ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction
The surface embeddings are implemented as coordinate-based MLPs that are fit to each video via consistency and contrastive reconstruction losses. Experimental results show that ViSER compares favorably against prior work on challenging videos of humans with loose clothing and unusual poses as well as animals videos from DAVIS and YTVOS.
Pyramid Adversarial Training Improves ViT Performance
In this work, we present pyramid adversarial training (PyramidAT), a simple and effective technique to improve ViT's overall performance.
Ranked #9 on
Domain Generalization
on ImageNet-C
(using extra training data)
ViDT: An Efficient and Effective Fully Transformer-based Object Detector
Transformers are transforming the landscape of computer vision, especially for recognition tasks.
Ranked #16 on
Object Detection
on COCO 2017 val
LASR: Learning Articulated Shape Reconstruction from a Monocular Video
Remarkable progress has been made in 3D reconstruction of rigid structures from a video or a collection of images.
AutoFlow: Learning a Better Training Set for Optical Flow
Synthetic datasets play a critical role in pre-training CNN models for optical flow, but they are painstaking to generate and hard to adapt to new applications.
Adaptive Prototype Learning and Allocation for Few-Shot Segmentation
By integrating the SGC and GPA together, we propose the Adaptive Superpixel-guided Network (ASGNet), which is a lightweight model and adapts to object scale and shape variation.
Ranked #63 on
Few-Shot Semantic Segmentation
on COCO-20i (5-shot)
HumanGPS: Geodesic PreServing Feature for Dense Human Correspondences
In this paper, we address the problem of building dense correspondences between human images under arbitrary camera viewpoints and body poses.
Improving Deep Stereo Network Generalization with Geometric Priors
End-to-end deep learning methods have advanced stereo vision in recent years and obtained excellent results when the training and test data are similar.
Learnable Cost Volume Using the Cayley Representation
Cost volume is an essential component of recent deep models for optical flow estimation and is usually constructed by calculating the inner product between two feature vectors.
Hierarchical Contrastive Motion Learning for Video Action Recognition
Thus, the motion features at higher levels are trained to gradually capture semantic dynamics and evolve more discriminative for action recognition.
SENSE: a Shared Encoder Network for Scene-flow Estimation
We introduce a compact network for holistic scene flow estimation, called SENSE, which shares common encoder features among four closely-related tasks: optical flow estimation, disparity estimation from stereo, occlusion estimation, and semantic segmentation.
Image Formation Model Guided Deep Image Super-Resolution
We present a simple and effective image super-resolution algorithm that imposes an image formation constraint on the deep neural networks via pixel substitution.
Video Stitching for Linear Camera Arrays
Despite the long history of image and video stitching research, existing academic and commercial solutions still produce strong artifacts.
Unsupervised Video Interpolation Using Cycle Consistency
We further introduce a pseudo supervised loss term that enforces the interpolated frames to be consistent with predictions of a pre-trained interpolation model.
Ranked #1 on
Video Frame Interpolation
on UCF101
(PSNR (sRGB) metric)
Pixel-Adaptive Convolutional Neural Networks
In addition, we also demonstrate that PAC can be used as a drop-in replacement for convolution layers in pre-trained networks, resulting in consistent performance improvements.
A Fusion Approach for Multi-Frame Optical Flow Estimation
To date, top-performing optical flow estimation methods only take pairs of consecutive frames into account.
Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation
We investigate two crucial and closely related aspects of CNNs for optical flow estimation: models and training.
Ranked #7 on
Optical Flow Estimation
on KITTI 2012
Learning Data Terms for Non-blind Deblurring
We propose a simple and effective discriminative framework to learn data terms that can adaptively handle blurred images in the presence of severe noise and outliers.
Rendering Portraitures from Monocular Camera and Beyond
Specifically, we first exploit Convolutional Neural Networks to estimate the relative depth and portrait segmentation maps from a single input image.
Superpixel Sampling Networks
Superpixels provide an efficient low/mid-level representation of image data, which greatly reduces the number of image primitives for subsequent vision tasks.
Learning Superpixels With Segmentation-Aware Affinity Loss
Specifically, we propose a new loss function that takes the segmentation error into account for affinity learning.
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation
We address the unsupervised learning of several interconnected problems in low-level vision: single view depth prediction, camera motion estimation, optical flow, and segmentation of a video into the static scene and moving regions.
Ranked #73 on
Monocular Depth Estimation
on KITTI Eigen split
Learning Dual Convolutional Neural Networks for Low-Level Vision
These problems usually involve the estimation of two components of the target signals: structures and details.
Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
Estimation of 3D motion in a dynamic scene from a temporal pair of images is a core task in many scene understanding problems.
SPLATNet: Sparse Lattice Networks for Point Cloud Processing
We present a network architecture for processing point clouds that directly operates on a collection of points represented as a sparse set of samples in a high-dimensional lattice.
Ranked #31 on
Semantic Segmentation
on ScanNet
(test mIoU metric)
Learning Binary Residual Representations for Domain-specific Video Streaming
Specifically, we target a streaming setting where the videos to be streamed from a server to a client are all in the same domain and they have to be compressed to a small size for low-latency transmission.
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
Finally, the two input images are warped and linearly fused to form each intermediate frame.
Learning to Super-Resolve Blurry Face and Text Images
We present an algorithm to directly restore a clear high-resolution image from a blurry low-resolution input.
PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
It then uses the warped features and features of the first image to construct a cost volume, which is processed by a CNN to estimate the optical flow.
Ranked #3 on
Dense Pixel Correspondence Estimation
on HPatches
Dense Pixel Correspondence Estimation
Optical Flow Estimation
Cascaded Scene Flow Prediction using Semantic Segmentation
Given two consecutive frames from a pair of stereo cameras, 3D scene flow methods simultaneously estimate the 3D geometry and motion of the observed scene.
Blind Image Deblurring Using Dark Channel Prior
Therefore, enforcing the sparsity of the dark channel helps blind deblurring on various scenarios, including natural, face, text, and low-illumination images.
Ranked #11 on
Deblurring
on RealBlur-R (trained on GoPro)
Optical Flow with Semantic Segmentation and Localized Layers
Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow.
Layered RGBD Scene Flow Estimation
As consumer depth sensors become widely available, estimating scene flow from RGBD sequences has received increasing attention.
Local Layering for Joint Motion Estimation and Occlusion Detection
To handle such situations, we propose a local layering model where motion and occlusion relationships are inferred jointly.
A Fully-Connected Layered Model of Foreground and Background Flow
Layered models allow scene segmentation and motion estimation to be formulated together and to inform one another.
Layered image motion with explicit occlusions, temporal consistency, and depth ordering
We present a new probabilistic model of optical flow in layers that addresses many of the shortcomings of previous approaches.
