Search Results for author:
Found 55 papers, 25 papers with code
Movie Gen: A Cast of Media Foundation Models
Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation.
Dynamic PDB: A New Dataset and a SE(3) Model Extension by Integrating Dynamic Behaviors and Physical Properties in Protein Structures
Despite significant progress in static protein structure collection and prediction, the dynamic behavior of proteins, one of their most vital characteristics, has been largely overlooked in prior research.
AlphaFolding: 4D Diffusion for Dynamic Protein Structure Prediction with Reference and Motion Guidance
Protein structure prediction is pivotal for understanding the structure-function relationship of proteins, advancing biological research, and facilitating pharmaceutical development and experimental design.
The Llama 3 Herd of Models
This paper presents a new set of foundation models, called Llama 3.
Ranked #3 on
Multi-task Language Understanding
on MMLU
(using extra training data)
Stereo Risk: A Continuous Modeling Approach to Stereo Matching
Stereo Risk departs from the conventional discretization approach by formulating the scene disparity as an optimal solution to a continuous risk minimization problem, hence the name "stereo risk".
Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation
Moving away from traditional paradigms that rely on parametric models for intermediate facial representations, our innovative approach embraces the end-to-end diffusion paradigm and introduces a hierarchical audio-driven visual synthesis module to enhance the precision of alignment between audio inputs and visual outputs, encompassing lip, expression, and pose motion.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
We introduce phi-3-mini, a 3. 8 billion parameter language model trained on 3. 3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3. 5 (e. g., phi-3-mini achieves 69% on MMLU and 8. 38 on MT-bench), despite being small enough to be deployed on a phone.
Ranked #5 on
MMR total
on MRR-Benchmark
(using extra training data)
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
We introduce Florence-2, a novel vision foundation model with a unified, prompt-based representation for a variety of computer vision and vision-language tasks.
Ranked #1 on
Visual Grounding
on RefCOCO+ val
(using extra training data)
MM-VID: Advancing Video Understanding with GPT-4V(ision)
We present MM-VID, an integrated system that harnesses the capabilities of GPT-4V, combined with specialized tools in vision, audio, and speech, to facilitate advanced video understanding.
Self-Enforced Job Matching
The classic two-sided many-to-one job matching model assumes that firms treat workers as substitutes and workers ignore colleagues when choosing where to work.
Single Image Depth Prediction Made Better: A Multivariate Gaussian Take
Accordingly, we introduce an approach that performs continuous modeling of per-pixel depth, where we can predict and reason about the per-pixel depth and its distribution.
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action.
Ranked #84 on
Visual Question Answering
on MM-Vet
VA-DepthNet: A Variational Approach to Single Image Depth Prediction
While state-of-the-art deep neural network methods for SIDP learn the scene depth from images in a supervised setting, they often overlook the invaluable invariances and priors in the rigid scene space, such as the regularity of the scene.
Ranked #28 on
Monocular Depth Estimation
on NYU-Depth V2
Learning Customized Visual Models with Retrieval-Augmented Knowledge
Image-text contrastive learning models such as CLIP have demonstrated strong task transfer ability.
Ranked #1 on
Semi-Supervised Image Classification
on ImageNet - 1% labeled data
(using extra training data)
X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusion
We demonstrate for the first time that using a text2image model to generate images or zero-shot recognition model to filter noisily crawled images for different object categories is a feasible way to make Copy-Paste truly scalable.
Ranked #8 on
Instance Segmentation
on LVIS v1.0 val
ReCo: Region-Controlled Text-to-Image Generation
Human evaluation on PaintSkill shows that ReCo is +19. 28% and +17. 21% more accurate in generating images with correct object count and spatial relationship than the T2I model.
Conditional Text-to-Image Synthesis
Layout-to-Image Generation
+1
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
This paper presents OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture.
Ranked #4 on
Cross-Modal Retrieval
on Flickr30k
(using extra training data)
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Vision-language (VL) pre-training has recently received considerable attention.
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Dev
LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling
In this work, we explore a unified VidL framework LAVENDER, where Masked Language Modeling (MLM) is used as the common interface for all pre-training and downstream tasks.
Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning
Thanks to the strong zero-shot capability of foundation models, we start by constructing a rich semantic representation of the image (e. g., image tags, object attributes / locations, captions) as a structured textual prompt, called visual clues, using a vision foundation model.
GIT: A Generative Image-to-text Transformer for Vision and Language
In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering.
Ranked #1 on
Image Captioning
on nocaps-XD near-domain
K-LITE: Learning Transferable Visual Models with External Knowledge
We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts.
Unified Contrastive Learning in Image-Text-Label Space
Particularly, it attains gains up to 9. 2% and 14. 5% in average on zero-shot recognition benchmarks over the language-image contrastive learning and supervised learning methods, respectively.
MaskGIT: Masked Generative Image Transformer
At inference time, the model begins with generating all tokens of an image simultaneously, and then refines the image iteratively conditioned on the previous generation.
Ranked #2 on
Text-to-Image Generation
on LHQC
ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction
The surface embeddings are implemented as coordinate-based MLPs that are fit to each video via consistency and contrastive reconstruction losses. Experimental results show that ViSER compares favorably against prior work on challenging videos of humans with loose clothing and unusual poses as well as animals videos from DAVIS and YTVOS.
Pyramid Adversarial Training Improves ViT Performance
In this work, we present pyramid adversarial training (PyramidAT), a simple and effective technique to improve ViT's overall performance.
Ranked #9 on
Domain Generalization
on ImageNet-C
(using extra training data)
Florence: A New Foundation Model for Computer Vision
Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications.
Ranked #1 on
Action Recognition In Videos
on Kinetics-600
Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition
Decomposing a scene into its shape, reflectance and illumination is a fundamental problem in computer vision and graphics.
SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting
We present SLIDE, a modular and unified system for single image 3D photography that uses a simple yet effective soft layering strategy to better preserve appearance details in novel views.
ViTGAN: Training GANs with Vision Transformers
Recently, Vision Transformers (ViTs) have shown competitive performance on image recognition while requiring less vision-specific inductive biases.
Ranked #30 on
Image Generation
on CIFAR-10
OCONet: Image Extrapolation by Object Completion
Existing methods struggle to extrapolate images with salient objects in the foreground or are limited to very specific objects such as humans, but tend to work well on indoor/outdoor scenes.
LASR: Learning Articulated Shape Reconstruction from a Monocular Video
Remarkable progress has been made in 3D reconstruction of rigid structures from a video or a collection of images.
COMISR: Compression-Informed Video Super-Resolution
Most video super-resolution methods focus on restoring high-resolution video frames from low-resolution videos without taking into account compression.
AutoFlow: Learning a Better Training Set for Optical Flow
Synthetic datasets play a critical role in pre-training CNN models for optical flow, but they are painstaking to generate and hard to adapt to new applications.
Deep 3D-to-2D Watermarking: Embedding Messages in 3D Meshes and Extracting Them from 2D Renderings
Digital watermarking is widely used for copyright protection.
Regularizing Generative Adversarial Networks under Limited Data
Recent years have witnessed the rapid progress of generative adversarial networks (GANs).
Ranked #1 on
Image Generation
on CIFAR-100
NeRD: Neural Reflectance Decomposition from Image Collections
This problem is inherently more challenging when the illumination is not a single light source under laboratory conditions but is instead an unconstrained environmental illumination.
Ranked #5 on
Image Relighting
on Stanford-ORB
Robust image stitching with multiple registrations
Here, we observe that the use of a single registration often leads to errors, especially in scenes with significant depth variation or object motion.
Stability in Repeated Matching Markets
This paper develops a framework for repeated matching markets.
Supervised Contrastive Learning
Contrastive learning applied to self-supervised representation learning has seen a resurgence in recent years, leading to state of the art performance in the unsupervised training of deep image models.
Ranked #2 on
Class Incremental Learning
on cifar100
Microwave Photonic Imaging Radar with a Millimeter-level Resolution
Microwave photonic radars enable fast or even real-time high-resolution imaging thanks to its broad bandwidth.
Depth Extraction from Video Using Non-parametric Sampling
We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling.
DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling
We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling.
Boundless: Generative Adversarial Networks for Image Extension
Image extension models have broad applications in image editing, computational photography and computer graphics.
Ranked #2 on
Uncropping
on Places2 val
Learning the Depths of Moving People by Watching Frozen People
We present a method for predicting dense depth in scenarios where both a monocular camera and people in the scene are freely moving.
Sparse, Smart Contours to Represent and Edit Images
We study the problem of reconstructing an image from information stored at contour locations.
Smart, Sparse Contours to Represent and Edit Images
We study the problem of reconstructing an image from information stored at contour locations.
On the Effectiveness of Visible Watermarks
Since such an attack relies on the consistency of watermarks across image collection, we explore and evaluate how it is affected by various types of inconsistencies in the watermark embedding that could potentially be used to make watermarking more secured.
Deep Convolutional Neural Network for Image Deconvolution
Many fundamental image-related problems involve deconvolution operators.
Ranked #1 on
Image Compression
on FER2013
Local Layering for Joint Motion Estimation and Occlusion Detection
To handle such situations, we propose a local layering model where motion and occlusion relationships are inferred jointly.
A Compositional Model for Low-Dimensional Image Set Representation
Learning a low-dimensional representation of images is useful for various applications in graphics and computer vision.
Unsupervised Joint Object Discovery and Segmentation in Internet Images
In contrast to previous co-segmentation methods, our algorithm performs well even in the presence of significant amounts of noise images (images not containing a common object), as typical for datasets collected from Internet search.
Probabilistic Label Trees for Efficient Large Scale Image Classification
Large-scale recognition problems with thousands of classes pose a particular challenge because applying the classifier requires more computation as the number of classes grows.
Deformable Spatial Pyramid Matching for Fast Dense Correspondences
We introduce a fast deformable spatial pyramid (DSP) matching algorithm for computing dense pixel correspondences.
