Search Results for author:
Found 114 papers, 69 papers with code
Dynamic Head: Unifying Object Detection Heads with Attentions
In this paper, we present a novel dynamic head framework to unify object detection heads with attentions.
Ranked #2 on
Object Detection
on COCO 2017 val
(AP75 metric)
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
By further pretraining on the larger dataset ImageNet-21K, we achieve 87. 5% Top-1 accuracy on ImageNet-1K and high segmentation performance on ADE20K with 55. 7 mIoU.
Ranked #25 on
Semantic Segmentation
on ADE20K val
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields
Furthermore, we propose an inverse optimization method that accurately projects an input image to the latent codes for manipulation to enable editing on real images.
Mobile-Former: Bridging MobileNet and Transformer
This structure leverages the advantages of MobileNet at local processing and transformer at global interaction.
Designing a Better Asymmetric VQGAN for StableDiffusion
The training cost of our asymmetric VQGAN is cheap, and we only need to retrain a new asymmetric decoder while keeping the vanilla VQGAN encoder and StableDiffusion unchanged.
Vector Quantized Diffusion Model for Text-to-Image Synthesis
Our experiments indicate that the VQ-Diffusion model with the reparameterization is fifteen times faster than traditional AR methods while achieving a better image quality.
Ranked #1 on
Text-to-Image Generation
on Oxford 102 Flowers
(using extra training data)
Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models
Text-to-Image diffusion models have made tremendous progress over the past two years, enabling the generation of highly realistic images based on open-domain text descriptions.
HairCLIP: Design Your Hair by Text and Reference Image
Hair editing is an interesting and challenging problem in computer vision and graphics.
Bringing Old Films Back to Life
We present a learning-based framework, recurrent transformer network (RTN), to restore heavily degraded old films.
Ranked #6 on
Analog Video Restoration
on TAPE
Florence: A New Foundation Model for Computer Vision
Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications.
Ranked #1 on
Action Recognition In Videos
on Kinetics-600
General Facial Representation Learning in a Visual-Linguistic Manner
In this paper, we study the transfer performance of pre-trained models on face analysis tasks and introduce a framework, called FaRL, for general Facial Representation Learning in a visual-linguistic manner.
Ranked #1 on
Face Parsing
on CelebAMask-HQ
(using extra training data)
MicroNet: Improving Image Recognition with Extremely Low FLOPs
This paper aims at addressing the problem of substantial performance degradation at extremely low computational cost (e. g. 5M FLOPs on ImageNet classification).
Equivariant Multi-Modality Image Fusion
These components enable the net training to follow the principles of the natural sensing-imaging process while satisfying the equivariant imaging prior.
High-Fidelity Pluralistic Image Completion with Transformers
Image completion has made tremendous progress with convolutional neural networks (CNNs), because of their powerful texture modeling capacity.
Ranked #6 on
Image Inpainting
on CelebA-HQ
MichiGAN: Multi-Input-Conditioned Hair Image Generation for Portrait Editing
In this paper, we present MichiGAN (Multi-Input-Conditioned Hair Image GAN), a novel conditional image generation method for interactive portrait hair manipulation.
SinDiffusion: Learning a Diffusion Model from a Single Natural Image
We present SinDiffusion, leveraging denoising diffusion models to capture internal distribution of patches from a single natural image.
Ranked #1 on
Image Generation
on Places50
Multi-attentional Deepfake Detection
Most of them model deepfake detection as a vanilla binary classification problem, i. e, first use a backbone network to extract a global feature and then feed it into a binary classifier (real/fake).
Gated Context Aggregation Network for Image Dehazing and Deraining
Image dehazing aims to recover the uncorrupted content from a hazy image.
Ranked #1 on
Rain Removal
on DID-MDN
Unsupervised Pre-training for Person Re-identification
In this paper, we present a large scale unlabeled person re-identification (Re-ID) dataset "LUPerson" and make the first attempt of performing unsupervised pre-training for improving the generalization ability of the learned person Re-ID feature representation.
Ranked #1 on
Person Re-Identification
on Market-1501
(using extra training data)
Large-Scale Pre-training for Person Re-identification with Noisy Labels
Since theses ID labels automatically derived from tracklets inevitably contain noises, we develop a large-scale Pre-training framework utilizing Noisy Labels (PNL), which consists of three learning modules: supervised Re-ID learning, prototype-based contrastive learning, and label-guided contrastive learning.
Ranked #7 on
Person Re-Identification
on CUHK03
Semantic Image Synthesis via Diffusion Models
Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks compared with Generative Adversarial Nets (GANs).
CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet
Recent studies have shown that CLIP has achieved remarkable success in performing zero-shot inference while its fine-tuning performance is not satisfactory.
AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control
Neural implicit fields are powerful for representing 3D scenes and generating high-quality novel views, but it remains challenging to use such implicit representations for creating a 3D human avatar with a specific identity and artistic style that can be easily animated.
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
This paper explores a better prediction target for BERT pre-training of vision transformers.
BEVT: BERT Pretraining of Video Transformers
This design is motivated by two observations: 1) transformers learned on image datasets provide decent spatial priors that can ease the learning of video transformers, which are often times computationally-intensive if trained from scratch; 2) discriminative clues, i. e., spatial and temporal information, needed to make correct predictions vary among different videos due to large intra-class and inter-class variations.
Ranked #8 on
Action Recognition
on Diving-48
NeRF-Art: Text-Driven Neural Radiance Fields Stylization
As a powerful representation of 3D scenes, the neural radiance field (NeRF) enables high-quality novel view synthesis from multi-view images.
Reduce Information Loss in Transformers for Pluralistic Image Inpainting
The indices of quantized pixels are used as tokens for the inputs and prediction targets of transformer.
Ranked #6 on
Seeing Beyond the Visible
on KITTI360-EX
Transformer based Pluralistic Image Completion with Reduced Information Loss
The indices of quantized pixels are used as tokens for the inputs and prediction targets of the transformer.
E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion
This paper studies the problem of StyleGAN inversion, which plays an essential role in enabling the pretrained StyleGAN to be used for real image editing tasks.
HairCLIPv2: Unifying Hair Editing via Proxy Feature Blending
Even though they can enable very fine-grained local control, such interaction modes are inefficient for the editing conditions that can be easily specified by language descriptions or reference images.
Revisiting Dynamic Convolution via Matrix Decomposition
It has two limitations: (a) it increases the number of convolutional weights by K-times, and (b) the joint optimization of dynamic attention and static convolution kernels is challenging.
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis
Diffusion models (DMs) have shown great potential for high-quality image synthesis.
Cross-Domain and Disentangled Face Manipulation with 3D Guidance
Face image manipulation via three-dimensional guidance has been widely applied in various interactive scenarios due to its semantically-meaningful understanding and user-friendly controllability.
Bootstrapped Masked Autoencoders for Vision BERT Pretraining
The first design is motivated by the observation that using a pretrained MAE to extract the features as the BERT prediction target for masked tokens can achieve better pretraining performance.
Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models
Video-based large language models (Video-LLMs) have been recently introduced, targeting both fundamental improvements in perception and comprehension, and a diverse range of user inquiries.
Protecting Celebrities from DeepFake with Identity Consistency Transformer
In this work we propose Identity Consistency Transformer, a novel face forgery detection method that focuses on high-level semantics, specifically identity information, and detecting a suspect face by finding identity inconsistency in inner and outer face regions.
StyleBank: An Explicit Representation for Neural Image Style Transfer
It also enables us to conduct incremental learning to add a new image style by learning a new filter bank while holding the auto-encoder fixed.
Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning
For the choice of teacher models, we observe that students taught by video teachers perform better on temporally-heavy video tasks, while image teachers transfer stronger spatial representations for spatially-heavy video tasks.
Ranked #1 on
Self-Supervised Action Recognition
on HMDB51
Robust Equivariant Imaging: a fully unsupervised framework for learning to image from noisy and partial measurements
Deep networks provide state-of-the-art performance in multiple imaging inverse problems ranging from medical imaging to computational photography.
Streaming Video Model
We believe that the concept of streaming video model and the implementation of S-ViT are solid steps towards a unified deep learning architecture for video understanding.
HQ-50K: A Large-scale, High-quality Dataset for Image Restoration
This paper introduces a new large-scale image restoration dataset, called HQ-50K, which contains 50, 000 high-quality images with rich texture details and semantic diversity.
Diversity-Aware Meta Visual Prompting
We present Diversity-Aware Meta Visual Prompting~(DAM-VP), an efficient and effective prompting method for transferring pre-trained models to downstream tasks with frozen backbone.
Deep Model Intellectual Property Protection via Deep Watermarking
By jointly training the target model and watermark embedding, the extra barrier can even be absorbed into the target model.
Learning with Noisy Labels for Robust Point Cloud Segmentation
Point cloud segmentation is a fundamental task in 3D.
Robust Point Cloud Segmentation with Noisy Annotations
To handle boundary-level label noise, we also propose a variant ``PNAL-boundary " with a progressive boundary label cleaning strategy.
Efficient Semantic Image Synthesis via Class-Adaptive Normalization
Spatially-adaptive normalization (SPADE) is remarkably successful recently in conditional semantic image synthesis \cite{park2019semantic}, which modulates the normalized activation with spatially-varying transformations learned from semantic layouts, to prevent the semantic information from being washed away.
Equivariant Imaging: Learning Beyond the Range Space
In various imaging problems, we only have access to compressed measurements of the underlying signals, hindering most learning-based strategies which usually require pairs of signals and associated measurements for training.
Sensing Theorems for Unsupervised Learning in Linear Inverse Problems
In this paper, we present necessary and sufficient sensing conditions for learning the signal model from measurement data alone which only depend on the dimension of the model and the number of operators or properties of the group action that the model is invariant to.
Diverse Semantic Image Synthesis via Probability Distribution Modeling
In this paper, we propose a novel diverse semantic image synthesis framework from the perspective of semantic class distributions, which naturally supports diverse generation at semantic or even instance level.
Ranked #1 on
Image-to-Image Translation
on ADE20K Labels-to-Photos
(LPIPS metric)
Shape-invariant 3D Adversarial Point Clouds
In this paper, we propose a novel Point-Cloud Sensitivity Map to boost both the efficiency and imperceptibility of point perturbations.
GreedyFool: Distortion-Aware Sparse Adversarial Attack
Sparse adversarial samples are a special branch of adversarial samples that can fool the target model by only perturbing a few pixels.
REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering
Specifically, we observe that in most state-of-the-art knowledge-based VQA methods: 1) visual features are extracted either from the whole image or in a sliding window manner for retrieving knowledge, and the important relationship within/among object regions is neglected; 2) visual features are not well utilized in the final answering model, which is counter-intuitive to some extent.
Ranked #11 on
Visual Question Answering (VQA)
on OK-VQA
Generative Enhancement for 3D Medical Images
Our method begins with a 2D slice, noted as the informed slice to serve the patient prior, and propagates the generation process using a 3D segmentation mask.
Should All Proposals be Treated Equally in Object Detection?
We formulate this as a learning problem where the goal is to assign operators to proposals, in the detection head, so that the total computational cost is constrained and the precision is maximized.
Meta-PU: An Arbitrary-Scale Upsampling Network for Point Cloud
Thus, Meta-PU even outperforms the existing methods trained for a specific scale factor only.
Graphics
Stronger NAS with Weaker Predictors
We propose a paradigm shift from fitting the whole architecture space using one strong predictor, to progressively fitting a search path towards the high-performance sub-space through a set of weaker predictors.
X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusion
We demonstrate for the first time that using a text2image model to generate images or zero-shot recognition model to filter noisily crawled images for different object categories is a feasible way to make Copy-Paste truly scalable.
Ranked #7 on
Instance Segmentation
on LVIS v1.0 val
Deep Decomposition Learning for Inverse Imaging Problems
Deep learning is emerging as a new paradigm for solving inverse imaging problems.
MADAv2: Advanced Multi-Anchor Based Active Domain Adaptation Segmentation
Unsupervised domain adaption has been widely adopted in tasks with scarce annotated data.
Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles
Through our analysis, we find one important reason is that existing large-scale VL datasets do not contain much commonsense knowledge, which motivates us to improve the commonsense of VL-models from the data perspective.
Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations
This motivates us to shift the paradigm from combining loss at the end, to choosing the proper learning method per network layer.
Passport-aware Normalization for Deep Model Protection
Only when the model IP is suspected to be stolen by someone, the private passport-aware branch is added back for ownership verification.
Compressive MR Fingerprinting reconstruction with Neural Proximal Gradient iterations
Consistency of the predictions with respect to the physical forward model is pivotal for reliably solving inverse problems.
OmniVid: A Generative Framework for Universal Video Understanding
The core of video understanding tasks, such as recognition, captioning, and tracking, is to automatically detect objects or actions in a video and analyze their temporal evolution.
Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Prompting
Based on our analysis, we provide a simple yet effective way to boost the adversarial robustness of MAE.
Poison Ink: Robust and Invisible Backdoor Attack
As the image structure can keep its semantic meaning during the data transformation, such trigger pattern is inherently robust to data transformations.
Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation
We hypothesize that the latent representation learned from a pretrained generative T2V model encapsulates rich semantics and coherent temporal correspondences, thereby naturally facilitating video understanding.
Referring Video Object Segmentation
Semantic Segmentation
+2
Unsupervised Learning From Incomplete Measurements for Inverse Problems
In many real-world inverse problems, only incomplete measurement data are available for training which can pose a problem for learning a reconstruction function.
Real-Time Image Segmentation via Hybrid Convolutional-Transformer Architecture Search
Manually replacing convolution layers with multi-head self-attention is non-trivial due to the costly overhead in memory to maintain high resolution.
Stereoscopic Neural Style Transfer
This paper presents the first attempt at stereoscopic neural style transfer, which responds to the emerging demand for 3D movies or AR/VR.
Coherent Online Video Style Transfer
Training a feed-forward network for fast neural style transfer of images is proven to be successful.
Learning Discriminative Representation with Signed Laplacian Restricted Boltzmann Machine
We investigate the potential of a restricted Boltzmann Machine (RBM) for discriminative representation learning.
Rethinking Spatially-Adaptive Normalization
Despite its impressive performance, a more thorough understanding of the true advantages inside the box is still highly demanded, to help reduce the significant computation and parameter overheads introduced by these new structures.
Weak NAS Predictor Is All You Need
Rather than expecting a single strong predictor to model the whole space, we seek a progressive line of weak predictors that can connect a path to the best architecture, thus greatly simplifying the learning task of each predictor.
LG-GAN: Label Guided Adversarial Network for Flexible Targeted Attack of Point Cloud-based Deep Networks
To overcome these shortcomings, this paper proposes a novel label guided adversarial network (LG-GAN) for real-time flexible targeted point cloud attack.
MicroNet: Towards Image Recognition with Extremely Low FLOPs
In this paper, we present MicroNet, which is an efficient convolutional neural network using extremely low computational cost (e. g. 6 MFLOPs on ImageNet classification).
Identity-Driven DeepFake Detection
Our approach takes as input the suspect image/video as well as the target identity information (a reference image or video).
Are Fewer Labels Possible for Few-shot Learning?
We conduct experiments on 10 different few-shot target datasets, and our average few-shot performance outperforms both vanilla inductive unsupervised transfer and supervised transfer by a large margin.
Improved Image Matting via Real-time User Clicks and Uncertainty Estimation
Image matting is a fundamental and challenging problem in computer vision and graphics.
COIN: Contrastive Identifier Network for Breast Mass Diagnosis in Mammography
Computer-aided breast cancer diagnosis in mammography is a challenging problem, stemming from mammographical data scarcity and data entanglement.
Improve Unsupervised Pretraining for Few-label Transfer
Unsupervised pretraining has achieved great success and many recent works have shown unsupervised pretraining can achieve comparable or even slightly better transfer performance than supervised pretraining on downstream target datasets.
Exploring Structure Consistency for Deep Model Watermarking
However, little attention has been devoted to the protection of DNNs in image processing tasks.
Unsupervised Finetuning
This problem is more challenging than the supervised counterpart, as the low data density in the small-scale target data is not friendly for unsupervised learning, leading to the damage of the pretrained representation and poor representation in the target domain.
Reinforced Pipeline Optimization: Behaving Optimally with Non-Differentiabilities
Many machine learning systems are implemented as pipelines.
3D Question Answering
To this end, we propose a novel transformer-based 3DQA framework "3DQA-TR", which consists of two encoders for exploiting the appearance and geometry information, respectively.
Online Multi-Object Tracking with Unsupervised Re-Identification Learning and Occlusion Estimation
In addition, such practice of re-identification still can not track those highly occluded objects when they are missed by the detector.
Ranked #7 on
Multi-Object Tracking
on MOT16
(using extra training data)
Deep Unrolling for Magnetic Resonance Fingerprinting
Magnetic Resonance Fingerprinting (MRF) has emerged as a promising quantitative MR imaging approach.
Magnetic Resonance Fingerprinting
Rolling Shutter Correction
Self-supervised Transformer for Deepfake Detection
After pre-training with our method, the model will then be partially fine-tuned for deepfake detection task.
Residual Mixture of Experts
In this paper, we propose Residual Mixture of Experts (RMoE), an efficient training pipeline for MoE vision transformers on downstream tasks, such as segmentation and detection.
i-Code: An Integrative and Composable Multimodal Learning Framework
Human intelligence is multimodal; we integrate visual, linguistic, and acoustic signals to maintain a holistic worldview.
Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embedding
Detection Hub further achieves SoTA performance on UODB benchmark with wide variety of datasets.
MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining
Second, masked self-distillation is also consistent with vision-language contrastive from the perspective of training objective as both utilize the visual encoder for feature aligning, and thus is able to learn local semantics getting indirect supervision from the language.
Video Mobile-Former: Video Recognition with Efficient Global Spatial-temporal Modeling
To avoid significant computational cost incurred by computing self-attention between the large number of local patches in videos, we propose to use very few global tokens (e. g., 6) for a whole video in Transformers to exchange information with 3D-CNNs with a cross-attention mechanism.
Imaging with Equivariant Deep Learning
From early image processing to modern computational imaging, successful models and algorithms have relied on a fundamental property of natural signals: symmetry.
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
This paper presents OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture.
Ranked #4 on
Cross-Modal Retrieval
on Flickr30k
(using extra training data)
PointCAT: Contrastive Adversarial Training for Robust Point Cloud Recognition
Notwithstanding the prominent performance achieved in various applications, point cloud recognition models have often suffered from natural corruptions and adversarial perturbations.
Self-Supervised Learning based on Heat Equation
When transferring to object detection with frozen backbone, QB-Heat outperforms MoCo-v2 and supervised pre-training on ImageNet by 7. 9 and 4. 5 AP respectively.
Look Before You Match: Instance Understanding Matters in Video Object Segmentation
Towards this goal, we present a two-branch network for VOS, where the query-based instance segmentation (IS) branch delves into the instance details of the current frame and the VOS branch performs spatial-temporal matching with the memory bank.
Ranked #1 on
Semi-Supervised Video Object Segmentation
on Long Video Dataset
(using extra training data)
PA-GM: Position-Aware Learning of Embedding Networks for Deep Graph Matching
In this paper, we introduce a novel end-to-end neural network that can map the linear assignment problem into a high-dimensional space augmented with node-level relative position information, which is crucial for improving the method's performance for similar content matching.
OmniTracker: Unifying Object Tracking by Tracking-with-Detection
Object tracking (OT) aims to estimate the positions of target objects in a video sequence.
Domain Generalization for Mammographic Image Analysis with Contrastive Learning
The training of an efficacious deep learning model requires large data with diverse styles and qualities.
ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System
Existing deep video models are limited by specific tasks, fixed input-output spaces, and poor generalization capabilities, making it difficult to deploy them in real-world scenarios.
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data
The convergence of text, visual, and audio data is a key step towards human-like artificial intelligence, however the current Vision-Language-Speech landscape is dominated by encoder-only models which lack generative abilities.
Album Storytelling with Iterative Story-aware Captioning and Large Language Models
One natural approach is to use caption models to describe each photo in the album, and then use LLMs to summarize and rewrite the generated captions into an engaging story.
Image as First-Order Norm+Linear Autoregression: Unveiling Mathematical Invariance
This paper introduces a novel mathematical property applicable to diverse images, referred to as FINOLA (First-Order Norm+Linear Autoregressive).
On the Hidden Waves of Image
We term this phenomenon hidden waves, as it reveals that, although the speeds of the set of wave equations and autoregressive coefficient matrices are latent, they are both learnable and shared across images.
PersonMAE: Person Re-Identification Pre-Training with Masked AutoEncoders
Pre-training is playing an increasingly important role in learning generic feature representation for Person Re-identification (ReID).
Uni-COAL: A Unified Framework for Cross-Modality Synthesis and Super-Resolution of MR Images
Cross-modality synthesis (CMS), super-resolution (SR), and their combination (CMSR) have been extensively studied for magnetic resonance imaging (MRI).
Traffic Video Object Detection using Motion Prior
In this work, we propose two innovative methods to exploit the motion prior and boost the performance of both fully-supervised and semi-supervised traffic video object detection.
TrafficMOT: A Challenging Dataset for Multi-Object Tracking in Complex Traffic Scenarios
Multi-object tracking in traffic videos is a crucial research area, offering immense potential for enhancing traffic monitoring accuracy and promoting road safety measures through the utilisation of advanced machine learning algorithms.
Mesh-Guided Neural Implicit Field Editing
We first introduce a differentiable method using marching tetrahedra for polygonal mesh extraction from the neural implicit field and then design a differentiable color extractor to assign colors obtained from the volume renderings to this extracted mesh.
Towards More Unified In-context Visual Understanding
Thanks to this design, the model is capable of handling in-context vision understanding tasks with multimodal output in a unified pipeline. Experimental results demonstrate that our model achieves competitive performance compared with specialized models and previous ICL baselines.
Image Fusion via Vision-Language Model
Therefore, we introduce a novel fusion paradigm named image Fusion via vIsion-Language Model (FILM), for the first time, utilizing explicit textual information in different source images to guide image fusion.
Diffusion Posterior Proximal Sampling for Image Restoration
Specifically, we opt for a sample consistent with the measurement identity at each generative step, exploiting the sampling selection as an avenue for output stability and enhancement.
