Search Results for author:
Found 104 papers, 66 papers with code
OmniTracker: Unifying Object Tracking by Tracking-with-Detection
Object tracking (OT) aims to estimate the positions of target objects in a video sequence.
Cognitive Semantic Communication Systems Driven by Knowledge Graph: Principle, Implementation, and Performance Evaluation
Extensive simulation results conducted on a public dataset demonstrate that our proposed single-user and multi-user cognitive semantic communication systems are superior to benchmark communication systems in terms of the data compression rate and communication reliability.
Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations
This motivates us to shift the paradigm from combining loss at the end, to choosing the proper learning method per network layer.
LC-NeRF: Local Controllable Face Generation in Neural Randiance Field
Since changes to the latent code affect global generation results, these methods do not allow for fine-grained control of local facial regions.
Generalized Decoding for Pixel, Image, and Language
We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly.
Ranked #3 on
Instance Segmentation
on ADE20K val
(using extra training data)
Look Before You Match: Instance Understanding Matters in Video Object Segmentation
Towards this goal, we present a two-branch network for VOS, where the query-based instance segmentation (IS) branch delves into the instance details of the current frame and the VOS branch performs spatial-temporal matching with the memory bank.
Ranked #1 on
Semi-Supervised Video Object Segmentation
on Long Video Dataset
(using extra training data)
CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet
Recent studies have shown that CLIP has achieved remarkable success in performing zero-shot inference while its fine-tuning performance is not satisfactory.
Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning
For the choice of teacher models, we observe that students taught by video teachers perform better on temporally-heavy video tasks, while image teachers transfer stronger spatial representations for spatially-heavy video tasks.
Ranked #1 on
Action Recognition
on Something-Something V2
(using extra training data)
X-Paste: Revisit Copy-Paste at Scale with CLIP and StableDiffusion
We demonstrate for the first time that using a text2image model to generate images or zero-shot recognition model to filter noisily crawled images for different object categories is a feasible way to make Copy-Paste truly scalable.
Ranked #4 on
Instance Segmentation
on LVIS v1.0 val
Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles
Through our analysis, we find one important reason is that existing large-scale VL datasets do not contain much commonsense knowledge, which motivates us to improve the commonsense of VL-models from the data perspective.
Self-Supervised Learning based on Heat Equation
When transferring to object detection with frozen backbone, QB-Heat outperforms MoCo-v2 and supervised pre-training on ImageNet by 7. 9 and 4. 5 AP respectively.
SinDiffusion: Learning a Diffusion Model from a Single Natural Image
We present SinDiffusion, leveraging denoising diffusion models to capture internal distribution of patches from a single natural image.
Ranked #1 on
Image Generation
on Places50
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
This paper presents OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture.
Ranked #2 on
Zero-Shot Video Retrieval
on MSR-VTT
(using extra training data)
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis
Diffusion models (DMs) have shown great potential for high-quality image synthesis.
MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining
This paper presents a simple yet effective framework MaskCLIP, which incorporates a newly proposed masked self-distillation into contrastive language-image pretraining.
Video Mobile-Former: Video Recognition with Efficient Global Spatial-temporal Modeling
To avoid significant computational cost incurred by computing self-attention between the large number of local patches in videos, we propose to use very few global tokens (e. g., 6) for a whole video in Transformers to exchange information with 3D-CNNs with a cross-attention mechanism.
Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training
Large-scale multi-modal contrastive pre-training has demonstrated great utility to learn transferable features for a range of downstream tasks by mapping multiple modalities into a shared embedding space.
TinyViT: Fast Pretraining Distillation for Small Vision Transformers
It achieves a top-1 accuracy of 84. 8% on ImageNet-1k with only 21M parameters, being comparable to Swin-B pretrained on ImageNet-21k while using 4. 2 times fewer parameters.
Ranked #120 on
Image Classification
on ImageNet
(using extra training data)
Bootstrapped Masked Autoencoders for Vision BERT Pretraining
The first design is motivated by the observation that using a pretrained MAE to extract the features as the BERT prediction target for masked tokens can achieve better pretraining performance.
Should All Proposals be Treated Equally in Object Detection?
We formulate this as a learning problem where the goal is to assign operators to proposals, in the detection head, so that the total computational cost is constrained and the precision is maximized.
Semantic Image Synthesis via Diffusion Models
Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks compared with Generative Adversarial Nets (GANs).
GLIPv2: Unifying Localization and Vision-Language Understanding
We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e. g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e. g., VQA, image captioning).
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Test
(using extra training data)
Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embedding
Detection Hub further achieves SoTA performance on UODB benchmark with wide variety of datasets.
Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning
Thanks to the strong zero-shot capability of foundation models, we start by constructing a rich semantic representation of the image (e. g., image tags, object attributes / locations, captions) as a structured textual prompt, called visual clues, using a vision foundation model.
REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering
Specifically, we observe that in most state-of-the-art knowledge-based VQA methods: 1) visual features are extracted either from the whole image or in a sliding window manner for retrieving knowledge, and the important relationship within/among object regions is neglected; 2) visual features are not well utilized in the final answering model, which is counter-intuitive to some extent.
Ranked #4 on
Visual Question Answering (VQA)
on OK-VQA
Reduce Information Loss in Transformers for Pluralistic Image Inpainting
The indices of quantized pixels are used as tokens for the inputs and prediction targets of transformer.
Ranked #6 on
Seeing Beyond the Visible
on KITTI360-EX
i-Code: An Integrative and Composable Multimodal Learning Framework
Human intelligence is multimodal; we integrate visual, linguistic, and acoustic signals to maintain a holistic worldview.
Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks
Experiments demonstrate that MAD leads to consistent gains in the low-shot, domain-shifted, and fully-supervised conditions on VCR, SNLI-VE, and VQA, achieving SOTA performance on VCR compared to other single models pretrained with image-text data.
Ranked #3 on
Visual Question Answering (VQA)
on VCR (Q-A) test
Residual Mixture of Experts
In this paper, we propose Residual Mixture of Experts (RMoE), an efficient training pipeline for MoE vision transformers on downstream tasks, such as segmentation and detection.
K-LITE: Learning Transferable Visual Models with External Knowledge
We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts.
MiniViT: Compressing Vision Transformers with Weight Multiplexing
The central idea of MiniViT is to multiplex the weights of consecutive transformer blocks.
Ranked #186 on
Image Classification
on ImageNet
Unified Contrastive Learning in Image-Text-Label Space
Particularly, it attains gains up to 9. 2% and 14. 5% in average on zero-shot recognition benchmarks over the language-image contrastive learning and supervised learning methods, respectively.
DaViT: Dual Attention Vision Transformers
We show that these two self-attentions complement each other: (i) since each channel token contains an abstract representation of the entire image, the channel attention naturally captures global interactions and representations by taking all spatial positions into account when computing attention scores between channels; (ii) the spatial attention refines the local representations by performing fine-grained interactions across spatial locations, which in turn helps the global information modeling in channel attention.
Ranked #11 on
Image Classification
on ImageNet
(using extra training data)
Large-Scale Pre-training for Person Re-identification with Noisy Labels
Since theses ID labels automatically derived from tracklets inevitably contain noises, we develop a large-scale Pre-training framework utilizing Noisy Labels (PNL), which consists of three learning modules: supervised Re-ID learning, prototype-based contrastive learning, and label-guided contrastive learning.
Ranked #6 on
Person Re-Identification
on CUHK03
Focal Modulation Networks
For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2. 4, and beats Swin at multi-scale (50. 5 v. s.
Ranked #5 on
Object Detection
on COCO minival
(using extra training data)
CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
Experiments demonstrate that our proposed CLIP-TD leads to exceptional gains in the low-shot (up to 51. 9%) and domain-shifted (up to 71. 3%) conditions of VCR, while simultaneously improving performance under standard fully-supervised conditions (up to 2%), achieving state-of-art performance on VCR compared to other single models that are pretrained with image-text data only.
Online Multi-Object Tracking with Unsupervised Re-Identification Learning and Occlusion Estimation
In addition, such practice of re-identification still can not track those highly occluded objects when they are missed by the detector.
Ranked #7 on
Multi-Object Tracking
on MOT16
(using extra training data)
RegionCLIP: Region-based Language-Image Pretraining
However, we show that directly applying such models to recognize image regions for object detection leads to poor performance due to a domain shift: CLIP was trained to match an image as a whole to a text description, without capturing the fine-grained alignment between image regions and text spans.
Ranked #4 on
Open Vocabulary Object Detection
on MSCOCO
(using extra training data)
HairCLIP: Design Your Hair by Text and Reference Image
Hair editing is an interesting and challenging problem in computer vision and graphics.
Grounded Language-Image Pre-training
The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representation semantic-rich.
Ranked #1 on
2D object detection
on RF100
General Facial Representation Learning in a Visual-Linguistic Manner
In this paper, we study the transfer performance of pre-trained models on face analysis tasks and introduce a framework, called FaRL, for general Facial Representation Learning in a visual-linguistic manner.
Ranked #1 on
Face Parsing
on CelebAMask-HQ
(using extra training data)
BEVT: BERT Pretraining of Video Transformers
This design is motivated by two observations: 1) transformers learned on image datasets provide decent spatial priors that can ease the learning of video transformers, which are often times computationally-intensive if trained from scratch; 2) discriminative clues, i. e., spatial and temporal information, needed to make correct predictions vary among different videos due to large intra-class and inter-class variations.
Ranked #4 on
Action Recognition
on Diving-48
Focal Attention for Long-Range Interactions in Vision Transformers
With focal attention, we propose a new variant of Vision Transformer models, called Focal Transformers, which achieve superior performance over the state-of-the-art (SoTA) Vision Transformers on a range of public image classification and object detection benchmarks.
Vector Quantized Diffusion Model for Text-to-Image Synthesis
Our experiments indicate that the VQ-Diffusion model with the reparameterization is fifteen times faster than traditional AR methods while achieving a better image quality.
Ranked #1 on
Text-to-Image Generation
on Oxford 102 Flowers
(using extra training data)
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
This paper explores a better prediction target for BERT pre-training of vision transformers.
Florence: A New Foundation Model for Computer Vision
Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications.
Ranked #1 on
Action Recognition In Videos
on Kinetics-600
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks.
Ranked #15 on
Cross-Modal Retrieval
on COCO 2014
Unsupervised Finetuning
This problem is more challenging than the supervised counterpart, as the low data density in the small-scale target data is not friendly for unsupervised learning, leading to the damage of the pretrained representation and poor representation in the target domain.
MA-CLIP: Towards Modality-Agnostic Contrastive Language-Image Pre-training
Large-scale multimodal contrastive pretraining has demonstrated great utility to support high performance in a range of downstream tasks by mapping multiple modalities into a shared embedding space.
Automatic Modulation Classification Using Involution Enabled Residual Networks
Automatic modulation classification (AMC) is of crucial importance for realizing wireless intelligence communications.
MicroNet: Improving Image Recognition with Extremely Low FLOPs
This paper aims at addressing the problem of substantial performance degradation at extremely low computational cost (e. g. 5M FLOPs on ImageNet classification).
Mobile-Former: Bridging MobileNet and Transformer
This structure leverages the advantages of MobileNet at local processing and transformer at global interaction.
Improve Unsupervised Pretraining for Few-label Transfer
Unsupervised pretraining has achieved great success and many recent works have shown unsupervised pretraining can achieve comparable or even slightly better transfer performance than supervised pretraining on downstream target datasets.
Focal Self-attention for Local-Global Interactions in Vision Transformers
With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers on a range of public image classification and object detection benchmarks.
Ranked #14 on
Instance Segmentation
on COCO test-dev
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
By further pretraining on the larger dataset ImageNet-21K, we achieve 87. 5% Top-1 accuracy on ImageNet-1K and high segmentation performance on ADE20K with 55. 7 mIoU.
Ranked #23 on
Semantic Segmentation
on ADE20K val
Efficient Self-supervised Vision Transformers for Representation Learning
This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning.
Ranked #3 on
Self-Supervised Image Classification
on ImageNet
Representation Learning
Self-Supervised Image Classification
Dynamic Head: Unifying Object Detection Heads with Attentions
In this paper, we present a novel dynamic head framework to unify object detection heads with attentions.
Ranked #1 on
Object Detection
on COCO 2017 val
(AP75 metric)
Chasing Sparsity in Vision Transformers: An End-to-End Exploration
For example, our sparsified DeiT-Small at (5%, 50%) sparsity for (data, architecture), improves 0. 28% top-1 accuracy, and meanwhile enjoys 49. 32% FLOPs and 4. 40% running time savings.
E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion
This paper studies the problem of StyleGAN inversion, which plays an essential role in enabling the pretrained StyleGAN to be used for real image editing tasks.
Lite-HRNet: A Lightweight High-Resolution Network
We introduce a lightweight unit, conditional channel weighting, to replace costly pointwise (1x1) convolutions in shuffle blocks.
Ranked #37 on
Pose Estimation
on COCO test-dev
CvT: Introducing Convolutions to Vision Transformers
We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs.
Ranked #2 on
Image Classification
on Flowers-102
(using extra training data)
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding
This paper presents a new Vision Transformer (ViT) architecture Multi-Scale Vision Longformer, which significantly enhances the ViT of \cite{dosovitskiy2020image} for encoding high-resolution images using two techniques.
Ranked #32 on
Instance Segmentation
on COCO minival
Dynamic Transfer for Multi-Source Domain Adaptation
However, such a static model is difficult to handle conflicts across multiple domains, and suffers from a performance degradation in both source domains and target domain.
Revisiting Dynamic Convolution via Matrix Decomposition
It has two limitations: (a) it increases the number of convolutional weights by K-times, and (b) the joint optimization of dynamic attention and static convolution kernels is challenging.
Stronger NAS with Weaker Predictors
We propose a paradigm shift from fitting the whole architecture space using one strong predictor, to progressively fitting a search path towards the high-performance sub-space through a set of weaker predictors.
Weak NAS Predictor Is All You Need
Rather than expecting a single strong predictor to model the whole space, we seek a progressive line of weak predictors that can connect a path to the best architecture, thus greatly simplifying the learning task of each predictor.
Dynamic DETR: End-to-End Object Detection With Dynamic Attention
To mitigate the second limitation of learning difficulty, we introduce a dynamic decoder by replacing the cross-attention module with a ROI-based dynamic attention in the Transformer decoder.
Are Fewer Labels Possible for Few-shot Learning?
We conduct experiments on 10 different few-shot target datasets, and our average few-shot performance outperforms both vanilla inductive unsupervised transfer and supervised transfer by a large margin.
Efficient Semantic Image Synthesis via Class-Adaptive Normalization
Spatially-adaptive normalization (SPADE) is remarkably successful recently in conditional semantic image synthesis \cite{park2019semantic}, which modulates the normalized activation with spatially-varying transformations learned from semantic layouts, to prevent the semantic information from being washed away.
Unsupervised Pre-training for Person Re-identification
In this paper, we present a large scale unlabeled person re-identification (Re-ID) dataset "LUPerson" and make the first attempt of performing unsupervised pre-training for improving the generalization ability of the learned person Re-ID feature representation.
Ranked #2 on
Person Re-Identification
on Market-1501
(using extra training data)
MicroNet: Towards Image Recognition with Extremely Low FLOPs
In this paper, we present MicroNet, which is an efficient convolutional neural network using extremely low computational cost (e. g. 6 MFLOPs on ImageNet classification).
MichiGAN: Multi-Input-Conditioned Hair Image Generation for Portrait Editing
In this paper, we present MichiGAN (Multi-Input-Conditioned Hair Image GAN), a novel conditional image generation method for interactive portrait hair manipulation.
GreedyFool: Distortion-Aware Sparse Adversarial Attack
Sparse adversarial samples are a special branch of adversarial samples that can fool the target model by only perturbing a few pixels.
LSM: Learning Subspace Minimization for Low-level Vision
We study the energy minimization problem in low-level vision tasks from a novel perspective.
Cross-domain Correspondence Learning for Exemplar-based Image Translation
The output has the style (e. g., color, texture) in consistency with the semantically corresponding objects in the exemplar.
Ranked #1 on
Image-to-Image Translation
on ADE20K-Outdoor Labels-to-Photos
(FID metric)
Rethinking Spatially-Adaptive Normalization
Despite its impressive performance, a more thorough understanding of the true advantages inside the box is still highly demanded, to help reduce the significant computation and parameter overheads introduced by these new structures.
Density-Aware Graph for Deep Semi-Supervised Visual Recognition
Motivated by these limitations, this paper proposes to solve the SSL problem by building a novel density-aware graph, based on which the neighborhood information can be easily leveraged and the feature learning and label propagation can also be trained in an end-to-end way.
DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search
One common way is searching on a smaller proxy dataset (e. g., CIFAR-10) and then transferring to the target task (e. g., ImageNet).
Dynamic Convolution: Attention over Convolution Kernels
Light-weight convolutional neural networks (CNNs) suffer performance degradation as their low computational budgets constrain both the depth (number of convolution layers) and the width (number of channels) of CNNs, resulting in limited representation capability.
Ranked #801 on
Image Classification
on ImageNet
A General Decoupled Learning Framework for Parameterized Image Operators
To overcome this limitation, we propose a new decoupled learning algorithm to learn from the operator parameters to dynamically adjust the weights of a deep network for image operators, denoted as the base network.
Deep Exemplar-based Video Colorization
This paper presents the first end-to-end network for exemplar-based video colorization.
Face Parsing with RoI Tanh-Warping
It uses hierarchical local based method for inner facial components and global methods for outer facial components.
Mask-Guided Portrait Editing with Conditional GANs
Portrait editing is a popular subject in photo manipulation.
Bidirectional Learning for Domain Adaptation of Semantic Segmentation
In this paper, we propose a novel bidirectional learning framework for domain adaptation of segmentation.
Ranked #7 on
Semantic Segmentation
on DADA-seg
Rethinking Classification and Localization for Object Detection
Two head structures (i. e. fully connected head and convolution head) have been widely used in R-CNN based detectors for classification and localization tasks.
Gated Context Aggregation Network for Image Dehazing and Deraining
Image dehazing aims to recover the uncorrupted content from a hazy image.
Ranked #1 on
Rain Removal
on DID-MDN
CariGANs: Unpaired Photo-to-Caricature Translation
Facial caricature is an art form of drawing faces in an exaggerated way to convey humor or sarcasm.
Reinforced Pipeline Optimization: Behaving Optimally with Non-Differentiabilities
Many machine learning systems are implemented as pipelines.
Decouple Learning for Parameterized Image Operators
Many different deep networks have been used to approximate, accelerate or improve traditional image operators, such as image smoothing, super-resolution and denoising.
Deep Exemplar-based Colorization
More importantly, as opposed to other learning-based colorization methods, our network allows the user to achieve customizable results by simply feeding different references.
Arbitrary Style Transfer with Deep Feature Reshuffle
We theoretically prove that our new style loss based on reshuffle connects both global and local style losses respectively used by most parametric and non-parametric neural style transfer methods.
Towards High Performance Video Object Detection for Mobiles
In this paper, we present a light weight network architecture for video object detection on mobiles.
Stereoscopic Neural Style Transfer
This paper presents the first attempt at stereoscopic neural style transfer, which responds to the emerging demand for 3D movies or AR/VR.
Towards High Performance Video Object Detection
There has been significant progresses for image object detection in recent years.
Progressive Color Transfer with Dense Semantic Correspondences
The proposed method can be successfully extended from one-to-one to one-to-many color transfer.
Visual Attribute Transfer through Deep Image Analogy
We propose a new technique for visual attribute transfer across images that may have very different appearance but have perceptually similar semantic structure.
Flow-Guided Feature Aggregation for Video Object Detection
The accuracy of detection suffers from degenerated object appearances in videos, e. g., motion blur, video defocus, rare poses, etc.
Ranked #19 on
Video Object Detection
on ImageNet VID
Coherent Online Video Style Transfer
Training a feed-forward network for fast neural style transfer of images is proven to be successful.
StyleBank: An Explicit Representation for Neural Image Style Transfer
It also enables us to conduct incremental learning to add a new image style by learning a new filter bank while holding the auto-encoder fixed.
Deep Feature Flow for Video Recognition
Yet, it is non-trivial to transfer the state-of-the-art image recognition networks to videos as per-frame evaluation is too slow and unaffordable.
Ranked #9 on
Video Semantic Segmentation
on Cityscapes val
Image Deblurring Using Smartphone Inertial Sensors
Removing image blur caused by camera shake is an ill-posed problem, as both the latent image and the point spread function (PSF) are unknown.
Dual-Feature Warping-Based Motion Model Estimation
Line segment is a prominent feature in artificial environments and it can supply sufficient geometrical and structural information of scenes, which not only helps guild to a correct warp in low-texture condition, but also prevents the undesired distortion induced by warping.
SteadyFlow: Spatially Smooth Optical Flow for Video Stabilization
We propose a novel motion model, SteadyFlow, to represent the motion between neighboring video frames for stabilization.
