Search Results for author:
Found 337 papers, 146 papers with code
Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding
We present Pixel-SAIL, a single transformer for pixel-wise MLLM tasks.
The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer
This paper introduces SAIL, a single transformer unified multimodal large language model (MLLM) that integrates raw pixel encoding and language decoding within a singular architecture.
Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model
This technical report highlights the key design decisions that enhance the performance of the medium-sized diffusion model.
GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation
In autoregressive (AR) image generation, visual tokenizers compress images into compact discrete latent tokens, enabling efficient training of downstream autoregressive models for visual generation via next-token prediction.
4th PVUW MeViS 3rd Place Report: Sa2VA
Referring video object segmentation (RVOS) is a challenging task that requires the model to segment the object in a video given the language description.
Bridging Continuous and Discrete Tokens for Autoregressive Visual Generation
Autoregressive visual generation models typically rely on tokenizers to compress images into tokens that can be predicted sequentially.
MagicArticulate: Make Your 3D Models Articulation-Ready
With the explosive growth of 3D content creation, there is an increasing demand for automatically converting static 3D models into articulation-ready versions that support realistic animation.
Video Depth Anything: Consistent Depth Estimation for Super-Long Videos
The model is trained on a joint dataset of video depth and unlabeled images, similar to Depth Anything V2.
VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
This work explores whether a deep generative model can learn complex knowledge solely from visual input, in contrast to the prevalent focus on text-based models like large language models (LLMs).
Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos
This work presents Sa2VA, the first unified model for dense grounded understanding of both images and videos.
Ranked #1 on
Referring Video Object Segmentation
on MeViS
Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoders
However, the widely adopted uniform point sampling strategy in Shape VAE training often leads to a significant loss of geometric details, limiting the quality of shape reconstruction and downstream generation tasks.
Parallelized Autoregressive Visual Generation
Autoregressive models have emerged as a powerful approach for visual generation but suffer from slow inference speed due to their sequential token-by-token prediction process.
Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation
Prompts play a critical role in unleashing the power of language and vision foundation models for specific tasks.
Image Understanding Makes for A Good Tokenizer for Image Generation
Currently, pixel reconstruction (e. g., VQGAN) dominates the training objective for image tokenizers.
Classification Done Right for Vision-Language Pre-Training
Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does.
How Far is Video Generation from World Model: A Physical Law Perspective
Our scaling experiments show perfect generalization within the distribution, measurable scaling behavior for combinatorial generalization, but failure in out-of-distribution scenarios.
DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution
MLLMs have demonstrated remarkable comprehension and reasoning capabilities with complex language and visual data.
LVD-2M: A Long-take Video Dataset with Temporally Dense Captions
The efficacy of video generation models heavily depends on the quality of their training datasets.
Loong: Generating Minute-level Long Videos with Autoregressive Language Models
Autoregressive large language models (LLMs) have achieved great success in generating coherent and long sequences of tokens in the domain of natural language processing, while the exploration of autoregressive LLMs for video generation is limited to generating short videos of several seconds.
High Quality Human Image Animation using Regional Supervision and Motion Blur Condition
Although generating reasonable results, existing methods often overlook the need for regional supervision in crucial areas such as the face and hands, and neglect the explicit modeling for motion blur, leading to unrealistic low-quality synthesis.
Hierarchical Memory for Long Video QA
This paper describes our champion solution to the LOVEU Challenge @ CVPR'24, Track 1 (Long Video VQA).
Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams
Our model is able to process extremely long video streams in real-time and respond to user queries simultaneously.
Ranked #1 on
Zero-Shot Video Question Answer
on MSRVTT-QA
DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention
In this paper, we aim to incorporate the sub-quadratic modeling capability of Gated Linear Attention (GLA) into the 2D diffusion backbone.
LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videos
Accuracy and speed are critical in image editing tasks.
PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Accelerator
We present Piecewise Rectified Flow (PeRFlow), a flow-based method for accelerating diffusion models.
StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation
This module converts the generated sequence of images into videos with smooth transitions and consistent subjects that are significantly more stable than the modules based on latent spaces only, especially in the context of long video generation.
PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning
PLLaVA achieves new state-of-the-art performance on modern benchmark datasets for both video question-answer and captioning tasks.
Magic-Boost: Boost 3D Generation with Multi-View Conditioned Diffusion
In detail, we first propose a novel multi-view conditioned diffusion model which extracts 3d prior from the synthesized multi-view images to synthesize high-fidelity novel view images and then introduce a novel iterative-update strategy to adopt it to provide precise guidance to refine the coarse generated results through a fast optimization process.
Magic-Me: Identity-Specific Video Customized Diffusion
To achieve this, we propose three novel components that are essential for high-quality identity preservation and stable video generation: 1) a noise initialization method with 3D Gaussian Noise Prior for better inter-frame stability; 2) an ID module based on extended Textual Inversion trained with the cropped identity to disentangle the ID information from the background 3) Face VCD and Tiled VCD modules to reinforce faces and upscale the video to higher resolution while preserving the identity's features.
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
To this end, we scale up the dataset by designing a data engine to collect and automatically annotate large-scale unlabeled data (~62M), which significantly enlarges the data coverage and thus is able to reduce the generalization error.
Ranked #4 on
Monocular Depth Estimation
on ETH3D
MagicVideo-V2: Multi-Stage High-Aesthetic Video Generation
The growing demand for high-fidelity video generation from textual descriptions has catalyzed significant research in this field.
VISTA-LLAMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens
This amplifies the effect of visual tokens on text generation especially when the relative distance is longer between visual and text tokens.
Harnessing Diffusion Models for Visual Perception with Meta Prompts
Our key insight is to introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception.
Ranked #2 on
Semantic Segmentation
on Cityscapes test
Video Recognition in Portrait Mode
While existing datasets mainly comprise landscape mode videos, our paper seeks to introduce portrait mode videos to the research community and highlight the unique challenges associated with this video format.
DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generation
The generation of emotional talking faces from a single portrait image remains a significant challenge.
Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method
However, since the off-the-shelf pre-trained networks are trained on clean images, the one-step estimation procedure of the clean image may be inaccurate, especially in the early stages of the generation process in diffusion models.
Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens
This amplifies the effect of visual tokens on text generation, especially when the relative distance is longer between visual and text tokens.
Ranked #2 on
Question Answering
on NExT-QA (Open-ended VideoQA)
AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text
We study the problem of creating high-fidelity and animatable 3D avatars from only textual descriptions.
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion.
MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration
Large Language Models (LLMs) have significantly advanced natural language processing, demonstrating exceptional reasoning, tool usage, and memory capabilities.
EPIM: Efficient Processing-In-Memory Accelerators based on Epitome
On the hardware side, we modify the datapath of current PIM accelerators to accommodate epitomes and implement a feature map reuse technique to reduce computation cost.
ChatAnything: Facetime Chat with LLM-Enhanced Personas
For MoV, we utilize the text-to-speech (TTS) algorithms with a variety of pre-defined tones and select the most matching one based on the user-provided text description automatically.
Low-Resolution Self-Attention for Semantic Segmentation
Semantic segmentation tasks naturally require high-resolution information for pixel-wise segmentation and global context information for class prediction.
GETAvatar: Generative Textured Meshes for Animatable Human Avatars
We study the problem of 3D-aware full-body human generation, aiming at creating animatable human avatars with high-quality textures and geometries.
MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask
In this work, we identify that a crucial factor leading to the text-image mismatch issue is the inadequate cross-modality relation learning between the prompt and the output image.
MagicProp: Diffusion-based Video Editing via Motion-aware Appearance Propagation
The flexibility of these techniques enables the editing of arbitrary regions within the frame.
MagicEdit: High-Fidelity and Temporally Coherent Video Editing
In this report, we present MagicEdit, a surprisingly simple yet effective solution to the text-guided video editing task.
MagicAvatar: Multimodal Avatar Generation and Animation
This report presents MagicAvatar, a framework for multimodal video generation and animation of human avatars.
Dataset Quantization
Extensive experiments demonstrate that DQ is able to generate condensed small datasets for training unseen network architectures with state-of-the-art compression ratios for lossless model training.
AdjointDPM: Adjoint Sensitivity Method for Gradient Backpropagation of Diffusion Probabilistic Models
Existing customization methods require access to multiple reference examples to align pre-trained diffusion probabilistic models (DPMs) with user-provided concepts.
BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
Our experiments show that BuboGPT achieves impressive multi-modality understanding and visual grounding abilities during the interaction with human.
COSA: Concatenated Sample Pretrained Vision-Language Foundation Model
Due to the limited scale and quality of video-text training corpus, most vision-language foundation models employ image-text datasets for pretraining and primarily focus on modeling visually semantic representations while disregarding temporal semantic representations and correlations.
Ranked #1 on
TGIF-Frame
on TGIF-QA
(using extra training data)
Delving Deeper into Data Scaling in Masked Image Modeling
The study reveals that: 1) MIM can be viewed as an effective method to improve the model capacity when the scale of the training data is relatively small; 2) Strong reconstruction targets can endow the models with increased capacities on downstream tasks; 3) MIM pre-training is data-agnostic under most scenarios, which means that the strategy of sampling pre-training data is non-critical.
VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blending
Towards this goal, we propose a novel video-text pre-training method dubbed VLAB: Video Language pre-training by feature Adapting and Blending, which transfers CLIP representations to video pre-training tasks and develops unified video multimodal models for a wide range of video-text tasks.
Ranked #1 on
Visual Question Answering (VQA)
on MSVD-QA
(using extra training data)
Associating Spatially-Consistent Grouping with Text-supervised Semantic Segmentation
In this work, we investigate performing semantic segmentation solely through the training on image-sentence pairs.
DOAD: Decoupled One Stage Action Detection Network
Specifically, one branch focuses on detection representation for actor detection, and the other one for action recognition.
OmniAvatar: Geometry-Guided Controllable 3D Head Synthesis
We present OmniAvatar, a novel geometry-guided 3D head synthesis model trained from in-the-wild unstructured images that is capable of synthesizing diverse identity-preserved 3D heads with compelling dynamic details under full disentangled control over camera poses, facial expressions, head shapes, articulated neck and jaw poses.
AgileGAN3D: Few-Shot 3D Portrait Stylization by Augmented Transfer Learning
Capitalizing on the recent advancement of 3D-aware GAN models, we perform \emph{guided transfer learning} on a pretrained 3D GAN generator to produce multi-view-consistent stylized renderings.
TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision
We conduct extensive experiments to analyze each of our proposed components and show the efficacy of our framework in generating high-fidelity 3D textured and text-relevant shapes.
Global Knowledge Calibration for Fast Open-Vocabulary Segmentation
Recent advancements in pre-trained vision-language models, such as CLIP, have enabled the segmentation of arbitrary concepts solely from textual inputs, a process commonly referred to as open-vocabulary semantic segmentation (OVS).
Knowledge Distillation
Open Vocabulary Semantic Segmentation
+4
Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring
In this paper, based on the CLIP model, we revisit temporal modeling in the context of image-to-video knowledge transferring, which is the key point for extending image-text pretrained models to the video domain.
Ranked #7 on
Video Retrieval
on MSR-VTT-1kA
(using extra training data)
MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval
The first is a Temporal Adaptation Module that is incorporated in the video branch to introduce global and local temporal contexts.
Temporal Perceiving Video-Language Pre-training
Specifically, text-video localization consists of moment retrieval, which predicts start and end boundaries in videos given the text description, and text localization which matches the subset of texts with the video features.
CMAE-V: Contrastive Masked Autoencoders for Video Action Recognition
Contrastive Masked Autoencoder (CMAE), as a new self-supervised framework, has shown its potential of learning expressive feature representations in visual image recognition.
Class Prototype-based Cleaner for Label Noise Learning
Semi-supervised learning based methods are current SOTA solutions to the noisy-label learning problem, which rely on learning an unsupervised label cleaner first to divide the training samples into a labeled set for clean data and an unlabeled set for noise data.
Ranked #3 on
Image Classification
on Clothing1M
(using extra training data)
PV3D: A 3D Generative Model for Portrait Video Generation
While some prior works have applied such image GANs to unconditional 2D portrait video generation and static 3D portrait synthesis, there are few works successfully extending GANs for generating 3D-aware portrait videos.
Diffusion Probabilistic Model Made Slim
Despite the recent visually-pleasing results achieved, the massive computational cost has been a long-standing flaw for diffusion probabilistic models (DPMs), which, in turn, greatly limits their applications on resource-limited platforms.
AvatarGen: A 3D Generative Model for Animatable Human Avatars
Specifically, we decompose the generative 3D human synthesis into pose-guided mapping and canonical representation with predefined human pose and shape, such that the canonical representation can be explicitly driven to different poses and shapes with the guidance of a 3D parametric human model SMPL.
Expanding Small-Scale Datasets with Guided Imagination
Specifically, GIF conducts data imagination by optimizing the latent features of the seed data in the semantically meaningful space of the prior model, resulting in the creation of photo-realistic images with new content.
Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition
This paper does not attempt to design a state-of-the-art method for visual recognition but investigates a more efficient way to make use of convolutions to encode spatial features.
MagicVideo: Efficient Video Generation With Latent Diffusion Models
In specific, unlike existing works that directly train video models in the RGB space, we use a pre-trained VAE to map video clips into a low-dimensional latent space and learn the distribution of videos' latent codes via a diffusion model.
Ranked #10 on
Text-to-Video Generation
on UCF-101
MagicMix: Semantic Mixing with Diffusion Models
Unlike style transfer, where an image is stylized according to the reference style without changing the image content, semantic blending mixes two different concepts in a semantic manner to synthesize a novel concept while preserving the spatial layout and geometry.
Reachability-Aware Laplacian Representation in Reinforcement Learning
In Reinforcement Learning (RL), Laplacian Representation (LapRep) is a task-agnostic state representation that encodes the geometry of the environment.
MetaFormer Baselines for Vision
By simply applying depthwise separable convolutions as token mixer in the bottom stages and vanilla self-attention in the top stages, the resulting model CAFormer sets a new record on ImageNet-1K: it achieves an accuracy of 85. 5% at 224x224 resolution, under normal supervised training without external data or distillation.
Ranked #2 on
Domain Generalization
on ImageNet-C
(using extra training data)
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning
With the proposed SSF, our model obtains 2. 46% (90. 72% vs. 88. 54%) and 11. 48% (73. 10% vs. 65. 57%) performance improvement on FGVC and VTAB-1k in terms of Top-1 accuracy compared to the full fine-tuning but only fine-tuning about 0. 3M parameters.
ManiCLIP: Multi-Attribute Face Manipulation from Text
In this paper we present a novel multi-attribute face manipulation method based on textual descriptions.
AvatarGen: a 3D Generative Model for Animatable Human Avatars
Unsupervised generation of clothed virtual humans with various appearance and animatable poses is important for creating 3D human avatars and other AR/VR applications.
Contrastive Masked Autoencoders are Stronger Vision Learners
The momentum encoder, fed with the full images, enhances the feature discriminability via contrastive learning with its online counterpart.
Clover: Towards A Unified Video-Language Alignment and Fusion Model
We then introduce \textbf{Clover}\textemdash a Correlated Video-Language pre-training method\textemdash towards a universal Video-Language model for solving multiple video understanding tasks with neither performance nor efficiency compromise.
Ranked #1 on
Video Question Answering
on LSMDC-FiB
Divide to Adapt: Mitigating Confirmation Bias for Domain Adaptation of Black-Box Predictors
Domain Adaptation of Black-box Predictors (DABP) aims to learn a model on an unlabeled target domain supervised by a black-box predictor trained on a source domain.
Sharpness-Aware Training for Free
Intuitively, SAF achieves this by avoiding sudden drops in the loss in the sharp local minima throughout the trajectory of the updates of the weights.
Tyger: Task-Type-Generic Active Learning for Molecular Property Prediction
However, existing best deep AL methods are mostly developed for a single type of learning task (e. g., single-label classification), and hence may not perform well in molecular property prediction that involves various task types.
Understanding The Robustness in Vision Transformers
Our study is motivated by the intriguing properties of the emerging visual grouping in Vision Transformers, which indicates that self-attention may promote robustness through improved mid-level representations.
Ranked #4 on
Domain Generalization
on ImageNet-R
(using extra training data)
PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision
Existing self-supervised 3D human pose estimation schemes have largely relied on weak supervisions like consistency loss to guide the learning, which, inevitably, leads to inferior results in real-world scenarios with unseen poses.
Ranked #42 on
3D Human Pose Estimation
on MPI-INF-3DHP
Generalizing Few-Shot NAS with Gradient Matching
Efficient performance estimation of architectures drawn from large search spaces is essential to Neural Architecture Search.
SODAR: Segmenting Objects by DynamicallyAggregating Neighboring Mask Representations
Unlike the original per grid cell object masks, SODAR is implicitly supervised to learn mask representations that encode geometric structure of nearby objects and complement adjacent representations with context.
The Geometry of Robust Value Functions
The key of this perspective is to decompose the value space, in a state-wise manner, into unions of hypersurfaces.
Towards Adversarially Robust Deep Image Denoising
Secondly, to robustify DIDs, we propose an adversarial training strategy, hybrid adversarial training ({\sc HAT}), that jointly trains DIDs with adversarial and non-adversarial noisy data to ensure that the reconstruction quality is high and the denoisers around non-adversarial data are locally smooth.
UMAD: Universal Model Adaptation under Domain and Category Shift
To achieve bilateral adaptation in the target domain, we further maximize localized mutual information to align known samples with the source classifier and employ an entropic loss to push unknown samples far away from the source classification boundary, respectively.
Ranked #7 on
Universal Domain Adaptation
on VisDA2017
Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning
Specifically, we experimentally show that directly encouraging CIL Learner at the initial phase to output similar representations as the model jointly trained on all classes can greatly boost the CIL performance.
Geometry-Guided Progressive NeRF for Generalizable and Efficient Neural Human Rendering
Meanwhile, for achieving higher rendering efficiency, we introduce a progressive rendering pipeline through geometry guidance, which leverages the geometric feature volume and the predicted density values to progressively reduce the number of sampling points and speed up the rendering process.
Towards Understanding Why Lookahead Generalizes Better Than SGD and Beyond
Specifically, we prove that lookahead using SGD as its inner-loop optimizer can better balance the optimization error and generalization error to achieve smaller excess risk error than vanilla SGD on (strongly) convex problems and nonconvex problems with Polyak-{\L}ojasiewicz condition which has been observed/proved in neural networks.
Shunted Self-Attention via Multi-Scale Token Aggregation
This novel merging scheme enables the self-attention to learn relationships between objects with different sizes and simultaneously reduces the token numbers and the computational cost.
MetaFormer Is Actually What You Need for Vision
Based on this observation, we hypothesize that the general architecture of the Transformers, instead of the specific token mixer module, is more essential to the model's performance.
Ranked #9 on
Semantic Segmentation
on DensePASS
Direct Multi-view Multi-person 3D Pose Estimation
Instead of estimating 3D joint locations from costly volumetric representation or reconstructing the per-person 3D pose from multiple detected 2D poses as in previous methods, MvP directly regresses the multi-person 3D poses in a clean and efficient way, without relying on intermediate tasks.
Ranked #3 on
3D Multi-Person Pose Estimation
on Panoptic
(using extra training data)
Deep Long-Tailed Learning: A Survey
Deep long-tailed learning, one of the most challenging problems in visual recognition, aims to train well-performing deep models from a large number of images that follow a long-tailed class distribution.
Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
Recently, the relation between the sharpness of the loss landscape and the generalization error has been established by Foret et al. (2020), in which the Sharpness Aware Minimizer (SAM) was proposed to mitigate the degradation of the generalization.
How Well Does Self-Supervised Pre-Training Perform with Streaming ImageNet?
Prior works on self-supervised pre-training focus on the joint training scenario, where massive unlabeled data are assumed to be given as input all at once, and only then is a learner trained.
PnP-DETR: Towards Efficient Visual Analysis with Transformers
Recently, DETR pioneered the solution of vision tasks with transformers, it directly translates the image feature map into the object detection result.
DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
In view of these, we propose DHA, which achieves joint optimization of Data augmentation policy, Hyper-parameter and Architecture.
Voxel Transformer for 3D Object Detection
We present Voxel Transformer (VoTr), a novel and effective voxel-based Transformer backbone for 3D object detection from point clouds.
Ranked #3 on
3D Object Detection
on waymo vehicle
(L1 mAP metric)
Triplet Contrastive Learning for Brain Tumor Classification
Brain tumor is a common and fatal form of cancer which affects both adults and children.
Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition
Existing long-tailed recognition methods, aiming to train class-balanced models from long-tailed data, generally assume the models would be evaluated on the uniform test class distribution.
Ranked #9 on
Long-tail Learning
on iNaturalist 2018
Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing
It enables learning high-quality Laplacian representations that faithfully approximate the ground truth.
Recovering the Unbiased Scene Graphs from the Biased Ones
Given input images, scene graph generation (SGG) aims to produce comprehensive, graphical representations describing visual relationships among salient objects.
Ranked #3 on
Unbiased Scene Graph Generation
on Visual Genome
VOLO: Vision Outlooker for Visual Recognition
Though recently the prevailing vision transformers (ViTs) have shown great potential of self-attention based models in ImageNet classification, their performance is still inferior to that of the latest SOTA CNNs if no extra data are provided.
Ranked #1 on
Image Classification
on VizWiz-Classification
Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
By realizing the importance of the positional information carried by 2D feature representations, unlike recent MLP-like models that encode the spatial information along the flattened spatial dimensions, Vision Permutator separately encodes the feature representations along the height and width dimensions with linear projections.
LV-BERT: Exploiting Layer Variety for BERT
In this paper, beyond this stereotyped layer pattern, we aim to improve pre-trained models by exploiting layer variety from two aspects: the layer type set and the layer order.
No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
Motivated by the above findings, we propose a novel and simple algorithm called Classifier Calibration with Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated gaussian mixture model.
Refiner: Refining Self-attention for Vision Transformers
Vision Transformers (ViTs) have shown competitive accuracy in image classification tasks compared with CNNs.
Ranked #176 on
Image Classification
on ImageNet
Image-to-Video Generation via 3D Facial Dynamics
Video generation from a single face image is an interesting problem and usually tackled by utilizing Generative Adversarial Networks (GANs) to integrate information from the input face image and a sequence of sparse facial landmarks.
PSGAN++: Robust Detail-Preserving Makeup Transfer and Removal
In this paper, we address the makeup transfer and removal tasks simultaneously, which aim to transfer the makeup from a reference image to a source image and remove the makeup from the with-makeup image respectively.
ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videos
Detecting human-object interactions (HOI) is an important step toward a comprehensive visual understanding of machines.
Ranked #3 on
Human-Object Interaction Anticipation
on VidHOI
Joint Face Image Restoration and Frontalization for Recognition
First, MDFR is a well-designed encoder-decoder architecture which extracts feature representation from an input face image with arbitrary low-quality factors and restores it to a high-quality counterpart.
Continual Learning via Bit-Level Information Preserving
Continual learning tackles the setting of learning different tasks sequentially.
PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation
To address this problem, we present PoseAug, a new auto-augmentation framework that learns to augment the available training poses towards a greater diversity and thus improve generalization of the trained 2D-to-3D pose estimator.
Ranked #1 on
Monocular 3D Human Pose Estimation
on Human3.6M
(Use Video Sequence metric)
Body Meshes as Points
In this work, we present a single-stage model, Body Meshes as Points (BMP), to simplify the pipeline and lift both efficiency and performance.
Ranked #9 on
3D Multi-Person Pose Estimation
on MuPoTS-3D
3D Human Shape Estimation
3D Multi-Person Pose Estimation
+1
How Well Does Self-Supervised Pre-Training Perform with Streaming Data?
Prior works on self-supervised pre-training focus on the joint training scenario, where massive unlabeled data are assumed to be given as input all at once, and only then is a learner trained.
All Tokens Matter: Token Labeling for Training Better Vision Transformers
In this paper, we present token labeling -- a new training objective for training high-performance vision transformers (ViTs).
Ranked #3 on
Efficient ViTs
on ImageNet-1K (With LV-ViT-S)
DINE: Domain Adaptation from Single and Multiple Black-box Predictors
To ease the burden of labeling, unsupervised domain adaptation (UDA) aims to transfer knowledge in previous and related labeled datasets (sources) to a new unlabeled dataset (target).
Augmented Transformer with Adaptive Graph for Temporal Action Proposal Generation
Temporal action proposal generation (TAPG) is a fundamental and challenging task in video understanding, especially in temporal action detection.
AutoSpace: Neural Architecture Search with Less Human Interference
Current neural architecture search (NAS) algorithms still require expert knowledge and effort to design a search space for network construction.
DeepViT: Towards Deeper Vision Transformer
In this paper, we show that, unlike convolution neural networks (CNNs)that can be improved by stacking more convolutional layers, the performance of ViTs saturate fast when scaled to be deeper.
Ranked #464 on
Image Classification
on ImageNet
Coordinate Attention for Efficient Mobile Network Design
Recent studies on mobile network design have demonstrated the remarkable effectiveness of channel attention (e. g., the Squeeze-and-Excitation attention) for lifting model performance, but they generally neglect the positional information, which is important for generating spatially selective attention maps.
Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning
In this paper, we investigate whether applying contrastive learning to fine-tuning would bring further benefits, and analytically find that optimizing the contrastive loss benefits both discriminative representation learning and model optimization during fine-tuning.
CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection
By comparing \textit{non-robust} (normally trained) and \textit{robustified} (adversarially trained) models, we observe that adversarial training (AT) robustifies CNNs by aligning the channel-wise activations of adversarial data with those of their natural counterparts.
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
To overcome such limitations, we propose a new Tokens-To-Token Vision Transformer (T2T-ViT), which incorporates 1) a layer-wise Tokens-to-Token (T2T) transformation to progressively structurize the image to tokens by recursively aggregating neighboring Tokens into one Token (Tokens-to-Token), such that local structure represented by surrounding tokens can be modeled and tokens length can be reduced; 2) an efficient backbone with a deep-narrow structure for vision transformer motivated by CNN architecture design after empirical study.
Ranked #437 on
Image Classification
on ImageNet
ORDNet: Capturing Omni-Range Dependencies for Scene Parsing
Our ORDNet is able to extract more comprehensive context information and well adapt to complex spatial variance in scene images.
Learning Safe Policies with Cost-sensitive Advantage Estimation
Reinforcement Learning (RL) with safety guarantee is critical for agents performing tasks in risky environments.
DiffAutoML: Differentiable Joint Optimization for Efficient End-to-End Automated Machine Learning
Our method performs co-optimization of the neural architectures, training hyper-parameters and data augmentation policies in an end-to-end fashion without the need of model retraining.
AggMask: Exploring locally aggregated learning of mask representations for instance segmentation
Recently proposed one-stage instance segmentation models (\emph{e. g.}, SOLO) learn to directly predict location-specific object mask with fully-convolutional networks.
Exploring Balanced Feature Spaces for Representation Learning
Motivated by this question, we conduct a series of studies on the performance of self-supervised contrastive learning and supervised learning methods over multiple datasets where training instance distributions vary from a balanced one to a long-tailed one.
Ranked #40 on
Long-tail Learning
on CIFAR-10-LT (ρ=10)
Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer
Furthermore, we propose a new labeling transfer strategy, which separates the target data into two splits based on the confidence of predictions (labeling information), and then employ semi-supervised learning to improve the accuracy of less-confident predictions in the target domain.
Ranked #5 on
Source-Free Domain Adaptation
on VisDA-2017
Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts
In particular, we propose a novel joint-training framework to train plain CNN by leveraging the gradients of the ResNet counterpart.
Fooling the primate brain with minimal, targeted image manipulation
Here we propose an array of methods for creating minimal, targeted image perturbations that lead to changes in both neuronal activity and perception as reflected in behavior.
Improving Generalization in Reinforcement Learning with Mixture Regularization
Deep reinforcement learning (RL) agents trained in a limited set of environments tend to suffer overfitting and fail to generalize to unseen testing environments.
Toward Accurate Person-level Action Recognition in Videos of Crowded Scenes
Prior works always fail to deal with this problem in two aspects: (1) lacking utilizing information of the scenes; (2) lacking training data in the crowd and complex scenes.
Towards Accurate Human Pose Estimation in Videos of Crowded Scenes
In this paper, we focus on improving human pose estimation in videos of crowded scenes from the perspectives of exploiting temporal context and collecting new data.
A Simple Baseline for Pose Tracking in Videos of Crowded Scenes
This paper presents our solution to ACM MM challenge: Large-scale Human-centric Video Analysis in Complex Events\cite{lin2020human}; specifically, here we focus on Track3: Crowd Pose Tracking in Complex Events.
Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
The result shows that (1) the escaping time of both SGD and ADAM~depends on the Radon measure of the basin positively and the heaviness of gradient noise negatively; (2) for the same basin, SGD enjoys smaller escaping time than ADAM, mainly because (a) the geometry adaptation in ADAM~via adaptively scaling each gradient coordinate well diminishes the anisotropic structure in gradient noise and results in larger Radon measure of a basin; (b) the exponential gradient average in ADAM~smooths its gradient and leads to lighter gradient noise tails than SGD.
Visual Relationship Detection with Visual-Linguistic Knowledge from Multimodal Representations
Visual relationship detection aims to reason over relationships among salient objects in images, which has drawn increasing attention over the past few years.
Dual Adversarial Auto-Encoders for Clustering
By performing variational inference on the objective function of Dual-AAE, we derive a new reconstruction loss which can be optimized by training a pair of Auto-encoders.
ConvBERT: Improving BERT with Span-based Dynamic Convolution
The novel convolution heads, together with the rest self-attention heads, form a new mixed attention block that is more efficient at both global and local context learning.
Few-shot Classification via Adaptive Attention
To be specific, we devise a simple and efficient meta-reweighting strategy to adapt the sample representations and generate soft attention to refine the representation such that the relevant features from the query and support samples can be extracted for a better few-shot classification.
The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentation
Specifically, we systematically investigate performance drop of the state-of-the-art two-stage instance segmentation model Mask R-CNN on the recent long-tail LVIS dataset, and unveil that a major cause is the inaccurate classification of object proposals.
Adversarial Self-Supervised Learning for Semi-Supervised 3D Action Recognition
Self-supervised learning (SSL) has been proved very effective at learning representations from unlabeled data in the image domain.
Domain Adaptation with Auxiliary Target Domain-Oriented Classifier
ATDOC alleviates the classifier bias by introducing an auxiliary classifier for target data only, to improve the quality of pseudo labels.
Rethinking Bottleneck Structure for Efficient Mobile Network Design
In this paper, we rethink the necessity of such design changes and find it may bring risks of information loss and gradient confusion.
Local Grid Rendering Networks for 3D Object Detection in Point Clouds
The performance of 3D object detection models over point clouds highly depends on their capability of modeling local geometric patterns.
Inference Stage Optimization for Cross-scenario 3D Human Pose Estimation
In this work, we propose a novel framework, Inference Stage Optimization (ISO), for improving the generalizability of 3D pose models when source and target data come from different pose distributions.
Ranked #131 on
3D Human Pose Estimation
on 3DPW
(PA-MPJPE metric)
Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmax
Solving long-tail large vocabulary object detection with deep learning based models is a challenging and demanding task, which is however under-explored. In this work, we provide the first systematic analysis on the underperformance of state-of-the-art models in front of long-tail distribution.
Multi-Miner: Object-Adaptive Region Mining for Weakly-Supervised Semantic Segmentation
In this paper, we propose a novel multi-miner framework to perform a region mining process that adapts to diverse object sizes and is thus able to mine more integral and finer object regions.
Understanding and Resolving Performance Degradation in Graph Convolutional Networks
In this work, we study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works.
Boosting Few-Shot Learning With Adaptive Margin Loss
Few-shot learning (FSL) has attracted increasing attention in recent years but remains challenging, due to the intrinsic difficulty in learning to generalize from a few examples.
Ranked #1 on
Few-Shot Image Classification
on ImageNet (1-shot)
RAIN: A Simple Approach for Robust and Accurate Image Classification Networks
However, similar to existing preprocessing-based methods, the randomized process will degrade the prediction accuracy.
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing
Spatial pooling has been proven highly effective in capturing long-range contextual information for pixel-wise prediction tasks, such as scene parsing.
Ranked #32 on
Semantic Segmentation
on Cityscapes test
Semantic Domain Adversarial Networks for Unsupervised Domain Adaptation
To successfully align the multi-modal data structures across domains, the following works exploit discriminative information in the adversarial training process, e. g., using multiple class-wise discriminators and introducing conditional information in input or output of the domain discriminator.
A Balanced and Uncertainty-aware Approach for Partial Domain Adaptation
On one hand, negative transfer results in misclassification of target samples to the classes only present in the source domain.
Ranked #2 on
Partial Domain Adaptation
on ImageNet-Caltech
Cross-layer Feature Pyramid Network for Salient Object Detection
Feature pyramid network (FPN) based models, which fuse the semantics and salient details in a progressive manner, have been proven highly effective in salient object detection.
Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation
Unsupervised domain adaptation (UDA) aims to leverage the knowledge learned from a labeled source dataset to solve similar tasks in a new unlabeled domain.
Ranked #2 on
Domain Adaptation
on SVNH-to-MNIST
ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set.
Ranked #1 on
Logical Reasoning Question Answering
on ReClor
The Alzheimer's Disease Prediction Of Longitudinal Evolution (TADPOLE) Challenge: Results after 1 Year Follow-up
TADPOLE's unique results suggest that current prediction algorithms provide sufficient accuracy to exploit biomarkers related to clinical diagnosis and ventricle volume, for cohort refinement in clinical trials for Alzheimer's disease.
MetaSelector: Meta-Learning for Recommendation with User-Level Adaptive Model Selection
Recommender systems often face heterogeneous datasets containing highly personalized historical data of users, where no single model could give the best recommendation for every user.
PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection
Human and object points are the center of the detection boxes, and the interaction point is the midpoint of the human and object points.
Ranked #25 on
Human-Object Interaction Detection
on V-COCO
RC-DARTS: Resource Constrained Differentiable Architecture Search
In this paper, we propose the resource constrained differentiable architecture search (RC-DARTS) method to learn architectures that are significantly smaller and faster while achieving comparable accuracy.
Zoom in to where it matters: a hierarchical graph based model for mammogram analysis
In clinical practice, human radiologists actually review medical images with high resolution monitors and zoom into region of interests (ROIs) for a close-up examination.
Efficient Differentiable Neural Architecture Search with Meta Kernels
We demonstrate the effectiveness of our search strategy by conducting extensive experiments.
Efficient Meta Learning via Minibatch Proximal Update
We address the problem of meta-learning which learns a prior over hypothesis from a sample of meta-training tasks for fast adaptation on meta-testing tasks.
Classification Calibration for Long-tail Instance Segmentation
In this report, we investigate the performance drop phenomenon of state-of-the-art two-stage instance segmentation models when processing extreme long-tail training data based on the LVIS [5] dataset, and find a major cause is the inaccurate classification of object proposals.
Decoupling Representation and Classifier for Long-Tailed Recognition
The long-tail distribution of the visual world poses great challenges for deep learning based classification models on how to handle the class imbalance problem.
Ranked #3 on
Long-tail learning with class descriptors
on CUB-LT
On Robustness of Neural Ordinary Differential Equations
We then provide an insightful understanding of this phenomenon by exploiting a certain desirable property of the flow of a continuous-time ODE, namely that integral curves are non-intersecting.
Compressed Video Action Recognition with Refined Motion Vector
Although CNN has reached satisfactory performance in image-related tasks, using CNN to process videos is much more challenging due to the enormous size of raw video streams.
Adaptive ROI Generation for Video Object Segmentation Using Reinforcement Learning
In this paper, we aim to tackle the task of semi-supervised video object segmentation across a sequence of frames where only the ground-truth segmentation of the first frame is provided.
Hierarchical Neural Architecture Search via Operator Clustering
Recently, the efficiency of automatic neural architecture design has been significantly improved by gradient-based search methods such as DARTS.
Revisiting Knowledge Distillation via Label Smoothing Regularization
Without any extra computation cost, Tf-KD achieves up to 0. 65\% improvement on ImageNet over well-established baseline models, which is superior to label smoothing regularization.
Towards Disentangling Non-Robust and Robust Components in Performance Metric
Then, we show by experiments that DNNs under standard training rely heavily on optimizing the non-robust component in achieving decent performance.
Prototype Recalls for Continual Learning
Without tampering with the performance on initial tasks, our method learns novel concepts given a few training examples of each class in new tasks.
PROTOTYPE-ASSISTED ADVERSARIAL LEARNING FOR UNSUPERVISED DOMAIN ADAPTATION
Previous adversarial learning methods condition domain alignment only on pseudo labels, but noisy and inaccurate pseudo labels may perturb the multi-class distribution embedded in probabilistic predictions, hence bringing insufficient alleviation to the latent mismatch problem.
Hierarchic Neighbors Embedding
Manifold learning now plays a very important role in machine learning and many relevant applications.
PSGAN: Pose and Expression Robust Spatial-Aware GAN for Customizable Makeup Transfer
In this paper, we address the makeup transfer task, which aims to transfer the makeup from a reference image to a source image.
Single-Stage Multi-Person Pose Machines
Based on SPR, we develop the SPM model that can directly predict structured poses for multiple persons in a single stage, and thus offer a more compact pipeline and attractive efficiency advantage over two-stage methods.
Ranked #3 on
Keypoint Detection
on MPII Multi-Person
Dynamic Kernel Distillation for Efficient Pose Estimation in Videos
Existing video-based human pose estimation methods extensively apply large networks onto every frame in the video to localize body joints, which suffer high computational cost and hardly meet the low-latency requirement in realistic applications.
Ranked #4 on
2D Human Pose Estimation
on JHMDB (2D poses only)
PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment
In this paper, we tackle the challenging few-shot segmentation problem from a metric learning perspective and present PANet, a novel prototype alignment network to better utilize the information of the support set.
Central Similarity Quantization for Efficient Image and Video Retrieval
In this work, we propose a new \emph{global} similarity metric, termed as \emph{central similarity}, with which the hash codes of similar data pairs are encouraged to approach a common center and those for dissimilar pairs to converge to different centers, to improve hash learning efficiency and retrieval accuracy.
Neural Epitome Search for Architecture-Agnostic Network Compression
The recent WSNet [1] is a new model compression method through sampling filterweights from a compact set and has demonstrated to be effective for 1D convolutionneural networks (CNNs).
Delving into 3D Action Anticipation from Streaming Videos
Action anticipation, which aims to recognize the action with a partial observation, becomes increasingly popular due to a wide range of applications.
PVRED: A Position-Velocity Recurrent Encoder-Decoder for Human Motion Prediction
We therefore propose a novel Position-Velocity Recurrent Encoder-Decoder (PVRED) for human motion prediction, which makes full use of pose velocities and temporal positional information.
Unsupervised Image Noise Modeling with Self-Consistent GAN
They jointly facilitate unsupervised learning of a noise model for various noise types.
Distilling Object Detectors with Fine-grained Feature Imitation
To address the challenge of distilling knowledge in detection model, we propose a fine-grained feature imitation method exploiting the cross-location discrepancy of feature response.
Query-efficient Meta Attack to Deep Neural Networks
Black-box attack methods aim to infer suitable attack patterns to targeted DNN models by only using output feedback of the models and the corresponding input queries.
Understanding Adversarial Behavior of DNNs by Disentangling Non-Robust and Robust Components in Performance Metric
Despite many previous works studying the reason behind such adversarial behavior, the relationship between the generalization performance and adversarial behavior of DNNs is still little understood.
Panoptic Edge Detection
In this work, we address a new finer-grained task, termed panoptic edge detection (PED), which aims at predicting semantic-level boundaries for stuff categories and instance-level boundaries for instance categories, in order to provide more comprehensive and unified scene understanding from the perspective of edges. We then propose a versatile framework, Panoptic Edge Network (PEN), which aggregates different tasks of object detection, semantic and instance edge detection into a single holistic network with multiple branches.
Deep Face Recognition Model Compression via Knowledge Transfer and Distillation
However, deploying such high performing models to resource constraint devices or real-time applications is challenging.
Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation
Recent deep learning based face recognition methods have achieved great performance, but it still remains challenging to recognize very low-resolution query face like 28x28 pixels when CCTV camera is far from the captured subject.
Variational Prototype Replays for Continual Learning
In each classification task, our method learns a set of variational prototypes with their means and variances, where embedding of the samples from the same class can be represented in a prototypical distribution and class-representative prototypes are separated apart.
A Simple Pooling-Based Design for Real-Time Salient Object Detection
We further design a feature aggregation module (FAM) to make the coarse-level semantic information well fused with the fine-level features from the top-down pathway.
Ranked #1 on
RGB Salient Object Detection
on SOD
Hierarchical Meta Learning
Extensive experiments on few-shot classification and regression problems clearly demonstrate the superiority of HML over fine-tuning and state-of-the-art meta learning approaches in terms of generalization across heterogeneous tasks.
Cycle-SUM: Cycle-consistent Adversarial LSTM Networks for Unsupervised Video Summarization
The evaluator defines a learnable information preserving metric between original video and summary video and "supervises" the selector to identify the most informative frames to form the summary video.
Ranked #7 on
Unsupervised Video Summarization
on TvSum
