Search Results for author:
Found 270 papers, 155 papers with code
Adapting LLaMA Decoder to Vision Transformer
We first "LLaMAfy" a standard ViT step-by-step to align with LLaMA's architecture, and find that directly applying a casual mask to the self-attention brings an attention collapse issue, resulting in the failure to the network training.
End-to-End Autonomous Driving through V2X Cooperation
Cooperatively utilizing both ego-vehicle and infrastructure sensor data via V2X communication has emerged as a promising approach for advanced autonomous driving.
DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model
Text-to-image (T2I) generative models have attracted significant attention and found extensive applications within and beyond academic research.
Accelerating Federated Learning by Selecting Beneficial Herd of Local Gradients
Federated Learning (FL) is a distributed machine learning framework in communication network systems.
DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving
We utilize the challenging driving scenarios from the CARLA leaderboard 2. 0, which involve high-speed driving and lane-changing, and propose a rule-based expert policy to control the vehicle and generate ground truth labels for its reasoning process across different driving aspects and the final decisions.
FlashFace: Human Image Personalization with High-fidelity Identity Preservation
This work presents FlashFace, a practical tool with which users can easily personalize their own photos on the fly by providing one or a few reference face images and a text prompt.
Zero-shot Generative Linguistic Steganography
Generative linguistic steganography attempts to hide secret messages into covertext.
Generalized Predictive Model for Autonomous Driving
In this paper, we introduce the first large-scale video prediction model in the autonomous driving discipline.
AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Adversarial Visual-Instructions
To bridge this gap, we introduce AVIBench, a framework designed to analyze the robustness of LVLMs when facing various adversarial visual-instructions (AVIs), including four types of image-based AVIs, ten types of text-based AVIs, and nine types of content bias AVIs (such as gender, violence, cultural, and racial biases, among others).
ACT-MNMT Auto-Constriction Turning for Multilingual Neural Machine Translation
Large language model (LLM) has achieved promising performance in multilingual machine translation tasks through zero/few-shot prompts or prompt-tuning.
PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation
In this paper, we introduce PixArt-\Sigma, a Diffusion Transformer model~(DiT) capable of directly generating images at 4K resolution.
RegionGPT: Towards Region Understanding Vision Language Model
Vision language models (VLMs) have experienced rapid advancements through the integration of large language models (LLMs) with image-text pairs, yet they struggle with detailed regional visual understanding due to limited spatial awareness of the vision encoder, and the use of coarse-grained training data that lacks detailed, region-specific captions.
Towards Implicit Prompt For Text-To-Image Models
We call for increased attention to the potential and risks of implicit prompts in the T2I community and further investigation into the capabilities and impacts of implicit prompts, advocating for a balanced approach that harnesses their benefits while mitigating their risks.
RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis
Robotic behavior synthesis, the problem of understanding multimodal inputs and generating precise physical control for robots, is an important part of Embodied AI.
Ranked #70 on
Visual Question Answering
on MM-Vet
AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks
Automated machine learning (AutoML) is a collection of techniques designed to automate the machine learning development process.
RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation
To bridge this ``ideal-to-real'' gap, this paper presents \textbf{RobotScript}, a platform for 1) a deployable robot manipulation pipeline powered by code generation; and 2) a code generation benchmark for robot manipulation tasks in free-form natural language.
BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
Large language models (LLMs) have demonstrated outstanding performance in various tasks, such as text summarization, text question-answering, and etc.
OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
A significant challenge arises from the scarcity of diverse medical images spanning various modalities and anatomical regions, which is essential in real-world medical applications.
PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models
As a state-of-the-art, open-source image generation model, PIXART-{\delta} offers a promising alternative to the Stable Diffusion family of models, contributing significantly to text-to-image synthesis.
ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning
Charts play a vital role in data visualization, understanding data patterns, and informed decision-making.
LLaMA Pro: Progressive LLaMA with Block Expansion
Humans generally acquire new skills without compromising the old; however, the opposite holds for Large Language Models (LLMs), e. g., from LLaMA to CodeLLaMA.
Video Understanding with Large Language Models: A Survey
With the burgeoning growth of online video platforms and the escalating volume of video content, the demand for proficient video understanding tools has intensified markedly.
UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces
We evaluate our unified models on various benchmarks.
DriveLM: Driving with Graph Visual Question Answering
The experiments demonstrate that Graph VQA provides a simple, principled framework for reasoning about a driving scene, and DriveLM-Data provides a challenging benchmark for this task.
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs.
Ranked #1 on
Zero-Shot Video Retrieval
on MSR-VTT-full
(using extra training data)
Cached Transformers: Improving Transformers with Differentiable Memory Cache
This work introduces a new Transformer model called Cached Transformer, which uses Gated Recurrent Cached (GRC) attention to extend the self-attention mechanism with a differentiable memory cache of tokens.
SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution
Experiments on multi-task robotic manipulation benchmarks like Meta-World and LOReL demonstrate state-of-the-art performance and human-interpretable skill representations from SkillDiffuser.
A Survey of Reasoning with Foundation Models
Reasoning, a crucial ability for complex problem-solving, plays a pivotal role in various real-world settings such as negotiation, medical diagnosis, and criminal investigation.
You Only Learn One Query: Learning Unified Human Query for Single-Stage Multi-Person Multi-Task Human-Centric Perception
Human-centric perception (e. g. pedetrian detection, segmentation, pose estimation, and attribute analysis) is a long-standing problem for computer vision.
GenTron: Delving Deep into Diffusion Transformers for Image and Video Generation
In this study, we explore Transformer-based diffusion models for image and video generation.
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation
Therefore, this paper presents MotionCtrl, a unified and flexible motion controller for video generation designed to effectively and independently control camera and object motion.
MLLMs-Augmented Visual-Language Representation Learning
Visual-language pre-training has achieved remarkable success in many multi-modal tasks, largely attributed to the availability of large-scale image-text datasets.
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
With the rapid development of Multi-modal Large Language Models (MLLMs), a number of diagnostic benchmarks have recently emerged to evaluate the comprehension capabilities of these models.
Ranked #1 on
Zero-Shot Video Question Answer
on STAR Benchmark
Advancing Vision Transformers with Group-Mix Attention
The attention map is computed based on the mixtures of tokens and group proxies and used to re-combine the tokens and groups in Value.
Large Language Models as Automated Aligners for benchmarking Vision-Language Models
In this work, we address the limitations via Auto-Bench, which delves into exploring LLMs as proficient aligners, measuring the alignment between VLMs and human intelligence and value through automatic data curation and assessment.
DiffusionMat: Alpha Matting as Sequential Refinement Learning
In this paper, we introduce DiffusionMat, a novel image matting framework that employs a diffusion model for the transition from coarse to refined alpha mattes.
Flow-Based Feature Fusion for Vehicle-Infrastructure Cooperative 3D Object Detection
To address these issues in vehicle-infrastructure cooperative 3D (VIC3D) object detection, we propose the Feature Flow Net (FFNet), a novel cooperative detection framework.
Harvest Video Foundation Models via Efficient Post-Pretraining
Building video-language foundation models is costly and difficult due to the redundant nature of video data and the lack of high-quality video-language datasets.
Tree-Planner: Efficient Close-loop Task Planning with Large Language Models
This paper studies close-loop task planning, which refers to the process of generating a sequence of skills (a plan) to accomplish a specific goal while adapting the plan based on real-time observations.
MeanAP-Guided Reinforced Active Learning for Object Detection
Active learning presents a promising avenue for training high-performance models with minimal labeled data, achieved by judiciously selecting the most informative instances to label and incorporating them into the task learner.
Guideline Learning for In-context Information Extraction
In this paper, we propose a Guideline Learning (GL) framework for In-context IE which reflectively learns and follows guidelines.
Open-Vocabulary Animal Keypoint Detection with Semantic-feature Matching
Current image-based keypoint detection methods for animal (including human) bodies and faces are generally divided into full-supervised and few-shot class-agnostic approaches.
LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving
Existing learning-based autonomous driving (AD) systems face challenges in comprehending high-level information, generalizing to rare events, and providing interpretability.
PixArt-$α$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
We hope PIXART-$\alpha$ will provide new insights to the AIGC community and startups to accelerate building their own high-quality yet low-cost generative models from scratch.
SPOT: Scalable 3D Pre-training via Occupancy Prediction for Autonomous Driving
Our contributions are threefold: (1) Occupancy prediction is shown to be promising for learning general representations, which is demonstrated by extensive experiments on plenty of datasets and tasks.
StyleAdapter: A Single-Pass LoRA-Free Model for Stylized Image Generation
StyleAdapter can generate high-quality images that match the content of the prompts and adopt the style of the references (even for unseen styles) in a single pass, which is more flexible and efficient than previous methods.
MedShapeNet -- A Large-Scale Dataset of 3D Medical Shapes for Computer Vision
For the medical domain, we present a large collection of anatomical shapes (e. g., bones, organs, vessels) and 3D models of surgical instrument, called MedShapeNet, created to facilitate the translation of data-driven vision algorithms to medical applications and to adapt SOTA vision algorithms to medical problems.
GKGNet: Group K-Nearest Neighbor based Graph Convolutional Network for Multi-Label Image Recognition
Multi-Label Image Recognition (MLIR) is a challenging task that aims to predict multiple object labels in a single image while modeling the complex relationships between labels and image regions.
Beyond One-to-One: Rethinking the Referring Image Segmentation
In this paper, we address this issue from two perspectives.
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
LWC modulates the extreme values of weights by optimizing the clipping threshold.
RestoreFormer++: Towards Real-World Blind Face Restoration from Undegraded Key-Value Pairs
In this work, we propose RestoreFormer++, which on the one hand introduces fully-spatial attention mechanisms to model the contextual information and the interplay with the priors, and on the other hand, explores an extending degrading model to help generate more realistic degraded face images to alleviate the synthetic-to-real-world gap.
Foundation Model is Efficient Multimodal Multitask Model Selector
This paper investigates an under-explored but important problem: given a collection of pre-trained neural networks, predicting their performance on each multi-modal task without fine-tuning them, such as image recognition, referring, captioning, visual question answering, and text question answering.
RIGID: Recurrent GAN Inversion and Editing of Real Face Videos
In this paper, we propose a unified recurrent framework, named \textbf{R}ecurrent v\textbf{I}deo \textbf{G}AN \textbf{I}nversion and e\textbf{D}iting (RIGID), to explicitly and simultaneously enforce temporally coherent GAN inversion and facial editing of real videos.
Exploring Transformers for Open-world Instance Segmentation
Open-world instance segmentation is a rising task, which aims to segment all objects in the image by learning from a limited number of base-category objects.
Tiny LVLM-eHub: Early Multimodal Experiments with Bard
Secondly, it conducts an in-depth analysis of LVLMs' predictions using the ChatGPT Ensemble Evaluation (CEE), which leads to a robust and accurate evaluation and exhibits improved alignment with human evaluation compared to the word matching approach.
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
Specifically, we utilize a multi-scale approach to generate video-related descriptions.
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
Before sending to LLM, the reference is replaced by RoI features and interleaved with language embeddings as a sequence.
Ranked #1 on
Visual Question Answering (VQA)
on VCR (Q-AR) test
ChiPFormer: Transferable Chip Placement via Offline Decision Transformer
To resolve these challenges, we cast the chip placement as an offline RL formulation and present ChiPFormer that enables learning a transferable placement policy from fixed offline data.
Align, Adapt and Inject: Sound-guided Unified Image Generation
Then, we propose the audio adapter to adapt audio representation into an audio token enriched with specific semantics, which can be injected into a frozen T2I model flexibly.
LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Large Vision-Language Models (LVLMs) have recently played a dominant role in multimodal vision-language learning.
RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths
Text-to-image generation has recently witnessed remarkable achievements.
Ranked #11 on
Text-to-Image Generation
on MS COCO
DiffRate : Differentiable Compression Rate for Efficient Vision Transformers
Token compression aims to speed up large-scale vision transformers (e. g. ViTs) by pruning (dropping) or merging tokens.
Ranked #4 on
Efficient ViTs
on ImageNet-1K (with DeiT-S)
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi-modal understanding and execution capabilities.
SyNDock: N Rigid Protein Docking via Learnable Group Synchronization
Firstly, SyNDock formulates multimeric protein docking as a problem of learning global transformations to holistically depict the placement of chain units of a complex, enabling a learning-centric solution.
VDT: General-purpose Video Diffusion Transformers via Mask Modeling
We also propose a unified spatial-temporal mask modeling mechanism, seamlessly integrated with the model, to cater to diverse video generation scenarios.
Going Denser with Open-Vocabulary Part Segmentation
In this paper, we propose a detector with the ability to predict both open-vocabulary objects and their part segmentation.
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
We hope this model can set a new baseline for generalist vision and language models.
VideoChat: Chat-Centric Video Understanding
In this paper, we initiate an attempt of developing an end-to-end chat-centric video understanding system, coined as VideoChat.
Ranked #6 on
Video Question Answering
on MVBench
Video-based Generative Performance Benchmarking (Consistency)
Video-based Generative Performance Benchmarking (Contextual Understanding)
+5
V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting
Utilizing infrastructure and vehicle-side information to track and forecast the behaviors of surrounding traffic participants can significantly improve decision-making and safety in autonomous driving.
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language
Different from existing interactive systems that rely on pure language, by incorporating pointing instructions, the proposed iGPT significantly improves the efficiency of communication between users and chatbots, as well as the accuracy of chatbots in vision-centric tasks, especially in complicated visual scenarios where the number of objects is greater than 2.
MultiModal-GPT: A Vision and Language Model for Dialogue with Humans
To further enhance the ability to chat with humans of the MultiModal-GPT, we utilize language-only instruction-following data to train the MultiModal-GPT jointly.
$π$-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation
Foundation models have achieved great advances in multi-task learning with a unified interface of unimodal and multimodal tasks.
OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mapping
Accurately depicting the complex traffic scene is a vital component for autonomous vehicles to execute correct judgments.
MetaBEV: Solving Sensor Failures for BEV Detection and Map Segmentation
These queries are then processed iteratively by a BEV-Evolving decoder, which selectively aggregates deep features from either LiDAR, cameras, or both modalities.
EC^2: Emergent Communication for Embodied Control
We learn embodied representations of video trajectories, emergent language, and natural language using a language model, which is then used to finetune a lightweight policy network for downstream control.
RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixer
Extensive experiments and ablative analysis also demonstrate that the inductive bias of network architecture, can be incorporated into simple network structure with appropriate optimization strategy.
Graph-based Topology Reasoning for Driving Scenes
Understanding the road genome is essential to realize autonomous driving.
Ranked #5 on
3D Lane Detection
on OpenLane-V2 val
Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following
ECL consists of: (i) an instruction parser that translates the natural languages into executable programs; (ii) an embodied concept learner that grounds visual concepts based on language descriptions; (iii) a map constructor that estimates depth and constructs semantic maps by leveraging the learned concepts; and (iv) a program executor with deterministic policies to execute each program.
Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention
When looking at an image, we can decompose the scene into entities and their parts as well as obtain the dependencies between them.
Multi-Level Contrastive Learning for Dense Prediction Task
We extend our pretext task to supervised pre-training, which achieves a similar performance to self-supervised learning.
EGC: Image Generation and Classification via a Diffusion Energy-Based Model
Learning image classification and image generation using the same set of network parameters is a challenging problem.
DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Driving
In addition, we propose a new task, end-to-end motion and accident prediction, which can be used to directly evaluate the accident prediction ability for different autonomous driving algorithms.
DDP: Diffusion Model for Dense Visual Prediction
We propose a simple, efficient, yet powerful framework for dense visual predictions based on the conditional diffusion pipeline.
Ranked #2 on
Monocular Depth Estimation
on SUN-RGBD
Soft Neighbors are Positive Supporters in Contrastive Visual Representation Learning
We evaluate our soft neighbor contrastive learning method (SNCLR) on standard visual recognition benchmarks, including image classification, object detection, and instance segmentation.
Real-time Controllable Denoising for Image and Video
Controllable image denoising aims to generate clean samples with human perceptual priors and balance sharpness and smoothness.
Accelerating Vision-Language Pretraining with Free Language Modeling
FLM successfully frees the prediction rate from the tie-up with the corruption rate while allowing the corruption spans to be customized for each token to be predicted.
Dense Distinct Query for End-to-End Object Detection
Concretely, we first lay dense queries like traditional detectors and then select distinct ones for one-to-one assignments.
Ranked #3 on
Object Detection
on CrowdHuman (full body)
Vehicle-Infrastructure Cooperative 3D Object Detection via Feature Flow Prediction
Cooperatively utilizing both ego-vehicle and infrastructure sensor data can significantly enhance autonomous driving perception abilities.
Universal Instance Perception as Object Discovery and Retrieval
All instance perception tasks aim at finding certain objects specified by some queries such as category names, language expressions, and target annotations, but this complete field has been split into multiple independent subtasks.
Ranked #1 on
Referring Expression Segmentation
on RefCoCo val
(using extra training data)
Described Object Detection
Generalized Referring Expression Comprehension
+15
Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos
Our framework is easily extensible to tasks covering visually-grounded language understanding and generation.
AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners
For example, AdaptDiffuser not only outperforms the previous art Diffuser by 20. 8% on Maze2D and 7. 5% on MuJoCo locomotion, but also adapts better to new tasks, e. g., KUKA pick-and-place, by 27. 9% without requiring additional expert data.
Understanding Self-Supervised Pretraining with Part-Aware Representation Learning
The study is mainly motivated by that random views, used in contrastive learning, and random masked (visible) patches, used in masked image modeling, are often about object parts.
Fast-BEV: Towards Real-time On-vehicle Bird's-Eye View Perception
Recently, the pure camera-based Bird's-Eye-View (BEV) perception removes expensive Lidar sensors, making it a feasible solution for economical autonomous driving.
Segment Every Reference Object in Spatial and Temporal Spaces
In this work, we end the current fragmented situation and propose UniRef to unify the three reference-based object segmentation tasks with a single architecture.
EC2: Emergent Communication for Embodied Control
We learn embodied representations of video trajectories, emergent language, and natural language using a language model, which is then used to finetune a lightweight policy network for downstream control.
MetaBEV: Solving Sensor Failures for 3D Detection and Map Segmentation
These queries are then processed iteratively by a BEV-Evolving decoder, which selectively aggregates deep features from either LiDAR, cameras, or both modalities.
RIFormer: Keep Your Vision Backbone Effective but Removing Token Mixer
Extensive experiments and ablative analysis also demonstrate that the inductive bias of network architecture, can be incorporated into simple network structure with appropriate optimization strategy.
Policy Adaptation from Foundation Model Feedback
When deploying the trained policy to a new task or a new environment, we first let the policy play with randomly generated instructions to record the demonstrations.
Learning Object-Language Alignments for Open-Vocabulary Object Detection
In this paper, we propose a novel open-vocabulary object detection framework directly learning from image-text pair data.
MaskPlace: Fast Chip Placement via Reinforced Visual Representation Learning
Firstly, MaskPlace recasts placement as a problem of learning pixel-level visual representation to comprehensively describe millions of modules on a chip, enabling placement in a high-resolution canvas and a large action space.
Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learning
The latent world model provides a promising way to learn policies in a compact latent space for tasks with high-dimensional observations, however, its generalization across diverse environments with unseen dynamics remains challenging.
Model-based Reinforcement Learning
reinforcement-learning
+1
Large-batch Optimization for Dense Visual Predictions
To address this challenge, we propose a simple yet effective algorithm, named Adaptive Gradient Variance Modulator (AGVM), which can train dense visual predictors with very large batch size, enabling several benefits more appealing than prior arts.
Decomposed Mutual Information Optimization for Generalized Context in Meta-Reinforcement Learning
This paper addresses such a challenge by Decomposed Mutual INformation Optimization (DOMINO) for context learning, which explicitly learns a disentangled context to maximize the mutual information between the context and historical trajectories, while minimizing the state transition prediction error.
Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model
End-to-end autonomous driving provides a feasible way to automatically maximize overall driving system performance by directly mapping the raw pixels from a front-facing camera to control signals.
Learning Transferable Spatiotemporal Representations from Natural Script Knowledge
Pre-training on large-scale video data has become a common recipe for learning transferable spatiotemporal representations in recent years.
FedVeca: Federated Vectorized Averaging on Non-IID Data with Adaptive Bi-directional Global Objective
However, the clients' Non-Independent and Identically Distributed (Non-IID) data negatively affect the trained model, and clients with different numbers of local updates may cause significant gaps to the local gradients in each communication round.
Rethinking Resolution in the Context of Efficient Video Recognition
Our key finding is that the major cause of degradation is not information loss in the down-sampling process, but rather the mismatch between network architecture and input scale.
ZoomNAS: Searching for Whole-body Human Pose Estimation in the Wild
We propose a single-network approach, termed ZoomNet, to take into account the hierarchical structure of the full human body and solve the scale variation of different body parts.
Ranked #2 on
2D Human Pose Estimation
on COCO-WholeBody
PoseTrans: A Simple Yet Effective Pose Transformation Augmentation for Human Pose Estimation
Human pose estimation aims to accurately estimate a wide variety of human poses.
3D Interacting Hand Pose Estimation by Hand De-occlusion and Removal
Unlike most previous works that directly predict the 3D poses of two interacting hands simultaneously, we propose to decompose the challenging interacting hand pose estimation task and estimate the pose of each hand separately.
Pose for Everything: Towards Category-Agnostic Pose Estimation
In this paper, we introduce the task of Category-Agnostic Pose Estimation (CAPE), which aims to create a pose estimation model capable of detecting the pose of any class of object given only a few samples with keypoint definition.
Ranked #4 on
2D Pose Estimation
on MP-100
Real-time End-to-End Video Text Spotter with Contrastive Representation Learning
Our contributions are three-fold: 1) CoText simultaneously address the three tasks (e. g., text detection, tracking, recognition) in a real-time end-to-end trainable framework.
Towards Grand Unification of Object Tracking
We present a unified method, termed Unicorn, that can simultaneously solve four tracking problems (SOT, MOT, VOS, MOTS) with a single network using the same model parameters.
Multi-Object Tracking
Multi-Object Tracking and Segmentation
+3
Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space
It is challenging because the ground-truth model ranking for each task can only be generated by fine-tuning the pre-trained models on the target dataset, which is brute-force and computationally expensive.
Ranked #2 on
Transferability
on classification benchmark
Exploiting Context Information for Generic Event Boundary Captioning
Previous methods only process the information of a single boundary at a time, which lacks utilization of video context information.
VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix
Existing vision-language pre-training (VLP) methods primarily rely on paired image-text datasets, which are either annotated by enormous human labors, or crawled from the internet followed by elaborate data cleaning techniques.
CtrlFormer: Learning Transferable State Representation for Visual Control via Transformer
In visual control, learning transferable state representation that can transfer between different control tasks is important to reduce the training sample size.
AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
Constraint by the high cost of collecting and labeling 3D medical data, most of the deep learning models to date are driven by datasets with a limited number of organs of interest or samples, which still limits the power of modern deep models and makes it difficult to provide a fully comprehensive and fair estimate of various methods.
CO^3: Cooperative Unsupervised 3D Representation Learning for Autonomous Driving
In this paper, we propose CO^3, namely Cooperative Contrastive Learning and Contextual Shape Prediction, to learn 3D representation for outdoor-scene point clouds in an unsupervised manner.
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
To address this challenge, we propose an effective adaptation approach for Transformer, namely AdaptFormer, which can adapt the pre-trained ViTs into many different image and video tasks efficiently.
Flow-based Recurrent Belief State Learning for POMDPs
Furthermore, we show that the learned belief states can be plugged into downstream RL algorithms to improve performance.
An Empirical Investigation of Representation Learning for Imitation
We propose a modular framework for constructing representation learning algorithms, then use our framework to evaluate the utility of representation learning for imitation across several environment suites.
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval
Dominant pre-training work for video-text retrieval mainly adopt the "dual-encoder" architectures to enable efficient retrieval, where two separate encoders are used to contrast global video and text representations, but ignore detailed local semantics.
Ranked #7 on
Zero-Shot Video Retrieval
on MSVD
Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer
Vision transformers have achieved great successes in many computer vision tasks.
Ranked #4 on
2D Human Pose Estimation
on COCO-WholeBody
Semantic-Aware Pretraining for Dense Video Captioning
This report describes the details of our approach for the event dense-captioning task in ActivityNet Challenge 2021.
M$^2$BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation
In this paper, we propose M$^2$BEV, a unified framework that jointly performs 3D object detection and map segmentation in the Birds Eye View~(BEV) space with multi-camera image inputs.
DaViT: Dual Attention Vision Transformers
We show that these two self-attentions complement each other: (i) since each channel token contains an abstract representation of the entire image, the channel attention naturally captures global interactions and representations by taking all spatial positions into account when computing attention scores between channels; (ii) the spatial attention refines the local representations by performing fine-grained interactions across spatial locations, which in turn helps the global information modeling in channel attention.
Scale-Equivalent Distillation for Semi-Supervised Object Detection
Further, we overcome these challenges by introducing a novel approach, Scale-Equivalent Distillation (SED), which is a simple yet effective end-to-end knowledge distillation framework robust to large object size variance and class imbalance.
Compression of Generative Pre-trained Language Models via Quantization
We find that previous quantization methods fail on generative tasks due to the \textit{homogeneous word embeddings} caused by reduced capacity, and \textit{varied distribution of weights}.
End-to-End Video Text Spotting with Transformer
Recent video text spotting methods usually require the three-staged pipeline, i. e., detecting text in individual images, recognizing localized text, tracking text streams with post-processing to generate final results.
WegFormer: Transformers for Weakly Supervised Semantic Segmentation
Although convolutional neural networks (CNNs) have achieved remarkable progress in weakly supervised semantic segmentation (WSSS), the effective receptive field of CNN is insufficient to capture global context information, leading to sub-optimal results.
Don't Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learning
One of the key challenges in visual Reinforcement Learning (RL) is to learn policies that can generalize to unseen environments.
Context Autoencoder for Self-Supervised Representation Learning
The pretraining tasks include two tasks: masked representation prediction - predict the representations for the masked patches, and masked patch reconstruction - reconstruct the masked patches.
DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations
AI-aided drug discovery (AIDD) is gaining increasing popularity due to its promise of making the search for new pharmaceuticals quicker, cheaper and more efficient.
Pseudo-Labeled Auto-Curriculum Learning for Semi-Supervised Keypoint Localization
PL approaches apply pseudo-labels to unlabeled data, and then train the model with a combination of the labeled and pseudo-labeled data iteratively.
RestoreFormer: High-Quality Blind Face Restoration from Undegraded Key-Value Pairs
Blind face restoration is to recover a high-quality face image from unknown degradations.
Bridging Video-text Retrieval with Multiple Choice Questions
As an additional benefit, our method achieves competitive results with much shorter pre-training videos on single-modality downstream tasks, e. g., action recognition with linear evaluation.
Ranked #8 on
Zero-Shot Video Retrieval
on MSVD
MetaDance: Few-shot Dancing Video Retargeting via Temporal-aware Meta-learning
Dancing video retargeting aims to synthesize a video that transfers the dance movements from a source video to a target person.
Language as Queries for Referring Video Object Segmentation
Referring video object segmentation (R-VOS) is an emerging cross-modal task that aims to segment the target object referred by a language expression in all video frames.
Ranked #3 on
Referring Expression Segmentation
on A2D Sentences
(using extra training data)
MetaCloth: Learning Unseen Tasks of Dense Fashion Landmark Detection from a Few Samples
This work proposes a novel framework named MetaCloth via meta-learning, which is able to learn unseen tasks of dense fashion landmark detection with only a few annotated samples.
Dynamic Token Normalization Improves Vision Transformers
It is difficult for Transformers to capture inductive bias such as the positional context in an image with LN.
Revitalizing CNN Attention via Transformers in Self-Supervised Visual Representation Learning
Motivated by the transformers that explore visual attention effectively in recognition scenarios, we propose a CNN Attention REvitalization (CARE) framework to train attentive CNN encoders guided by transformers in SSL.
Model-Based Reinforcement Learning via Imagination with Derived Memory
Model-based reinforcement learning aims to improve the sample efficiency of policy learning by modeling the dynamics of the environment.
Model-based Reinforcement Learning
reinforcement-learning
+1
Compressed Video Contrastive Learning
To enhance the representation ability of the motion vectors, hence the effectiveness of our method, we design a cross guidance contrastive learning algorithm based on multi-instance InfoNCE loss, where motion vectors can take supervision signals from RGB frames and vice versa.
DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion
A typical pipeline for multi-object tracking (MOT) is to use a detector for object localization, and following re-identification (re-ID) for object association.
FAST: Faster Arbitrarily-Shaped Text Detector with Minimalist Kernel Representation
We propose an accurate and efficient scene text detection framework, termed FAST (i. e., faster arbitrarily-shaped text detector).
Ranked #2 on
Scene Text Detection
on MSRA-TD500
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language
This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine.
ByteTrack: Multi-Object Tracking by Associating Every Detection Box
ByteTrack also achieves state-of-the-art performance on MOT20, HiEve and BDD100K tracking benchmarks.
Ranked #1 on
Multiple Object Tracking
on BDD100K val
Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
Motivated by the transformers that explore visual attention effectively in recognition scenarios, we propose a CNN Attention REvitalization (CARE) framework to train attentive CNN encoders guided by transformers in SSL.
Objects in Semantic Topology
A more realistic object detection paradigm, Open-World Object Detection, has arisen increasing research interests in the community recently.
Scale-Invariant Teaching for Semi-Supervised Object Detection
Furthermore, the limited annotations in semi-supervised learning scale up the challenges: large variance of object sizes and class imbalance (i. e., the extreme ratio between background and object), hindering the performance of prior arts.
Towards High-Quality Temporal Action Detection with Sparse Proposals
Towards high-quality temporal action detection, we introduce Sparse Proposals to interact with the hierarchical features.
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers
Specifically, we supervise the attention modules in the mask decoder in a layer-wise manner.
Ranked #4 on
Panoptic Segmentation
on COCO test-dev
Adversarial Robustness for Unsupervised Domain Adaptation
Extensive Unsupervised Domain Adaptation (UDA) studies have shown great success in practice by learning transferable representations across a labeled source domain and an unlabeled target domain with deep models.
End-to-End Dense Video Captioning with Parallel Decoding
Dense video captioning aims to generate multiple associated captions with their temporal locations from the video.
Ranked #5 on
Dense Video Captioning
on YouCook2
CycleMLP: A MLP-like Architecture for Dense Prediction
We build a family of models which surpass existing MLPs and even state-of-the-art Transformer-based models, e. g., Swin Transformer, while using fewer parameters and FLOPs.
Ranked #15 on
Semantic Segmentation
on DensePASS
Multi-frame Collaboration for Effective Endoscopic Video Polyp Detection via Spatial-Temporal Feature Transformation
For example, STFT improves the still image baseline FCOS by 10. 6% and 20. 6% on the comprehensive F1-score of the polyp localization task in CVC-Clinic and ASUMayo datasets, respectively, and outperforms the state-of-the-art video-based method by 3. 6% and 8. 0%, respectively.
Multi-Compound Transformer for Accurate Biomedical Image Segmentation
The recent vision transformer(i. e. for image classification) learns non-local attentive interaction of different patch tokens.
PVT v2: Improved Baselines with Pyramid Vision Transformer
We hope this work will facilitate state-of-the-art Transformer researches in computer vision.
Ranked #23 on
Object Detection
on COCO-O
HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
Last, we proposed an efficient fine-grained search strategy to train HR-NAS, which effectively explores the search space, and finds optimal architectures given various tasks and computation resources.
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders.
Ranked #1 on
Semantic Segmentation
on COCO-Stuff full
ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search
Human pose estimation has achieved significant progress in recent years.
Ranked #23 on
Pose Estimation
on COCO test-dev
(using extra training data)
Extracting Variable-Depth Logical Document Hierarchy from Long Documents: Method, Evaluation, and Application
In this paper, we study the problem of extracting variable-depth "logical document hierarchy" from long documents, namely organizing the recognized "physical document objects" into hierarchical structures.
BWCP: Probabilistic Learning-to-Prune Channels for ConvNets via Batch Whitening
To address this problem, we develop a probability-based pruning algorithm, called batch whitening channel pruning (BWCP), which can stochastically discard unimportant channels by modeling the probability of a channel being activated.
When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks
However, unlike human vision that is robust to various data corruptions such as blur and pixelation, current pose estimators are easily confused by these corruptions.
PolarMask++: Enhanced Polar Representation for Single-Shot Instance Segmentation and Beyond
Extensive experiments demonstrate the effectiveness of both PolarMask and PolarMask++, which achieve competitive results on instance segmentation in the challenging COCO dataset with single-model and single-scale training and testing, as well as new state-of-the-art results on rotate text detection and cell segmentation.
Ranked #81 on
Instance Segmentation
on COCO test-dev
(using extra training data)
Going Deeper Into Face Detection: A Survey
In this work, we provide a detailed overview of some of the most representative deep learning based face detection methods by grouping them into a few major categories, and present their core architectural designs and accuracies on popular benchmarks.
Learning Versatile Neural Architectures by Propagating Network Codes
(4) Thorough studies of NCP on inter-, cross-, and intra-tasks highlight the importance of cross-task neural architecture design, i. e., multitask neural architectures and architecture transferring between different tasks.
Towards Ultra-Resolution Neural Style Transfer via Thumbnail Instance Normalization
(1) We divide input image into small patches and adopt TIN, successfully transferring image style with arbitrary high-resolution.
Disentangled Cycle Consistency for Highly-realistic Virtual Try-On
To this end, DCTON can be naturally trained in a self-supervised manner following cycle consistency learning.
Parser-Free Virtual Try-on via Distilling Appearance Flows
A recent pioneering work employed knowledge distillation to reduce the dependency of human parsing, where the try-on images produced by a parser-based method are used as supervisions to train a "student" network without relying on segmentation, making the student mimic the try-on ability of the parser-based model.
Ranked #1 on
Virtual Try-on
on MPV
Deeply Unsupervised Patch Re-Identification for Pre-training Object Detectors
Unsupervised pre-training aims at learning transferable features that are beneficial for downstream tasks.
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Unlike the recently-proposed Transformer model (e. g., ViT) that is specially designed for image classification, we propose Pyramid Vision Transformer~(PVT), which overcomes the difficulties of porting Transformer to various dense prediction tasks.
Ranked #5 on
Semantic Segmentation
on SynPASS
FAT: Learning Low-Bitwidth Parametric Representation via Frequency-Aware Transformation
Prior arts often discretize the network weights by carefully tuning hyper-parameters of quantization (e. g. non-uniform stepsize and layer-wise bitwidths), which are complicated and sub-optimal because the full-precision and low-precision models have a large discrepancy.
DetCo: Unsupervised Contrastive Learning for Object Detection
Unlike most recent methods that focused on improving accuracy of image classification, we present a novel contrastive learning approach, named DetCo, which fully explores the contrasts between global image and local image patches to learn discriminative representations for object detection.
Segmenting Transparent Object in the Wild with Transformer
This work presents a new fine-grained transparent object segmentation dataset, termed Trans10K-v2, extending Trans10K-v1, the first large-scale transparent object segmentation dataset.
Ranked #3 on
Semantic Segmentation
on Trans10K
Watch Only Once: An End-to-End Video Action Detection Framework
WOO takes a unified video backbone to simultaneously extract features for actor location and action classification.
Rethinking the Pruning Criteria for Convolutional Neural Network
Channel pruning is a popular technique for compressing convolutional neural networks (CNNs), and various pruning criteria have been proposed to remove the redundant filters of CNNs.
Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw
With the constrained jigsaw puzzles, instead of solving them directly, which could still be extremely hard, we carefully design four surrogate tasks that are more solvable but meanwhile still ensure that the learned representation is sensitive to spatiotemporal continuity at both the local and global levels.
STAR: A Structure-Aware Lightweight Transformer for Real-Time Image Enhancement
STAR is a general architecture that can be easily adapted to different image enhancement tasks.
Bringing Events Into Video Deblurring With Non-Consecutively Blurry Frames
EFM can also be easily incorporated into existing deblurring networks, making event-driven deblurring task benefit from state-of-the-art deblurring methods.
TransTrack: Multiple Object Tracking with Transformer
In this work, we propose TransTrack, a simple but efficient scheme to solve the multiple object tracking problems.
Ranked #7 on
Multi-Object Tracking
on SportsMOT
(using extra training data)
Multi-Object Tracking
Multiple Object Tracking with Transformer
+3
What Makes for End-to-End Object Detection?
We identify that classification cost in matching cost is the main ingredient: (1) previous detectors only consider location cost, (2) by additionally introducing classification cost, previous detectors immediately produce one-to-one prediction during inference.
Polygon-free: Unconstrained Scene Text Detection with Box Annotations
For example, without using polygon annotations, PSENet achieves an 80. 5% F-score on TotalText [3] (vs. 80. 9% of fully supervised counterpart), 31. 1% better than training directly with upright bounding box annotations, and saves 80%+ labeling costs.
Sparse R-CNN: End-to-End Object Detection with Learnable Proposals
In our method, however, a fixed sparse set of learned object proposals, total length of $N$, are provided to object recognition head to perform classification and location.
Ranked #5 on
2D Object Detection
on CeyMo
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs
Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner.
UXNet: Searching Multi-level Feature Aggregation for 3D Medical Image Segmentation
Unlike the recent neural architecture search (NAS) methods that typically searched the optimal operators in each network layer, but missed a good strategy to search for feature aggregations, this paper proposes a novel NAS method for 3D medical image segmentation, named UXNet, which searches both the scale-wise feature aggregation strategies as well as the block-wise operators in the encoder-decoder network.
RelativeNAS: Relative Neural Architecture Search via Slow-Fast Learning
Despite the remarkable successes of Convolutional Neural Networks (CNNs) in computer vision, it is time-consuming and error-prone to manually design a CNN.
Compensation Tracker: Reprocessing Lost Object for Multi-Object Tracking
Based on simple and traditional methods, we propose a compensation tracker to further alleviate the lost tracking problem caused by missing detection.
Webly Supervised Image Classification with Self-Contained Confidence
Therefore, a simple yet effective WSL framework is proposed.
Ranked #7 on
Image Classification
on WebVision-1000
Dynamic and Static Context-aware LSTM for Multi-agent Motion Prediction
Existing work addressed this challenge by either learning social spatial interactions represented by the positions of a group of pedestrians, while ignoring their temporal coherence (\textit{i. e.} dependencies between different long trajectories), or by understanding the complicated scene layout (\textit{e. g.} scene segmentation) to ensure safe navigation.
AE TextSpotter: Learning Visual and Linguistic Representation for Ambiguous Text Spotting
Unlike previous works that merely employed visual features for text detection, this work proposes a novel text spotter, named Ambiguity Eliminating Text Spotter (AE TextSpotter), which learns both visual and linguistic features to significantly reduce ambiguity in text detection.
Differentiable Hierarchical Graph Grouping for Multi-Person Pose Estimation
The modules of HGG can be trained end-to-end with the keypoint detection network and is able to supervise the grouping process in a hierarchical manner.
Ranked #3 on
Keypoint Detection
on OCHuman
Whole-Body Human Pose Estimation in the Wild
This paper investigates the task of 2D human whole-body pose estimation, which aims to localize dense landmarks on the entire human body including face, hands, body, and feet.
Ranked #8 on
2D Human Pose Estimation
on COCO-WholeBody
3D Human Mesh Regression with Dense Correspondence
This paper proposes a model-free 3D human mesh estimation framework, named DecoMR, which explicitly establishes the dense correspondence between the mesh and the local image features in the UV space (i. e. a 2D space used for texture mapping of 3D mesh).
Ranked #1 on
3D Human Reconstruction
on Surreal
Learning a Reinforced Agent for Flexible Exposure Bracketing Selection
Automatically selecting exposure bracketing (images exposed differently) is important to obtain a high dynamic range image by using multi-exposure fusion.
Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning
Channel pruning is a popular technique for compressing convolutional neural networks (CNNs), where various pruning criteria have been proposed to remove the redundant filters.
AdaX: Adaptive Gradient Descent with Exponential Long Term Memory
Although adaptive optimization algorithms such as Adam show fast convergence in many machine learning tasks, this paper identifies a problem of Adam by analyzing its performance in a simple non-convex synthetic problem, showing that Adam's fast convergence would possibly lead the algorithm to local minimums.
