Search Results for author:
Found 95 papers, 53 papers with code
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
A robot foundation model, trained on massive and diverse data sources, is essential for enabling the robots to reason about novel situations, robustly handle real-world variability, and rapidly learn new tasks.
Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training
Hydra-NeXt surpasses the previous state-of-the-art by 22. 98 DS and 17. 49 SR, marking a significant advancement in autonomous driving.
Centaur: Robust End-to-End Autonomous Driving with Test-Time Training
In this work, we propose Centaur (Cluster Entropy for Test-time trAining using Uncertainty) which updates a planner's behavior via test-time training, without relying on hand-engineered rules or cost functions.
Enhancing Autonomous Driving Safety with Collision Scenario Integration
In this paper, we propose SafeFusion, a training framework to learn from collision data.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
To address this challenge, we present RocketKV, a training-free KV cache compression strategy designed specifically to reduce both memory bandwidth and capacity demand of KV cache during the decode phase.
AIDE: Agentically Improve Visual Language Model with Domain Experts
The enhancement of Visual Language Models (VLMs) has traditionally relied on knowledge distillation from larger, more capable models.
QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation
We validate the effectiveness of QLIP for multimodal understanding and text-conditioned image generation with a single model.
Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models
Recently, promising progress has been made by open-source vision-language models (VLMs) in bringing their capabilities closer to those of proprietary frontier models.
StreamChat: Chatting with Streaming Video
This paper presents StreamChat, a novel approach that enhances the interaction capabilities of Large Multimodal Models (LMMs) with streaming video content.
Neural Eulerian Scene Flow Fields
We reframe scene flow as the task of estimating a continuous space-time ODE that describes motion for an entire observation sequence, represented with a neural prior.
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs).
Exploring Camera Encoder Designs for Autonomous Driving Perception
The cornerstone of autonomous vehicles (AV) is a solid perception system, where camera encoders play a crucial role.
Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation
We propose Hydra-MDP, a novel paradigm employing multiple teachers in a teacher-student model.
X-VILA: Cross-Modality Alignment for Large Language Model
We introduce X-VILA, an omni-modality model designed to extend the capabilities of large language models (LLMs) by incorporating image, video, and audio modalities.
Memorize What Matters: Emergent Scene Decomposition from Multitraverse
Humans naturally retain memories of permanent elements, while ephemeral moments often slip through the cracks of memory.
OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning
We further propose OmniDrive-nuScenes, a new visual question-answering dataset challenging the true 3D situational awareness of a model with comprehensive visual question-answering (VQA) tasks, including scene description, traffic regulation, 3D grounding, counterfactual reasoning, decision making and planning.
What is Point Supervision Worth in Video Instance Segmentation?
Video instance segmentation (VIS) is a challenging vision task that aims to detect, segment, and track objects in videos.
LITA: Language Instructed Temporal-Localization Assistant
In addition to leveraging existing video datasets with timestamps, we propose a new task, Reasoning Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for learning and evaluating this task.
Improving Distant 3D Object Detection Using 2D Box Supervision
This mapping allows the depth estimation of distant objects conditioned on their 2D boxes, making long-range 3D detection with 2D supervision feasible.
T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching
Sampling from diffusion probabilistic models (DPMs) is often expensive for high-quality image generation and typically requires many steps with a large model.
Fully Attentional Networks with Self-emerging Token Labeling
With the proposed STL framework, our best model based on FAN-L-Hybrid (77. 3M parameters) achieves 84. 8% Top-1 accuracy and 42. 1% mCE on ImageNet-1K and ImageNet-C, and sets a new state-of-the-art for ImageNet-A (46. 1%) and ImageNet-R (56. 6%) without using extra data, outperforming the original FAN counterpart by significant margins.
Ranked #16 on
Domain Generalization
on ImageNet-C
A Semantic Space is Worth 256 Language Descriptions: Make Stronger Segmentation Models with Descriptive Properties
Instead of relying solely on category-specific annotations, ProLab uses descriptive properties grounded in common sense knowledge for supervising segmentation models.
Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?
We initially observed that the nuScenes dataset, characterized by relatively simple driving scenarios, leads to an under-utilization of perception information in end-to-end models incorporating ego status, such as the ego vehicle's velocity.
FocalFormer3D : Focusing on Hard Instance for 3D Object Detection
For 3D object detection, we instantiate this method as FocalFormer3D, a simple yet effective detector that excels at excavating difficult objects and improving prediction recall.
Ranked #8 on
3D Object Detection
on nuScenes
FB-BEV: BEV Representation from Forward-Backward View Transformations
Currently, the two most prominent VTM paradigms are forward projection and backward projection.
FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation
This technical report summarizes the winning solution for the 3D Occupancy Prediction Challenge, which is held in conjunction with the CVPR 2023 Workshop on End-to-End Autonomous Driving and CVPR 23 Workshop on Vision-Centric Autonomous Driving Workshop.
Ranked #2 on
Prediction Of Occupancy Grid Maps
on Occ3D-nuScenes
Differentially Private Video Activity Recognition
In recent years, differential privacy has seen significant advancements in image classification; however, its application to video activity recognition remains under-explored.
SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Driving
Monocular scene understanding is a foundational component of autonomous systems.
3D Semantic Scene Completion
3D Semantic Scene Completion from a single 2D image
Real-Time Radiance Fields for Single-Image Portrait View Synthesis
We present a one-shot method to infer and render a photorealistic 3D representation from a single unposed image (e. g., face portrait) in real-time.
Prismer: A Vision-Language Model with Multi-Task Experts
Recent vision-language models have shown impressive multi-modal generation capabilities.
Ranked #1 on
Image Captioning
on nocaps val
VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion
To enable such capability in AI systems, we propose VoxFormer, a Transformer-based semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images.
3D geometry
3D Semantic Scene Completion from a single RGB image
+1
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
Augmenting pretrained language models (LMs) with a vision encoder (e. g., Flamingo) has obtained the state-of-the-art results in image-to-text generation.
Vision Transformers Are Good Mask Auto-Labelers
We propose Mask Auto-Labeler (MAL), a high-quality Transformer-based mask auto-labeling framework for instance segmentation using only box annotations.
End-to-end 3D Tracking with Decoupled Queries
In this work, we present an end-to-end framework for camera-based 3D multi-object tracking, called DQTrack.
FocalFormer3D: Focusing on Hard Instance for 3D Object Detection
For 3D object detection, we instantiate this method as FocalFormer3D, a simple yet effective detector that excels at excavating difficult objects and improving prediction recall.
1st Place Solution of The Robust Vision Challenge 2022 Semantic Segmentation Track
The model is trained on a composite dataset consisting of images from 9 datasets (ADE20K, Cityscapes, Mapillary Vistas, ScanNet, VIPER, WildDash 2, IDD, BDD, and COCO) with a simple dataset balancing strategy.
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data.
PointDP: Diffusion-driven Purification against Adversarial Attacks on 3D Point Cloud Recognition
3D Point cloud is becoming a critical data representation in many real-world applications like autonomous driving, robotics, and medical imaging.
MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training
By only training a query-based image instance segmentation model, MinVIS outperforms the previous best result on the challenging Occluded VIS dataset by over 10% AP.
How Much More Data Do I Need? Estimating Requirements for Downstream Tasks
Given a small training data set and a learning algorithm, how much more data is necessary to reach a target validation or test performance?
Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions
A significant gap remains between today's visual pattern recognition models and human-level visual cognition especially when it comes to few-shot learning and compositional reasoning of novel concepts.
Ranked #1 on
Few-Shot Image Classification
on Bongard-HOI
Understanding The Robustness in Vision Transformers
Our study is motivated by the intriguing properties of the emerging visual grouping in Vision Transformers, which indicates that self-attention may promote robustness through improved mid-level representations.
Ranked #4 on
Domain Generalization
on ImageNet-R
(using extra training data)
RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning
This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of entities, and 3) generalizing to novel object-relation combinations, i. e., systematic generalization.
Ranked #1 on
Zero-Shot Human-Object Interaction Detection
on HICO
M$^2$BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation
In this paper, we propose M$^2$BEV, a unified framework that jointly performs 3D object detection and map segmentation in the Birds Eye View~(BEV) space with multi-camera image inputs.
CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs
We represent the correspondence maps of different images as warped coordinate frames transformed from a canonical coordinate frame, i. e., the correspondence map, which describes the structure (e. g., the shape of a face), is controlled via a transformation.
FreeSOLO: Learning to Segment Objects without Annotations
FreeSOLO further demonstrates superiority as a strong pre-training method, outperforming state-of-the-art self-supervised pre-training methods by +9. 8% AP when fine-tuning instance segmentation with only 5% COCO masks.
Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions
Deep neural networks on 3D point cloud data have been widely used in the real world, especially in safety-critical applications.
Ranked #1 on
3D Point Cloud Data Augmentation
on ModelNet40-C
3D Point Cloud Classification
3D Point Cloud Data Augmentation
+2
Adversarially Robust 3D Point Cloud Recognition Using Self-Supervisions
In this paper, we systematically study the impact of various self-supervised learning proxy tasks on different architectures and threat models for 3D point clouds with adversarial training.
Coupled Segmentation and Edge Learning via Dynamic Graph Propagation
It is therefore interesting to study how these two tasks can be coupled to benefit each other.
AugMax: Adversarial Composition of Random Augmentations for Robust Training
Diversity and hardness are two complementary dimensions of data augmentation to achieve robustness.
Learning Contrastive Representation for Semantic Correspondence
Dense correspondence across semantically related images has been extensively studied, but still faces two challenges: 1) large variations in appearance, scale and pose exist even for objects from the same category, and 2) labeling pixel-level dense correspondences is labor intensive and infeasible to scale.
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers
Specifically, we supervise the attention modules in the mask decoder in a layer-wise manner.
Ranked #4 on
Panoptic Segmentation
on COCO test-dev
Not All Labels Are Equal: Rationalizing The Labeling Costs for Training Object Detection
Deep neural networks have reached high accuracy on object detection but their success hinges on large amounts of labeled data.
SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
A student network then learns to mimic the expert policy by supervised learning with strong augmentations, making its representation more robust against visual variations compared to the expert.
Taxonomy of Machine Learning Safety: A Survey and Primer
The open-world deployment of Machine Learning (ML) algorithms in safety-critical applications such as autonomous vehicles needs to address a variety of ML vulnerabilities such as interpretability, verifiability, and performance limitations.
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders.
Ranked #1 on
Semantic Segmentation
on COCO-Stuff full
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision
We introduce DiscoBox, a novel framework that jointly learns instance segmentation and semantic correspondence using bounding box supervision.
Image-Level or Object-Level? A Tale of Two Resampling Strategies for Long-Tailed Detection
As a result, image resampling alone is not enough to yield a sufficiently balanced distribution at the object level.
Contrastive Syn-to-Real Generalization
Training on synthetic data can be beneficial for label or data-scarce scenarios.
Transferable Unsupervised Robust Representation Learning
We upend this view and show that URRL improves both the natural accuracy of unsupervised representation learning and its robustness to corruptions and adversarial noise.
UFO$^2$: A Unified Framework towards Omni-supervised Object Detection
Existing work on object detection often relies on a single form of annotation: the model is trained using either accurate yet costly bounding boxes or cheaper but less expressive image-level tags.
Learning Calibrated Uncertainties for Domain Shift: A Distributionally Robust Learning Approach
We propose a framework for learning calibrated uncertainties under domain shifts, where the source (training) distribution differs from the target (test) distribution.
Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning
Inspired by the original one hundred BPs, we propose a new benchmark Bongard-LOGO for human-level concept learning and reasoning.
Distributionally Robust Learning for Unsupervised Domain Adaptation
This formulation motivates the use of two neural networks that are jointly trained --- a discriminative network between the source and target domains for density-ratio estimation, in addition to the standard classification network.
Delving Deeper into Anti-aliasing in ConvNets
Aliasing refers to the phenomenon that high frequency signals degenerate into completely different ones after sampling.
Joint Disentangling and Adaptation for Cross-Domain Person Re-Identification
To this end, we propose a joint learning framework that disentangles id-related/unrelated features and enforces adaptation to work on the id-related feature space exclusively.
Ranked #7 on
Unsupervised Domain Adaptation
on Market to MSMT
Unsupervised Controllable Generation with Self-Training
Instead, we propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training.
Neural Networks with Recurrent Generative Feedback
This mechanism can be interpreted as a form of self-consistency between the maximum a posteriori (MAP) estimation of an internal generative model and the external environment.
Transposer: Universal Texture Synthesis Using Feature Maps as Transposed Convolution Filter
Specifically, we directly treat the whole encoded feature map of the input texture as transposed convolution filters and the features' self-similarity map, which captures the auto-correlation information, as input to the transposed convolution.
Automated Synthetic-to-Real Generalization
Models trained on synthetic images often face degraded generalization to real data.
Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation
Experiments on the NIH pancreas segmentation dataset and a multi-organ segmentation dataset show state-of-the-art performance of the proposed framework on semi-supervised medical image segmentation.
Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detection
Weakly supervised learning has emerged as a compelling tool for object detection by reducing the need for strong supervision during training.
Ranked #1 on
Weakly Supervised Object Detection
on COCO test-dev
Angular Visual Hardness
We also find that AVH has a statistically significant correlation with human visual hardness.
Confidence Regularized Self-Training
Recent advances in domain adaptation show that deep self-training presents a powerful means for unsupervised domain adaptation.
Ranked #18 on
Image-to-Image Translation
on SYNTHIA-to-Cityscapes
Regularizing Neural Networks via Minimizing Hyperspherical Energy
Inspired by the Thomson problem in physics where the distribution of multiple propelling electrons on a unit sphere can be modeled via minimizing some potential energy, hyperspherical energy minimization has demonstrated its potential in regularizing neural networks and improving their generalization power.
Joint Discriminative and Generative Learning for Person Re-identification
To this end, we propose a joint learning framework that couples re-id learning and data generation end-to-end.
Ranked #1 on
Person Re-Identification
on UAV-Human
Image-to-Image Translation
Unsupervised Domain Adaptation
+1
Partial Convolution based Padding
In this paper, we present a simple yet effective padding scheme that can be used as a drop-in module for existing convolutional neural networks.
Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training
In this paper, we propose a novel UDA framework based on an iterative self-training procedure, where the problem is formulated as latent variable loss minimization, and can be solved by alternatively generating pseudo labels on target data and re-training the model with these labels.
Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training
Recent deep networks achieved state of the art performanceon a variety of semantic segmentation tasks.
Simultaneous Edge Alignment and Learning
Edge detection is among the most fundamental vision problems for its role in perceptual grouping and its wide applications.
Learning towards Minimum Hyperspherical Energy
In light of this intuition, we reduce the redundancy regularization problem to generic energy minimization, and propose a minimum hyperspherical energy (MHE) objective as generic regularization for neural networks.
Decoupled Networks
Inner product-based convolution has been a central component of convolutional neural networks (CNNs) and the key to learning visual representations.
Learning Strict Identity Mappings in Deep Residual Networks
A family of super deep networks, referred to as residual networks or ResNet, achieved record-beating performance in various visual tasks such as image recognition, object detection, and semantic segmentation.
Deep Hyperspherical Learning
In light of such challenges, we propose hyperspherical convolution (SphereConv), a novel learning framework that gives angular representations on hyperspheres.
CASENet: Deep Category-Aware Semantic Edge Detection
To this end, we propose a novel end-to-end deep semantic edge learning architecture based on ResNet and a new skip-layer architecture where category-wise edge activations at the top convolution layer share and are fused with the same set of bottom layer features.
Ranked #1 on
Edge Detection
on SBD
SphereFace: Deep Hypersphere Embedding for Face Recognition
This paper addresses deep face recognition (FR) problem under open-set protocol, where ideal face features are expected to have smaller maximal intra-class distance than minimal inter-class distance under a suitably chosen metric space.
Ranked #1 on
Face Verification
on CK+
Large-Margin Softmax Loss for Convolutional Neural Networks
Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs).
Jointly Learning Non-negative Projection and Dictionary with Discriminative Graph Constraints for Classification
Sparse coding with dictionary learning (DL) has shown excellent classification performance.
Structured Hough Voting for Vision-based Highway Border Detection
We propose a vision-based highway border detection algorithm using structured Hough voting.
KCRC-LCD: Discriminative Kernel Collaborative Representation with Locality Constrained Dictionary for Visual Categorization
The LCD similarity measure can be kernelized under KCRC, which theoretically links CRC and LCD under the kernel method.
Transitive Distance Clustering with K-Means Duality
We propose a very intuitive and simple approximation for the conventional spectral clustering methods.
Multi-Task Regularization with Covariance Dictionary for Linear Classifiers
In this paper we propose a multi-task linear classifier learning problem called D-SVM (Dictionary SVM).
Constructing the L2-Graph for Robust Subspace Learning and Subspace Clustering
Under the framework of graph-based learning, the key to robust subspace clustering and subspace learning is to obtain a good similarity graph that eliminates the effects of errors and retains only connections between the data points from the same subspace (i. e., intra-subspace data points).
