Search Results for author:
Found 104 papers, 50 papers with code
Generating Comprehensive Lithium Battery Charging Data with Generative AI
Through preprocessing data into a quasi-video format, our study achieves an integrated synthesis of electrochemical data, including voltage, current, temperature, and charging capacity, which is then processed by the RCVAE model.
P-MapNet: Far-seeing Map Generator Enhanced by both SDMap and HDMap Priors
On the other hand, we exploit a masked autoencoder to capture the prior distribution of HDMap, which can serve as a refinement module to mitigate occlusions and artifacts.
PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors
Autonomous vehicles rely extensively on perception systems to navigate and interpret their surroundings.
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
We introduce DriveVLM, an autonomous driving system leveraging Vision-Language Models (VLMs) for enhanced scene understanding and planning capabilities.
PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models
As a state-of-the-art, open-source image generation model, PIXART-{\delta} offers a promising alternative to the Stable Diffusion family of models, contributing significantly to text-to-image synthesis.
LCM-LoRA: A Universal Stable-Diffusion Acceleration Module
Latent Consistency Models (LCMs) have achieved impressive performance in accelerating text-to-image generative tasks, producing high-quality images with minimal inference steps.
Large Trajectory Models are Scalable Motion Predictors and Planners
STR reformulates the motion prediction and motion planning problems by arranging observations, states, and actions into one unified sequence modeling task.
LiDAR-based 4D Occupancy Completion and Forecasting
In this paper, we introduce a novel LiDAR perception task of Occupancy Completion and Forecasting (OCF) in the context of autonomous driving to unify these aspects into a cohesive framework.
What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?
With the growing success of multi-modal learning, research on the robustness of multi-modal models, especially when facing situations with missing modalities, is receiving increased attention.
Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable Environments
In this work, we focus on imitator learning based on only one expert demonstration.
Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models
To this end, we introduce Multi-Modal Low-Rank Adaptation learning (MMLoRA).
Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
Inspired by Consistency Models (song et al.), we propose Latent Consistency Models (LCMs), enabling swift inference with minimal steps on any pre-trained LDMs, including Stable Diffusion (rombach et al).
GPT-Driver: Learning to Drive with GPT
In this paper, we propose a novel approach to motion planning that capitalizes on the strong reasoning capabilities and generalization potential inherent to Large Language Models (LLMs).
Ranked #1 on
Motion Planning
on nuScenes
Uncertainty-Aware Decision Transformer for Stochastic Driving Environments
Offline Reinforcement Learning (RL) has emerged as a promising framework for learning policies without active interactions, making it especially appealing for autonomous driving tasks.
AutoEncoding Tree for City Generation and Applications
Based on this dataset, we propose AETree, a tree-structured auto-encoder neural network, for city generation.
Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills
Learning-based vehicle planning is receiving increasing attention with the emergence of diverse driving simulators and large-scale driving datasets.
Robot Parkour Learning
Parkour is a grand challenge for legged locomotion that requires robots to overcome various obstacles rapidly in complex environments.
StreamMapNet: Streaming Mapping Network for Vectorized Online HD Map Construction
This approach limits their stability and performance in complex scenarios such as occlusions, largely due to the absence of temporal information.
Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals
Millimeter wave (mmWave) based speech recognition provides more possibility for audio-related applications, such as conference speech transcription and eavesdropping.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Learning-based Control for PMSM Using Distributed Gaussian Processes with Optimal Aggregation Strategy
The growing demand for accurate control in varying and unknown environments has sparked a corresponding increase in the requirements for power supply components, including permanent magnet synchronous motors (PMSMs).
Reconstructing Three-decade Global Fine-Grained Nighttime Light Observations by a New Super-Resolution Framework
We provide the long-term and fine-grained nighttime light observations to promote research on human activities.
Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models
The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in video/film production.
BEVScope: Enhancing Self-Supervised Depth Estimation Leveraging Bird's-Eye-View in Dynamic Scenarios
Depth estimation is a cornerstone of perception in autonomous driving and robotic systems.
A Universal Semantic-Geometric Representation for Robotic Manipulation
To this end, we present $\textbf{Semantic-Geometric Representation} (\textbf{SGR})$, a universal perception module for robotics that leverages the rich semantic information of large-scale pre-trained 2D models and inherits the merits of 3D spatial reasoning.
SSCBench: Monocular 3D Semantic Scene Completion Benchmark in Street Views
Monocular scene understanding is a foundational component of autonomous systems.
3D Semantic Scene Completion
3D Semantic Scene Completion from a single 2D image
ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory
Large language models (LLMs) with memory are computationally universal.
GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Training
This paper tries to address a fundamental question in point cloud self-supervised learning: what is a good signal we should leverage to learn features from point clouds without annotations?
On Uni-Modal Feature Learning in Supervised Multi-Modal Learning
We abstract the features (i. e. learned representations) of multi-modal data into 1) uni-modal features, which can be learned from uni-modal training, and 2) paired features, which can only be learned from cross-modal interactions.
Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving
3D occupancy prediction, which estimates the detailed occupancy states and semantics of a scene, is an emerging task to overcome these limitations.
What Happened 3 Seconds Ago? Inferring the Past with Thermal Imaging
Inferring past human motion from RGB images is challenging due to the inherent uncertainty of the prediction problem.
Programmatically Grounded, Compositionally Generalizable Robotic Manipulation
Robots operating in the real world require both rich manipulation skills as well as the ability to semantically reason about when to apply those skills.
Neural Map Prior for Autonomous Driving
To the best of our knowledge, this is the first learning-based system for creating a global map prior.
SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer
High-resolution images enable neural networks to learn richer visual representations.
INT2: Interactive Trajectory Prediction at Intersections
One of the most challenging problems in motion forecasting is interactive trajectory prediction, whose goal is to jointly forecasts the future trajectories of interacting agents.
AsyInst: Asymmetric Affinity with DepthGrad and Color for Box-Supervised Instance Segmentation
Through systematic analysis, we found that the commonly used pairwise affinity loss has two limitations: (1) it works with color affinity but leads to inferior performance with other modalities such as depth gradient, (2)the original affinity loss does not prevent trivial predictions as intended but actually accelerates this process due to the affinity loss term being symmetric.
Learning Physically Realizable Skills for Online Packing of General 3D Shapes
We study the problem of learning online packing skills for irregular 3D shapes, which is arguably the most challenging setting of bin packing problems.
PVT++: A Simple End-to-End Latency-Aware Visual Tracking Framework
Visual object tracking is essential to intelligent robots.
P4P: Conflict-Aware Motion Prediction for Planning in Autonomous Driving
Motion prediction is crucial in enabling safe motion planning for autonomous vehicles in interactive scenarios.
VectorFlow: Combining Images and Vectors for Traffic Occupancy and Flow Prediction
Recently, the occupancy flow fields representation was proposed to represent joint future states of road agents through a combination of occupancy grid and flow, which supports efficient and consistent joint predictions.
ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries
In this work, we propose ViP3D, a query-based visual trajectory prediction pipeline that exploits rich information from raw videos to directly predict future trajectories of agents in a scene.
Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Vision
This paper focuses on perceiving and navigating 3D environments using echoes and RGB image.
Augmented Imagefication: A Data-driven Fault Detection Method for Aircraft Air Data Sensors
In this paper, a novel data-driven approach named Augmented Imagefication for Fault detection (FD) of aircraft air data sensors (ADS) is proposed.
VectorMapNet: End-to-end Vectorized HD Map Learning
To the best of our knowledge, VectorMapNet is the first work designed towards end-to-end vectorized map learning from onboard observations.
Ranked #1 on
HD semantic map learning
on nuScenes
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation
Crossmodal knowledge distillation (KD) extends traditional knowledge distillation to the area of multimodal learning and demonstrates great success in various applications.
R4D: Utilizing Reference Objects for Long-Range Distance Estimation
An edge in the graph encodes the relative distance information between a pair of target and reference objects.
Depth Estimation Matters Most: Improving Per-Object Depth Estimation for Monocular 3D Detection and Tracking
Monocular image-based 3D perception has become an active research area in recent years owing to its applications in autonomous driving.
Learning Visual Styles from Audio-Visual Associations
Our model learns to manipulate the texture of a scene to match a sound, a problem we term audio-driven image stylization.
Sound2Synth: Interpreting Sound via FM Synthesizer Parameters Estimation
Synthesizer is a type of electronic musical instrument that is now widely used in modern music production and sound design.
MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries
In contrast to prior works, MUTR3D does not explicitly rely on the spatial and appearance similarity of objects.
Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization
Multimodal fusion emerges as an appealing technique to improve model performances on many tasks.
Egocentric Prediction of Action Target in 3D
We are interested in anticipating as early as possible the target location of a person's object manipulation action in a 3D workspace from egocentric vision.
Self-supervision through Random Segments with Autoregressive Coding (RandSAC)
We illustrate that randomized serialization of the segments significantly improves the performance and results in distribution over spatially-long (across-segments) and -short (within-segment) predictions which are effective for feature learning.
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection
Sensor fusion is an essential topic in many perception systems, such as autonomous driving and robotics.
CYBORGS: Contrastively Bootstrapping Object Representations by Grounding in Segmentation
Many recent approaches in contrastive learning have worked to close the gap between pretraining on iconic images like ImageNet and pretraining on complex scenes like COCO.
M2I: From Factored Marginal Trajectory Prediction to Interactive Prediction
Predicting future motions of road participants is an important task for driving autonomously in urban scenes.
S3T: Self-Supervised Pre-training with Swin Transformer for Music Classification
To our knowledge, S3T is the first method combining the Swin Transformer with a self-supervised learning method for music classification.
AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detection
This map enables our model to automate the alignment of non-homogenous features in a dynamic and data-driven manner.
Embracing Single Stride 3D Object Detector with Sparse Transformer
In LiDAR-based 3D object detection for autonomous driving, the ratio of the object size to input scene size is significantly smaller compared to 2D detection cases.
Ranked #3 on
3D Object Detection
on waymo cyclist
IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes
While most works focus on single-object or agent-object visual functionality and affordances, our work proposes to study a new kind of visual relationship that is also important to perceive and model -- inter-object functional relationships (e. g., a switch on the wall turns on or off the light, a remote control operates the TV).
SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations
To bridge this gap, we aim to learn a spatial-aware visual representation that can describe the three-dimensional space and is more suitable and effective for these tasks.
Neural Dubber: Dubbing for Videos According to Scripts
Neural Dubber is a multi-modal text-to-speech (TTS) model that utilizes the lip movement in the video to control the prosody of the generated speech.
DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
This top-down approach outperforms its bottom-up counterpart in which object bounding box prediction follows per-pixel depth estimation, since it does not suffer from the compounding error introduced by a depth prediction model.
Ranked #7 on
Robust Camera Only 3D Object Detection
on nuScenes-C
Modality Laziness: Everybody's Business is Nobody's Business
We name this problem of multi-modal training, \emph{Modality Laziness}.
Learning Efficient Online 3D Bin Packing on Packing Configuration Trees
PCT is a full-fledged description of the state and action space of bin packing which can support packing policy learning based on deep reinforcement learning (DRL).
Learning Practically Feasible Policies for Online 3D Bin Packing
In this problem, the items are delivered to the agent without informing the full sequence information.
DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets
In this work, we propose an anchor-free and end-to-end trajectory prediction model, named DenseTNT, that directly outputs a set of trajectories from dense goal candidates.
HDMapNet: An Online HD Map Construction and Evaluation Framework
By introducing the method and metrics, we invite the community to study this novel map learning problem.
HDMapGen: A Hierarchical Graph Generative Model of High Definition Maps
To address this issue, we introduce a new challenging task to generate HD maps.
DenseTNT: Waymo Open Dataset Motion Prediction Challenge 1st Place Solution
In autonomous driving, goal-based multi-trajectory prediction methods are proved to be effective recently, where they first score goal candidates, then select a final set of goals, and finally complete trajectories based on the selected goals.
Intrinsically Motivated Self-supervised Learning in Reinforcement Learning
In vision-based reinforcement learning (RL) tasks, it is prevalent to assign auxiliary tasks with a surrogate self-supervised loss so as to obtain more semantic representations and improve sample efficiency.
Co-advise: Cross Inductive Bias Distillation
Transformers recently are adapted from the community of natural language processing as a promising substitute of convolution-based neural networks for visual learning tasks.
Improving Multi-Modal Learning with Uni-Modal Teachers
We name this problem Modality Failure, and hypothesize that the imbalance of modalities and the implicit bias of common objectives in fusion method prevent encoders of each modality from sufficient feature learning.
Ranked #60 on
Semantic Segmentation
on NYU Depth v2
What Makes Multi-modal Learning Better than Single (Provably)
The world provides us with data of multiple modalities.
On Feature Decorrelation in Self-Supervised Learning
In self-supervised representation learning, a common idea behind most of the state-of-the-art approaches is to enforce the robustness of the representations to predefined augmentations.
Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset
Furthermore, we introduce a new set of metrics that provides a comprehensive evaluation of both single agent and joint agent interaction motion forecasting models.
Multimodal Knowledge Expansion
The popularity of multimodal sensors and the accessibility of the Internet have brought us a massive amount of unlabeled multimodal data.
Ranked #63 on
Semantic Segmentation
on NYU Depth v2
Predictive Visual Tracking: A New Benchmark and Baseline Approach
As a crucial robotic perception capability, visual tracking has been intensively studied recently.
Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset
Furthermore, we introduce a new set of metrics that provides a comprehensive evaluation of both single agent and joint agent interaction motion forecasting models.
AETree: Areal Spatial Data Generation
Areal spatial data represent not only geographical locations but also sizes and shapes of physical objects such as buildings in a city.
UnModNet: Learning to Unwrap a Modulo Image for High Dynamic Range Imaging
A conventional camera often suffers from over- or under-exposure when recording a real-world scene with a very high dynamic range (HDR).
Unsupervised Monocular Depth Learning in Dynamic Scenes
We present a method for jointly training the estimation of depth, ego-motion, and a dense 3D translation field of objects relative to the scene, with monocular photometric consistency being the sole source of supervision.
CLOUD: Contrastive Learning of Unsupervised Dynamics
Developing agents that can perform complex control tasks from high dimensional observations such as pixels is challenging due to difficulties in learning dynamics efficiently.
LID 2020: The Learning from Imperfect Data Challenge Results
The purpose of the Learning from Imperfect Data (LID) workshop is to inspire and facilitate the research in developing novel approaches that would harness the imperfect data and improve the data-efficiency during training.
SEMI: Self-supervised Exploration via Multisensory Incongruity
The variance of actions is further used to measure action incongruity.
Multivariate Time-series Anomaly Detection via Graph Attention Network
Anomaly detection on multivariate time-series is of great importance in both data mining research and industrial applications.
TNT: Target-driveN Trajectory Prediction
Our key insight is that for prediction within a moderate time horizon, the future modes can be effectively captured by a set of target states.
Online 3D Bin Packing with Constrained Deep Reinforcement Learning
We solve a challenging yet practically useful variant of 3D Bin Packing Problem (3D-BPP).
VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation
Behavior prediction in dynamic, multi-agent systems is an important problem in the context of self-driving cars, due to the complex representations and interactions of road components, including moving agents (e. g. pedestrians and vehicles) and road context information (e. g. lanes, traffic lights).
Music Gesture for Visual Sound Separation
Recent deep learning approaches have achieved impressive performance on visual sound separation tasks.
AlignNet: A Unifying Approach to Audio-Visual Alignment
We present AlignNet, a model that synchronizes videos with reference audios under non-uniform and irregular misalignments.
Neural network with data augmentation in multi-objective prediction of multi-stage pump
Finally, a neural network model based on data augmentation (NNDA) is proposed for the reason that simulation cost is too high and data is scarce in mechanical simulation field especially in CFD problems.
Scalability in Perception for Autonomous Driving: Waymo Open Dataset
In an effort to help align the research community's contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset.
Self-supervised Moving Vehicle Tracking with Stereo Sound
At test time, the stereo-sound student network can work independently to perform object localization us-ing just stereo audio and camera meta-data, without any visual input.
Active Scene Understanding via Online Semantic Reconstruction
In our method, the exploratory robot scanning is both driven by and targeting at the recognition and segmentation of semantic objects from the scene.
Self-Supervised Audio-Visual Co-Segmentation
Segmenting objects in images and separating sound sources in audio are challenging tasks, in part because traditional approaches require large amounts of labeled data.
RF-based 3D skeletons
It maintains this accuracy even in the presence of multiple people, and in new environments that it has not seen in the training set.
Through-Wall Human Pose Estimation Using Radio Signals
Yet, unlike vision-based pose estimation, the radio-based system can estimate 2D poses through walls despite never trained on such scenarios.
The Sound of Pixels
We introduce PixelPlayer, a system that, by leveraging large amounts of unlabeled videos, learns to locate image regions which produce sounds and separate the input sounds into a set of components that represents the sound from each pixel.
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization
This paper presents a new large-scale dataset for recognition and temporal localization of human actions collected from Web videos.
Ranked #10 on
Temporal Action Localization
on HACS
Scene Parsing Through ADE20K Dataset
A novel network design called Cascade Segmentation Module is proposed to parse a scene into stuff, objects, and object parts in a cascade and improve over the baselines.
Open Vocabulary Scene Parsing
Recognizing arbitrary objects in the wild has been a challenging problem due to the limitations of existing classification models and datasets.
Semantic Understanding of Scenes through the ADE20K Dataset
Scene parsing, or recognizing and segmenting objects and stuff in an image, is one of the key problems in computer vision.
Loss Functions for Neural Networks for Image Processing
Neural networks are becoming central in several areas of computer vision and image processing and different architectures have been proposed to solve specific problems.
