Search Results for author:
Found 53 papers, 13 papers with code
RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins
In the rapidly advancing field of robotics, dual-arm coordination and complex object manipulation are essential capabilities for developing advanced autonomous systems.
Dexterous Manipulation through Imitation Learning: A Survey
This survey provides an overview of dexterous manipulation methods based on imitation learning (IL), details recent advances, and addresses key challenges in the field.
Adaptive Interactive Navigation of Quadruped Robots using Large Language Models
Robotic navigation in complex environments remains a critical research challenge.
Post-interactive Multimodal Trajectory Prediction for Autonomous Driving
Next, we build a hypergraph neural network-based Trajectory Proposal Network to generate trajectory proposals, where the high-order interaction features are learned by the hypergraphs.
AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems
Introducing AgiBot World, a large-scale platform comprising over 1 million trajectories across 217 tasks in five deployment scenarios, we achieve an order-of-magnitude increase in data scale compared to existing datasets.
RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete
To enhance the robotic brain's core capabilities from abstract to concrete, we introduce ShareRobot, a high-quality heterogeneous dataset that labels multi-dimensional information such as task planning, object affordance, and end-effector trajectory.
Interpretable Droplet Digital PCR Assay for Trustworthy Molecular Diagnostics
Droplet digital PCR (ddPCR) has emerged as a gold standard for achieving absolute quantification.
SafeDrive: Knowledge- and Data-Driven Risk-Sensitive Decision-Making for Autonomous Vehicles with Large Language Models
SafeDrive establishes a novel paradigm for integrating knowledge- and data-driven methods, highlighting significant potential to improve safety and adaptability of autonomous driving in high-risk traffic scenarios.
M$^3$PC: Test-time Model Predictive Control for Pretrained Masked Trajectory Model
Recent work in Offline Reinforcement Learning (RL) has shown that a unified Transformer trained under a masked auto-encoding objective can effectively capture the relationships between different modalities (e. g., states, actions, rewards) within given trajectory datasets.
DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation
Our framework achieves an average of 70. 7% success rate on goal adaptive dexterous tasks, highlighting its robustness and flexibility in contact-rich manipulation.
G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulation
Our results demonstrate the effectiveness of G3Flow in enhancing real-time dynamic semantic feature understanding for robotic manipulation policies.
Autoregressive Models in Vision: A Survey
Autoregressive modeling has been a huge success in the field of natural language processing (NLP).
EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents
Heterogeneous multi-robot systems (HMRS) have emerged as a powerful approach for tackling complex tasks that single robots cannot manage alone.
Articulated Object Manipulation using Online Axis Estimation with SAM2-Based Tracking
To address this limitation, we present a closed-loop pipeline integrating interactive perception with online axis estimation from segmented 3D point clouds.
Cog-GA: A Large Language Models-based Generative Agent for Vision-Language Navigation in Continuous Environments
Firstly, it constructs a cognitive map, integrating temporal, spatial, and semantic elements, thereby facilitating the development of spatial memory within LLMs.
RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)
To address this, we introduce RoboTwin, a generative digital twin framework that uses 3D generative foundation models and large language models to produce diverse expert datasets and provide a real-world-aligned evaluation platform for dual-arm robotic tasks.
HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model
Specifically, HiAgent prompts LLMs to formulate subgoals before generating executable actions and enables LLMs to decide proactively to replace previous subgoals with summarized observations, retaining only the action-observation pairs relevant to the current subgoal.
CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversations
Visual navigation tasks are critical for household service robots.
DAG-Plan: Generating Directed Acyclic Dependency Graphs for Dual-Arm Cooperative Planning
Extensive experiments demonstrate the superiority of DAG-Plan over directly using LLM to generate linear task sequence, achieving 52. 8% higher efficiency compared to the single-arm task planning and 48% higher success rate of the dual-arm task planning.
Learning Reward for Robot Skills Using Large Language Models via Self-Alignment
Learning reward functions remains the bottleneck to equip a robot with a broad repertoire of skills.
ManiPose: A Comprehensive Benchmark for Pose-aware Object Manipulation in Robotics
Robotic manipulation in everyday scenarios, especially in unstructured environments, requires skills in pose-aware object manipulation (POM), which adapts robots' grasping and handling according to an object's 6D pose.
DOZE: A Dataset for Open-Vocabulary Zero-Shot Object Navigation in Dynamic Environments
Zero-Shot Object Navigation (ZSON) requires agents to autonomously locate and approach unseen objects in unfamiliar environments and has emerged as a particularly challenging task within the domain of Embodied AI.
RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis
Robotic behavior synthesis, the problem of understanding multimodal inputs and generating precise physical control for robots, is an important part of Embodied AI.
Ranked #198 on
Visual Question Answering
on MM-Vet
RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation
To bridge this ``ideal-to-real'' gap, this paper presents \textbf{RobotScript}, a platform for 1) a deployable robot manipulation pipeline powered by code generation; and 2) a code generation benchmark for robot manipulation tasks in free-form natural language.
VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Model
In the realm of household robotics, the Zero-Shot Object Navigation (ZSON) task empowers agents to adeptly traverse unfamiliar environments and locate objects from novel categories without prior explicit training.
SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution
Experiments on multi-task robotic manipulation benchmarks like Meta-World and LOReL demonstrate state-of-the-art performance and human-interpretable skill representations from SkillDiffuser.
Tree-Planner: Efficient Close-loop Task Planning with Large Language Models
This paper studies close-loop task planning, which refers to the process of generating a sequence of skills (a plan) to accomplish a specific goal while adapting the plan based on real-time observations.
Human-oriented Representation Learning for Robotic Manipulation
We introduce Task Fusion Decoder as a plug-and-play embedding translator that utilizes the underlying relationships among these perceptual skills to guide the representation learning towards encoding meaningful structure for what's important for all perceptual skills, ultimately empowering learning of downstream robotic manipulation tasks.
LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving
Existing learning-based autonomous driving (AD) systems face challenges in comprehending high-level information, generalizing to rare events, and providing interpretability.
AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model
Aligning agent behaviors with diverse human preferences remains a challenging problem in reinforcement learning (RL), owing to the inherent abstractness and mutability of human preferences.
SEPT: Towards Efficient Scene Representation Learning for Motion Prediction
Motion prediction is crucial for autonomous vehicles to operate safely in complex traffic environments.
MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL
Recently, diffusion model shines as a promising backbone for the sequence modeling paradigm in offline reinforcement learning(RL).
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi-modal understanding and execution capabilities.
EC^2: Emergent Communication for Embodied Control
We learn embodied representations of video trajectories, emergent language, and natural language using a language model, which is then used to finetune a lightweight policy network for downstream control.
AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners
For example, AdaptDiffuser not only outperforms the previous art Diffuser by 20. 8% on Maze2D and 7. 5% on MuJoCo locomotion, but also adapts better to new tasks, e. g., KUKA pick-and-place, by 27. 9% without requiring additional expert data.
EC2: Emergent Communication for Embodied Control
We learn embodied representations of video trajectories, emergent language, and natural language using a language model, which is then used to finetune a lightweight policy network for downstream control.
MaskPlace: Fast Chip Placement via Reinforced Visual Representation Learning
Firstly, MaskPlace recasts placement as a problem of learning pixel-level visual representation to comprehensively describe millions of modules on a chip, enabling placement in a high-resolution canvas and a large action space.
Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learning
The latent world model provides a promising way to learn policies in a compact latent space for tasks with high-dimensional observations, however, its generalization across diverse environments with unseen dynamics remains challenging.
Model-based Reinforcement Learning
reinforcement-learning
+1
Decomposed Mutual Information Optimization for Generalized Context in Meta-Reinforcement Learning
This paper addresses such a challenge by Decomposed Mutual INformation Optimization (DOMINO) for context learning, which explicitly learns a disentangled context to maximize the mutual information between the context and historical trajectories, while minimizing the state transition prediction error.
Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model
End-to-end autonomous driving provides a feasible way to automatically maximize overall driving system performance by directly mapping the raw pixels from a front-facing camera to control signals.
EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Model
Unsupervised reinforcement learning (URL) poses a promising paradigm to learn useful behaviors in a task-agnostic environment without the guidance of extrinsic rewards to facilitate the fast adaptation of various downstream tasks.
CtrlFormer: Learning Transferable State Representation for Visual Control via Transformer
In visual control, learning transferable state representation that can transfer between different control tasks is important to reduce the training sample size.
CO^3: Cooperative Unsupervised 3D Representation Learning for Autonomous Driving
In this paper, we propose CO^3, namely Cooperative Contrastive Learning and Contextual Shape Prediction, to learn 3D representation for outdoor-scene point clouds in an unsupervised manner.
Flow-based Recurrent Belief State Learning for POMDPs
Furthermore, we show that the learned belief states can be plugged into downstream RL algorithms to improve performance.
Scale-Equivalent Distillation for Semi-Supervised Object Detection
Further, we overcome these challenges by introducing a novel approach, Scale-Equivalent Distillation (SED), which is a simple yet effective end-to-end knowledge distillation framework robust to large object size variance and class imbalance.
Don't Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learning
One of the key challenges in visual Reinforcement Learning (RL) is to learn policies that can generalize to unseen environments.
Model-Based Reinforcement Learning via Imagination with Derived Memory
Model-based reinforcement learning aims to improve the sample efficiency of policy learning by modeling the dynamics of the environment.
Model-based Reinforcement Learning
reinforcement-learning
+2
Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian
Based on this, the penalty method is formulated as a proportional controller, and the Lagrangian method is formulated as an integral controller.
Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning
Taking a control perspective, we first interpret the penalty method and the Lagrangian method as proportional feedback and integral feedback control, respectively.
Steadily Learn to Drive with Virtual Memory
Reinforcement learning has shown great potential in developing high-level autonomous driving.
Robust Memory Augmentation by Constrained Latent Imagination
The latent dynamics model summarizes an agent’s high dimensional experiences in a compact way.
Model-Based Actor-Critic with Chance Constraint for Stochastic System
Safety is essential for reinforcement learning (RL) applied in real-world situations.
Mixed Reinforcement Learning with Additive Stochastic Uncertainty
This paper presents a mixed reinforcement learning (mixed RL) algorithm by simultaneously using dual representations of environmental dynamics to search the optimal policy with the purpose of improving both learning accuracy and training speed.
