Search Results for author:
Found 46 papers, 24 papers with code
A Touch, Vision, and Language Dataset for Multimodal Alignment
This is partially due to the difficulty of obtaining natural language labels for tactile data and the complexity of aligning tactile readings with both visual observations and language descriptions.
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation
Our neural representation driven by multimodal sensing can serve as a perception backbone towards advancing robot dexterity.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation
To overcome this shortcoming, we study the idea of replacing the RGB camera with an event-based camera and introduce a new event-based optical tactile sensor called Evetac.
The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback
Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) more capable in complex settings.
A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning
Model-based Reinforcement Learning (MBRL) aims to make agents more sample-efficient, adaptive, and explainable by learning an explicit model of the environment.
Deep Reinforcement Learning for the Joint Control of Traffic Light Signaling and Vehicle Speed Advice
We show this to improve the efficacy of traffic systems.
General In-Hand Object Rotation with Vision and Touch
We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs.
In-Hand Object Rotation via Rapid Motor Adaptation
Generalized in-hand manipulation has long been an unsolved challenge of robotics.
Investigating Compounding Prediction Errors in Learned Dynamics Models
In this paper, we explore the effects of subcomponents of a control problem on long term prediction error: including choosing a system, collecting data, and training a model.
Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable agents.
What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networks
The co-adaptation of robot morphology and behaviour becomes increasingly important with the advent of fast 3D-manufacturing methods and efficient deep reinforcement learning algorithms.
Active 3D Shape Reconstruction from Vision and Touch
In this paper, we focus on this problem and introduce a system composed of: 1) a haptic simulator leveraging high spatial resolution vision-based tactile sensors for active touching of 3D objects; 2)a mesh-based 3D shape reconstruction model that relies on tactile or visuotactile signals; and 3) a set of data-driven solutions with either tactile or visuotactile priors to guide the shape exploration.
Towards Learning to Play Piano with Dexterous Hands and Touch
The virtuoso plays the piano with passion, poetry and extraordinary technical ability.
PyTouch: A Machine Learning Library for Touch Processing
With the increased availability of rich tactile sensors, there is an equally proportional need for open-source and integrated software capable of efficiently and effectively processing raw touch measurements into high-level signals that can be used for control and decision-making.
MBRL-Lib: A Modular Library for Model-based Reinforcement Learning
MBRL-Lib is designed as a platform for both researchers, to easily develop, debug and compare new algorithms, and non-expert user, to lower the entry-bar of deploying state-of-the-art algorithms.
Model-based Reinforcement Learning
reinforcement-learning
+1
On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
We demonstrate that this problem can be tackled effectively with automated HPO, which we demonstrate to yield significantly improved performance compared to human experts.
Hyperparameter Optimization
Model-based Reinforcement Learning
+2
Model-Invariant State Abstractions for Model-Based Reinforcement Learning
Unlike previous forms of state abstractions, a model-invariance state abstraction leverages causal sparsity over state variables.
Invariant Representations for Reinforcement Learning without Reconstruction
We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction.
Learning Accurate Long-term Dynamics for Model-based Reinforcement Learning
Accurately predicting the dynamics of robotic systems is crucial for model-based control and reinforcement learning.
Model-based Reinforcement Learning
reinforcement-learning
+1
TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
We believe that TACTO is a step towards the widespread adoption of touch sensing in robotic applications, and to enable machine learning practitioners interested in multi-modal learning and control.
Active MR k-space Sampling with Reinforcement Learning
Deep learning approaches have recently shown great promise in accelerating magnetic resonance image (MRI) acquisition.
3D Shape Reconstruction from Vision and Touch
When a toddler is presented a new toy, their instinctual behaviour is to pick it upand inspect it with their hand and eyes in tandem, clearly searching over its surface to properly understand what they are playing with.
Learning Invariant Representations for Reinforcement Learning without Reconstruction
We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction.
Learning to Play Table Tennis From Scratch using Muscular Robots
This work is the first to (a) fail-safe learn of a safety-critical dynamic task using anthropomorphic robot arms, (b) learn a precision-demanding problem with a PAM-driven system despite the control challenges and (c) train robots to play table tennis without real balls.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
Despite decades of research, general purpose in-hand manipulation remains one of the unsolved challenges of robotics.
Plan2Vec: Unsupervised Representation Learning by Latent Plans
In this paper we introduce plan2vec, an unsupervised representation learning approach that is inspired by reinforcement learning.
Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads
Our experiments demonstrate that our online adaptation approach outperforms non-adaptive methods on a series of challenging suspended payload transportation tasks.
Adversarial Continual Learning
We show that shared features are significantly less prone to forgetting and propose a novel hybrid continual learning framework that learns a disjoint representation for task-invariant and task-specific features required to solve a sequence of tasks.
OmniTact: A Multi-Directional High Resolution Touch Sensor
We compare with a state-of-the-art tactile sensor that is only sensitive on one side, as well as a state-of-the-art multi-directional tactile sensor, and find that OmniTact's combination of high-resolution and multi-directional sensing is crucial for reliably inserting the electrical connector and allows for higher accuracy in the state estimation task.
Objective Mismatch in Model-based Reinforcement Learning
In our experiments, we study this objective mismatch issue and demonstrate that the likelihood of one-step ahead predictions is not always correlated with control performance.
Model-based Reinforcement Learning
reinforcement-learning
+1
Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimization
We show empirically that properly addressing these issues significantly improves the efficacy of linear embeddings for BO on a range of problems, including learning a gait policy for robot locomotion.
Data-efficient Co-Adaptation of Morphology and Behaviour with Deep Reinforcement Learning
Key to our approach is the possibility of leveraging previously tested morphologies and behaviors to estimate the performance of new candidate morphologies.
Data-efficient Learning of Morphology and Controller for a Microrobot
Robot design is often a slow and difficult process requiring the iterative construction and testing of prototypes, with the goal of sequentially optimizing the design.
Learning to Control Highly Accelerated Ballistic Movements on Muscular Robots
High-speed and high-acceleration movements are inherently hard to control.
Manipulation by Feel: Touch-Based Control with Deep Predictive Models
Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging.
Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
We propose a novel framing of the problem as multi-modal recognition: the goal of our system is to recognize, given a visual and tactile observation, whether or not these observations correspond to the same object.
Fast Neural Network Verification via Shadow Prices
To use neural networks in safety-critical settings it is paramount to provide assurances on their runtime operation.
Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning
Designing effective low-level robot controllers often entail platform-specific implementations that require manual heuristic parameter tuning, significant system knowledge, or long design times.
Model-based Reinforcement Learning
reinforcement-learning
+1
Unsupervised Exploration with Deep Model-Based Reinforcement Learning
We show that both challenges can be addressed by representing model-uncertainty, which can both guide exploration in the unsupervised phase and ensure that the errors in the model are not exploited by the planner in the goal-directed phase.
Model-based Reinforcement Learning
reinforcement-learning
+1
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Model-based reinforcement learning (RL) algorithms can attain excellent sample efficiency, but often lag behind the best model-free algorithms in terms of asymptotic performance.
Model-based Reinforcement Learning
reinforcement-learning
+1
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
This model -- a deep, multimodal convolutional network -- predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions.
Learning Flexible and Reusable Locomotion Primitives for a Microrobot
This approach formalizes locomotion as a contextual policy search task to collect data, and subsequently uses that data to learn multi-objective locomotion primitives that can be used for planning.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
In this work, we investigate the question of whether touch sensing aids in predicting grasp outcomes within a multimodal sensing framework that combines vision and touch.
MBMF: Model-Based Priors for Model-Free Reinforcement Learning
Reinforcement Learning is divided in two main paradigms: model-free and model-based.
Goal-Driven Dynamics Learning via Bayesian Optimization
Real-world robots are becoming increasingly complex and commonly act in poorly understood environments where it is extremely challenging to model or learn their true dynamics.
Manifold Gaussian Processes for Regression
This feature space is often learned in an unsupervised way, which might lead to data representations that are not useful for the overall regression task.
