Search Results for author:
Found 47 papers, 5 papers with code
Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs
The high-level policy consists of a long-context VLM that takes the demonstration tour video and the multimodal user instruction as input to find the goal frame in the tour video.
Modeling the Real World with High-Density Visual Particle Dynamics
We present High-Density Visual Particle Dynamics (HD-VPD), a learned world model that can emulate the physical dynamics of real scenes by processing massive latent point clouds containing 100K+ particles.
Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning
Parameter-efficient fine-tuning (PEFT) approaches have emerged as a viable alternative by allowing us to fine-tune models by updating only a small number of parameters.
SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention
We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment.
Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models
In this work, we revisit the choice of energy-based models (EBM) as a policy class.
Robotic Table Tennis: A Case Study into a High Speed Learning System
We present a deep-dive into a real-world robotic learning system that, in previous work, was shown to be capable of hundreds of table tennis rallies with a human and has the ability to precisely return the ball to desired targets.
Barkour: Benchmarking Animal-level Agility with Quadruped Robots
In the second approach, we distill the specialist skills into a Transformer-based generalist locomotion policy, named Locomotion-Transformer, that can handle various terrains and adjust the robot's gait based on the perceived environment and robot states.
Safely Learning Dynamical Systems
For $T = \infty$, we provide SDP-representable inner approximations of the set of safe initial conditions and show that one trajectory generically suffices for safe learning.
A Contextual Bandit Approach for Learning to Plan in Environments with Probabilistic Goal Configurations
Object-goal navigation (Object-nav) entails searching, recognizing and navigating to a target object.
Robotic Table Wiping via Reinforcement Learning and Whole-body Trajectory Optimization
This problem is challenging, as it requires planning wiping actions while reasoning over uncertain latent dynamics of crumbs and spills captured via high-dimensional visual observations.
Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation
Despite decades of research, existing navigation systems still face real-world challenges when deployed in the wild, e. g., in cluttered home environments or in human-occupied public spaces.
Implicit Two-Tower Policies
We present a new class of structured reinforcement learning policy-architectures, Implicit Two-Tower (ITT) policies, where the actions are chosen based on the attention scores of their learnable latent representations with those of the input states.
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Large pretrained (e. g., "foundation") models exhibit distinct capabilities depending on the domain of data they are trained on.
Ranked #21 on
Video Retrieval
on MSR-VTT-1kA
(video-to-text R@1 metric)
Multiscale Sensor Fusion and Continuous Control with Neural CDEs
Though robot learning is often formulated in terms of discrete-time Markov decision processes (MDPs), physical robots require near-continuous multiscale feedback control.
Hybrid Random Features
We propose a new class of random feature methods for linearizing softmax and Gaussian kernels called hybrid random features (HRFs) that automatically adapt the quality of kernel estimation to provide most accurate approximation in the defined regions of interest.
Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks
Goals cannot be as easily specified as rigid object poses, and may involve complex relative spatial relations such as "place the item inside the bag".
Ode to an ODE
We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the orthogonal group O(d).
Safely Learning Dynamical Systems from Short Trajectories
For our first two results, we consider the setting of safely learning linear dynamics.
Learning Stability Certificates from Data
Many existing tools in nonlinear control theory for establishing stability or safety of a dynamical system can be distilled to the construction of a certificate function that guarantees a desired property.
An Ode to an ODE
We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the orthogonal group O(d).
Time Dependence in Non-Autonomous Neural ODEs
Neural Ordinary Differential Equations (ODEs) are elegant reinterpretations of deep networks where continuous time can replace the discrete notion of depth, ODE solvers perform forward propagation, and the adjoint method enables efficient, constant memory backpropagation.
Robotic Table Tennis with Model-Free Reinforcement Learning
We propose a model-free algorithm for learning efficient policies capable of returning table tennis balls by controlling robot joints at a rate of 100Hz.
Stochastic Flows and Geometric Optimization on the Orthogonal Group
We present a new class of stochastic, geometrically-driven optimization algorithms on the orthogonal group $O(d)$ and naturally reductive homogeneous manifolds obtained from the action of the rotation group $SO(d)$.
Policies Modulating Trajectory Generators
We propose an architecture for learning complex controllable behaviors by having simple Policies Modulate Trajectory Generators (PMTG), a powerful combination that can provide both memory and prior knowledge to the controller.
Learning Stabilizable Nonlinear Dynamics with Contraction-Based Regularization
We propose a novel framework for learning stabilizable nonlinear dynamical systems for continuous control tasks in robotics.
Data Efficient Reinforcement Learning for Legged Robots
We present a model-based framework for robot locomotion that achieves walking based on only 4. 5 minutes (45, 000 control steps) of data collected on a quadruped robot.
Provably Robust Blackbox Optimization for Reinforcement Learning
Interest in derivative-free optimization (DFO) and "evolutionary strategies" (ES) has recently surged in the Reinforcement Learning (RL) community, with growing evidence that they can match state of the art methods for policy optimization problems in Robotics.
Learning Stabilizable Dynamical Systems via Control Contraction Metrics
We propose a novel framework for learning stabilizable nonlinear dynamical systems for continuous control tasks in robotics.
Learning Contracting Vector Fields For Stable Imitation Learning
We propose a new non-parametric framework for learning incrementally stable dynamical systems x' = f(x) from a set of sampled trajectories.
Structured Evolution with Compact Architectures for Scalable Policy Optimization
We present a new method of blackbox optimization via gradient approximation with the use of structured random orthogonal matrices, providing more accurate estimators than baselines and with provable theoretical guarantees.
On Blackbox Backpropagation and Jacobian Sensing
From a small number of calls to a given “blackbox" on random input perturbations, we show how to efficiently recover its unknown Jacobian, or estimate the left action of its Jacobian on a given vector.
Manifold Regularization for Kernelized LSTD
Policy evaluation or value function or Q-function approximation is a key procedure in reinforcement learning (RL).
Geometry of 3D Environments and Sum of Squares Polynomials
Motivated by applications in robotics and computer vision, we study problems related to spatial reasoning of a 3D environment using sublevel sets of polynomials.
Hierarchically Compositional Kernels for Scalable Nonparametric Learning
We propose a novel class of kernels to alleviate the high computational cost of large-scale nonparametric learning with kernel methods.
Recycling Randomness with Structure for Sublinear time Kernel Expansions
We propose a scheme for recycling Gaussian random vectors into structured matrices to approximate various kernel functions in sublinear time via random embeddings.
Learning Compact Recurrent Neural Networks
Recurrent neural networks (RNNs), including long short-term memory (LSTM) RNNs, have produced state-of-the-art results on a variety of speech recognition tasks.
Structured Transforms for Small-Footprint Deep Learning
We consider the task of building compact deep learning pipelines suitable for deployment on storage and power constrained mobile devices.
Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernels
These approximate feature maps arise as Monte Carlo approximations to integral representations of shift-invariant kernel functions (e. g., Gaussian kernel).
Learning Machines Implemented on Non-Deterministic Hardware
This paper highlights new opportunities for designing large-scale machine learning systems as a consequence of blurring traditional boundaries that have allowed algorithm designers and application-level practitioners to stay -- for the most part -- oblivious to the details of the underlying hardware-level implementations.
High-performance Kernel Machines with Implicit Distributed Optimization and Randomization
In order to fully utilize "big data", it is often required to use "big models".
Scalable Matrix-valued Kernel Learning for High-dimensional Nonlinear Multivariate Regression and Granger Causality
We propose a general matrix-valued multiple kernel learning framework for high-dimensional nonlinear multivariate regression problems.
Random Laplace Feature Maps for Semigroup Kernels on Histograms
With the goal of accelerating the training and testing complexity of nonlinear kernel methods, several recent papers have proposed explicit embeddings of the input data into low-dimensional feature spaces, where fast linear methods can instead be used to generate approximate solutions.
Near-separable Non-negative Matrix Factorization with $\ell_1$- and Bregman Loss Functions
Recently, a family of tractable NMF algorithms have been proposed under the assumption that the data matrix satisfies a separability condition Donoho & Stodden (2003); Arora et al. (2012).
Sketching Structured Matrices for Faster Nonlinear Regression
Motivated by the desire to extend fast randomized techniques to nonlinear $l_p$ regression, we consider a class of structured regression problems.
Non-parametric Group Orthogonal Matching Pursuit for Sparse Learning with Multiple Kernels
We consider regularized risk minimization in a large dictionary of Reproducing kernel Hilbert Spaces (RKHSs) over which the target function has a sparse representation.
Block Variable Selection in Multivariate Regression and High-dimensional Causal Inference
We consider multivariate regression problems involving high-dimensional predictor and response spaces.
Regularized Co-Clustering with Dual Supervision
By attempting to simultaneously partition both the rows (examples) and columns (features) of a data matrix, Co-clustering algorithms often demonstrate surpris- ingly impressive performance improvements over traditional one-sided (row) clustering techniques.
