Search Results for author:
Found 74 papers, 23 papers with code
From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
Multimodal Large Language Models have made significant strides in integrating visual and textual information, yet they often struggle with effectively aligning these modalities.
Cross-Embodiment Dexterous Grasping with Reinforcement Learning
While recent studies have primarily focused on learning policies for specific robotic hands, the development of a universal policy that controls diverse dexterous hands remains largely unexplored.
Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Grasping
To overcome these challenges, we introduce ResDex, a novel approach that integrates residual policy learning with a mixture-of-experts (MoE) framework.
Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrations
With BiDexHD, scalable learning of numerous bimanual dexterous skills from auto-constructed tasks becomes feasible, offering promising advances toward universal bimanual dexterous manipulation.
Egocentric Vision Language Planning
LMMs excel in planning long-horizon tasks over symbolic abstractions but struggle with grounding in the physical world, often failing to accurately identify object positions in images.
Visual Grounding for Object-Level Generalization in Reinforcement Learning
To approach this goal, we leverage a vision-language model (VLM) for visual grounding and transfer its vision-language knowledge into reinforcement learning (RL) for object-centric tasks, which makes the agent capable of zero-shot generalization to unseen objects and instructions.
NOLO: Navigate Only Look Once
The in-context learning ability of Transformer models has brought new possibilities to visual navigation.
Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learning
Efficient fine-tuning plays a fundamental role in modern large models, with low-rank adaptation emerging as a particularly promising approach.
SMPLX-Lite: A Realistic and Drivable Avatar Benchmark with Rich Geometry and Texture Annotations
We present SMPLX-Lite dataset, the most comprehensive clothing avatar dataset with multi-view RGB sequences, keypoints annotations, textured scanned meshes, and textured SMPLX-Lite-D models.
Cross-Domain Policy Adaptation by Capturing Representation Mismatch
We perform representation learning only in the target domain and measure the representation deviations on the transitions from the source domain, which we show can be a signal of dynamics mismatch.
OccGaussian: 3D Gaussian Splatting for Occluded Human Rendering
OccGaussian initializes 3D Gaussian distributions in the canonical space, and we perform occlusion feature query at occluded regions, the aggregated pixel-align feature is extracted to compensate for the missing information.
MTLight: Efficient Multi-Task Reinforcement Learning for Traffic Signal Control
Traffic signal control has a great impact on alleviating traffic congestion in modern cities.
UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling
Therefore, we propose UV Gaussians, which models the 3D human body by jointly learning mesh deformations and 2D UV-space Gaussian textures.
UniCode: Learning a Unified Codebook for Multimodal Large Language Models
In this paper, we propose \textbf{UniCode}, a novel approach within the domain of multimodal large language models (MLLMs) that learns a unified codebook to efficiently tokenize visual, text, and potentially other types of signals.
Cradle: Empowering Foundation Agents Towards General Computer Control
To handle this issue, we propose the General Computer Control (GCC) setting to restrict foundation agents to interact with software through the most unified and standardized interface, i. e., using screenshots as input and keyboard and mouse actions as output.
RL-GPT: Integrating Reinforcement Learning and Code-as-policy
To seamlessly integrate both modalities, we introduce a two-level hierarchical framework, RL-GPT, comprising a slow agent and a fast agent.
SEABO: A Simple Search-Based Method for Offline Imitation Learning
Offline reinforcement learning (RL) has attracted much attention due to its ability in learning from static offline datasets and eliminating the need of interacting with the environment.
Understanding What Affects Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence
Recently, there are many efforts attempting to learn useful policies for continuous control in visual reinforcement learning (RL).
Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing
The other is how agents can explore in a coordinated way.
Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey
Cooperative multi-agent reinforcement learning is a powerful tool to solve many real-world cooperative tasks, but restrictions of real-world applications may require training the agents in a fully decentralized manner.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Creative Agents: Empowering Agents with Imagination for Creative Tasks
We benchmark creative tasks with the challenging open-world game Minecraft, where the agents are asked to create diverse buildings given free-form language instructions.
Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds
Steve-Eye integrates the LLM with a visual encoder which enables it to process visual-text inputs and generate multimodal feedback.
LLaMA Rider: Spurring Large Language Models to Explore the Open World
Recently, various studies have leveraged Large Language Models (LLMs) to help decision-making and planning in environments, and try to align the LLMs' knowledge with the world conditions.
AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback
Large Language Models (LLMs) have demonstrated significant success across various domains.
Elucidating the solution space of extended reverse-time SDE for diffusion models
In this work, we formulate the sampling process as an extended reverse-time SDE (ER SDE), unifying prior explorations into ODEs and SDEs.
Learning from Visual Observation via Offline Pretrained State-to-Go Transformer
Learning from visual observation (LfVO), aiming at recovering policies from only visual observation data, is promising yet a challenging problem.
Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representation
COREP primarily employs a guided updating mechanism to learn a stable graph representation for the state, termed as causal-origin representation.
Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse
Empirical results show that SMR significantly boosts the sample efficiency of the base methods across most of the evaluated tasks without any hyperparameter tuning or additional tricks.
LLA-FLOW: A Lightweight Local Aggregation on Cost Volume for Optical Flow Estimation
Lack of texture often causes ambiguity in matching, and handling this issue is an important challenge in optical flow estimation.
Skill Reinforcement Learning and Planning for Open-World Long-Horizon Tasks
Our method outperforms baselines by a large margin and is the most sample-efficient demonstration-free RL method to solve Minecraft Tech Tree tasks.
Reinforcement Learning Friendly Vision-Language Model for Minecraft
One of the essential missions in the AI research community is to build an autonomous embodied agent that can achieve high-level performance across a wide spectrum of tasks.
Model-Based Decentralized Policy Optimization
To help the policy improvement be stable and monotonic, we propose model-based decentralized policy optimization (MDPO), which incorporates a latent variable function to help construct the transition and reward function from an individual perspective.
Learning Multi-Object Positional Relationships via Emergent Communication
The study of emergent communication has been dedicated to interactive artificial intelligence.
Best Possible Q-Learning
To tackle this challenge, we propose best possible operator, a novel decentralized operator, and prove that the policies of agents will converge to the optimal joint policy if each agent independently updates its individual state-action value by the operator.
A Survey on Transformers in Reinforcement Learning
Transformer has been considered the dominating neural architecture in NLP and CV, mostly under supervised settings.
ReLeaPS : Reinforcement Learning-based Illumination Planning for Generalized Photometric Stereo
Illumination planning in photometric stereo aims to find a balance between tween surface normal estimation accuracy and image capturing efficiency by selecting optimal light configurations.
Decentralized Policy Optimization
In this paper, we propose \textit{decentralized policy optimization} (DPO), a decentralized actor-critic algorithm with monotonic improvement and convergence guarantee.
Entity Divider with Language Grounding in Multi-Agent Reinforcement Learning
We investigate the use of natural language to drive the generalization of policies in multi-agent settings.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Multi-Agent Automated Machine Learning
In this paper, we propose multi-agent automated machine learning (MA2ML) with the aim to effectively handle joint optimization of modules in automated machine learning (AutoML).
State Advantage Weighting for Offline RL
We present state advantage weighting for offline reinforcement learning (RL).
Multi-Agent Sequential Decision-Making via Communication
Communication helps agents to obtain information about others so that better coordinated behavior can be learned.
More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorization
To this end, we propose multi-agent conditional policy factorization (MACPF), which takes more centralized training but still enables decentralized execution.
MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning
Decentralized learning has shown great promise for cooperative multi-agent reinforcement learning (MARL).
Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning
We study offline meta-reinforcement learning, a practical reinforcement learning paradigm that learns from offline data to adapt to new tasks.
Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning
In this study, we propose the Bimanual Dexterous Hands Benchmark (Bi-DexHands), a simulator that involves two dexterous hands with tens of bimanual manipulation tasks and thousands of target objects.
Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination
Model-based RL methods offer a richer dataset and benefit generalization by generating imaginary trajectories with either trained forward or reverse dynamics model.
Mildly Conservative Q-Learning for Offline Reinforcement Learning
The distribution shift between the learned policy and the behavior policy makes it necessary for the value function to stay conservative such that out-of-distribution (OOD) actions will not be severely overestimated.
Learning to Share in Multi-Agent Reinforcement Learning
Inspired by the fact that sharing plays a key role in human's learning of cooperation, we propose LToS, a hierarchically decentralized MARL framework that enables agents to learn to dynamically share reward with neighbors so as to encourage agents to cooperate on the global objective through collectives.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
APANet: Adaptive Prototypes Alignment Network for Few-Shot Semantic Segmentation
In practice, it can adaptively generate multiple class-agnostic prototypes for query images and learn feature alignment in a self-contrastive manner.
Ranked #48 on
Few-Shot Semantic Segmentation
on COCO-20i (1-shot)
Divergence-Regularized Multi-Agent Actor-Critic
Though divergence regularization has been proposed to settle this problem, it cannot be trivially applied to cooperative multi-agent reinforcement learning (MARL).
Multi-agent Reinforcement Learning
reinforcement-learning
+3
Multi-Agent Language Learning: Symbolic Mapping
We find that symbolic mapping learned in simple referential games can notably promote language learning in difficult tasks.
Sequential Communication in Multi-Agent Reinforcement Learning
Agents determine the priority of decision-making by comparing the value of intention.
Online Tuning for Offline Decentralized Multi-Agent Reinforcement Learning
OTC is simple yet effective to increase data efficiency and improve agent policies in online tuning.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Offline Decentralized Multi-Agent Reinforcement Learning
In this paper, we propose a framework for offline decentralized multi-agent reinforcement learning, which exploits value deviation and transition normalization to deliberately modify the transition probabilities.
Model-Based Opponent Modeling
When one agent interacts with a multi-agent environment, it is challenging to deal with various opponents unseen before.
Metric Policy Representations for Opponent Modeling
In multi-agent reinforcement learning, the inherent non-stationarity of the environment caused by other agents' actions posed significant difficulties for an agent to learn a good policy independently.
SCNet: Enhancing Few-Shot Semantic Segmentation by Self-Contrastive Background Prototypes
To this end, we generate self-contrastive background prototypes directly from the query image, with which we enable the construction of complete sample pairs and thus a complementary and auxiliary segmentation task to achieve the training of a better segmentation model.
Revisiting Prioritized Experience Replay: A Value Perspective
Furthermore, we successfully extend our theoretical framework to maximum-entropy RL by deriving the lower and upper bounds of these value metrics for soft Q-learning, which turn out to be the product of $|\text{TD}|$ and "on-policyness" of the experiences.
A region-based descriptor network for uniformly sampled keypoints
By the new descriptor, we can obtain more high confidence matching points without extremum operation.
MetaVIM: Meta Variationally Intrinsic Motivated Reinforcement Learning for Decentralized Traffic Signal Control
To make the policy learned from a training scenario generalizable to new unseen scenarios, a novel Meta Variationally Intrinsic Motivated (MetaVIM) RL method is proposed to learn the decentralized policy for each intersection that considers neighbor information in a latent way.
Adaptive Learning Rates for Multi-Agent Reinforcement Learning
In multi-agent reinforcement learning (MARL), the learning rates of actors and critic are mostly hand-tuned and fixed.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
The Emergence of Individuality in Multi-Agent Reinforcement Learning
EOI learns a probabilistic classifier that predicts a probability distribution over agents given their observation and gives each agent an intrinsic reward of being correctly predicted by the classifier.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Learning Individually Inferred Communication for Multi-Agent Cooperation
Empirically, we show that I2C can not only reduce communication overhead but also improve the performance in a variety of multi-agent cooperative scenarios, comparing to existing methods.
The Emergence of Individuality
EOI learns a probabilistic classifier that predicts a probability distribution over agents given their observation and gives each agent an intrinsic reward of being correctly predicted by the classifier.
Noise-Sampling Cross Entropy Loss: Improving Disparity Regression Via Cost Volume Aware Regularizer
Recent end-to-end deep neural networks for disparity regression have achieved the state-of-the-art performance.
XSepConv: Extremely Separated Convolution
Depthwise convolution has gradually become an indispensable operation for modern efficient neural networks and larger kernel sizes ($\ge5$) have been applied to it recently.
Learning Fairness in Multi-Agent Systems
Fairness is essential for human society, contributing to stability and productivity.
Towards More Accurate Automatic Sleep Staging via Deep Transfer Learning
We employ the Montreal Archive of Sleep Studies (MASS) database consisting of 200 subjects as the source domain and study deep transfer learning on three different target domains: the Sleep Cassette subset and the Sleep Telemetry subset of the Sleep-EDF Expanded database, and the Surrey-cEEGrid database.
Ranked #1 on
Multimodal Sleep Stage Detection
on Surrey-PSG
Automatic Sleep Stage Classification
Multimodal Sleep Stage Detection
+2
Generative Exploration and Exploitation
Sparse reward is one of the biggest challenges in reinforcement learning (RL).
Graph Convolutional Reinforcement Learning
The key is to understand the mutual interplay between agents.
Optical Flow Super-Resolution Based on Image Guidence Using Convolutional Neural Network
With the motivation of various convolutional neural network(CNN) structures succeeded in single image super-resolution(SISR) task, an end-to-end convolutional neural network is proposed to reconstruct the high resolution(HR) optical flow field from initial LR optical flow with the guidence of the first frame used in optical flow estimation.
Learning Attentional Communication for Multi-Agent Cooperation
Our model leads to efficient and effective communication for large-scale multi-agent cooperation.
Modeling the Resource Requirements of Convolutional Neural Networks on Mobile Devices
In doing so Augur tackles several challenges: (i) how to overcome pro ling and measurement overhead; (ii) how to capture the variance in different mobile platforms with different processors, memory, and cache sizes; and (iii) how to account for the variance in the number, type and size of layers of the different CNN configurations.
