Search Results for author:
Found 64 papers, 19 papers with code
Learning Predictive Visuomotor Coordination
Understanding and predicting human visuomotor coordination is crucial for applications in robotics, human-computer interaction, and assistive technologies.
A Generalist Hanabi Agent
Traditional multi-agent reinforcement learning (MARL) systems can develop cooperative strategies through repeated interactions.
X-LeBench: A Benchmark for Extremely Long Egocentric Video Understanding
Leveraging the advanced text processing capabilities of large language models (LLMs), X-LeBench develops a life-logging simulation pipeline that produces realistic, coherent daily plans aligned with real-world video data.
Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMs
Long-form video understanding with Large Vision Language Models is challenged by the need to analyze temporally dispersed yet spatially concentrated key moments within limited context windows.
Can Large Language Models Adapt to Other Agents In-Context?
As the research community aims to build better AI assistants that are more dynamic and personalized to the diversity of humans that they interact with, there is increased interest in evaluating the theory of mind capabilities of large language models (LLMs).
Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation
We also introduce a relation regularization method to further disentangle image transformation features from irrelevant contents in exemplar images.
Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction
For Scenario II, based on the reduction strategy from G-Search, we design a parametric sigmoid function (P-Sigmoid) to guide the reduction at each layer of the MLLM, whose parameters are optimized by Bayesian Optimization.
Evaluating the Prompt Steerability of Large Language Models
Achieving this requires first being able to evaluate the degree to which a given model is capable of reflecting various personas.
OpenLS-DGF: An Adaptive Open-Source Dataset Generation Framework for Machine Learning Tasks in Logic Synthesis
This paper introduces OpenLS-DGF, an adaptive logic synthesis dataset generation framework, to enhance machine learning~(ML) applications within the logic synthesis process.
Human-inspired Perspectives: A Survey on AI Long-term Memory
With the rapid advancement of AI systems, their abilities to store, retrieve, and utilize information over the long term - referred to as long-term memory - have become increasingly significant.
Human Action Anticipation: A Survey
Predicting future human behavior is an increasingly popular topic in computer vision, driven by the interest in applications such as autonomous vehicles, digital assistants and human-robot interactions.
Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs
The growing demand for AI training data has transformed data annotation into a global industry, but traditional approaches relying on human annotators are often time-consuming, labor-intensive, and prone to inconsistent quality.
The Llama 3 Herd of Models
This paper presents a new set of foundation models, called Llama 3.
Ranked #3 on
Question Answering
on PeerQA
SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning
This paper studies the transfer reinforcement learning (RL) problem where multiple RL problems have different reward functions but share the same underlying transition dynamics.
BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation
We introduce the BEHAVIOR Vision Suite (BVS), a set of tools and assets to generate fully customized synthetic data for systematic evaluation of computer vision models, based on the newly developed embodied AI benchmark, BEHAVIOR-1K.
Contextual Moral Value Alignment Through Context-Based Aggregation
Developing value-aligned AI agents is a complex undertaking and an ongoing challenge in the field of AI.
A Neuro-Symbolic Approach to Multi-Agent RL for Interpretability and Probabilistic Decision Making
To enable decision-making under uncertainty and partial observability, we developed a novel probabilistic neuro-symbolic framework, Probabilistic Logical Neural Networks (PLNN), which combines the capabilities of logical reasoning with probabilistic graphical models.
Animated Stickers: Bringing Stickers to Life with Video Diffusion
Our model is built on top of the state-of-the-art Emu text-to-image model, with the addition of temporal layers to model motion.
Context-Former: Stitching via Latent Conditioned Sequence Modeling
Offline reinforcement learning (RL) algorithms can learn better decision-making compared to behavior policies by stitching the suboptimal trajectories to derive more optimal ones.
The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective
We propose a unified multi-modal framework -- Audio-Visual Conversational Attention (AV-CONV), for the joint prediction of conversation behaviors -- speaking and listening -- for both the camera wearer as well as all other social partners present in the egocentric video.
LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning
Additionally, existing diffusion-based image manipulation models are sub-optimal in controlling the state transition of an action in egocentric image pixel space because of the domain gap.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
On the Convergence and Sample Complexity Analysis of Deep Q-Networks with $ε$-Greedy Exploration
This paper provides the first theoretical convergence and sample complexity analysis of the practical setting of DQNs with $\epsilon$-greedy policy.
MatChat: A Large Language Model and Application Service Platform for Materials Science
The prediction of chemical synthesis pathways plays a pivotal role in materials science research.
Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi
In this work, we show empirically that state-of-the-art ZSC algorithms have poor performance when paired with agents trained with different learning methods, and they require millions of interaction samples to adapt to these new partners.
Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation
Motivated by this observation, we introduce the first model that leverages both the video and audio modalities for egocentric gaze anticipation.
Learning in Factored Domains with Information-Constrained Visual Representations
This raises the question of how humans learn to efficiently represent visual information in a manner useful for learning tasks.
Joint Edge-Model Sparse Learning is Provably Efficient for Graph Neural Networks
Due to the significant computational challenge of training large-scale graph neural networks (GNNs), various sparse learning techniques have been exploited to reduce memory and storage costs.
Werewolf Among Us: A Multimodal Dataset for Modeling Persuasion Behaviors in Social Deduction Games
We also explore the generalization ability of language models for persuasion modeling and the role of persuasion strategies in predicting social deduction game outcomes.
Game-Theoretical Perspectives on Active Equilibria: A Preferred Solution Concept over Nash Equilibria
By directly comparing active equilibria to Nash equilibria in these examples, we find that active equilibria find more effective solutions than Nash equilibria, concluding that an active equilibrium is the desired solution for multiagent learning settings.
Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approach
Machine learning problems with multiple objective functions appear either in learning with multiple criteria where learning has to make a trade-off between multiple performance metrics such as fairness, safety and accuracy; or, in multi-task learning where multiple tasks are optimized jointly, sharing inductive bias between them.
In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation
To this end, we design the transformer encoder to embed the global context as one additional visual token and further propose a novel Global-Local Correlation (GLC) module to explicitly model the correlation of the global token and each local token.
Generative Adversarial Network for Future Hand Segmentation from Egocentric Video
We introduce the novel problem of anticipating a time series of future hand masks from egocentric video.
Influencing Long-Term Behavior in Multiagent Reinforcement Learning
An effective approach that has recently emerged for addressing this non-stationarity is for each agent to anticipate the learning of other agents and influence the evolution of future policies towards desirable behavior for its own benefit.
Hierarchical Reinforcement Learning with AI Planning Models
Our approach defines options in hierarchical reinforcement learning (HRL) from AIP operators by establishing a correspondence between the state transition model of AI planning problem and the abstract state transition system of a Markov Decision Process (MDP).
Local Explanations for Reinforcement Learning
Many works in explainable AI have focused on explaining black-box classification models.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Learning Multi-Objective Curricula for Robotic Policy Learning
Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of deep reinforcement learning (DRL).
Interpreting Reinforcement Policies through Local Behaviors
Many works in explainable AI have focused on explaining black-box classification models.
Context-Specific Representation Abstraction for Deep Option Learning
Hierarchical reinforcement learning has focused on discovering temporally extended actions, such as options, that can provide benefits in problems requiring extensive exploration.
Egocentric Activity Recognition and Localization on a 3D Map
Given a video captured from a first person perspective and the environment context of where the video is recorded, can we recognize what the person is doing and identify where the action occurs in the 3D space?
4D Human Body Capture from Egocentric Video via 3D Scene Grounding
We introduce a novel task of reconstructing a time series of second-person 3D human body meshes from monocular egocentric videos.
Consolidation via Policy Information Regularization in Deep RL for Multi-Agent Games
This paper introduces an information-theoretic constraint on learned policy complexity in the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) reinforcement learning algorithm.
A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
A fundamental challenge in multiagent reinforcement learning is to learn beneficial behaviors in a shared environment with other simultaneously learning agents.
Deep RL With Information Constrained Policies: Generalization in Continuous Control
We focus on the model-free reinforcement learning (RL) setting and formalize our approach in terms of an information-theoretic constraint on the complexity of learned policies.
In the Eye of the Beholder: Gaze and Actions in First Person Video
Moving beyond the dataset, we propose a novel deep model for joint gaze estimation and action recognition in FPV.
Coagent Networks Revisited
Coagent networks formalize the concept of arbitrary networks of stochastic agents that collaborate to take actions in a reinforcement learning environment.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
On the Role of Weight Sharing During Deep Option Learning
In this work we note that while this key assumption of the policy gradient theorems of option-critic holds in the tabular case, it is always violated in practice for the deep function approximation setting.
Forecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video
Motivated by this, we adopt intentional hand movement as a future representation and propose a novel deep network that jointly models and predicts the egocentric hand motion, interaction hotspots and future action.
CAPACITY-LIMITED REINFORCEMENT LEARNING: APPLICATIONS IN DEEP ACTOR-CRITIC METHODS FOR CONTINUOUS CONTROL
We formalize this type of bounded rationality in terms of an information-theoretic constraint on the complexity of policies that agents seek to learn.
Attention Distillation for Learning Video Representations
To this end, we make use of attention modules that learn to highlight regions in the video and aggregate features for recognition.
Ranked #42 on
Action Recognition
on UCF101
Learning Hierarchical Teaching Policies for Cooperative Agents
Collective learning can be greatly enhanced when agents effectively exchange knowledge with their peers.
Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
In this work we propose a new conceptualization of the continual learning problem in terms of a temporally symmetric trade-off between transfer and interference that can be optimized by enforcing gradient alignment across examples.
Learning Abstract Options
Building systems that autonomously create temporal abstractions from data is a key challenge in scaling learning and planning in reinforcement learning.
In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video
We address the task of jointly determining what a person is doing and where they are looking based on the analysis of video captured by a headworn camera.
Learning to Teach in Cooperative Multiagent Reinforcement Learning
The problem of teaching to improve agent learning has been investigated by prior works, but these approaches make assumptions that prevent application of teaching to general multiagent problems, or require domain expertise for problems they can apply to.
Faster Reinforcement Learning with Expert State Sequences
In this paper, we consider a realistic and more difficult sce- nario where a reinforcement learning agent only has access to the state sequences of an expert, while the expert actions are not available.
The Eigenoption-Critic Framework
Eigenoptions (EOs) have been recently introduced as a promising idea for generating a diverse set of options through the graph Laplacian, having been shown to allow efficient exploration.
Efficient Exploration
Hierarchical Reinforcement Learning
+2
Eigenoption Discovery through the Deep Successor Representation
Options in reinforcement learning allow agents to hierarchically decompose a task into subtasks, having the potential to speed up learning and planning.
Learning for Multi-robot Cooperation in Partially Observable Stochastic Environments with Macro-actions
We implement two variants of multi-robot Search and Rescue (SAR) domains (with and without obstacles) on hardware to demonstrate the learned policies can effectively control a team of distributed robots to cooperate in a partially observable stochastic environment.
Socially Aware Motion Planning with Deep Reinforcement Learning
For robotic vehicles to navigate safely and efficiently in pedestrian-rich environments, it is important to model subtle human behaviors and navigation rules (e. g., passing on the right).
Decentralized Non-communicating Multiagent Collision Avoidance with Deep Reinforcement Learning
Finding feasible, collision-free paths for multiagent systems can be challenging, particularly in non-communicating scenarios where each agent's intent (e. g. goal) is unobservable to the others.
Multiagent Systems
Stick-Breaking Policy Learning in Dec-POMDPs
Expectation maximization (EM) has recently been shown to be an efficient algorithm for learning finite-state controllers (FSCs) in large decentralized POMDPs (Dec-POMDPs).
Dynamic Clustering via Asymptotics of the Dependent Dirichlet Process Mixture
This paper presents a novel algorithm, based upon the dependent Dirichlet process mixture model (DDPMM), for clustering batch-sequential data containing an unknown number of evolving clusters.
