Search Results for author:
Found 184 papers, 72 papers with code
GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation
Additionally, we provide a comprehensive analysis of 17 baselines, which includes statistical models, deep learning models, and foundation models.
Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Frequency-level specialization is too coarse-grained to capture this level of diversity.
SFR-RAG: Towards Contextually Faithful LLMs
Retrieval Augmented Generation (RAG), a paradigm that integrates external contextual information with large language models (LLMs) to enhance factual accuracy and relevance, has emerged as a pivotal area in generative AI.
xLAM: A Family of Large Action Models to Empower AI Agent Systems
By releasing the xLAM series, we aim to advance the performance of open-source LLMs for autonomous AI agents, potentially accelerating progress and democratizing access to high-performance models for agent tasks.
xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations
We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions.
xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs.
Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents
For instance, a group of open-source SWE agents, with a maximum individual resolve rate of 27. 3% on SWE-Bench Lite, can achieve a 34. 3% resolve rate with DEI, making a 25% improvement and beating most closed-source solutions.
Enabling High Data Throughput Reinforcement Learning on GPUs: A Domain Agnostic Framework for Data-Driven Scientific Research
We introduce WarpSci, a domain agnostic framework designed to overcome crucial system bottlenecks encountered in the application of reinforcement learning to intricate environments with vast datasets featuring high-dimensional observation or action spaces.
Shared Imagination: LLMs Hallucinate Alike
In this paper, we propose a novel setting, imaginary question answering (IQA), to better understand model similarity.
APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets
The advancement of function-calling agent models requires diverse, reliable, and high-quality datasets.
INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulness
In this work, we introduce INDICT: a new framework that empowers LLMs with Internal Dialogues of Critiques for both safety and helpfulness guidance.
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
Multimodal interleaved datasets featuring free-form interleaved sequences of images and text are crucial for training frontier large multimodal models (LMMs).
MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases
The deployment of Large Language Models (LLMs) and Large Multimodal Models (LMMs) on mobile devices has gained significant attention due to the benefits of enhanced privacy, stability, and personalization.
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity.
BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation
We present BEHAVIOR-1K, a comprehensive simulation benchmark for human-centered robotics.
AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning
It meticulously standardizes and unifies these trajectories into a consistent format, streamlining the creation of a generic data loader optimized for agent training.
AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent System
Thus, we open-source a new AI agent library, AgentLite, which simplifies this process by offering a lightweight, user-friendly platform for innovating LLM agent reasoning, architectures, and applications with ease.
Unified Training of Universal Time Series Forecasting Transformers
Deep learning for time series forecasting has traditionally operated within a one-model-per-dataset framework, limiting its potential to leverage the game-changing impact of large pre-trained models.
Ranked #12 on
Time Series Forecasting
on ETTh1 (336) Multivariate
Causal Layering via Conditional Entropy
Under appropriate assumptions and conditioning, we can separate the sources or sinks from the remainder of the nodes by comparing their conditional entropy to the unconditional entropy of their noise.
Editing Arbitrary Propositions in LLMs without Subject Labels
On datasets of binary propositions derived from the CounterFact dataset, we show that our method -- without access to subject labels -- performs close to state-of-the-art L\&E methods which has access subject labels.
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning
To enable this framework, we devise a scalable pipeline that automatically generates high-quality, instruction-tuning datasets from readily available captioning data across different modalities, and contribute 24K QA data for audio and 250K QA data for 3D.
Nothing Stands Still: A Spatiotemporal Benchmark on 3D Point Cloud Registration Under Large Geometric and Temporal Change
To this end, we introduce the Nothing Stands Still (NSS) benchmark, which focuses on the spatiotemporal registration of 3D scenes undergoing large spatial and temporal change, ultimately creating one coherent spatiotemporal map.
How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations
Through extensive probing and a new pasting experiment, we further reveal several mechanisms within the trained transformers, such as concrete copying behaviors on both the inputs and the representations, linear ICL capability of the upper layers alone, and a post-ICL representation selection mechanism in a harder mixture setting.
XGen-7B Technical Report
Most open-source LLMs, on the other hand, are limited in their ability to support longer sequence lengths, which is a key requirement for many tasks that require inference over an input context.
Enhancing Performance on Seen and Unseen Dialogue Scenarios using Retrieval-Augmented End-to-End Task-Oriented System
End-to-end task-oriented dialogue (TOD) systems have achieved promising performance by leveraging sophisticated natural language understanding and natural language generation capabilities of pre-trained models.
BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents
The massive successes of large language models (LLMs) encourage the emerging exploration of LLM-augmented Autonomous Agents (LAAs).
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
This demonstrates that using policy gradient optimization to improve language agents, for which we believe our work is one of the first, seems promising and can be applied to optimize other models in the agent architecture to enhance agent performances over time.
DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI
Despite advancements in conversational AI, language models encounter challenges to handle diverse conversational tasks, and existing dialogue dataset collections often lack diversity and comprehensiveness.
REX: Rapid Exploration and eXploitation for AI Agents
In this paper, we propose an enhanced approach for Rapid Exploration and eXploitation for AI Agents called REX.
Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear
We introduce Sonicverse, a multisensory simulation platform with integrated audio-visual simulation for training household agents that can both see and hear.
Modeling Dynamic Environments with Scene Graph Memory
We evaluate our method in the Dynamic House Simulator, a new benchmark that creates diverse dynamic graphs following the semantic patterns typically seen at homes, and show that NEP can be trained to predict the locations of objects in a variety of environments with diverse object movement dynamics, outperforming baselines both in terms of new scene adaptability and overall accuracy.
UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild
Visual generative foundation models such as Stable Diffusion show promise in navigating these goals, especially when prompted with arbitrary languages.
ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding
It achieves a new SOTA of 50. 6% (top-1) on Objaverse-LVIS and 84. 7% (top-1) on ModelNet40 in zero-shot classification.
Ranked #9 on
3D Point Cloud Classification
on ScanObjectNN
(using extra training data)
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages
In this study, we attempt to render the training of LLMs for program synthesis more efficient by unifying four key components: (1) model architectures, (2) learning methods, (3) infill sampling, and, (4) data distributions.
An Extensible Multimodal Multi-task Object Dataset with Materials
For example, we can train a model to predict the object category from the listing text, or the mass and price from the product listing image.
Procedure-Aware Pretraining for Instructional Video Understanding
This graph can then be used to generate pseudo labels to train a video representation that encodes the procedural knowledge in a more accessible form to generalize to multiple procedure understanding tasks.
HIVE: Harnessing Human Feedback for Instructional Visual Editing
Incorporating human feedback has been shown to be crucial to align text generated by large language models to human preferences.
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models.
Ranked #1 on
Image-to-Text Retrieval
on MS COCO
ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets.
Ranked #3 on
Training-free 3D Point Cloud Classification
on ModelNet40
(using extra training data)
Best-$k$ Search Algorithm for Neural Text Generation
We first investigate the vanilla best-first search (BFS) algorithm and then propose the Best-$k$ Search algorithm.
Online Distribution Shift Detection via Recency Prediction
When deploying modern machine learning-enabled robotic systems in high-stakes applications, detecting distribution shift is critical.
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training
Visual question answering (VQA) is a hallmark of vision and language reasoning and a challenging task under the zero-shot setting.
Ranked #2 on
Visual Question Answering (VQA)
on VQA v2 val
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
LAVIS: A Library for Language-Vision Intelligence
We introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications.
Minkowski Tracker: A Sparse Spatio-Temporal R-CNN for Joint Object Detection and Tracking
In this work, we present Minkowski Tracker, a sparse spatio-temporal R-CNN that jointly solves object detection and tracking.
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
To address the limitations, we propose "CodeRL", a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning (RL).
Ranked #7 on
Code Generation
on APPS
Masked Unsupervised Self-training for Label-free Image Classification
We demonstrate the efficacy of MUST on a variety of downstream tasks, where it improves upon CLIP by a large margin.
OmniXAI: A Library for Explainable AI
We introduce OmniXAI (short for Omni eXplainable AI), an open-source Python library of eXplainable AI (XAI), which offers omni-way explainable AI capabilities and various interpretable machine learning techniques to address the pain points of understanding and interpreting the decisions made by machine learning (ML) in practice.
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
To democratize this, we train and release a family of large language models up to 16. 1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER.
Ranked #96 on
Code Generation
on HumanEval
Long Document Summarization with Top-down and Bottom-up Inference
Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents.
Ranked #1 on
Text Summarization
on Pubmed
ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation
Manipulating volumetric deformable objects in the real world, like plush toys and pizza dough, bring substantial challenges due to infinite shape variations, non-rigid motions, and partial observability.
Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation
Doing this is challenging for two reasons: on the data side, current interfaces make collecting high-quality human demonstrations difficult, and on the learning side, policies trained on limited data can suffer from covariate shift when deployed.
Long Document Summarization with Top-Down and Bottom-Up Representation Inference
Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents.
Sample-Efficient Safety Assurances using Conformal Prediction
When deploying machine learning models in high-stakes robotics applications, the ability to detect unsafe situations is crucial.
Merlion: A Machine Learning Library for Time Series
We introduce Merlion, an open-source machine learning library for time series.
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
However, goal images also have a number of drawbacks: they are inconvenient for humans to provide, they can over-specify the desired behavior leading to a sparse reward signal, or under-specify task information in the case of non-goal reaching tasks.
Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration
Our method co-optimizes a human policy and a robot policy in an interactive learning process: the human policy learns to generate diverse and plausible collaborative behaviors from demonstrations while the robot policy learns to assist by estimating the unobserved latent strategy of its human collaborator.
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation.
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
We evaluate the new capabilities of iGibson 2. 0 to enable robot learning of novel tasks, in the hope of demonstrating the potential of this new simulator to support new research in embodied AI.
BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation, spanning a range of everyday household chores such as cleaning, maintenance, and food preparation.
Discovering Generalizable Skills via Automated Generation of Diverse Tasks
To encourage generalizable skills to emerge, our method trains each skill to specialize in the paired task and maximizes the diversity of the generated tasks.
JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection
However, learning to recognise human actions and their social interactions in an unconstrained real-world environment comprising numerous people, with potentially highly unbalanced and long-tailed distributed action labels from a stream of sensory data captured from a mobile robot platform remains a significant challenge, not least owing to the lack of a reflective large-scale dataset.
TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild
Joint forecasting of human trajectory and pose dynamics is a fundamental building block of various applications ranging from robotics and autonomous driving to surveillance systems.
LASER: Learning a Latent Action Space for Efficient Reinforcement Learning
Additionally, similar tasks or instances of the same task family impose latent manifold constraints on the most effective action space: the task family can be best solved with actions in a manifold of the entire action space of the robot.
Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control
Key to such capability is hand-eye coordination, a cognitive ability that enables humans to adaptively direct their movements at task-relevant objects and be invariant to the objects' absolute spatial location.
Local Calibration: Metrics and Recalibration
In this work, we propose the local calibration error (LCE) to span the gap between average and individual reliability.
Embodied Intelligence via Learning and Evolution
However, the principles governing relations between environmental complexity, evolved morphology, and the learnability of intelligent control, remain elusive, partially due to the substantial challenge of performing large-scale in silico experiments on evolution and learning.
Learning Multi-Arm Manipulation Through Collaborative Teleoperation
To address these challenges, we present Multi-Arm RoboTurk (MART), a multi-user data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms and collect demonstrations for multi-arm tasks.
Human-in-the-Loop Imitation Learning using Remote Teleoperation
We develop a simple and effective algorithm to train the policy iteratively on new data collected by the system that encourages the policy to learn how to traverse bottlenecks through the interventions.
Topological Planning with Transformers for Vision-and-Language Navigation
Conventional approaches to vision-and-language navigation (VLN) are trained end-to-end but struggle to perform well in freely traversable environments.
Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Search
We propose to use a 3D scene graph representation to capture the hierarchical, semantic, and geometric aspects of this problem.
IGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
We present iGibson 1. 0, a novel simulation environment to develop robotic solutions for interactive tasks in large-scale realistic scenes.
Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
Vision-based robotics often separates the control loop into one module for perception and a separate module for control.
Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learning
Navigating fluently around pedestrians is a necessary capability for mobile robots deployed in human environments, such as buildings and homes.
Privacy Preserving Recalibration under Domain Shift
In an extensive empirical study, we find that our algorithm improves calibration on domain-shift benchmarks under the constraints of differential privacy.
ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
To validate our method, we apply ReLMoGen to two types of tasks: 1) Interactive Navigation tasks, navigation problems where interactions with the environment are required to reach the destination, and 2) Mobile Manipulation tasks, manipulation tasks that require moving the robot base.
Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter
When searching for objects in cluttered environments, it is often necessary to perform complex interactions in order to move occluding objects out of the way and fully reveal the object of interest and make it graspable.
How Trustworthy are Performance Evaluations for Basic Vision Tasks?
This paper examines performance evaluation criteria for basic vision tasks involving sets of objects namely, object detection, instance-level segmentation and multi-object tracking.
Goal-Aware Prediction: Learning to Model What Matters
In this paper, we propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space, resulting in a learning objective that more closely matches the downstream task.
Adaptive Procedural Task Generation for Hard-Exploration Problems
To enable curriculum learning in the absence of a direct indicator of learning progress, we propose to train the task generator by balancing the agent's performance in the generated tasks and the similarity to the target tasks.
Generative Sparse Detection Networks for 3D Single-shot Object Detection
3D object detection has been widely studied due to its potential applicability to many promising areas such as robotics and augmented reality.
Ranked #7 on
3D Object Detection
on S3DIS
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
In the second stage of GTI, we collect a small set of rollouts from the unconditioned stochastic policy of the first stage, and train a goal-directed agent to generalize to novel start and goal configurations.
JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset
In this work we present JRMOT, a novel 3D MOT system that integrates information from RGB images and 3D point clouds to achieve real-time, state-of-the-art tracking performance.
Ranked #10 on
Multiple Object Tracking
on KITTI Tracking test
Learning to Navigate Using Mid-Level Visual Priors
How much does having visual priors about the world (e. g. the fact that the world is 3D) assist in learning to perform downstream motor tasks (e. g. navigating a complex environment)?
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data
For simple short-horizon manipulation tasks with modest variation in task instances, offline learning from a small set of demonstrations can produce controllers that successfully solve the task.
Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity
We evaluate the quality of our platform, the diversity of demonstrations in our dataset, and the utility of our dataset via quantitative and qualitative analysis.
Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments
We present Interactive Gibson Benchmark, the first comprehensive benchmark for training and evaluating Interactive Navigation: robot navigation strategies where physical interaction with objects is allowed and even encouraged to accomplish a task.
Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
The fundamental challenge of planning for multi-step manipulation is to find effective and plausible action sequences that lead to the task goal.
KETO: Learning Keypoint Representations for Tool Manipulation
For this purpose, we present KETO, a framework of learning keypoint representations of tool-based manipulation.
Robotics
JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments
We present JRDB, a novel egocentric dataset collected from our social mobile manipulator JackRabbot.
HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators
Different from other HRL solutions, HRL4IN handles the heterogeneous nature of the Interactive Navigation task by creating subgoals in different spaces in different phases of the task.
Hierarchical Reinforcement Learning
reinforcement-learning
+2
6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints
We present 6-PACK, a deep learning approach to category-level 6D object pose tracking on RGB-D data.
Ranked #1 on
6D Pose Estimation using RGBD
on REAL275
(Rerr metric)
3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera
Given a 3D mesh and registered panoramic images, we construct a graph that spans the entire building and includes semantics on objects (e. g., class, material, and other attributes), rooms (e. g., scene category, volume, etc.)
Causal Induction from Visual Observations for Goal Directed Tasks
Causal reasoning has been an indispensable capability for humans and other intelligent animals to interact with the physical world.
Regression Planning Networks
Recent learning-to-plan methods have shown promising results on planning directly from observation space.
SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning
We present an overview of SURREAL-System, a reproducible, flexible, and scalable framework for distributed reinforcement learning (RL).
AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
The exploration mechanism used by a Deep Reinforcement Learning (RL) agent plays a key role in determining its sample efficiency.
Situational Fusion of Visual Representation for Visual Navigation
A complex visual navigation task puts an agent in different situations which call for a diverse range of visual perception abilities.
Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning
The key technical challenge is that the symbol grounding is prone to error with limited training data and leads to subsequent symbolic planning failures.
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback.
Improving Social Awareness Through DANTE: A Deep Affinity Network for Clustering Conversational Interactants
We propose a data-driven approach to detect conversational groups by identifying spatial arrangements typical of these focused social encounters.
Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks
This problem is compounded by the presence of social interactions between humans and their physical interactions with the scene.
Ranked #17 on
Trajectory Prediction
on ETH/UCY
Time-Varying Interaction Estimation Using Ensemble Methods
Directed information (DI) is a useful tool to explore time-directed interactions in multivariate data.
Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks
This paper studies the effect of different action spaces in deep RL and advocates for Variable Impedance Control in End-effector Space (VICES) as an advantageous action space for constrained and contact-rich tasks.
Which Tasks Should Be Learned Together in Multi-task Learning?
Many computer vision applications require solving multiple tasks in real-time.
Deep Local Trajectory Replanning and Control for Robot Navigation
We present a navigation system that combines ideas from hierarchical planning and machine learning.
4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks
To overcome challenges in the 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and the trilateral-stationary conditional random field that enforces spatio-temporal consistency in the 7D space-time-chroma space.
Ranked #1 on
Robust 3D Semantic Segmentation
on WOD-C
4D Spatio Temporal Semantic Segmentation
Robust 3D Semantic Segmentation
Machine Vision for Natural Gas Methane Emissions Detection Using an Infrared Camera
We find that the detection accuracy can reach as high as 99%, the overall detection accuracy can exceed 95% for a case across all leak sizes and imaging distances.
Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
Many robotic applications require the agent to perform long-horizon tasks in partially observable environments.
Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter
In this paper, we formalize Mechanical Search and study a version where distractor objects are heaped over the target object in a bin.
Robotics
A Behavioral Approach to Visual Navigation with Graph Localization Networks
Inspired by research in psychology, we introduce a behavioral approach for visual navigation using topological maps.
Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression
By incorporating this generalized $IoU$ ($GIoU$) as a loss into the state-of-the art object detection frameworks, we show a consistent improvement on their performance using both the standard, $IoU$ based, and new, $GIoU$ based, performance measures on popular object detection benchmarks such as PASCAL VOC and MS COCO.
DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion
A key technical challenge in performing 6D object pose estimation from RGB-D image is to fully leverage the two complementary data sources.
Ranked #4 on
6D Pose Estimation
on LineMOD
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies
This skill set (hereafter mid-level perception) provides the policy with a more processed state of the world compared to raw images.
Coupled Recurrent Network (CRN)
To learn from these heterogenous input sources, existing methods reply on two-stream architectural designs that contain independent, parallel streams of Recurrent Neural Networks (RNNs).
Action Recognition In Videos
Multi-Person Pose Estimation
+2
RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
Imitation Learning has empowered recent advances in learning robotic manipulation tasks by addressing shortcomings of Reinforcement Learning such as exploration and reward specification.
Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback.
Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation
We propose an end-to-end deep learning model for translating free-form natural language instructions to a high-level plan for behavioral robot navigation.
Gibson Env: Real-World Perception for Embodied Agents
Developing visual perception models for active agents and sensorimotor control are cumbersome to be done in the physical world, as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly.
Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model.
Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision
We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering.
VUNet: Dynamic Scene View Synthesis for Traversability Estimation using an RGB Camera
We present VUNet, a novel view(VU) synthesis method for mobile robots in dynamic environments, and its application to the estimation of future traversability.
SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints
Whereas, the social attention component aggregates information across the different agent interactions and extracts the most important trajectory information from the surrounding neighbors.
Ranked #4 on
Trajectory Prediction
on Stanford Drone
(ADE (8/12) @K=5 metric)
Demo2Vec: Reasoning Object Affordances From Online Videos
Watching expert demonstrations is an important way for humans and robots to reason about affordances of unseen objects.
Ranked #2 on
Video-to-image Affordance Grounding
on OPRA (28x28)
Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation ( <=50%) in the form of an RGB-D image.
Generalizing to Unseen Domains via Adversarial Data Augmentation
Only using training data from a single source distribution, we propose an iterative procedure that augments the dataset with examples from a fictitious target domain that is "hard" under the current model.
Deep Learning under Privileged Information Using Heteroscedastic Dropout
This is what the Learning Under Privileged Information (LUPI) paradigm endeavors to model by utilizing extra knowledge only available during training.
Taskonomy: Disentangling Task Transfer Learning
The product is a computational taxonomic map for task transfer learning.
Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks
Understanding human motion behavior is critical for autonomous moving platforms (like self-driving cars and social robots) if they are to navigate human-centric environments.
Ranked #4 on
Trajectory Prediction
on ETH
Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings
To this end, we first learn joint embeddings of freeform text descriptions and colored 3D shapes.
GONet: A Semi-Supervised Deep Learning Approach For Traversability Estimation
We present semi-supervised deep learning approaches for traversability estimation from fisheye images.
Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation (<= 50%) in the form of an RGB-D image.
CAR-Net: Clairvoyant Attentive Recurrent Network
We exploit two sources of information: the past motion trajectory of the agent of interest and a wide top-view image of the navigation scene.
Adversarial Feature Augmentation for Unsupervised Domain Adaptation
Recent works showed that Generative Adversarial Networks (GANs) can be successfully applied in unsupervised domain adaptation, where, given a labeled source dataset and an unlabeled target dataset, the goal is to train powerful classifiers for the target samples.
Recurrent Autoregressive Networks for Online Multi-Object Tracking
The external memory explicitly stores previous inputs of each trajectory in a time window, while the internal memory learns to summarize long-term tracking history and associate detections by processing the external memory.
Generic 3D Representation via Pose Estimation and Matching
Though a large body of computer vision research has investigated developing generic semantic representations, efforts towards developing a similar representation for 3D has been limited.
SEGCloud: Semantic Segmentation of 3D Point Clouds
Coarse voxel predictions from a 3D Fully Convolutional NN are transferred back to the raw 3D points via trilinear interpolation.
Ranked #13 on
Semantic Segmentation
on Semantic3D
Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
We introduce a large-scale 3D shape understanding benchmark using data and annotation from ShapeNet 3D object database.
Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation
Learning-based approaches to robotic manipulation are limited by the scalability of data collection and accessibility of labels.
Neural Task Programming: Learning to Generalize Across Hierarchical Tasks
In this work, we propose a novel robot learning framework called Neural Task Programming (NTP), which bridges the idea of few-shot learning from demonstration and neural program induction.
To Go or Not To Go? A Near Unsupervised Learning Approach For Robot Navigation
It is important for robots to be able to decide whether they can go through a space or not, as they navigate through a dynamic environment.
Lattice Long Short-Term Memory for Human Action Recognition
This method effectively enhances the ability to model dynamics across time and addresses the non-stationary issue of long-term motion dynamics without significantly increasing the model complexity.
