Search Results for author:
Found 103 papers, 57 papers with code
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
We examine the capability of Multimodal Large Language Models (MLLMs) to tackle diverse domains that extend beyond the traditional language and vision tasks these models are typically trained on.
Grounding Descriptions in Images informs Zero-Shot Visual Recognition
In this paper, we propose GRAIN, a new pretraining strategy aimed at aligning representations at both fine and coarse levels simultaneously.
Adversarial Attacks Using Differentiable Rendering: A Survey
Differentiable rendering methods have emerged as a promising means for generating photo-realistic and physically plausible adversarial attacks by manipulating 3D objects and scenes that can deceive deep neural networks (DNNs).
Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models
This exploration is beneficial for finding good loss basins when training from scratch.
Neural Fields in Robotics: A Survey
Neural Fields have emerged as a transformative approach for 3D scene representation in computer vision and robotics, enabling accurate inference of geometry, 3D semantics, and dynamics from posed 2D data.
ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AI
Intelligent embodied agents need to quickly adapt to new scenarios by integrating long histories of experience into decision-making.
Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge
In order to develop robots that can effectively serve as versatile and capable home assistants, it is crucial for them to reliably perceive and interact with a wide variety of objects across diverse environments.
Reinforcement Learning via Auxiliary Task Distillation
We present Reinforcement Learning via Auxiliary Task Distillation (AuxDistill), a new method that enables reinforcement learning (RL) to perform long-horizon robot control problems by distilling behaviors from auxiliary RL tasks.
ICE-G: Image Conditional Editing of 3D Gaussian Splats
Our framework enables a wide variety of editing tasks such as manual local edits, correspondence based style transfer from any example image, and a combination of different styles from multiple example images.
Grounding Multimodal Large Language Models in Actions
For discrete actions, we demonstrate that semantically aligning these actions with the native output token space of the MLLM leads to the strongest performance.
Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control
This has led to the emergence of pre-trained vision-language models as a tool for transferring representations learned from internet-scale data to downstream tasks and new domains.
Adaptive Memory Replay for Continual Learning
This continual learning (CL) phenomenon has been extensively studied, but primarily in a setting where only a small amount of past data can be stored.
GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation
The Embodied AI community has made significant strides in visual navigation tasks, exploring targets from 3D coordinates, objects, language descriptions, and images.
NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fields
Given the capabilities of neural fields in densely representing a 3D scene from 2D images, we ask the question: Can we scale their self-supervised pretraining, specifically using masked autoencoders, to generate effective 3D representations from posed RGB images.
Seeing the Unseen: Visual Common Sense for Semantic Placement
Datasets for image description are typically constructed by curating relevant images and asking humans to annotate the contents of the image; neither of those two steps are straightforward for objects not present in the image.
Diffuse Attend and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion
The proposed method does not require any training or language dependency to extract quality segmentation for any images.
Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis
Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models (i. e., foundation models) in research fields such as Natural Language Processing (NLP) and Computer Vision (CV), we devote this survey to exploring (i) how these existing foundation models from NLP and CV can be applied to the field of general-purpose robotics, and also exploring (ii) what a robotics-specific foundation model would look like.
Continual Diffusion with STAMINA: STack-And-Mask INcremental Adapters
We show that STAMINA outperforms the prior SOTA for the setting of text-to-image continual customization on a 50-concept benchmark composed of landmarks and human faces, with no stored replay data.
ConstraintMatch for Semi-constrained Clustering
Constrained clustering allows the training of classification models using pairwise constraints only, which are weak and relatively easy to mine, while still yielding full-supervision-level model performance.
DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets
Construction of a universal detector poses a crucial question: How can we most effectively train a model on a large mixture of datasets?
FSD: Fast Self-Supervised Single RGB-D to Categorical 3D Objects
In this work, we address the challenging task of 3D object recognition without the reliance on real-world 3D labeled data.
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
We present Habitat 3. 0: a simulation platform for studying collaborative human-robot tasks in home environments.
Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models
The generative process of Diffusion Models (DMs) has recently set state-of-the-art on many AI generation benchmarks.
LatentDR: Improving Model Generalization Through Sample-Aware Latent Degradation and Restoration
To address this challenge, we propose a novel approach for distribution-aware latent augmentation that leverages the relationships across samples to guide the augmentation procedure.
NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes
NeO 360's representation allows us to learn from a large collection of unbounded 3D scenes while offering generalizability to new views and novel scenes from as few as a single image during inference.
Ranked #1 on
Generalizable Novel View Synthesis
on NERDS 360
Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion
The proposed method does not require any training or language dependency to extract quality segmentation for any images.
Ranked #1 on
Semantic Segmentation
on COCO-Stuff-27
HomeRobot: Open-Vocabulary Mobile Manipulation
HomeRobot (noun): An affordable compliant robot that navigates homes and manipulates a wide range of objects in order to complete everyday tasks.
Continual Adaptation of Vision Transformers for Federated Learning
In this paper, we attempt to tackle forgetting and heterogeneity while minimizing overhead costs and without requiring access to any stored data.
Adaptive Coordination in Social Embodied Rearrangement
We present the task of "Social Rearrangement", consisting of cooperative everyday tasks like setting up the dinner table, tidying a house or unpacking groceries in a simulated multi-agent environment.
HAAV: Hierarchical Aggregation of Augmented Views for Image Captioning
The image captioning model encodes each view independently with a shared encoder efficiently, and a contrastive loss is incorporated across the encoded views in a novel way to improve their representation quality and the model's data efficiency.
Training Energy-Based Normalizing Flow with Score-Matching Objectives
In this paper, we establish a connection between the parameterization of flow-based and energy-based generative models, and present a new flow-based modeling approach called energy-based normalizing flow (EBFlow).
CLIP-GCD: Simple Language Guided Generalized Category Discovery
Generalized Category Discovery (GCD) requires a model to both classify known categories and cluster unknown categories in unlabeled data.
Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation
We create the first benchmark for semi-supervised multi-modal semantic segmentation and also report the robustness to missing modalities.
Ranked #1 on
Semantic Segmentation
on SUN-RGBD
(Mean IoU (test) metric, using extra
training data)
RGBD Semantic Segmentation
Robust Semi-Supervised RGBD Semantic Segmentation
+3
Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA
We show that C-LoRA not only outperforms several baselines for our proposed setting of text-to-image continual customization, which we refer to as Continual Diffusion, but that we achieve a new state-of-the-art in the well-established rehearsal-free continual learning setting for image classification.
BC-IRL: Learning Generalizable Reward Functions from Demonstrations
How well do reward functions learned with inverse reinforcement learning (IRL) generalize?
Trainable Projected Gradient Method for Robust Fine-tuning
To solve this problem, we propose Trainable Projected Gradient Method (TPGM) to automatically learn the constraint imposed for each layer for a fine-grained fine-tuning regularization.
OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNav
We present a single neural network architecture composed of task-agnostic components (ViTs, convolutions, and LSTMs) that achieves state-of-art results on both the ImageNav ("go to location in <this picture>") and ObjectNav ("find a chair") tasks without any task-specific modules like object detection, segmentation, mapping, or planning modules.
Communication-Critical Planning via Multi-Agent Trajectory Exchange
This paper addresses the task of joint multi-agent perception and planning, especially as it relates to the real-world challenge of collision-free navigation for connected self-driving vehicles.
System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Games
In this paper, we introduce the Lifelong Reinforcement Learning Components Framework (L2RLCF), which standardizes L2RL systems and assimilates different continual learning components (each addressing different aspects of the lifelong learning problem) into a unified system.
CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning
Our experiments show that we outperform the current SOTA method DualPrompt on established benchmarks by as much as 4. 5% in average final accuracy.
Structure-Encoding Auxiliary Tasks for Improved Visual Representation in Vision-and-Language Navigation
In Vision-and-Language Navigation (VLN), researchers typically take an image encoder pre-trained on ImageNet without fine-tuning on the environments that the agent will be trained or tested on.
ConStruct-VL: Data-Free Continual Structured VL Concepts Learning
This leads to reasoning mistakes, which need to be corrected as they occur by teaching VL models the missing SVLC skills; often this must be done using private data where the issue was found, which naturally leads to a data-free continual (no task-id) VL learning setting.
Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Tasks
Specifically, Polyhistor achieves competitive accuracy compared to the state-of-the-art while only using ~10% of their trainable parameters.
FedFOR: Stateless Heterogeneous Federated Learning with First-Order Regularization
For the more typical applications of FL where the number of clients is large (e. g., edge-device and mobile applications), these methods cannot be applied, motivating the need for a stateless approach to heterogeneous FL which can be used for any number of clients.
On the Surprising Effectiveness of Transformers in Low-Labeled Video Recognition
Our work empirically explores the low data regime for video classification and discovers that, surprisingly, transformers perform extremely well in the low-labeled video setting compared to CNNs.
Open-Set Semi-Supervised Object Detection
To address this problem, we consider online and offline OOD detection modules, which are integrated with SSOD methods.
ShAPO: Implicit Representations for Multi-Object Shape, Appearance, and Pose Optimization
A novel disentangled shape and appearance database of priors is first learned to embed objects in their respective shape and appearance space.
3D Shape Reconstruction From A Single 2D Image
6D Pose Estimation
+4
Unbiased Teacher v2: Semi-supervised Object Detection for Anchor-free and Anchor-based Detectors
In this paper, we present Unbiased Teacher v2, which shows the generalization of SS-OD method to anchor-free detectors and also introduces Listen2Student mechanism for the unsupervised regression loss.
Lifelong Wandering: A realistic few-shot online continual learning setting
Online few-shot learning describes a setting where models are trained and evaluated on a stream of data while learning emerging classes.
Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning
A key limitation of such methods, however, is that the output of the model is conditioned only on the object detector's outputs.
Ranked #12 on
Image Captioning
on COCO Captions
A Closer Look at Rehearsal-Free Continual Learning
Next, we explore how to leverage knowledge from a pre-trained model in rehearsal-free continual learning and find that vanilla L2 parameter regularization outperforms EWC parameter regularization and feature distillation.
A Closer Look at Knowledge Distillation with Features, Logits, and Gradients
Knowledge distillation (KD) is a substantial strategy for transferring learned knowledge from one neural network model to another.
CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation
This paper studies the complex task of simultaneous multi-object 3D reconstruction, 6D pose and size estimation from a single-view RGB-D observation.
Ranked #1 on
6D Pose Estimation using RGBD
on CAMERA25
Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
We are the first to propose a method that works well across both OOD detection and calibration and under different types of shifts.
A Geometric Perspective towards Neural Calibration via Sensitivity Decomposition
It is well known that vision classification models suffer from poor calibration in the face of data distribution shifts.
Safe Reinforcement Learning Using Robust Control Barrier Functions
Reinforcement Learning (RL) has been shown to be effective in many scenarios.
Model-based Reinforcement Learning
reinforcement-learning
+3
Exploring Covariate and Concept Shift for Detection and Confidence Calibration of Out-of-Distribution Data
To this end, we theoretically derive two score functions for OOD detection, the covariate shift score and concept shift score, based on the decomposition of KL-divergence for both scores, and propose a geometrically-inspired method (Geometric ODIN) to improve OOD detection under both shifts with only in-distribution data.
CrossMatch: Improving Semi-Supervised Object Detection via Multi-Scale Consistency
Inspired by the fact that teacher/student pseudo-labeling approaches result in a weak and sparse gradient signal due to the difficulty of confidence-thresholding, CrossMatch leverages \textit{multi-scale feature extraction} in object detection.
Overcoming Obstructions via Bandwidth-Limited Multi-Agent Spatial Handshaking
In this paper, we address bandwidth-limited and obstruction-prone collaborative perception, specifically in the context of multi-agent semantic segmentation.
Enhancing Multi-Robot Perception via Learned Data Association
In this paper, we address the multi-robot collaborative perception problem, specifically in the context of multi-view infilling for distributed semantic segmentation.
Habitat 2.0: Training Home Assistants to Rearrange their Habitat
We introduce Habitat 2. 0 (H2. 0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios.
Striking the Right Balance: Recall Loss for Semantic Segmentation
There are two major drawbacks to these methods: 1) constantly up-weighting minority classes can introduce excessive false positives in semantic segmentation; 2) a minority class is not necessarily a hard class.
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning
Modern computer vision applications suffer from catastrophic forgetting when incrementally learning new concepts over time.
Ranked #5 on
Class Incremental Learning
on cifar100
LRGNet: Learnable Region Growing for Class-Agnostic Point Cloud Segmentation
3D point cloud segmentation is an important function that helps robots understand the layout of their surrounding environment and perform tasks such as grasping objects, avoiding obstacles, and finding landmarks.
Unbiased Teacher for Semi-Supervised Object Detection
To address this, we introduce Unbiased Teacher, a simple yet effective approach that jointly trains a student and a gradually progressing teacher in a mutually-beneficial manner.
Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer
Our work investigates whether we can significantly reduce this memory budget by leveraging unlabeled data from an agent's environment in a realistic and challenging continual learning paradigm.
Recall Loss for Imbalanced Image Classification and Semantic Segmentation
Many works have proposed to weigh the standard cross entropy loss function with pre-computed weights based on class statistics such as the number of samples and class margins.
Posterior Re-calibration for Imbalanced Datasets
Neural Networks can perform poorly when the training label distribution is heavily imbalanced, as well as when the testing data differs from the training distribution.
Ranked #31 on
Long-tail Learning
on CIFAR-100-LT (ρ=10)
3D for Free: Crossmodal Transfer Learning using HD Maps
In this work, we address the long-tail problem by leveraging both the large class-taxonomies of modern 2D datasets and the robustness of state-of-the-art 2D detection methods.
FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning
Recent state-of-the-art semi-supervised learning (SSL) methods use a combination of image-based transformations and consistency regularization as core components.
Frustratingly Simple Domain Generalization via Image Stylization
Convolutional Neural Networks (CNNs) show impressive performance in the standard classification setting where training and testing data are drawn i. i. d.
Ranked #57 on
Domain Generalization
on PACS
When2com: Multi-Agent Perception via Communication Graph Grouping
While significant advances have been made for single-agent perception, many applications require multiple sensing agents and cross-agent communication due to benefits such as coverage and robustness.
Who2com: Collaborative Perception via Learnable Handshake Communication
In this paper, we propose the problem of collaborative perception, where robots can combine their local observations with those of neighboring agents in a learnable way to improve accuracy on a perception task.
Multi-agent Reinforcement Learning
Reinforcement Learning
+2
Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation
Despite the recent progress of fully-supervised action segmentation techniques, the performance is still not fully satisfactory.
Ranked #13 on
Action Segmentation
on GTEA
Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
Deep neural networks have attained remarkable performance when applied to data that comes from the same distribution as that of the training set, but can significantly degrade otherwise.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation
Specifically, we analyze a number of uncertainty measures, each of which captures a different aspect of uncertainty, and we propose a novel way to fuse degraded inputs by scaling modality-specific output softmax probabilities.
Temporal Attentive Alignment for Large-Scale Video Domain Adaptation
Finally, we propose Temporal Attentive Adversarial Adaptation Network (TA3N), which explicitly attends to the temporal dynamics using domain discrepancy for more effective domain alignment, achieving state-of-the-art performance on four video DA datasets (e. g. 7. 9% accuracy gain over "Source only" from 73. 9% to 81. 8% on "HMDB --> UCF", and 10. 3% gain on "Kinetics --> Gameplay").
Manifold Graph with Learned Prototypes for Semi-Supervised Image Classification
We then show that when combined with these regularizers, the proposed method facilitates the propagation of information from generated prototypes to image data to further improve results.
Learning to Generate Grounded Visual Captions without Localization Supervision
When automatically generating a sentence description for an image or video, it often remains unclear how well the generated caption is grounded, that is whether the model uses the correct image regions to output particular words, or if the model is hallucinating based on priors in the dataset and/or the language model.
Leveraging Semantics for Incremental Learning in Multi-Relational Embeddings
Service robots benefit from encoding information in semantically meaningful ways to enable more robust task execution.
Path Ranking with Attention to Type Hierarchies
The objective of the knowledge base completion problem is to infer missing information from existing facts in a knowledge base.
Temporal Attentive Alignment for Video Domain Adaptation
Finally, we propose Temporal Attentive Adversarial Adaptation Network (TA3N), which explicitly attends to the temporal dynamics using domain discrepancy for more effective domain alignment, achieving state-of-the-art performance on three video DA datasets.
Ranked #1 on
Domain Adaptation
on UCF-to-Olympic
A Closer Look at Few-shot Classification
Few-shot classification aims to learn a classifier to recognize unseen classes during training with limited labeled examples.
Sub-Task Discovery with Limited Supervision: A Constrained Clustering Approach
Hierarchical reinforcement learning captures sub-task information to learn modular policies that can be quickly adapted to new tasks.
Constrained Clustering
Hierarchical Reinforcement Learning
+3
Unsupervised Continual Learning and Self-Taught Associative Memory Hierarchies
We first pose the Unsupervised Continual Learning (UCL) problem: learning salient representations from a non-stationary stream of unlabeled data in which the number of object classes varies with time.
RoboCSE: Robot Common Sense Embedding
Autonomous service robots require computational frameworks that allow them to generalize knowledge to new situations in a manner that models uncertainty while scaling to real-world problem sizes.
The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation
As deep learning continues to make progress for challenging perception tasks, there is increased interest in combining vision, language, and decision-making.
Ranked #115 on
Vision and Language Navigation
on VLN Challenge
The Regretful Navigation Agent for Vision-and-Language Navigation
As deep learning continues to make progress for challenging perception tasks, there is increased interest in combining vision, language, and decision-making.
Multi-view Incremental Segmentation of 3D Point Clouds for Mobile Robots
Mobile robots need to create high-definition 3D maps of the environment for applications such as remote surveillance and infrastructure mapping.
Robotics
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments.
Ranked #115 on
Vision and Language Navigation
on VLN Challenge
Natural Language Visual Grounding
Vision and Language Navigation
+2
Multi-class Classification without Multi-class Labels
This work presents a new strategy for multi-class classification that requires no class-specific labels, but instead leverages pairwise similarity between examples, which is a weaker form of annotation.
Data-Efficient Graph Embedding Learning for PCB Component Detection
This paper presents a challenging computer vision task, namely the detection of generic components on a PCB, and a novel set of deep-learning methods that are able to jointly leverage the appearance of individual components and the propagation of information across the structure of the board to accurately detect and identify various types of components on a PCB.
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
Continual learning has received a great deal of attention recently with several approaches being proposed.
A probabilistic constrained clustering for transfer learning and image category discovery
The proposed objective directly minimizes the negative log-likelihood of cluster assignment with respect to the pairwise constraints, has no hyper-parameters, and demonstrates improved scalability and performance on both supervised learning and unsupervised transfer learning.
Ranked #1 on
Ecg Risk Stratification
on ngm
Learning to Cluster for Proposal-Free Instance Segmentation
We utilize the most fundamental property of instance labeling -- the pairwise relationship between pixels -- as the supervision to formulate the learning objective, then apply it to train a fully convolutional network (FCN) for learning to perform pixel-wise clustering.
Ranked #14 on
Lane Detection
on TuSimple
Learning to cluster in order to transfer across domains and tasks
The key insight is that, in addition to features, we can transfer similarity information and this is sufficient to learn a similarity function and clustering network to perform both domain adaptation and cross-task transfer learning.
Attend and Interact: Higher-Order Object Interactions for Video Understanding
Human actions often involve complex interactions across several inter-related objects in the scene.
Grounded Objects and Interactions for Video Captioning
We address the problem of video captioning by grounding language generation on object interactions in the video.
On Convergence and Stability of GANs
We propose studying GAN training dynamics as regret minimization, which is in contrast to the popular view that there is consistent minimization of a divergence between real and generated distributions.
TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition
We demonstrate that using both RNNs (using LSTMs) and Temporal-ConvNets on spatiotemporal feature matrices are able to exploit spatiotemporal dynamics to improve the overall performance.
Ranked #56 on
Action Recognition
on HMDB-51
Deep Image Category Discovery using a Transferred Similarity Function
We propose that this network can be learned with contrastive loss which is only based on weak binary pair-wise constraints.
A Continuous Optimization Approach for Efficient and Accurate Scene Flow
We propose a continuous optimization method for solving dense 3D scene flow problems from stereo imagery.
Neural network-based clustering using pairwise constraints
Robustness analysis also shows that the method is largely insensitive to the number of clusters.
