Search Results for author:
Found 46 papers, 15 papers with code
Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks
To achieve consistent region representation across spatio-temporal dimensions, we introduce Token Mark, a set of tokens highlighting the target regions within the visual feature space.
NVILA: Efficient Frontier Visual Language Models
This paper introduces NVILA, a family of open VLMs designed to optimize both efficiency and accuracy.
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies.
ARDuP: Active Region Video Diffusion for Universal Policies
Sequential decision-making can be formulated as a text-conditioned video generation problem, where a video planner, guided by a text-defined goal, generates future frames visualizing planned actions, from which control actions are subsequently derived.
X-VILA: Cross-Modality Alignment for Large Language Model
We introduce X-VILA, an omni-modality model designed to extend the capabilities of large language models (LLMs) by incorporating image, video, and audio modalities.
What is Point Supervision Worth in Video Instance Segmentation?
Video instance segmentation (VIS) is a challenging vision task that aims to detect, segment, and track objects in videos.
LITA: Language Instructed Temporal-Localization Assistant
In addition to leveraging existing video datasets with timestamps, we propose a new task, Reasoning Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for learning and evaluating this task.
Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition
To tackle this issue, we propose content-motion latent diffusion model (CMD), a novel efficient extension of pretrained image diffusion models for video generation.
T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching
Sampling from diffusion probabilistic models (DPMs) is often expensive for high-quality image generation and typically requires many steps with a large model.
Deep Multimodal Fusion for Surgical Feedback Classification
This work offers an important first look at the feasibility of automated classification of real-world live surgical feedback based on text, audio, and video modalities.
Eureka: Human-Level Reward Design via Coding Large Language Models
The generality of Eureka also enables a new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF), readily incorporating human inputs to improve the quality and the safety of the generated rewards without model updating.
Differentially Private Video Activity Recognition
In recent years, differential privacy has seen significant advancements in image classification; however, its application to video activity recognition remains under-explored.
PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees
In this paper, we propose PerAda, a parameter-efficient pFL framework that reduces communication and computational costs and exhibits superior generalization performance, especially under test-time distribution shifts.
I$^2$SB: Image-to-Image Schrödinger Bridge
We propose Image-to-Image Schr\"odinger Bridge (I$^2$SB), a new class of conditional diffusion models that directly learn the nonlinear diffusion processes between two given distributions.
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
Augmenting pretrained language models (LMs) with a vision encoder (e. g., Flamingo) has obtained the state-of-the-art results in image-to-text generation.
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data.
Identifying Auxiliary or Adversarial Tasks Using Necessary Condition Analysis for Adversarial Multi-task Video Understanding
In this work, we propose a generalized notion of multi-task learning by incorporating both auxiliary tasks that the model should perform well on and adversarial tasks that the model should not perform well on.
MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training
By only training a query-based image instance segmentation model, MinVIS outperforms the previous best result on the challenging Occluded VIS dataset by over 10% AP.
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Autonomous agents have made great strides in specialist domains like Atari games and Go.
Pre-Trained Language Models for Interactive Decision-Making
Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.
Auditing AI models for Verified Deployment under Semantic Specifications
We enable such unit tests through variations in a semantically-interpretable latent space of a generative model.
SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
A student network then learns to mimic the expert policy by supervised learning with strong augmentations, making its representation more robust against visual variations compared to the expert.
Transferable Unsupervised Robust Representation Learning
We upend this view and show that URRL improves both the natural accuracy of unsupervised representation learning and its robustness to corruptions and adversarial noise.
Spatio-Temporal Graph for Video Captioning with Knowledge Distillation
In this paper, we propose a novel spatio-temporal graph model for video captioning that exploits object interactions in space and time.
Motion Reasoning for Goal-Based Imitation Learning
We further show that by using the automatically inferred goal from the video demonstration, our robot is able to reproduce the same task in a real kitchen environment.
Regression Planning Networks
Recent learning-to-plan methods have shown promising results on planning directly from observation space.
Imitation Learning for Human Pose Prediction
Modeling and prediction of human motion dynamics has long been a challenging problem in computer vision, and most existing methods rely on the end-to-end supervised training of various architectures of recurrent neural networks.
Ranked #2 on
Human Pose Forecasting
on Human3.6M
(MAR, walking, 1,000ms metric)
Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning
The key technical challenge is that the symbol grounding is prone to error with limited training data and leads to subsequent symbolic planning failures.
Procedure Planning in Instructional Videos
In this paper, we study the problem of procedure planning in instructional videos, which can be seen as a step towards enabling autonomous agents to plan for complex tasks in everyday settings such as cooking.
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation
The key technical challenge for discriminative modeling with weak supervision is that the loss function of the ordering supervision is usually formulated using dynamic programming and is thus not differentiable.
Action-Agnostic Human Pose Forecasting
In this paper, we propose a new action-agnostic method for short- and long-term human pose forecasting.
Ranked #5 on
Human Pose Forecasting
on Human3.6M
(MAR, walking, 1,000ms metric)
Dynamic Task Prioritization for Multitask Learning
We propose dynamic task prioritization for multitask learning.
Temporal Modular Networks for Retrieving Complex Compositional Activities in Videos
A major challenge in computer vision is scaling activity understanding to the long tail of complex activities without requiring collecting large quantities of data for new actions.
Neural Graph Matching Networks for Fewshot 3D Action Recognition
We propose Neural Graph Matching (NGM) Networks, a novel framework that can learn to recognize a previous unseen 3D action class with only a few examples.
Ranked #1 on
Skeleton Based Action Recognition
on CAD-120
Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model.
Learning to Decompose and Disentangle Representations for Video Prediction
Our goal is to predict future video frames given a sequence of input frames.
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos
In this work, we propose to tackle this new task with a weakly-supervised framework for reference-aware visual grounding in instructional videos, where only the temporal alignment between the transcription and the video segment are available for supervision.
What Makes a Video a Video: Analyzing Temporal Information in Video Understanding Models and Datasets
The ability to capture temporal information has been critical to the development of video understanding models.
Visual Forecasting by Imitating Dynamics in Natural Sequences
This allows us to apply IRL at scale and directly imitate the dynamics in high-dimensional continuous visual sequences from the raw pixel values.
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
We propose an unsupervised method for reference resolution in instructional videos, where the goal is to temporally link an entity (e. g., "dressing") to the action (e. g., "mix yogurt") that produced it.
Unsupervised Learning of Long-Term Motion Dynamics for Videos
We present an unsupervised representation learning approach that compactly encodes the motion dependencies in videos.
Connectionist Temporal Modeling for Weakly Supervised Action Labeling
We propose a weakly-supervised framework for action labeling in video, where only the order of occurring actions is required during training time.
Forecasting Interactive Dynamics of Pedestrians with Fictitious Play
We develop predictive models of pedestrian dynamics by encoding the coupled nature of multi-pedestrian interaction using game theory, and deep learning-based visual analysis to estimate person-specific behavior parameters.
Robust Classification by Pre-conditioned LASSO and Transductive Diffusion Component Analysis
Collecting datasets in this way, however, requires robust and efficient ways for detecting and excluding outliers that are common and prevalent.
How Do We Use Our Hands? Discovering a Diverse Set of Common Grasps
Furthermore, we develop a hierarchical extension to the DPP clustering algorithm and show that it can be used to discover appearance-based grasp taxonomies.
