Search Results for author:
Found 84 papers, 30 papers with code
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition
We analyze the suitability of our new primitive for video action recognition and explore several novel variations of our approach to enable stronger representational flexibility while maintaining an efficient design.
AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning
It meticulously standardizes and unifies these trajectories into a consistent format, streamlining the creation of a generic data loader optimized for agent training.
Causal Layering via Conditional Entropy
Under appropriate assumptions and conditioning, we can separate the sources or sinks from the remainder of the nodes by comparing their conditional entropy to the unconditional entropy of their noise.
Editing Arbitrary Propositions in LLMs without Subject Labels
On datasets of binary propositions derived from the CounterFact dataset, we show that our method -- without access to subject labels -- performs close to state-of-the-art L\&E methods which has access subject labels.
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning
Vision-language pre-training and instruction tuning have demonstrated general-purpose capabilities in 2D visual reasoning tasks by aligning visual encoders with state-of-the-art large language models (LLMs).
Temporally Disentangled Representation Learning under Unknown Nonstationarity
In unsupervised causal representation learning for sequential data with time-delayed latent causal influences, strong identifiability results for the disentanglement of causally-related latent variables have been established in stationary settings by leveraging temporal structure.
BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents
The massive successes of large language models (LLMs) encourage the emerging exploration of LLM-augmented Autonomous Agents (LAAs).
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
This demonstrates that using policy gradient optimization to improve language agents, for which we believe our work is one of the first, seems promising and can be applied to optimize other models in the agent architecture to enhance agent performances over time.
REX: Rapid Exploration and eXploitation for AI Agents
In this paper, we propose an enhanced approach for Rapid Exploration and eXploitation for AI Agents called REX.
HomE: Homography-Equivariant Video Representation Learning
In this work, we propose a novel method for representation learning of multi-view videos, where we explicitly model the representation space to maintain Homography Equivariance (HomE).
UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild
Visual generative foundation models such as Stable Diffusion show promise in navigating these goals, especially when prompted with arbitrary languages.
ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding
Recent advancements in multimodal pre-training methods have shown promising efficacy in 3D representation learning by aligning multimodal features across 3D shapes, their 2D counterparts, and language descriptions.
Ranked #4 on
3D Point Cloud Classification
on ScanObjectNN
(using extra training data)
Procedure-Aware Pretraining for Instructional Video Understanding
This graph can then be used to generate pseudo labels to train a video representation that encodes the procedural knowledge in a more accessible form to generalize to multiple procedure understanding tasks.
Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations
In summary, an OV method learns task-specific information using strong supervision from base annotations and novel category information using weak supervision from image-captions pairs.
On the Unlikelihood of D-Separation
We then provide an analytic average case analysis of the PC Algorithm for causal discovery, as well as a variant of the SGS Algorithm we call UniformSGS.
Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation
Accurately estimating 3D hand pose is crucial for understanding how humans interact with the world.
Salesforce CausalAI Library: A Fast and Scalable Framework for Causal Analysis of Time Series and Tabular Data
Finally, we provide a user interface (UI) that allows users to perform causal analysis on data without coding.
Model-Agnostic Hierarchical Attention for 3D Object Detection
By plugging our proposed modules into the state-of-the-art transformer-based 3D detector, we improve the previous best results on both benchmarks, with the largest improvement margin on small objects.
LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer
Graphic layout designs play an essential role in visual communication.
ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets.
Ranked #3 on
Training-free 3D Point Cloud Classification
on ModelNet40
(using extra training data)
Identifying Auxiliary or Adversarial Tasks Using Necessary Condition Analysis for Adversarial Multi-task Video Understanding
In this work, we propose a generalized notion of multi-task learning by incorporating both auxiliary tasks that the model should perform well on and adversarial tasks that the model should not perform well on.
PrivHAR: Recognizing Human Actions From Privacy-preserving Lens
The accelerated use of digital cameras prompts an increasing concern about privacy and security, particularly in applications such as action recognition.
Revisiting the "Video" in Video-Language Understanding
Building on recent progress in self-supervised image-language models, we revisit this question in the context of video and language tasks.
Ranked #1 on
Video Question Answering
on MSR-VTT-MC
Align and Prompt: Video-and-Language Pre-training with Entity Prompts
To achieve this, we first introduce an entity prompter module, which is trained with VTC to produce the similarity between a video crop and text prompts instantiated with entity names.
Ranked #19 on
Zero-Shot Video Retrieval
on DiDeMo
MOMA: Multi-Object Multi-Actor Activity Parsing
This paper introduces Activity Parsing as the overarching task of temporal segmentation and classification of activities, sub-activities, atomic actions, along with an instance-level understanding of actors, objects, and their relationships in videos.
PreViTS: Contrastive Pretraining with Video Tracking Supervision
In this work, we propose PreViTS, an SSL framework that utilizes an unsupervised tracking signal for selecting clips containing the same object, which helps better utilize temporal transformations of objects.
Open Vocabulary Object Detection with Pseudo Bounding-Box Labels
To enlarge the set of base classes, we propose a method to automatically generate pseudo bounding-box annotations of diverse objects from large-scale image-caption pairs.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
TNT: Text-Conditioned Network with Transductive Inference for Few-Shot Video Classification
Recently, few-shot learning has received increasing interest.
Home Action Genome: Cooperative Compositional Action Understanding
However, there remains a lack of studies that extend action composition and leverage multiple viewpoints and multiple modalities of data for representation learning.
Ranked #1 on
Video Classification
on Home Action Genome
Metadata Normalization
Batch Normalization (BN) and its variants have delivered tremendous success in combating the covariate shift induced by the training step of deep learning methods.
TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild
Joint forecasting of human trajectory and pose dynamics is a fundamental building block of various applications ranging from robotics and autonomous driving to surveillance systems.
Learning Privacy-Preserving Optics for Human Pose Estimation
However, we also want the camera to capture useful information to perform computer vision tasks.
Detecting Human-Object Relationships in Videos
We study a crucial problem in video analysis: human-object relationship detection.
Vision-based Estimation of MDS-UPDRS Gait Scores for Assessing Parkinson's Disease Motor Severity
This is the first benchmark for classifying PD patients based on MDS-UPDRS gait severity and could be an objective biomarker for disease severity.
Socially and Contextually Aware Human Motion and Pose Forecasting
In this paper, we propose a novel framework to tackle both tasks of human motion (or trajectory) and body skeleton pose forecasting in a unified end-to-end pipeline.
Spatio-Temporal Graph for Video Captioning with Knowledge Distillation
In this paper, we propose a novel spatio-temporal graph model for video captioning that exploits object interactions in space and time.
Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction
In addition, we introduce a new dataset designed specifically for autonomous-driving scenarios in areas with dense pedestrian populations: the Stanford-TRI Intent Prediction (STIP) dataset.
Adversarial Cross-Domain Action Recognition with Co-Attention
Action recognition has been a widely studied topic with a heavy focus on supervised learning involving sufficient labeled videos.
Action Genome: Actions as Composition of Spatio-temporal Scene Graphs
Next, by decomposing and learning the temporal changes in visual relationships that result in an action, we demonstrate the utility of a hierarchical event decomposition by enabling few-shot action recognition, achieving 42. 7% mAP using as few as 10 examples.
Motion Reasoning for Goal-Based Imitation Learning
We further show that by using the automatically inferred goal from the video demonstration, our robot is able to reproduce the same task in a real kitchen environment.
Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision
In contrast to the previous work that aims to solve either the task of pose prediction or trajectory forecasting in isolation, we propose a framework to unify the two problems and address the practically useful task of pedestrian locomotion prediction in the wild.
Representation Learning with Statistical Independence to Mitigate Bias
Presence of bias (in datasets or tasks) is inarguably one of the most critical challenges in machine learning applications that has alluded to pivotal debates in recent years.
Learning Temporal Action Proposals With Fewer Labels
Most current methods for training action proposal modules rely on fully supervised approaches that require large amounts of annotated temporal action intervals in long video sequences.
Ranked #3 on
Semi-Supervised Action Detection
on ActivityNet-1.3
Bias-Resilient Neural Network
We apply our method to a synthetic, a medical diagnosis, and a gender classification (Gender Shades) dataset.
Imitation Learning for Human Pose Prediction
Modeling and prediction of human motion dynamics has long been a challenging problem in computer vision, and most existing methods rely on the end-to-end supervised training of various architectures of recurrent neural networks.
Ranked #2 on
Human Pose Forecasting
on Human3.6M
(MAR, walking, 1,000ms metric)
Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning
The key technical challenge is that the symbol grounding is prone to error with limited training data and leads to subsequent symbolic planning failures.
Procedure Planning in Instructional Videos
In this paper, we study the problem of procedure planning in instructional videos, which can be seen as a step towards enabling autonomous agents to plan for complex tasks in everyday settings such as cooking.
Few-Shot Video Classification via Temporal Alignment
In this paper, we propose Temporal Alignment Module (TAM), a novel few-shot learning framework that can learn to classify a previous unseen video.
Ranked #5 on
Few Shot Action Recognition
on Kinetics-100
Segmenting the Future
While prior work attempts to predict future video pixels, anticipate activities or forecast future scene semantic segments from segmentation of the preceding frames, methods that predict future semantic segmentation solely from the previous frame RGB data in a single end-to-end trainable model do not exist.
Peeking into the Future: Predicting Future Person Activities and Locations in Videos
To facilitate the training, the network is learned with an auxiliary task of predicting future location in which the activity will happen.
Ranked #1 on
Activity Prediction
on ActEV
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation
The key technical challenge for discriminative modeling with weak supervision is that the loss function of the ordering supervision is usually formulated using dynamic programming and is thus not differentiable.
Action-Agnostic Human Pose Forecasting
In this paper, we propose a new action-agnostic method for short- and long-term human pose forecasting.
Ranked #5 on
Human Pose Forecasting
on Human3.6M
(MAR, walking, 1,000ms metric)
Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation
We propose an end-to-end deep learning model for translating free-form natural language instructions to a high-level plan for behavioral robot navigation.
End-to-End Joint Semantic Segmentation of Actors and Actions in Video
Traditional video understanding tasks include human action recognition and actor/object semantic segmentation.
Temporal Modular Networks for Retrieving Complex Compositional Activities in Videos
A major challenge in computer vision is scaling activity understanding to the long tail of complex activities without requiring collecting large quantities of data for new actions.
The ActivityNet Large-Scale Activity Recognition Challenge 2018 Summary
The guest tasks focused on complementary aspects of the activity recognition problem at large scale and involved three challenging and recently compiled datasets: the Kinetics-600 dataset from Google DeepMind, the AVA dataset from Berkeley and Google, and the Moments in Time dataset from MIT and IBM Research.
Liquid Pouring Monitoring via Rich Sensory Inputs
In the closed-loop system, the ability to monitor the state of the task via rich sensory information is important but often less studied.
Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Mining
A key aspect of VQA models that are interpretable is their ability to ground their answers to relevant regions in the image.
Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model.
Learning to Decompose and Disentangle Representations for Video Prediction
Our goal is to predict future video frames given a sequence of input frames.
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos
In this work, we propose to tackle this new task with a weakly-supervised framework for reference-aware visual grounding in instructional videos, where only the temporal alignment between the transcription and the video segment are available for supervision.
What Makes a Video a Video: Analyzing Temporal Information in Video Understanding Models and Datasets
The ability to capture temporal information has been critical to the development of video understanding models.
A Deep Learning Based Behavioral Approach to Indoor Autonomous Navigation
We present a semantically rich graph representation for indoor robotic navigation.
Graph Distillation for Action Detection with Privileged Modalities
We propose a technique that tackles action detection in multimodal videos under a realistic and challenging condition in which only limited training data and partially observed modalities are available.
ActivityNet Challenge 2017 Summary
The ActivityNet Large Scale Activity Recognition Challenge 2017 Summary: results and challenge participants papers.
Visual Forecasting by Imitating Dynamics in Natural Sequences
This allows us to apply IRL at scale and directly imitate the dynamics in high-dimensional continuous visual sequences from the raw pixel values.
SST: Single-Stream Temporal Action Proposals
Our paper presents a new approach for temporal detection of human actions in long, untrimmed video sequences.
Agent-Centric Risk Assessment: Accident Anticipation and Risky Region Localization
For survival, a living agent must have the ability to assess risk (1) by temporally anticipating accidents before they occur, and (2) by spatially localizing risky regions in the environment to move away from threats.
Dense-Captioning Events in Videos
We also introduce ActivityNet Captions, a large-scale benchmark for dense-captioning events.
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
We propose an unsupervised method for reference resolution in instructional videos, where the goal is to temporally link an entity (e. g., "dressing") to the action (e. g., "mix yogurt") that produced it.
Leveraging Video Descriptions to Learn Video Question Answering
Then, a large number of candidate QA pairs are automatically generated from descriptions rather than manually annotated.
Title Generation for User Generated Videos
Finally, our sentence augmentation method also outperforms the baselines on the M-VAD dataset.
Connectionist Temporal Modeling for Weakly Supervised Action Labeling
We propose a weakly-supervised framework for action labeling in video, where only the order of occurring actions is required during training time.
A Hierarchical Pose-Based Approach to Complex Action Understanding Using Dictionaries of Actionlets and Motion Poselets
In this paper, we introduce a new hierarchical model for human action recognition using body joint locations.
Fast Temporal Activity Proposals for Efficient Detection of Human Actions in Untrimmed Videos
In many large-scale video analysis scenarios, one is interested in localizing and recognizing human activities that occur in short temporal intervals within long untrimmed videos.
On the Relationship Between Visual Attributes and Convolutional Networks
One of the cornerstone principles of deep models is their abstraction capacity, i. e. their ability to learn abstract concepts from `simpler' ones.
Robust Manhattan Frame Estimation From a Single RGB-D Image
This paper proposes a new framework for estimating the Manhattan Frame (MF) of an indoor scene from a single RGB-D image.
ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding
In spite of many dataset efforts for human action recognition, current computer vision algorithms are still severely limited in terms of the variability and complexity of the actions that they can recognize.
Discriminative Hierarchical Modeling of Spatio-Temporally Composable Human Activities
Our method describes human activities in a hierarchical discriminative model that operates at three semantic levels.
