Search Results for author:
Found 17 papers, 8 papers with code
Video ReCap: Recursive Captioning of Hour-Long Videos
We utilize a curriculum learning training scheme to learn the hierarchical structure of videos, starting from clip-level captions describing atomic actions, then focusing on segment-level descriptions, and concluding with generating summaries for hour-long videos.
Detours for Navigating Instructional Videos
We introduce the video detours problem for navigating instructional videos.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
We present Any-Modality Augmented Language Model (AnyMAL), a unified model that reasons over diverse input modality signals (i. e. text, image, video, audio, IMU motion sensor), and generates textual responses.
Ranked #7 on
Video Question Answering
on STAR Benchmark
Shaping embodied agent behavior with activity-context priors from egocentric video
For a given object, an activity-context prior represents the set of other compatible objects that are required for activities to succeed (e. g., a knife and cutting board brought together with a tomato are conducive to cutting).
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Environment Predictive Coding for Visual Navigation
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents.
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets.
Environment Predictive Coding for Embodied Agents
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents.
Differentiable Causal Discovery Under Unmeasured Confounding
In this work, we derive differentiable algebraic constraints that fully characterize the space of ancestral ADMGs, as well as more general classes of ADMGs, arid ADMGs and bow-free ADMGs, that capture all equality restrictions on the observed variables.
Learning Affordance Landscapes for Interaction Exploration in 3D Environments
We introduce a reinforcement learning approach for exploration for interaction, whereby an embodied agent autonomously discovers the affordance landscape of a new unmapped 3D environment (such as an unfamiliar kitchen).
EGO-TOPO: Environment Affordances from Egocentric Video
We introduce a model for environment affordances that is learned directly from egocentric video.
Grounded Human-Object Interaction Hotspots from Video (Extended Abstract)
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements.
Grounded Human-Object Interaction Hotspots from Video
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements.
Ranked #3 on
Video-to-image Affordance Grounding
on EPIC-Hotspot
Attributes as Operators: Factorizing Unseen Attribute-Object Compositions
In addition, we show that not only can our model recognize unseen compositions robustly in an open-world setting, it can also generalize to compositions where objects themselves were unseen during training.
Ranked #5 on
Image Retrieval with Multi-Modal Query
on MIT-States
BlockDrop: Dynamic Inference Paths in Residual Networks
Very deep convolutional neural networks offer excellent recognition results, yet their computational expense limits their impact for many real-world applications.
CANDiS: Coupled & Attention-Driven Neural Distant Supervision
The unsupervised nature of this technique allows it to scale to web-scale relation extraction tasks, at the expense of noise in the training data.
