Search Results for author:
Found 26 papers, 6 papers with code
Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos
We propose a self-supervised method for learning representations based on spatial audio-visual correspondences in egocentric videos.
SpotEM: Efficient Video Search for Episodic Memory
The goal in episodic memory (EM) is to search a long egocentric video to answer a natural language query (e. g., "where did I leave my purse?").
A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems
Despite the advancement of machine learning techniques in recent years, state-of-the-art systems lack robustness to "real world" events, where the input distributions and tasks encountered by the deployed systems will not be limited to the original training context, and systems will instead need to adapt to novel distributions and tasks while deployed.
NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory
Searching long egocentric videos with natural language queries (NLQ) has compelling applications in augmented reality and robotics, where a fluid index into everything that a person (agent) has seen before could augment human memory and surface relevant information on demand.
Few-Shot Audio-Visual Learning of Environment Acoustics
Room impulse response (RIR) functions capture how the surrounding physical environment transforms the sounds heard by a listener, with implications for various applications in AR, VR, and robotics.
Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation
In reinforcement learning for visual navigation, it is common to develop a model for each new task, and train that model from scratch with task-specific interactions in 3D environments.
PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning
We propose Potential functions for ObjectGoal Navigation with Interaction-free learning (PONI), a modular approach that disentangles the skills of `where to look?'
Environment Predictive Coding for Visual Navigation
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents.
Move2Hear: Active Audio-Visual Source Separation
We introduce the active audio-visual source separation problem, where an agent must move intelligently in order to better isolate the sounds coming from an object of interest in its environment.
Environment Predictive Coding for Embodied Agents
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents.
Semantic Audio-Visual Navigation
We propose a transformer-based model to tackle this new semantic AudioGoal task, incorporating an inferred goal descriptor that captures both spatial and semantic properties of the target.
Modeling Fashion Influence from Photos
The discovered influence relationships reveal how both cities and brands exert and receive fashion influence for an array of visual styles inferred from the images.
Occupancy Anticipation for Efficient Exploration and Navigation
State-of-the-art navigation methods leverage a spatial memory to generalize to new environments, but their occupancy maps are limited to capturing the geometric structures directly observed by the agent.
Ranked #3 on
Robot Navigation
on Habitat 2020 Point Nav test-std
Learning to Set Waypoints for Audio-Visual Navigation
In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source (e. g., a phone ringing in another room).
VisualEchoes: Spatial Image Representation Learning through Echolocation
Several animal species (e. g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation: a biological sonar used to perceive spatial layout and locate objects in the world.
From Paris to Berlin: Discovering Fashion Style Influences Around the World
The evolution of clothing styles and their migration across the world is intriguing, yet difficult to describe quantitatively.
SoundSpaces: Audio-Visual Navigation in 3D Environments
Moving around in the world is naturally a multisensory experience, but today's embodied agents are deaf---restricted to solely their visual perception of the environment.
Smile, Be Happy :) Emoji Embedding for Visual Sentiment Analysis
Additionally, we introduce a novel emoji representation based on their visual emotional response which supports a deeper understanding of the emoji modality and their usage on social media.
Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback
We provide a detailed analysis of the characteristics of the Fashion IQ data, and present a transformer-based user simulator and interactive image retriever that can seamlessly integrate visual attributes with image features, user feedback, and dialog history, leading to improved performance over the state of the art in dialog-based image retrieval.
Traversing the Continuous Spectrum of Image Retrieval with Deep Dynamic Models
While the proposed approaches in the literature can be roughly categorized into two main groups: category- and instance-based retrieval, in this work we show that the retrieval task is much richer and more complex.
Informed Democracy: Voting-based Novelty Detection for Action Recognition
While it is common in activity recognition to assume a closed-set setting, i. e. test samples are always of training categories, this assumption is impractical in a real-world scenario.
Automatic Discovery, Association Estimation and Learning of Semantic Attributes for a Thousand Categories
Furthermore, we demonstrate that our model outperforms the state-of-the-art in zero-shot learning on three data sets: ImageNet, Animals with Attributes and aPascal/aYahoo.
Relaxed Earth Mover's Distances for Chain- and Tree-connected Spaces and their use as a Loss Function in Deep Learning
The Earth Mover's Distance (EMD) computes the optimal cost of transforming one distribution into another, given a known transport metric between them.
Recovering the Missing Link: Predicting Class-Attribute Associations for Unsupervised Zero-Shot Learning
In this work, we aim to carry out attribute-based zero-shot classification in an unsupervised manner.
How to Transfer? Zero-Shot Object Recognition via Hierarchical Transfer of Semantic Attributes
We propose to capture these variations in a hierarchical model that expands the knowledge source with additional abstraction levels of attributes.
