Search Results for author:
Found 76 papers, 36 papers with code
There’s a Time and Place for Reasoning Beyond the Image
Images are often more significant than only the pixels to human eyes, as we can infer, associate, and reason with contextual information from other sources to establish a more complete picture.
RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios
We introduce RESIN-11, a new schema-guided event extraction&prediction framework that can be applied to a large variety of newsworthy scenarios.
Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape
Specifically, we encode the input 3D textured shape into triplane feature maps that represent the signed distance and texture fields of the input.
Tracking through Containers and Occluders in the Wild
Tracking objects with persistence in cluttered and dynamic environments remains a difficult challenge for computer vision systems.
Humans as Light Bulbs: 3D Human Reconstruction from Thermal Reflection
The relatively hot temperature of the human body causes people to turn into long-wave infrared light sources.
SURFSUP: Learning Fluid Simulation for Novel Surfaces
Modeling the mechanics of fluid in complex scenes is vital to applications in design, graphics, and robotics.
Zero-1-to-3: Zero-shot One Image to 3D Object
We introduce Zero-1-to-3, a framework for changing the camera viewpoint of an object given just a single RGB image.
ViperGPT: Visual Inference via Python Execution for Reasoning
Answering visual queries is a complex task that requires both visual processing and reasoning.
Ranked #6 on
Video Question Answering
on NExT-QA
Affective Faces for Goal-Driven Dyadic Communication
We introduce a video framework for modeling the association between verbal and non-verbal communication during dyadic conversation.
What You Can Reconstruct From a Shadow
Experiments and visualizations show that the method is able to generate multiple possible solutions that are consistent with the observation of the shadow.
Understanding Zero-Shot Adversarial Robustness for Large-Scale Models
We apply this training loss to two adaption methods, model finetuning and visual prompt tuning.
Adversarially Robust Video Perception by Seeing Motion
Even under adaptive attacks where the adversary knows our defense, our algorithm is still effective.
Robust Perception through Equivariance
In this paper, we introduce a framework that uses the dense intrinsic constraints in natural images to robustify inference.
Doubly Right Object Recognition: A Why Prompt for Visual Rationales
We propose a ``doubly right'' object recognition benchmark, where the metric requires the model to simultaneously produce both the right labels as well as the right rationales.
Task Bias in Vision-Language Models
Incidental supervision from language has become a popular approach for learning generic visual representations that can be prompted to perform many recognition tasks in computer vision.
Muscles in Action
The dataset consists of 12. 5 hours of synchronized video and surface electromyography (sEMG) data of 10 subjects performing various exercises.
Private Multiparty Perception for Navigation
We introduce a framework for navigating through cluttered environments by connecting multiple cameras together while simultaneously preserving privacy.
FLEX: Full-Body Grasping Without Full-Body Grasps
Towards this goal, we address the task of generating a virtual human -- hands and full body -- grasping everyday objects.
Visual Classification via Description from Large Language Models
By basing decisions on these descriptors, we can provide additional cues that encourage using the features we want to be used.
Representing Spatial Trajectories as Distributions
We introduce a representation learning framework for spatial trajectories.
Forget-me-not! Contrastive Critics for Mitigating Posterior Collapse
Variational autoencoders (VAEs) suffer from posterior collapse, where the powerful neural networks used for modeling and inference optimize the objective without meaningfully using the latent representation.
Shadows Shed Light on 3D Objects
Experiments and visualizations show that the method is able to generate multiple possible solutions that are consistent with the observation of the shadow.
Landscape Learning for Neural Network Inversion
Many machine learning methods operate by inverting a neural network at inference time, which has become a popular technique for solving inverse problems in computer vision, robotics, and graphics.
It's Time for Artistic Correspondence in Music and Video
In order to capture the high-level concepts that are required to solve the task, we propose modeling the long-term temporal context of both the video and the music signals, using Transformer networks for each modality.
Causal Transportability for Visual Recognition
Visual representations underlie object recognition tasks, but they often contain both robust and non-robust features.
Revealing Occlusions with 4D Neural Fields
For computer vision systems to operate in dynamic situations, they need to be able to represent and reason about object permanence.
There is a Time and Place for Reasoning Beyond the Image
For example, in Figure 1, we can find a way to identify the news articles related to the picture through segment-wise understandings of the signs, the buildings, the crowds, and more.
UnweaveNet: Unweaving Activity Stories
Our lives can be seen as a complex weaving of activities; we switch from one activity to another, to maximise our achievements or in reaction to demands placed upon us.
Real-Time Neural Voice Camouflage
Automatic speech recognition systems have created exciting possibilities for applications, however they also enable opportunities for systematic eavesdropping.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Discrete Representations Strengthen Vision Transformer Robustness
Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition.
Ranked #3 on
Domain Generalization
on Stylized-ImageNet
Full-Body Visual Self-Modeling of Robot Morphologies
Internal computational models of physical bodies are fundamental to the ability of robots and animals alike to plan and control their actions.
Learning the Predictability of the Future
We introduce a framework for learning from unlabeled video what is predictable in the future.
Representation Learning
Self-Supervised Action Recognition
+1
RESIN: A Dockerized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System
We present a new information extraction system that can automatically construct temporal event graphs from a collection of news documents from multiple sources, multiple languages (English and Spanish for our experiment), and multiple data modalities (speech, text, image and video).
The Boombox: Visual Reconstruction from Acoustic Vibrations
Due to the many ways that robots use containers, we believe the box will have a number of applications in robotics.
Adversarial Attacks are Reversible with Natural Supervision
We find that images contain intrinsic structure that enables the reversal of many adversarial attacks.
Analogical Reasoning for Visually Grounded Compositional Generalization
Children acquire language subconsciously by observing the surrounding world and listening to descriptions.
Generative Interventions for Causal Learning
We introduce a framework for learning robust visual representations that generalize to new viewpoints, backgrounds, and scene contexts.
Ranked #43 on
Image Classification
on ObjectNet
(using extra training data)
Globetrotter: Connecting Languages by Connecting Images
Machine translation between many languages at once is highly challenging, since training with ground truth requires supervision between all language pairs, which is difficult to obtain.
Dissecting Image Crops
The elementary operation of cropping underpins nearly every computer vision system, ranging from data augmentation and translation invariance to computational photography and representation learning.
Listening to Sounds of Silence for Speech Denoising
We introduce a deep learning model for speech denoising, a long-standing challenge in audio analysis arising in numerous applications.
We Have So Much In Common: Modeling Semantic Relational Set Abstractions in Videos
This allows our model to perform cognitive tasks such as set abstraction (which general concept is in common among a set of videos?
Analogical Reasoning for Visually Grounded Language Acquisition
Children acquire language subconsciously by observing the surrounding world and listening to descriptions.
Multitask Learning Strengthens Adversarial Robustness
Although deep networks achieve strong accuracy on a range of computer vision benchmarks, they remain vulnerable to adversarial attacks, where imperceptible input perturbations fool the network.
Learning Goals from Failure
We introduce a framework that predicts the goals behind observable human action in video.
Learning to Learn Words from Visual Scenes
Language acquisition is the process of learning words from the surrounding scene.
Oops! Predicting Unintentional Action in Video
We train a supervised neural network as a baseline and analyze its performance compared to human consistency on the tasks.
Visual Hide and Seek
We train embodied agents to play Visual Hide and Seek where a prey must navigate in a simulated environment in order to avoid capture from a predator.
Metric Learning for Adversarial Robustness
Deep networks are well-known to be fragile to adversarial attacks.
Learning from Noisy Demonstration Sets via Meta-Learned Suitability Assessor
A noisy and diverse demonstration set may hinder the performances of an agent aiming to acquire certain skills via imitation learning.
VideoBERT: A Joint Model for Video and Language Representation Learning
Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube.
Ranked #1 on
Action Classification
on YouCook2
Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding
Following dedicated non-linear mappings for visual features at each level, word, and sentence embeddings, we obtain multiple instantiations of our common semantic space in which comparisons between any target text and the visual content is performed with cosine similarity.
Ranked #1 on
Phrase Grounding
on ReferIt
Actor-Centric Relation Network
A visualization of the learned relation features confirms that our approach is able to attend to the relevant relations for each action.
Ranked #14 on
Action Recognition
on AVA v2.1
Tracking Emerges by Colorizing Videos
We use large amounts of unlabeled video to learn models for visual tracking without manual human supervision.
The Sound of Pixels
We introduce PixelPlayer, a system that, by leveraging large amounts of unlabeled videos, learns to locate image regions which produce sounds and separate the input sounds into a set of components that represents the sound from each pixel.
Moments in Time Dataset: one million videos for event understanding
We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds.
Counterfactual Image Networks
However, when a layer is removed, the model learns to produce a different image that still looks natural to an adversary, which is possible by removing objects.
Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems
Finally, we show that retraining using the safety violations detected by VeriVis can reduce the average number of violations up to 60. 2%.
Following Gaze in Video
In this paper, we present an approach for following gaze in video by predicting where a person (in the video) is looking even when the object is in a different frame.
Generating the Future With Adversarial Transformers
We present a model that generates the future by transforming pixels in the past.
See, Hear, and Read: Deep Aligned Representations
We capitalize on large amounts of readily-available, synchronous data to learn a deep discriminative representations shared across three major natural modalities: vision, sound and language.
AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions
The AVA dataset densely annotates 80 atomic visual actions in 430 15-minute video clips, where actions are localized in space and time, resulting in 1. 58M action labels with multiple labels per person occurring frequently.
Ranked #3 on
Temporal Action Localization
on UCF101-24
Following Gaze Across Views
In this paper, we present an approach for following gaze across views by predicting where a particular person is looking throughout a scene.
Who is Mistaken?
We then create a representation of characters' beliefs for two tasks in human action understanding: predicting who is mistaken, and when they are mistaken.
SoundNet: Learning Sound Representations from Unlabeled Video
We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild.
Cross-Modal Scene Networks
Our experiments suggest that our scene representation can help transfer representations across modalities for retrieval.
Generating Videos with Scene Dynamics
We capitalize on large amounts of unlabeled video in order to learn a model of scene dynamics for both video recognition tasks (e. g. action classification) and video generation tasks (e. g. future prediction).
Learning Aligned Cross-Modal Representations from Weakly Aligned Data
Our experiments suggest that our scene representation can help transfer representations across modalities for retrieval.
Where are they looking?
Humans have the remarkable ability to follow the gaze of other people to identify what they are looking at.
Anticipating Visual Representations from Unlabeled Video
The key idea behind our approach is that we can train deep networks to predict the visual representation of images in the future.
Do We Need More Training Data?
Datasets for training object recognition systems are steadily increasing in size.
Visualizing Object Detection Features
We introduce algorithms to visualize feature spaces used by object detectors.
Learning visual biases from human imagination
Although the human visual system can recognize many concepts under challenging conditions, it still has some biases.
Predicting Motivations of Actions by Leveraging Text
In this paper, we introduce the problem of predicting why a person has performed an action in images.
Inverting and Visualizing Features for Object Detection
By visualizing feature spaces, we can gain a more intuitive understanding of our detection systems.
Video Annotation and Tracking with Active Learning
We introduce a novel active learning framework for video annotation.
