Search Results for author:
Found 202 papers, 90 papers with code
Understanding and Estimating the Adaptability of Domain-Invariant Representations
We also propose a method for estimating how well a model based on domain-invariant representations will perform on the target domain, without having seen any target labels.
Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models.
Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos
Recent advancements in static feed-forward scene reconstruction have demonstrated significant progress in high-quality novel view synthesis.
Art-Free Generative Models: Art Creation Without Graphic Art Knowledge
We explore the question: "How much prior art knowledge is needed to create art?"
SketchAgent: Language-Driven Sequential Sketch Generation
In this work, we introduce SketchAgent, a language-driven, sequential sketch generation method that enables users to create, modify, and refine sketches through dynamic, conversational interactions.
Adaptive Length Image Tokenization via Recurrent Allocation
Our encoder-decoder architecture recursively processes 2D image tokens, distilling them into 1D latent tokens over multiple iterations of recurrent rollouts.
LoRA vs Full Fine-tuning: An Illusion of Equivalence
Second, we show that LoRA models with intruder dimensions, despite achieving similar performance to full fine-tuning on the target task, become worse models of the pre-training distribution and adapt less robustly to multiple tasks sequentially.
Characterizing Model Robustness via Natural Input Gradients
Adversarially robust models are locally smooth around each data sample so that small perturbations cannot drastically change model outputs.
Compositional Physical Reasoning of Objects and Events from Videos
The model is evaluated based on its capability to unravel the compositional hidden properties, such as mass and charge, and use this knowledge to answer a set of questions.
L4GM: Large 4D Gaussian Reconstruction Model
We present L4GM, the first 4D Large Reconstruction Model that produces animated objects from a single-view video input -- in a single feed-forward pass that takes only a second.
Automatic Discovery of Visual Circuits
To date, most discoveries of network subcomponents that implement human-interpretable computations in deep vision models have involved close study of single units and large amounts of human labor.
A Multimodal Automated Interpretability Agent
Interpretability experiments proposed by MAIA compose these tools to describe and explain system behavior.
Efficient 3D Instance Mapping and Localization with Neural Fields
We tackle the problem of learning an implicit scene representation for 3D instance segmentation from a sequence of posed RGB images.
LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis
Recent text-to-3D generation approaches produce impressive 3D results but require time-consuming optimization that can take up to an hour per prompt.
GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignment
GOMA formulates verbal communication as a planning problem that minimizes the misalignment between the parts of agents' mental states that are relevant to the goals.
MMToM-QA: Multimodal Theory of Mind Question Answering
To engineer multimodal ToM capacity, we propose a novel method, BIP-ALM (Bayesian Inverse Planning Accelerated by Language Models).
Structure from Duplicates: Neural Inverse Graphics from a Pile of Objects
Our world is full of identical objects (\emphe. g., cans of coke, cars of same model).
A Vision Check-up for Language Models
Although LLM-generated images do not look like natural images, results on image generation and the ability of models to correct these generated images indicate that precise modeling of strings can teach language models about numerous aspects of the visual world.
Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models
We also propose a motion amplification mechanism as well as a new autoregressive synthesis scheme to generate and combine multiple 4D sequences for longer generation.
NewMove: Customizing text-to-video models with novel motions
To avoid overfitting to the new custom motion, we introduce an approach for regularization over videos.
Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models
We present a method to create interpretable concept sliders that enable precise control over attributes in image generations from diffusion models.
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning
We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts.
FIND: A Function Description Benchmark for Evaluating Interpretability Methods
FIND contains functions that resemble components of trained neural networks, and accompanying descriptions of the kind we seek to generate.
Follow Anything: Open-set detection, tracking, and following in real-time
We demonstrate FAn on a real-world robotic system (a micro aerial vehicle) and report its ability to seamlessly follow the objects of interest in a real-time control loop.
Multimodal Neurons in Pretrained Text-Only Transformers
Language models demonstrate remarkable capacity to generalize representations learned in one modality to downstream tasks in other modalities.
DreamTeacher: Pretraining Image Backbones with Deep Generative Models
In this work, we introduce a self-supervised feature representation learning framework DreamTeacher that utilizes generative networks for pre-training downstream image backbones.
Background Prompting for Improved Object Depth
To infer object depth on a real image, we place the segmented object into the learned background prompt and run off-the-shelf depth networks.
Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models
Text-to-image generative models have enabled high-resolution image synthesis across different domains, but require users to specify the content they wish to generate.
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks.
Generalizing Dataset Distillation via Deep Generative Prior
Dataset Distillation aims to distill an entire dataset's knowledge into a few synthetic images.
NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models
We first train a scene auto-encoder to express a set of image and pose pairs as a neural field, represented as density and feature voxel grids that can be projected to produce novel views of the scene.
Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning
In goal-reaching reinforcement learning (RL), the optimal value function has a particular geometry, called quasimetric structure.
Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos
Traditional methods of impact sound synthesis use physics simulation to obtain a set of physics parameters that could represent and synthesize the sound.
Detecting Everything in the Open World: Towards Universal Object Detection
In this paper, we formally address universal object detection, which aims to detect every scene and predict every category.
Open-vocabulary Panoptic Segmentation with Embedding Modulation
Open-vocabulary image segmentation is attracting increasing attention due to its critical applications in the real world.
FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation
We identify several challenges for fluid manipulation learning by evaluating a set of reinforcement learning and trajectory optimization methods on our platform.
ConceptFusion: Open-set Multimodal 3D Mapping
ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio.
Debiasing Vision-Language Models via Biased Prompts
Machine learning models have been shown to inherit biases from their training datasets.
NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants
Experiments show that our helper agent robustly updates its goal inference and adapts its helping plans to the changing level of uncertainty.
BT^2: Backward-compatible Training with Basis Transformation
In this way, the new representation can be directly compared with the old representation, in principle avoiding the need for any backfilling.
Aliasing is a Driver of Adversarial Attacks
In this work, we investigate the hypothesis that the existence of adversarial perturbations is due in part to aliasing in neural networks.
Procedural Image Programs for Representation Learning
Learning image representations using synthetic data allows training neural networks without some of the concerns associated with real images, such as privacy and bias.
$BT^2$: Backward-compatible Training with Basis Transformation
In this way, the new representation can be directly compared with the old representation, in principle avoiding the need for any backfilling.
Composing Ensembles of Pre-trained Models via Iterative Consensus
Such closed-loop communication enables models to correct errors caused by other models, significantly boosting performance on downstream tasks, e. g. improving accuracy on grade school math problems by 7. 5%, without requiring any model finetuning.
Ranked #1 on
Video Question Answering
on ActivityNet-QA
Totems: Physical Objects for Verifying Visual Integrity
We introduce a new approach to image forensics: placing physical refractive objects, which we call totems, into a scene so as to protect any photograph taken of that scene.
Finding Fallen Objects Via Asynchronous Audio-Visual Integration
The way an object looks and sounds provide complementary reflections of its physical properties.
Local Relighting of Real Scenes
We introduce the task of local relighting, which changes a photograph of a scene by switching on and off the light sources that are visible within the image.
Denoised MDPs: Learning World Models Better Than the World Itself
The ability to separate signal from noise, and reason with clean abstractions, is critical to intelligence.
Virtual Correspondence: Humans as a Cue for Extreme-View Geometry
Similar to classic correspondences, VCs conform with epipolar geometry; unlike classic correspondences, VCs do not need to be co-visible across views.
Disentangling visual and written concepts in CLIP
The CLIP network measures the similarity between natural text and images; in this work, we investigate the entanglement of the representation of word images and natural images in its image encoder.
Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps
Modern image generative models show remarkable sample quality when trained on a single domain or class of objects.
Generative Adversarial Network
Image-to-Image Translation
+1
Compositional Visual Generation with Composable Diffusion Models
Large text-guided diffusion models, such as DALLE-2, are able to generate stunning photorealistic images given natural language descriptions.
Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction
Specifically, FixNet consists of a perception module to extract the structured representation from the 3D point cloud, a physical dynamics prediction module to simulate the results of interactions on 3D objects, and a functionality prediction module to evaluate the functionality and choose the correct fix.
ComPhy: Compositional Physical Reasoning of Objects and Events from Videos
In this paper, we take an initial step to highlight the importance of inferring the hidden physical properties not directly observable from visual appearances, by introducing the Compositional Physical Reasoning (ComPhy) dataset.
Correcting Robot Plans with Natural Language Feedback
In this paper, we explore natural language as an expressive and flexible tool for robot correction.
Learning Neural Acoustic Fields
By modeling acoustic propagation in a scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds.
Learning Program Representations for Food Images and Cooking Recipes
To validate our idea, we crowdsource programs for cooking recipes and show that: (a) projecting the image-recipe embeddings into programs leads to better cross-modal retrieval results; (b) generating programs from images leads to better recognition results compared to predicting raw cooking instructions; and (c) we can generate food images by manipulating programs via optimizing the latent code of a GAN.
Dataset Distillation by Matching Training Trajectories
To efficiently obtain the initial and target network parameters for large-scale datasets, we pre-compute and store training trajectories of expert networks trained on the real dataset.
Ranked #3 on
Dataset Distillation - 1IPC
on CUB-200-2011
Pre-Trained Language Models for Interactive Decision-Making
Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.
Natural Language Descriptions of Deep Visual Features
Given a neuron, MILAN generates a description by searching for a natural language string that maximizes pointwise mutual information with the image regions in which the neuron is active.
Robust Contrastive Learning against Noisy Views
Contrastive learning relies on an assumption that positive pairs contain related views, e. g., patches of an image or co-occurring multimodal signals of a video, that share certain underlying information about an instance.
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations
By training an effective feature segmentation architecture on top of BigGAN, we turn BigGAN into a labeled dataset generator.
Incidents1M: a large-scale dataset of images with natural disasters, damage, and incidents
In this work, we present the Incidents1M Dataset, a large-scale multi-label dataset which contains 977, 088 images, with 43 incident and 49 place categories.
GAN-Supervised Dense Visual Alignment
We propose GAN-Supervised Learning, a framework for learning discriminative models and their GAN-generated training data jointly end-to-end.
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierarchies.
Editing a classifier by rewriting its prediction rules
We present a methodology for modifying the behavior of a classifier by directly rewriting its prediction rules.
Learning to Compose Visual Relations
The visual world around us can be described as a structured set of objects and their associated relations.
EditGAN: High-Precision Semantic Image Editing
EditGAN builds on a GAN framework that jointly models images and their semantic segmentations, requiring only a handful of labeled examples, making it a scalable tool for editing.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
OPEn: An Open-ended Physics Environment for Learning Without a Task
We test several existing RL-based exploration methods on this benchmark and find that an agent using unsupervised contrastive learning for representation learning, and impact-driven learning for exploration, achieved the best results.
Toward a Visual Concept Vocabulary for GAN Latent Space
A large body of recent work has identified transformations in the latent spaces of generative adversarial networks (GANs) that consistently and interpretably transform generated images.
Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
Our task poses unique challenges as a system does not know what types of human-object interactions are present in a video or the actual spatiotemporal location of the human and the object.
Scaling up instance annotation via label propagation
Through a large-scale experiment to populate 1M unlabeled images with object segmentation masks for 80 object classes, we show that (1) we obtain 1M object segmentation masks with an total annotation time of only 290 hours; (2) we reduce annotation time by 76x compared to manual annotation; (3) the segmentation quality of our masks is on par with those from manually annotated datasets.
Skill Induction and Planning with Latent Language
We evaluate this approach in the ALFRED household simulation environment, providing natural language annotations for only 10% of demonstrations.
Language Model Pre-training Improves Generalization in Policy Learning
Additional experiments explore the role of language-based encodings in these results; we find that it is possible to train a simple adapter layer that maps from observations and action histories to LM embeddings, and thus that language modeling provides an effective initializer even for tasks with no language as input or output.
Natural Language Descriptions of Deep Features
Given a neuron, MILAN generates a description by searching for a natural language string that maximizes pointwise mutual information with the image regions in which the neuron is active.
Dynamic Modeling of Hand-Object Interactions via Tactile Sensing.
This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing, which opens the door for future applications in activity learning, human-computer interactions, and imitation learning for robotics.
Dynamic Modeling of Hand-Object Interactions via Tactile Sensing
This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing, which opens the door for future applications in activity learning, human-computer interactions, and imitation learning for robotics.
What You Can Learn by Staring at a Blank Wall
We present a passive non-line-of-sight method that infers the number of people or activity of a person from the observation of a blank wall in an unknown room.
3D Neural Scene Representations for Visuomotor Control
Humans have a strong intuitive understanding of the 3D environment around us.
Intelligent Carpet: Inferring 3D Human Pose From Tactile Signals
In this work, leveraging such tactile interactions, we propose a 3D human pose estimation approach using the pressure maps recorded by a tactile carpet as input.
Learning to See by Looking at Noise
We investigate a suite of image generation models that produce images from simple random processes.
Measuring Generalization with Optimal Transport
Understanding the generalization of deep neural networks is one of the most important tasks in deep learning.
DriveGAN: Towards a Controllable High-Quality Neural Simulation
Realistic simulators are critical for training and verifying robotics systems.
Cetacean Translation Initiative: a roadmap to deciphering the communication of sperm whales
We posit that machine learning will be the cornerstone of future collection, processing, and analysis of multimodal streams of data in animal communication studies, including bioacoustic, behavioral, biological, and environmental data.
BARF: Bundle-Adjusting Neural Radiance Fields
In this paper, we propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect (or even unknown) camera poses -- the joint problem of learning neural 3D representations and registering camera frames.
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort
To showcase the power of our approach, we generated datasets for 7 image segmentation tasks which include pixel-level labels for 34 human face parts, and 32 car parts.
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
Training deep networks with limited labeled data while achieving a strong generalization ability is key in the quest to reduce human annotation efforts.
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI
To complete the task, an embodied agent must plan a sequence of actions to change the state of a large number of objects in the face of realistic physical constraints.
Deep Feedback Inverse Problem Solver
Specifically, at each iteration, the neural network takes the feedback as input and outputs an update on the current estimation.
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents.
Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
Key to our approach is to exploit GANs as a multi-view data generator to train an inverse graphics network using an off-the-shelf differentiable renderer, and the trained inverse graphics network as a teacher to disentangle the GAN's latent code into interpretable 3D properties.
LID 2020: The Learning from Imperfect Data Challenge Results
The purpose of the Learning from Imperfect Data (LID) workshop is to inspire and facilitate the research in developing novel approaches that would harness the imperfect data and improve the data-efficiency during training.
Improving Inversion and Generation Diversity in StyleGAN using a Gaussianized Latent Space
We show that, under a simple nonlinear operation, the data distribution can be modeled as Gaussian and therefore expressed using sufficient statistics.
Understanding the Role of Individual Units in a Deep Neural Network
Second, we use a similar analytic method to analyze a generative adversarial network (GAN) model trained to generate scenes.
The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement
In this paper, we propose the Hessian Penalty, a simple regularization term that encourages the Hessian of a generative model with respect to its input to be diagonal.
Detecting natural disasters, damage, and incidents in the wild
While most studies on social media are limited to text, images offer more information for understanding disaster and incident scenes.
Rewriting a Deep Generative Model
To address the problem, we propose a formulation in which the desired rule is changed by manipulating a layer of a deep network as a linear associative memory.
Noisy Agents: Self-supervised Exploration by Predicting Auditory Events
Humans integrate multiple sensory modalities (e. g. visual and audio) to build a causal understanding of the physical world.
Foley Music: Learning to Generate Music from Videos
In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments.
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation.
Estimating Generalization under Distribution Shifts via Domain-Invariant Representations
When machine learning models are deployed on a test distribution different from the training distribution, they can perform poorly, but overestimate their performance.
Debiased Contrastive Learning
A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples.
Causal Discovery in Physical Systems from Videos
We assume access to different configurations and environmental conditions, i. e., data from unknown interventions on the underlying system; thus, we can hope to discover the correct underlying causal graph without explicit interventions.
Diverse Image Generation via Self-Conditioned GANs
We introduce a simple but effective unsupervised method for generating realistic and diverse images.
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos
Further, we propose a tri-modal model that jointly processes raw audio, video, and text captions from videos to learn a multi-modal semantic embedding space useful for text-video retrieval.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Learning to Simulate Dynamic Environments with GameGAN
Simulation is a crucial component of any robotic system.
Semantic Photo Manipulation with a Generative Image Prior
First, it is hard for GANs to precisely reproduce an input image.
Deep Audio Priors Emerge From Harmonic Convolutional Networks
We show that networks using Harmonic Convolution can reliably model audio priors and achieve high performance in unsupervised audio restoration tasks.
Visual Grounding of Learned Physical Models
The abilities to perform physical reasoning and to adapt to new environments, while intrinsic to humans, remain challenging to state-of-the-art computational models.
Music Gesture for Visual Sound Separation
Recent deep learning approaches have achieved impressive performance on visual sound separation tasks.
Self-supervised Moving Vehicle Tracking with Stereo Sound
At test time, the stereo-sound student network can work independently to perform object localization us-ing just stereo audio and camera meta-data, without any visual input.
Seeing What a GAN Cannot Generate
Differences in statistics reveal object classes that are omitted by a GAN.
Gaze360: Physically Unconstrained Gaze Estimation in the Wild
Finally, we demonstrate an application of our model for estimating customer attention in a supermarket setting.
Ranked #4 on
Gaze Estimation
on Gaze360
Learning Compositional Koopman Operators for Model-Based Control
Finding an embedding space for a linear approximation of a nonlinear dynamical system enables efficient system identification and control synthesis.
The Role of Embedding Complexity in Domain-invariant Representations
In this work, we study, theoretically and empirically, the effect of the embedding complexity on generalization to the target domain.
Neural Turtle Graphics for Modeling City Road Layouts
We propose Neural Turtle Graphics (NTG), a novel generative model for spatial graphs, and demonstrate its applications in modeling city road layouts.
CLEVRER: CoLlision Events for Video REpresentation and Reasoning
While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations.
Connecting Touch and Vision via Cross-Modal Prediction
To connect vision and touch, we introduce new tasks of synthesizing plausible tactile signals from visual inputs as well as imagining how we interact with objects given tactile data as input.
How to make a pizza: Learning a compositional layer-based GAN model
From a visual perspective, every instruction step can be seen as a way to change the visual appearance of the dish by adding extra objects (e. g., adding an ingredient) or changing the appearance of the existing ones (e. g., cooking the dish).
Meta-Sim: Learning to Generate Synthetic Datasets
Training models to high-end performance requires availability of large labeled datasets, which are expensive to get.
Self-Supervised Audio-Visual Co-Segmentation
Segmenting objects in images and separating sound sources in audio are challenging tasks, in part because traditional approaches require large amounts of labeled data.
Visualizing and Understanding GANs
We present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level.
On the Units of GANs (Extended Abstract)
We quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.
Visual Object Networks: Image Generation with Disentangled 3D Representation
Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes.
Visual Object Networks: Image Generation with Disentangled 3D Representations
The VON not only generates images that are more realistic than the state-of-the-art 2D image synthesis methods but also enables many 3D operations such as changing the viewpoint of a generated image, shape and texture editing, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints.
Dataset Distillation
Model distillation aims to distill the knowledge of a complex model into a simpler one.
GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.
Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images
In this paper, we introduce Recipe1M+, a new large-scale, structured corpus of over one million cooking recipes and 13 million food images.
Ranked #2 on
Cross-Modal Retrieval
on Recipe1M+
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
Second, the model is more data- and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering.
Ranked #1 on
Visual Question Answering (VQA)
on CLEVR
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
In this paper, we propose to learn a particle-based simulator for complex control tasks.
Propagation Networks for Model-Based Control Under Partial Observation
There has been an increasing interest in learning dynamics simulators for model-based control.
Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks
We introduce a saliency-based distortion layer for convolutional neural networks that helps to improve the spatial sampling of input data for a given task.
Single Image Intrinsic Decomposition without a Single Intrinsic Image
At inference time, our model can be easily reduced to a single stream module that performs intrinsic decomposition on a single input image.
Interpretable Basis Decomposition for Visual Explanation
Explanations of the decisions made by a deep neural network are important for human end-users to be able to understand and diagnose the trustworthiness of the system.
3D-Aware Scene Manipulation via Inverse Graphics
In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model.
RF-based 3D skeletons
It maintains this accuracy even in the presence of multiple people, and in new environments that it has not seen in the training set.
Real-Time Object Pose Estimation with Pose Interpreter Networks
In this work, we introduce pose interpreter networks for 6-DoF object pose estimation.
VirtualHome: Simulating Household Activities via Programs
We then implement the most common atomic (inter)actions in the Unity3D game engine, and use our programs to "drive" an artificial agent to execute tasks in a simulated household environment.
Revisiting the Importance of Individual Units in CNNs via Ablation
We confirm that unit attributes such as class selectivity are a poor predictor for impact on overall accuracy as found previously in recent work \cite{morcos2018importance}.
Through-Wall Human Pose Estimation Using Radio Signals
Yet, unlike vision-based pose estimation, the radio-based system can estimate 2D poses through walls despite never trained on such scenarios.
Inferring Light Fields From Shadows
We present a method for inferring a 4D light field of a hidden scene from 2D shadows cast by a known occluder on a diffuse wall.
The Sound of Pixels
We introduce PixelPlayer, a system that, by leveraging large amounts of unlabeled videos, learns to locate image regions which produce sounds and separate the input sounds into a set of components that represents the sound from each pixel.
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input
In this paper, we explore neural network models that learn to associate segments of spoken audio captions with the semantically relevant portions of natural images that they refer to.
Ranked #2 on
Speech Prompted Semantic Segmentation
on ADE20K
3D Interpreter Networks for Viewer-Centered Wireframe Modeling
3D-INN is trained on real images to estimate 2D keypoint heatmaps from an input image; it then predicts 3D object structure from heatmaps using knowledge learned from synthetic 3D shapes.
Counterfactual Image Networks
However, when a layer is removed, the model learns to produce a different image that still looks natural to an adversary, which is possible by removing objects.
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization
This paper presents a new large-scale dataset for recognition and temporal localization of human actions collected from Web videos.
Ranked #12 on
Temporal Action Localization
on HACS
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
The sound of crashing waves, the roar of fast-moving cars -- sound conveys important information about the objects in our surroundings.
Learning to Act Properly: Predicting and Explaining Affordances from Images
We address the problem of affordance reasoning in diverse scenes that appear in the real world.
Temporal Relational Reasoning in Videos
Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is a fundamental property of intelligent species.
Ranked #2 on
Hand Gesture Recognition
on Jester test
Interpreting Deep Visual Representations via Network Dissection
In this work, we describe Network Dissection, a method that interprets networks by providing labels for the units of their deep visual representations.
Following Gaze in Video
In this paper, we present an approach for following gaze in video by predicting where a person (in the video) is looking even when the object is in a different frame.
Turning Corners Into Cameras: Principles and Methods
We show that walls and other obstructions with edges can be exploited as naturally-occurring "cameras" that reveal the hidden scenes beyond them.
Learning Cross-Modal Embeddings for Cooking Recipes and Food Images
In this paper, we introduce Recipe1M, a new large-scale, structured corpus of over 1m cooking recipes and 800k food images.
Scene Parsing Through ADE20K Dataset
A novel network design called Cascade Segmentation Module is proposed to parse a scene into stuff, objects, and object parts in a cascade and improve over the baselines.
Generating the Future With Adversarial Transformers
We present a model that generates the future by transforming pixels in the past.
See, Hear, and Read: Deep Aligned Representations
We capitalize on large amounts of readily-available, synchronous data to learn a deep discriminative representations shared across three major natural modalities: vision, sound and language.
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Given any CNN model, the proposed method draws on a broad data set of visual concepts to score the semantics of hidden units at each intermediate convolutional layer.
Open Vocabulary Scene Parsing
Recognizing arbitrary objects in the wild has been a challenging problem due to the limitations of existing classification models and datasets.
Face-to-BMI: Using Computer Vision to Infer Body Mass Index on Social Media
A person's weight status can have profound implications on their life, ranging from mental health, to longevity, to financial income.
SegICP: Integrated Deep Semantic Segmentation and Pose Estimation
Recent robotic manipulation competitions have highlighted that sophisticated robots still struggle to achieve fast and reliable perception of task-relevant objects in complex, realistic scenarios.
Is Saki #delicious? The Food Perception Gap on Instagram and Its Relation to Health
Studying how food is perceived in relation to what it actually is typically involves a laboratory setup.
Following Gaze Across Views
In this paper, we present an approach for following gaze across views by predicting where a particular person is looking throughout a scene.
Who is Mistaken?
We then create a representation of characters' beliefs for two tasks in human action understanding: predicting who is mistaken, and when they are mistaken.
A Compositional Object-Based Approach to Learning Physical Dynamics
By comparing to less structured architectures, we show that the NPE's compositional representation of the structure in physical interactions improves its ability to predict movement, generalize across variable object count and different scene configurations, and infer latent properties of objects such as mass.
Unsupervised Learning of Spoken Language with Visual Context
Humans learn to speak before they can read or write, so why can’t computers do the same?
SoundNet: Learning Sound Representations from Unlabeled Video
We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild.
Cross-Modal Scene Networks
Our experiments suggest that our scene representation can help transfer representations across modalities for retrieval.
Places: An Image Database for Deep Scene Understanding
The rise of multi-million-item dataset initiatives has enabled data-hungry machine learning algorithms to reach near-human semantic classification at tasks such as object and scene recognition.
Generating Videos with Scene Dynamics
We capitalize on large amounts of unlabeled video in order to learn a model of scene dynamics for both video recognition tasks (e. g. action classification) and video generation tasks (e. g. future prediction).
Ambient Sound Provides Supervision for Visual Learning
We show that, through this process, the network learns a representation that conveys information about objects and scenes.
Semantic Understanding of Scenes through the ADE20K Dataset
Scene parsing, or recognizing and segmenting objects and stuff in an image, is one of the key problems in computer vision.
Learning Aligned Cross-Modal Representations from Weakly Aligned Data
Our experiments suggest that our scene representation can help transfer representations across modalities for retrieval.
Eye Tracking for Everyone
We believe that we can put the power of eye tracking in everyone's palm by building eye tracking software that works on commodity hardware such as mobile phones and tablets, without the need for additional sensors or devices.
Single Image 3D Interpreter Network
In this work, we propose 3D INterpreter Network (3D-INN), an end-to-end framework which sequentially estimates 2D keypoint heatmaps and 3D object structure, trained on both real 2D-annotated images and synthetic 3D data.
What do different evaluation metrics tell us about saliency models?
How best to evaluate a saliency model's ability to predict where humans look in images is an open research question.
Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition
The complex multi-stage architecture of cortical visual pathways provides the neural basis for efficient visual object recognition in humans.
Learning Deep Features for Discriminative Localization
In this work, we revisit the global average pooling layer proposed in [13], and shed light on how it explicitly enables the convolutional neural network to have remarkable localization ability despite being trained on image-level labels.
MovieQA: Understanding Stories in Movies through Question-Answering
We introduce the MovieQA dataset which aims to evaluate automatic story comprehension from both video and text.
Understanding and Predicting Image Memorability at a Large Scale
Progress in estimating visual memorability has been limited by the small scale and lack of variety of benchmark data.
Where are they looking?
Humans have the remarkable ability to follow the gaze of other people to identify what they are looking at.
Skip-Thought Vectors
The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice.
Ranked #2 on
Semantic Similarity
on SICK
Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story.
Anticipating Visual Representations from Unlabeled Video
The key idea behind our approach is that we can train deep networks to predict the visual representation of images in the future.
Visualizing Object Detection Features
We introduce algorithms to visualize feature spaces used by object detectors.
Object Detectors Emerge in Deep Scene CNNs
With the success of new computational architectures for visual processing, such as convolutional neural networks (CNN) and access to image databases with millions of labeled examples (e. g., ImageNet, Places), the state of the art in computer vision is advancing rapidly.
Learning Deep Features for Scene Recognition using Places Database
Whereas the tremendous recent progress in object recognition tasks is due to the availability of large datasets like ImageNet and the rise of Convolutional Neural Networks (CNNs) for learning high-level features, performance at scene recognition has not attained the same level of success.
Learning visual biases from human imagination
Although the human visual system can recognize many concepts under challenging conditions, it still has some biases.
Predicting Motivations of Actions by Leveraging Text
In this paper, we introduce the problem of predicting why a person has performed an action in images.
Looking Beyond the Visible Scene
In this work, we propose to look beyond the visible elements of a scene; we demonstrate that a scene is not just a collection of objects and their configuration or the labels assigned to its pixels - it is so much more.
Are all training examples equally valuable?
When learning a new concept, not all training examples may prove equally useful for training: some may have higher or lower training value than others.
Inverting and Visualizing Features for Object Detection
By visualizing feature spaces, we can gain a more intuitive understanding of our detection systems.
Localizing 3D cuboids in single-view images
In this paper we seek to detect rectangular cuboids and localize their corners in uncalibrated single-view images depicting everyday scenes.
Learning to Learn with Compound HD Models
We introduce HD (or ``Hierarchical-Deep'') models, a new compositional learning architecture that integrates deep learning models with structured hierarchical Bayesian models.
Understanding the Intrinsic Memorability of Images
Artists, advertisers, and photographers are routinely presented with the task of creating an image that a viewer will remember.
Unsupervised Detection of Regions of Interest Using Iterative Link Analysis
This paper proposes a fast and scalable alternating optimization technique to detect regions of interest (ROIs) in cluttered Web images without labels.
Semi-Supervised Learning in Gigantic Image Collections
With the advent of the Internet it is now possible to collect hundreds of millions of images.
