Search Results for author:
Found 146 papers, 69 papers with code
Talk The Walk: Navigating Grids in New York City through Grounded Dialogue
We introduce `"Talk The Walk", the first large-scale dialogue dataset grounded in action and perception.
The Llama 3 Herd of Models
This paper presents a new set of foundation models, called Llama 3.
Ranked #3 on
Multi-task Language Understanding
on MMLU
Video Editing via Factorized Diffusion Distillation
We introduce Emu Video Edit (EVE), a model that establishes a new state-of-the art in video editing without relying on any supervised video editing data.
Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning
We present Emu Video, a text-to-video generation model that factorizes the generation into two steps: first generating an image conditioned on the text, and then generating a video conditioned on the text and the generated image.
Emu Edit: Precise Image Editing via Recognition and Generation Tasks
Lastly, to facilitate a more rigorous and informed assessment of instructable image editing models, we release a new challenging and versatile benchmark that includes seven different image editing tasks.
Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack
Training text-to-image models with web scale image-text pairs enables the generation of a wide range of visual concepts from text.
Make-An-Animation: Large-Scale Text-conditional 3D Human Motion Generation
However, existing approaches are limited by their reliance on relatively small-scale motion capture data, leading to poor performance on more diverse, in-the-wild prompts.
Ranked #23 on
Motion Synthesis
on HumanML3D
Text-Conditional Contextualized Avatars For Zero-Shot Personalization
Recent large-scale text-to-image generation models have made significant improvements in the quality, realism, and diversity of the synthesized images and enable users to control the created content through language.
Text-To-4D Dynamic Scene Generation
We present MAV3D (Make-A-Video3D), a method for generating three-dimensional dynamic scenes from text descriptions.
SpaText: Spatio-Textual Representation for Controllable Image Generation
Due to lack of large-scale datasets that have a detailed textual description for each region in the image, we choose to leverage the current large-scale text-to-image datasets and base our approach on a novel CLIP-based spatio-textual representation, and show its effectiveness on two state-of-the-art diffusion models: pixel-based and latent-based.
AudioGen: Textually Guided Audio Generation
Finally, we explore the ability of the proposed method to generate audio continuation conditionally and unconditionally.
Ranked #13 on
Audio Generation
on AudioCaps
Make-A-Video: Text-to-Video Generation without Text-Video Data
We propose Make-A-Video -- an approach for directly translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V).
Ranked #3 on
Text-to-Video Generation
on MSR-VTT
(CLIP-FID metric)
Episodic Memory Question Answering
Towards that end, we introduce (1) a new task - Episodic Memory Question Answering (EMQA) wherein an egocentric AI assistant is provided with a video sequence (the tour) and a question as an input and is asked to localize its answer to the question within the tour, (2) a dataset of grounded questions designed to probe the agent's spatio-temporal understanding of the tour, and (3) a model for the task that encodes the scene as an allocentric, top-down semantic feature map and grounds the question into the map to localize the answer.
MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration
Altogether, MUGEN can help progress research in many tasks in multimodal understanding and generation.
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer
Videos are created to express emotion, exchange information, and share experiences.
Ranked #17 on
Video Generation
on UCF-101
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors
Recent text-to-image generation methods provide a simple yet exciting conversion capability between text and image domains.
Ranked #20 on
Text-to-Image Generation
on MS COCO
(using extra training data)
Telling Creative Stories Using Generative Visual Aids
Can visual artworks created using generative visual algorithms inspire human creativity in storytelling?
Visual Conceptual Blending with Large-scale Language and Vision Models
We ask the question: to what extent can recent large-scale language and image generation models blend visual concepts?
Building Bridges: Generative Artworks to Explore AI Ethics
In recent years, there has been an increased emphasis on understanding and mitigating adverse impacts of artificial intelligence (AI) technologies on society.
Human-Adversarial Visual Question Answering
Human subjects interact with a state-of-the-art VQA model, and for each image in the dataset, attempt to find a question where the model's predicted answer is incorrect.
VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator
Interactive robots navigating photo-realistic environments need to be trained to effectively leverage and handle the dynamic nature of dialogue in addition to the challenges underlying vision-and-language navigation (VLN).
ForceNet: A Graph Neural Network for Large-Scale Quantum Calculations
By not imposing explicit physical constraints, we can flexibly design expressive models while maintaining their computational efficiency.
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs
We present \textsc{Vx2Text}, a framework for text generation from multimodal inputs consisting of video plus text, speech, or audio.
ForceNet: A Graph Neural Network for Large-Scale Quantum Chemistry Simulation
We use ForceNet to perform quantum chemistry simulations, where ForceNet is able to achieve 4x higher success rate than existing ML models.
Object-Centric Diagnosis of Visual Reasoning
In contrast, symbolic and modular models have a relatively better grounding and robustness, though at the cost of accuracy.
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA
One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image.
Ranked #7 on
Visual Question Answering (VQA)
on A-OKVQA
Creative Sketch Generation
Sketching or doodling is a popular creative activity that people engage in.
Where Are You? Localization from Embodied Dialog
In this paper, we focus on the LED task -- providing a strong baseline model with detailed ablations characterizing both dataset biases and the importance of various modeling choices.
Sim-to-Real Transfer for Vision-and-Language Navigation
We study the challenging problem of releasing a robot in a previously unseen environment, and having it follow unconstrained natural language navigation instructions.
SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency
Recent research in Visual Question Answering (VQA) has revealed state-of-the-art models to be inconsistent in their understanding of the world -- they answer seemingly difficult questions requiring reasoning correctly but get simpler associated sub-questions wrong.
The Open Catalyst 2020 (OC20) Dataset and Community Challenges
Catalyst discovery and optimization is key to solving many societal and energy challenges including solar fuels synthesis, long-term energy storage, and renewable fertilizer production.
An Introduction to Electrocatalyst Design using Machine Learning for Renewable Energy Storage
As we increase our reliance on renewable energy sources such as wind and solar, which produce intermittent power, storage is needed to transfer power from times of peak generation to peak demand.
Contrast and Classify: Training Robust VQA Models
Recent Visual Question Answering (VQA) models have shown impressive performance on the VQA benchmark but remain sensitive to small linguistic variations in input questions.
Integrating Egocentric Localization for More Realistic Point-Goal Navigation Agents
This enables a seamless adaption to changing dynamics (a different robot or floor type) by simply re-calibrating the visual odometry model -- circumventing the expense of re-training of the navigation policy.
Ranked #5 on
Robot Navigation
on Habitat 2020 Point Nav test-std
Dialog without Dialog Data: Learning Visual Dialog Agents from VQA Data
Can we develop visually grounded dialog agents that can efficiently adapt to new tasks without forgetting how to talk to people?
Spatially Aware Multimodal Transformers for TextVQA
Further, each head in our multi-head self-attention layer focuses on a different subset of relations.
Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation
We also demonstrate that reducing the task of room navigation to point navigation improves the performance further.
Neuro-Symbolic Generative Art: A Preliminary Study
There are two classes of generative art approaches: neural, where a deep model is trained to generate samples from a data distribution, and symbolic or algorithmic, where an artist designs the primary parameters and an autonomous system generates samples within these constraints.
Feel The Music: Automatically Generating A Dance For An Input Song
We encode intuitive, flexible heuristics for what a 'good' dance is: the structure of the dance should align with the structure of the music.
Extended Abstract: Improving Vision-and-Language Navigation with Image-Text Pairs from the Web
Following a navigation instruction such as 'Walk down the stairs and stop near the sofa' requires an agent to ground scene elements referenced via language (e. g.'stairs') to visual content in the environment (pixels corresponding to 'stairs').
Exploring Crowd Co-creation Scenarios for Sketches
As a first step towards studying the ability of human crowds and machines to effectively co-create, we explore several human-only collaborative co-creation scenarios.
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web
Following a navigation instruction such as 'Walk down the stairs and stop at the brown sofa' requires embodied AI agents to ground scene elements referenced via language (e. g. 'stairs') to visual content in the environment (pixels corresponding to 'stairs').
Ranked #6 on
Vision and Language Navigation
on VLN Challenge
Are we pretraining it right? Digging deeper into visio-linguistic pretraining
Surprisingly, we show that automatically generated data in a domain closer to the downstream task (e. g., VQA v2) is a better choice for pretraining than "natural" data but of a slightly different domain (e. g., Conceptual Captions).
Predicting A Creator's Preferences In, and From, Interactive Generative Art
These preferences could be in the specific generative art form (e. g., color palettes, density of the piece, thickness or curvatures of any lines in the piece); predicting them could lead to a smarter interactive tool.
SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions
We quantify the extent to which this phenomenon occurs by creating a new Reasoning split of the VQA dataset and collecting VQA-introspect, a new dataset1 which consists of 238K new perception questions which serve as sub questions corresponding to the set of perceptual tasks needed to effectively answer the complex reasoning questions in the Reasoning split.
12-in-1: Multi-Task Vision and Language Representation Learning
Much of vision-and-language research focuses on a small but diverse set of independent tasks and supporting datasets often studied in isolation; however, the visually-grounded language understanding skills required for success at these tasks overlap significantly.
Ranked #1 on
10-shot image generation
on MEAD
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline
Next, we find that additional finetuning using "dense" annotations in VisDial leads to even higher NDCG -- more than 10% over our base model -- but hurts MRR -- more than 17% below our base model!
RUBi: Reducing Unimodal Biases for Visual Question Answering
We propose RUBi, a new learning strategy to reduce biases in any VQA model.
Cross-channel Communication Networks
Convolutional neural networks process input data by sending channel-wise feature response maps to subsequent layers.
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
We leverage this scaling to train an agent for 2. 5 Billion steps of experience (the equivalent of 80 years of human experience) -- over 6 months of GPU-time training in under 3 days of wall-clock time with 64 GPUs.
Ranked #1 on
PointGoal Navigation
on Gibson PointGoal Navigation
DS-VIC: Unsupervised Discovery of Decision States for Transfer in RL
We learn to identify decision states, namely the parsimonious set of states where decisions meaningfully affect the future states an agent can reach in an environment.
Improving Generative Visual Dialog by Answering Diverse Questions
Prior work on training generative Visual Dialog models with reinforcement learning(Das et al.) has explored a Qbot-Abot image-guessing game and shown that this 'self-talk' approach can lead to improved performance at the downstream dialog-conditioned image-guessing task.
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language.
Ranked #5 on
Referring Expression Comprehension
on Talk2Car
IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL
We propose a novel framework to identify sub-goals useful for exploration in sequential decision making tasks under partial observability.
Chasing Ghosts: Instruction Following as Bayesian State Tracking
Our experiments show that our approach outperforms a strong LingUNet baseline when predicting the goal location on the map.
RUBi: Reducing Unimodal Biases in Visual Question Answering
We propose RUBi, a new learning strategy to reduce biases in any VQA model.
Ranked #7 on
Visual Question Answering (VQA)
on VQA-CP
SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigation
We propose SplitNet, a method for decoupling visual perception and policy learning.
Cross-Task Knowledge Transfer for Visually-Grounded Navigation
Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for two different tasks: learning to follow navigational instructions and embodied question answering.
Modeling the Long Term Future in Model-Based Reinforcement Learning
This paper focuses on building a model that reasons about the long-term future and demonstrates how to use this for efficient planning and exploration.
Fashion++: Minimal Edits for Outfit Improvement
We introduce Fashion++, an approach that proposes minimal adjustments to a full-body clothing outfit that will have maximal impact on its fashionability.
Emergence of Compositional Language with Deep Generational Transmission
In this paper, we introduce these cultural evolutionary dynamics into language emergence by periodically replacing agents in a population to create a knowledge gap, implicitly inducing cultural transmission of language.
Towards VQA Models That Can Read
We show that LoRRA outperforms existing state-of-the-art VQA models on our TextVQA dataset.
Ranked #3 on
Visual Question Answering (VQA)
on VizWiz 2018
Counterfactual Visual Explanations
In this work, we develop a technique to produce counterfactual visual explanations.
Embodied Visual Recognition
Passive visual systems typically fail to recognize objects in the amodal setting where they are heavily occluded.
Embodied Question Answering in Photorealistic Environments with Point Cloud Perception
To help bridge the gap between internet vision-style problems and the goal of vision for embodied perception we instantiate a large-scale navigation task -- Embodied Question Answering [1] in photo-realistic environments (Matterport 3D).
Habitat: A Platform for Embodied AI Research
We present Habitat, a platform for research in embodied artificial intelligence (AI).
Ranked #2 on
PointGoal Navigation
on Gibson PointGoal Navigation
Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment
We propose a novel end-to-end model that uses caption-to-image retrieval as a `downstream' task to guide the process of phrase localization.
Trick or TReAT: Thematic Reinforcement for Artistic Typography
An approach to make text visually appealing and memorable is semantic reinforcement - the use of visual cues alluding to the context or theme in which the word is being used to reinforce the message (e. g., Google Doodles).
Lemotif: An Affective Visual Journal Using Deep Neural Networks
We present Lemotif, an integrated natural language processing and image generation system that uses machine learning to (1) parse a text-based input journal entry describing the user's day for salient themes and emotions and (2) visualize the detected themes and emotions in creative and appealing image motifs.
CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
Specifically, we construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset.
Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
This paper focuses on building a model that reasons about the long-term future and demonstrates how to use this for efficient planning and exploration.
Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering
We propose a new class of probabilistic neural-symbolic models, that have symbolic functional programs as a latent, stochastic variable.
Cycle-Consistency for Robust Visual Question Answering
Despite significant progress in Visual Question Answering over the years, robustness of today's VQA models leave much to be desired.
Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
Many vision and language models suffer from poor visual grounding - often falling back on easy-to-learn language priors rather than basing their decisions on visual concepts in the image.
Embodied Multimodal Multitask Learning
In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks.
Response to "Visual Dialogue without Vision or Dialogue" (Massiceti et al., 2018)
In a recent workshop paper, Massiceti et al. presented a baseline model and subsequent critique of Visual Dialog (Das et al., CVPR 2017) that raises what we believe to be unfounded concerns about the dataset and evaluation.
Dialog System Technology Challenge 7
This paper introduces the Seventh Dialog System Technology Challenges (DSTC), which use shared datasets to explore the problem of building dialog systems.
nocaps: novel object captioning at scale
To encourage the development of image captioning models that can learn visual concepts from alternative data sources, such as object detection datasets, we present the first large-scale benchmark for this task.
Do Explanations make VQA Models more Predictable to a Human?
A rich line of research attempts to make deep neural networks more transparent by generating human-interpretable 'explanations' of their decision process, especially for interactive tasks like Visual Question Answering (VQA).
TarMAC: Targeted Multi-Agent Communication
We propose a targeted communication architecture for multi-agent reinforcement learning, where agents learn both what messages to send and whom to address them to while performing cooperative tasks in partially-observable environments.
Neural Modular Control for Embodied Question Answering
We use imitation learning to warm-start policies at each level of the hierarchy, dramatically increasing sample efficiency, followed by reinforcement learning.
Visual Curiosity: Learning to Ask Questions to Learn Visual Recognition
Our question generation policy generalizes to new environments and a new pair of eyes, i. e., new visual system.
Visual Coreference Resolution in Visual Dialog using Neural Module Networks
Visual dialog entails answering a series of questions grounded in an image, using dialog history as context.
Ranked #1 on
Common Sense Reasoning
on Visual Dialog v0.9
Choose Your Neuron: Incorporating Domain Knowledge through Neuron-Importance
Our approach, which we call Neuron Importance-AwareWeight Transfer (NIWT), learns to map domain knowledge about novel "unseen" classes onto this dictionary of learned concepts and then optimizes for network parameters that can effectively combine these concepts - essentially learning classifiers by discovering and composing learned semantic concepts in deep networks.
Graph R-CNN for Scene Graph Generation
We propose a novel scene graph generation model called Graph R-CNN, that is both effective and efficient at detecting objects and their relations in images.
Ranked #13 on
Scene Graph Generation
on Visual Genome
Pythia v0.1: the Winning Entry to the VQA Challenge 2018
We demonstrate that by making subtle but important changes to the model architecture and the learning rate schedule, fine-tuning image features, and adding data augmentation, we can significantly improve the performance of the up-down model on VQA v2. 0 dataset -- from 65. 67% to 70. 22%.
Ranked #11 on
Visual Question Answering (VQA)
on A-OKVQA
Talk the Walk: Navigating New York City through Grounded Dialogue
We introduce "Talk The Walk", the first large-scale dialogue dataset grounded in action and perception.
End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features
We introduce a new dataset of dialogs about videos of human behaviors.
Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7
Scene-aware dialog systems will be able to have conversations with users about the objects and events around them.
Neural Baby Talk
We introduce a novel framework for image captioning that can produce natural language explicitly grounded in entities that object detectors find in the image.
CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
The game involves two players: a Teller and a Drawer.
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Specifically, we present new splits of the VQA v1 and VQA v2 datasets, which we call Visual Question Answering under Changing Priors (VQA-CP v1 and VQA-CP v2 respectively).
Embodied Question Answering
We present a new AI task -- Embodied Question Answering (EmbodiedQA) -- where an agent is spawned at a random location in a 3D environment and asked a question ("What color is the car?").
Active Learning for Visual Question Answering: An Empirical Study
We present an empirical study of active learning for Visual Question Answering, where a deep VQA model selects informative question-image pairs from a pool and queries an oracle for answers to maximally improve its performance under a limited query budget.
Deal or No Deal? End-to-End Learning of Negotiation Dialogues
Much of human dialogue occurs in semi-cooperative settings, where agents with different goals attempt to agree on common decisions.
Evaluating Visual Conversational Agents via Cooperative Human-AI Games
This suggests a mismatch between benchmarking of AI in isolation and in the context of human-AI teams.
Deal or No Deal? End-to-End Learning for Negotiation Dialogues
Much of human dialogue occurs in semi-cooperative settings, where agents with different goals attempt to agree on common decisions.
Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
In contrast, discriminative dialog models (D) that are trained to rank a list of candidate human responses outperform their generative counterparts; in terms of automatic metrics, diversity, and informativeness of the responses.
Ranked #8 on
Visual Dialog
on VisDial v0.9 val
ParlAI: A Dialog Research Software Platform
We introduce ParlAI (pronounced "par-lay"), an open-source software platform for dialog research implemented in Python, available at http://parl. ai.
Cooperative Learning with Visual Attributes
Learning paradigms involving varying levels of supervision have received a lot of interest within the computer vision and machine learning communities.
C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
Finally, we evaluate several existing VQA models under this new setting and show that the performances of these models degrade by a significant amount compared to the original VQA setting.
Punny Captions: Witty Wordplay in Image Descriptions
Wit is a form of rich interaction that is often grounded in a specific situation (e. g., a comment in response to an event).
It Takes Two to Tango: Towards Theory of AI's Mind
Surprisingly, we find that having access to the model's internal states - its confidence in its top-k predictions, explicit or implicit attention maps which highlight regions in the image (and words in the question) the model is looking at (and listening to) while answering a question about an image - do not help people better predict its behavior.
Sound-Word2Vec: Learning Word Representations Grounded in Sounds
In this work, we treat sound as a first-class citizen, studying downstream textual tasks which require aural grounding.
LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
We present LR-GAN: an adversarial image generation model which takes scene structure and context into account.
Ranked #4 on
Image Generation
on Stanford Cars
Context-aware Captions from Context-agnostic Supervision
We introduce an inference technique to produce discriminative context-aware image captions (captions that describe differences between images or visual concepts) using only generic context-agnostic training data (captions that describe a concept or an image in isolation).
Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
The model decides whether to attend to the image and where, in order to extract meaningful information for sequential word generation.
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
We propose to counter these language priors for the task of Visual Question Answering (VQA) and make vision (the V in VQA) matter!
Visual Dialog
We introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content.
Ranked #15 on
Visual Dialog
on VisDial v0.9 val
Grad-CAM: Why did you say that?
We propose a technique for making Convolutional Neural Network (CNN)-based models more transparent by visualizing input regions that are 'important' for predictions -- or visual explanations.
Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
For captioning and VQA, we show that even non-attention based models can localize inputs.
Ranked #2 on
Image Attribution
on CUB-200-2011
Measuring Machine Intelligence Through Visual Question Answering
As machines have become more intelligent, there has been a renewed interest in methods for measuring their intelligence.
Towards Transparent AI Systems: Interpreting Visual Question Answering Models
In this paper, we address the problem of interpreting Visual Question Answering (VQA) models.
Deep Learning the City : Quantifying Urban Perception At A Global Scale
Computer vision methods that quantify the perception of urban environment are increasingly being used to study the relationship between a city's physical appearance and the behavior and health of its residents.
Analyzing the Behavior of Visual Question Answering Models
Recently, a number of deep-learning based models have been proposed for the task of Visual Question Answering (VQA).
Sort Story: Sorting Jumbled Images and Captions into Stories
Temporal common sense has applications in AI tasks such as QA, multi-document summarization, and human-AI communication.
Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions
We introduce the novel problem of determining the relevance of questions to images in VQA.
Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images.
Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images.
Hierarchical Question-Image Co-Attention for Visual Question Answering
In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN).
Ranked #3 on
Visual Question Answering (VQA)
on VQA v1 test-std
Leveraging Visual Question Answering for Image-Caption Ranking
This allows the model to interpret images and captions from a wide variety of perspectives.
Joint Unsupervised Learning of Deep Representations and Image Clusters
In this paper, we propose a recurrent framework for Joint Unsupervised LEarning (JULE) of deep representations and image clusters.
Ranked #1 on
Image Clustering
on Coil-20
Visual Storytelling
We introduce the first dataset for sequential vision-to-language, and explore how this data may be used for the task of visual storytelling.
Counting Everyday Objects in Everyday Scenes
In this work, we build dedicated models for counting designed to tackle the large variance in counts, appearances, and scales of objects found in natural scenes.
Ranked #1 on
Object Counting
on COCO count-test
A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories
We created a new corpus of ~50k five-sentence commonsense stories, ROCStories, to enable this evaluation.
We Are Humor Beings: Understanding and Predicting Visual Humor
We collect two datasets of abstract scenes that facilitate the study of humor at both the scene-level and the object-level.
Learning Common Sense Through Visual Abstraction
We show that the commonsense knowledge we learn is complementary to what can be learnt from sources of text.
Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes
While word embeddings trained using text have been extremely successful, they cannot uncover notions of semantic relatedness implicit in our visual world.
Yin and Yang: Balancing and Answering Binary Visual Questions
If the concept can be found in the image, the answer to the question is "yes", and otherwise "no".
WhittleSearch: Interactive Image Search with Relative Attribute Feedback
We propose a novel mode of feedback for image search, where a user describes which properties of exemplar images should be adjusted in order to more closely match his/her mental model of the image sought.
VQA: Visual Question Answering
Given an image and a natural language question about the image, the task is to provide an accurate natural language answer.
Understanding Image Virality
We train classifiers with state-of-the-art image features to predict virality of individual images, relative virality in pairs of images, and the dominant topic of a viral image.
Don't Just Listen, Use Your Imagination: Leveraging Visual Common Sense for Non-Visual Tasks
But much of common sense knowledge is unwritten - partly because it tends not to be interesting enough to talk about, and partly because some common sense is unnatural to articulate in text.
Image Specificity
For some images, descriptions written by multiple people are consistent with each other.
CIDEr: Consensus-based Image Description Evaluation
We propose a novel paradigm for evaluating image descriptions that uses human consensus.
Collecting Image Description Datasets using Crowdsourcing
We describe our two new datasets with images described by humans.