Search Results for author:
Found 46 papers, 27 papers with code
Visual Dialog
We introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content.
Ranked #15 on
Visual Dialog
on VisDial v0.9 val
Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
For captioning and VQA, we show that even non-attention based models can localize inputs.
Grad-CAM: Why did you say that?
We propose a technique for making Convolutional Neural Network (CNN)-based models more transparent by visualizing input regions that are 'important' for predictions -- or visual explanations.
Embodied Question Answering
We present a new AI task -- Embodied Question Answering (EmbodiedQA) -- where an agent is spawned at a random location in a 3D environment and asked a question ("What color is the car?").
Neural Modular Control for Embodied Question Answering
We use imitation learning to warm-start policies at each level of the hierarchy, dramatically increasing sample efficiency, followed by reinforcement learning.
The Open Catalyst 2020 (OC20) Dataset and Community Challenges
Catalyst discovery and optimization is key to solving many societal and energy challenges including solar fuels synthesis, long-term energy storage, and renewable fertilizer production.
Rotation Invariant Graph Neural Networks using Spin Convolutions
We introduce a novel approach to modeling angular information between sets of neighboring atoms in a graph neural network.
Ranked #3 on
Initial Structure to Relaxed Energy (IS2RE)
on OC20
GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets
This work investigates this question by first developing the GemNet-OC model based on the large Open Catalyst 2020 (OC20) dataset.
Ranked #1 on
Initial Structure to Relaxed Energy (IS2RE)
on OC20
The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts
The development of machine learning models for electrocatalysts requires a broad set of training data to enable their use across a wide variety of materials.
Spherical Channels for Modeling Atomic Interactions
We propose the Spherical Channel Network (SCN) to model atomic energies and forces.
AdsorbML: A Leap in Efficiency for Adsorption Energy Calculations using Generalizable Machine Learning Potentials
Computational catalysis is playing an increasingly significant role in the design of catalysts across a wide range of applications.
The Open DAC 2023 Dataset and Challenges for Sorbent Discovery in Direct Air Capture
We also trained state-of-the-art ML models on this dataset to approximate calculations at the DFT level.
Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
Specifically, we pose a cooperative 'image guessing' game between two agents -- Qbot and Abot -- who communicate in natural language dialog so that Qbot can select an unseen image from a lineup of images.
EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations
Equivariant Transformers such as Equiformer have demonstrated the efficacy of applying Transformers to the domain of 3D atomistic systems.
Towards Training Billion Parameter Graph Neural Networks for Atomic Simulations
Recent progress in Graph Neural Networks (GNNs) for modeling atomic simulations has the potential to revolutionize catalyst discovery, which is a key step in making progress towards the energy breakthroughs needed to combat climate change.
Ranked #2 on
Initial Structure to Relaxed Energy (IS2RE)
on OC20
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline
Next, we find that additional finetuning using "dense" annotations in VisDial leads to even higher NDCG -- more than 10% over our base model -- but hurts MRR -- more than 17% below our base model!
Feel The Music: Automatically Generating A Dance For An Input Song
We encode intuitive, flexible heuristics for what a 'good' dance is: the structure of the dance should align with the structure of the music.
Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7
Scene-aware dialog systems will be able to have conversations with users about the objects and events around them.
End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features
We introduce a new dataset of dialogs about videos of human behaviors.
PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav
We find that BC$\rightarrow$RL on human demonstrations outperforms BC$\rightarrow$RL on SP and FE trajectories, even when controlled for same BC-pretraining success on train, and even on a subset of val episodes where BC-pretraining success favors the SP or FE policies.
Multi-Image Steganography Using Deep Neural Networks
Steganography is the science of hiding a secret message within an ordinary public message.
Auxiliary Tasks and Exploration Enable ObjectNav
We instead re-enable a generic learned agent by adding auxiliary learning tasks and an exploration reward.
Ranked #2 on
Robot Navigation
on Habitat 2020 Object Nav test-std
Improving Generative Visual Dialog by Answering Diverse Questions
Prior work on training generative Visual Dialog models with reinforcement learning(Das et al.) has explored a Qbot-Abot image-guessing game and shown that this 'self-talk' approach can lead to improved performance at the downstream dialog-conditioned image-guessing task.
Auxiliary Tasks Speed Up Learning PointGoal Navigation
PointGoal Navigation is an embodied task that requires agents to navigate to a specified point in an unseen environment.
Detecting Hate Speech in Multi-modal Memes
A crucial characteristic of the challenge is that it includes "benign confounders" to counter the possibility of models exploiting unimodal priors.
Smart Refrigerator using Internet of Things and Android
The kitchen is regarded as the central unit of the traditional as well as modern homes.
Human-Computer Interaction
Evaluating Visual Conversational Agents via Cooperative Human-AI Games
This suggests a mismatch between benchmarking of AI in isolation and in the context of human-AI teams.
Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images.
Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images.
TarMAC: Targeted Multi-Agent Communication
We propose a targeted communication architecture for multi-agent reinforcement learning, where agents learn both what messages to send and whom to address them to while performing cooperative tasks in partially-observable environments.
Impact of Data Normalization on Deep Neural Network for Time Series Forecasting
The time series forecasting has a great impact on our socio-economic environment.
Connecting Language and Vision to Actions
A long-term goal of AI research is to build intelligent agents that can see the rich visual environment around us, communicate this understanding in natural language to humans and other agents, and act in a physical or embodied environment.
Response to "Visual Dialogue without Vision or Dialogue" (Massiceti et al., 2018)
In a recent workshop paper, Massiceti et al. presented a baseline model and subsequent critique of Visual Dialog (Das et al., CVPR 2017) that raises what we believe to be unfounded concerns about the dataset and evaluation.
Embodied Question Answering in Photorealistic Environments with Point Cloud Perception
To help bridge the gap between internet vision-style problems and the goal of vision for embodied perception we instantiate a large-scale navigation task -- Embodied Question Answering [1] in photo-realistic environments (Matterport 3D).
IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL
We propose a novel framework to identify sub-goals useful for exploration in sequential decision making tasks under partial observability.
Probing Emergent Semantics in Predictive Agents via Question Answering
Recent work has shown how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments.
ForceNet: A Graph Neural Network for Large-Scale Quantum Chemistry Simulation
We use ForceNet to perform quantum chemistry simulations, where ForceNet is able to achieve 4x higher success rate than existing ML models.
An Introduction to Electrocatalyst Design using Machine Learning for Renewable Energy Storage
As we increase our reliance on renewable energy sources such as wind and solar, which produce intermittent power, storage is needed to transfer power from times of peak generation to peak demand.
ForceNet: A Graph Neural Network for Large-Scale Quantum Calculations
By not imposing explicit physical constraints, we can flexibly design expressive models while maintaining their computational efficiency.
Auxiliary Tasks and Exploration Enable ObjectGoal Navigation
We instead re-enable a generic learned agent by adding auxiliary learning tasks and an exploration reward.
ABSA-Bench: Towards the Unified Evaluation of Aspect-based Sentiment Analysis Research
Aspect-Based Sentiment Analysis (ABSA)has gained much attention in recent years.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
+1
NarrationBot and InfoBot: A Hybrid System for Automated Video Description
To overcome the increasing gaps in video accessibility, we developed a hybrid system of two tools to 1) automatically generate descriptions for videos and 2) provide answers or additional descriptions in response to user queries on a video.
DS-VIC: Unsupervised Discovery of Decision States for Transfer in RL
We learn to identify decision states, namely the parsimonious set of states where decisions meaningfully affect the future states an agent can reach in an environment.
Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale
We present a large-scale study of imitating human demonstrations on tasks that require a virtual robot to search for objects in new environments -- (1) ObjectGoal Navigation (e. g. 'find & go to a chair') and (2) Pick&Place (e. g. 'find mug, pick mug, find counter, place mug on counter').
Generalizing Denoising to Non-Equilibrium Structures Improves Equivariant Force Fields
We study the effectiveness of training equivariant networks with DeNS on OC20, OC22 and MD17 datasets and demonstrate that DeNS can achieve new state-of-the-art results on OC20 and OC22 and significantly improve training efficiency on MD17.
