Search Results for author:
Found 26 papers, 15 papers with code
Fusing Pre-Trained Language Models With Multimodal Prompts Through Reinforcement Learning
Language models are capable of commonsense reasoning: while domain-specific models can learn from explicit knowledge (e. g. commonsense graphs [6], ethical norms [25]), and larger models like GPT-3 manifest broad commonsense reasoning capacity.
MAUVE Scores for Generative Models: Theory and Practice
We present MAUVE, a family of comparison measures between pairs of distributions such as those encountered in the generative modeling of text or images.
Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest
We challenge AI models to "demonstrate understanding" of the sophisticated multimodal humor of The New Yorker Caption Contest.
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
We propose Unified-IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region captioning and referring expression, to natural language processing tasks such as question answering and paraphrasing.
Ranked #1 on
Object Segmentation
on GRIT
Multimodal Knowledge Alignment with Reinforcement Learning
Large language models readily adapt to novel settings, even without task-specific training data.
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning
We present Sherlock, an annotated corpus of 103K images for testing machine capacity for abductive reasoning beyond literal image contents.
MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound
Given a video, we replace snippets of text and audio with a MASK token; the model learns by choosing the correct masked-out snippet.
Ranked #4 on
Action Classification
on Kinetics-600
(using extra training data)
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics
To enable constrained generation, we build on NeuroLogic decoding (Lu et al., 2021), combining its flexibility in incorporating logical constraints with A*esque estimates of future constraint satisfaction.
Ranked #1 on
Text Generation
on ROCStories
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer
In a difficult zero-shot setting with no paired audio-text data, our model demonstrates state-of-the-art zero-shot performance on the ESC50 and US8K audio classification tasks, and even surpasses the supervised state of the art for Clotho caption retrieval (with audio queries) by 2. 2\% R@1.
Edited Media Understanding Frames: Reasoning About the Intent and Implications of Visual Misinformation
Understanding manipulated media, from automatically generated {`}deepfakes{'} to manually edited ones, raises novel research challenges.
MERLOT: Multimodal Neural Script Knowledge Models
As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future.
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language.
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
As major progress is made in open-ended text generation, measuring how close machine-generated text is to human language remains a critical open problem.
Edited Media Understanding: Reasoning About Implications of Manipulated Images
The difference between this example, and harmful edits that spread disinformation, is one of intent.
NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints
While the dominant recipe for conditional text generation has been large-scale pretrained language models that are finetuned on the task-specific training data, such models do not learn to follow the underlying constraints reliably, even when supervised with large amounts of task-specific examples.
Probing Contextual Language Models for Common Ground with Visual Representations
The success of large-scale contextual language models has attracted great interest in probing what is encoded in their representations.
TuringAdvice: A Generative and Dynamic Evaluation of Language Use
We propose TuringAdvice, a new challenge task and dataset for language understanding models.
Adversarial Filters of Dataset Biases
Large neural models have demonstrated human-level performance on language and vision benchmarks, while their performance degrades considerably on adversarial or out-of-distribution samples.
PIQA: Reasoning about Physical Commonsense in Natural Language
Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems.
Natural Language Understanding
Physical Commonsense Reasoning
+1
Defending Against Neural Fake News
We find that best current discriminators can classify neural fake news from real, human-written, news with 73% accuracy, assuming access to a moderate level of training data.
Ranked #2 on
Fake News Detection
on Grover-Mega
HellaSwag: Can a Machine Really Finish Your Sentence?
In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset.
From Recognition to Cognition: Visual Commonsense Reasoning
While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world.
Multiple-choice
Multiple Choice Question Answering (MCQA)
+1
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Given a partial description like "she opened the hood of the car," humans can reason about the situation and anticipate what might come next ("then, she examined the engine").
Ranked #4 on
Common Sense Reasoning
on SWAG
Neural Motifs: Scene Graph Parsing with Global Context
We then introduce Stacked Motif Networks, a new architecture designed to capture higher order motifs in scene graphs that further improves over our strong baseline by an average 7. 1% relative gain.
Ranked #6 on
Panoptic Scene Graph Generation
on PSG Dataset
Zero-Shot Activity Recognition with Verb Attribute Induction
In this paper, we investigate large-scale zero-shot activity recognition by modeling the visual and linguistic attributes of action verbs.
MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
This paper introduces to the scientific community the first opinion-level annotated corpus of sentiment and subjectivity analysis in online videos called Multimodal Opinion-level Sentiment Intensity dataset (MOSI).
