Search Results for author:
Found 56 papers, 24 papers with code
Investigating Negation in Pre-trained Vision-and-language Models
Pre-trained vision-and-language models have achieved impressive results on a variety of tasks, including ones that require complex reasoning beyond object recognition.
PosterSum: A Multimodal Benchmark for Scientific Poster Summarization
We benchmark state-of-the-art Multimodal Large Language Models (MLLMs) on PosterSum and demonstrate that they struggle to accurately interpret and summarize scientific posters.
End-to-End Long Document Summarization using Gradient Caching
Training transformer-based encoder-decoder models for long document summarization poses a significant challenge due to the quadratic memory consumption during training.
Ranked #1 on
Long-Form Narrative Summarization
on SummScreen
Plancraft: an evaluation dataset for planning with LLM agents
We present Plancraft, a multi-modal evaluation dataset for LLM agents.
BookWorm: A Dataset for Character Description and Analysis
Using this dataset, we evaluate state-of-the-art long-context models in zero-shot and fine-tuning settings, utilizing both retrieval-based and hierarchical processing for book-length inputs.
Generating Visual Stories with Grounded and Coreferent Characters
We also propose new evaluation metrics to measure the richness of characters and coreference in stories.
MovieSum: An Abstractive Summarization Dataset for Movie Screenplays
Movie screenplay summarization is challenging, as it requires an understanding of long input contexts and various elements unique to movies.
Ranked #3 on
Long-Form Narrative Summarization
on MovieSum
Coarse or Fine? Recognising Action End States without Labels
We use our synthetic data to train a model based on UNet and test it on real images showing coarsely/finely cut objects.
Select and Summarize: Scene Saliency for Movie Script Summarization
Abstractive summarization for long-form narrative texts such as movie scripts is challenging due to the computational and memory constraints of current language models.
Ranked #6 on
Long-Form Narrative Summarization
on MENSA
Abstractive Text Summarization
Long-Form Narrative Summarization
Improving Explicit Spatial Relationships in Text-to-Image Generation through an Automatically Derived Dataset
We hypothesize that this is because explicit spatial relations rarely appear in the image captions used to train these models.
Efficient Pre-training for Localized Instruction Generation of Videos
To mitigate these issues, we propose a novel technique, Sieve-&-Swap, to automatically generate high-quality training data for the recipe domain: (i) Sieve: filters irrelevant transcripts and (ii) Swap: acquires high-quality text by replacing transcripts with human-written instruction from a text-only recipe dataset.
Semi-supervised multimodal coreference resolution in image narrations
In this paper, we study multimodal coreference resolution, specifically where a longer descriptive text, i. e., a narration is paired with an image.
Visual Storytelling with Question-Answer Plans
Our model translates the image sequence into a visual prefix, a sequence of continuous embeddings which language models can interpret.
Dynamic Planning with a LLM
While Large Language Models (LLMs) can solve many NLP tasks in zero-shot settings, applications involving embodied agents remain problematic.
Meta-learning For Vision-and-language Cross-lingual Transfer
Current pre-trained vison-language models (PVLMs) achieve excellent performance on a range of multi-modal datasets.
Detecting and Grounding Important Characters in Visual Stories
Characters are essential to the plot of any story.
Learning Action Changes by Measuring Verb-Adverb Textual Relationships
The goal of this work is to understand the way actions are performed in videos.
Ranked #2 on
Video-Adverb Retrieval
on HowTo100M Adverbs
Learning the Effects of Physical Actions in a Multi-modal Environment
However, predicting the effects of an action before it is executed is crucial in planning, where coherent sequences of actions are often needed to achieve a goal.
Who are you referring to? Coreference resolution in image narrations
Coreference resolution aims to identify words and phrases which refer to same entity in a text, a core task in natural language processing.
A Closer Look at Temporal Ordering in the Segmentation of Instructional Videos
We refer to this task as Procedure Segmentation and Summarization (PSS).
Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognition
We propose to learn what makes a good video for action recognition and select only high-quality samples for augmentation.
Ranked #2 on
Few Shot Action Recognition
on HMDB51
Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation
Scene graph generation (SGG) aims to capture a wide variety of interactions between pairs of objects, which is essential for full scene understanding.
Finding the Right Moment: Human-Assisted Trailer Creation via Task Composition
A main advantage of our algorithm is that it uses interpretable criteria, which allows us to deploy it in an interactive tool for trailer creation with a human in the loop.
Memory and Knowledge Augmented Language Models for Inferring Salience in Long-Form Stories
Measuring event salience is essential in the understanding of stories.
A New Split for Evaluating True Zero-Shot Action Recognition
We benchmark several recent approaches on the proposed True Zero-Shot(TruZe) Split for UCF101 and HMDB51, with zero-shot and generalized zero-shot evaluation.
CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition
Theproblem can be seen as learning a function which general-izes well to instances of unseen classes without losing dis-crimination between classes.
Ranked #2 on
Zero-Shot Action Recognition
on Olympics
Movie Summarization via Sparse Graph Construction
We summarize full-length movies by creating shorter videos containing their most informative scenes.
Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads
Here, we propose a novel fully unsupervised parsing approach that extracts constituency trees from PLM attention heads.
Modelling Suspense in Short Stories as Uncertainty Reduction over Neural Representation
Suspense is a crucial ingredient of narrative fiction, engaging readers and making stories compelling.
Screenplay Summarization Using Latent Narrative Structure
Most general-purpose extractive summarization models are trained on news articles, which are short and present all important information upfront.
Movie Plot Analysis via Turning Point Identification
According to screenwriting theory, turning points (e. g., change of plans, major setback, climax) are crucial narrative moments within a screenplay: they define the plot structure, determine its progression and segment the screenplay into thematic units (e. g., setup, complications, aftermath).
An Imitation Learning Approach to Unsupervised Parsing
In our work, we propose an imitation learning approach to unsupervised parsing, where we transfer the syntactic knowledge induced by the PRPN to a Tree-LSTM model with discrete parsing actions.
Cross-lingual Visual Verb Sense Disambiguation
We extend this line of work to the more challenging task of cross-lingual verb sense disambiguation, introducing the MultiSense dataset of 9, 504 images annotated with English, German, and Spanish verbs.
Character-based Surprisal as a Model of Reading Difficulty in the Presence of Error
Also, transpositions are more difficult than misspellings, and a high error rate increases difficulty for all words, including correct ones.
Dependency Grammar Induction with a Neural Variational Transition-based Parser
Transition-based models enable faster inference with $O(n)$ time complexity, but their performance still lags behind.
Modeling Task Effects in Human Reading with Neural Network-based Attention
Research on human reading has long documented that reading behavior shows task-specific effects, but it has been challenging to build general models predicting what reading behavior humans will show in a given task.
An Evaluation of Image-Based Verb Prediction Models against Human Eye-Tracking Data
Recent research in language and vision has developed models for predicting and disambiguating verbs from images.
Extreme clicking for efficient object annotation
We crowd-source extreme point annotations for PASCAL VOC 2007 and 2012 and show that (1) annotation time is only 7s per box, 5x faster than the traditional way of drawing boxes [62]; (2) the quality of the boxes is as good as the original ground-truth drawn the traditional way; (3) detectors trained on our annotations are as accurate as those trained on the original ground-truth.
Image Pivoting for Learning Multilingual Multimodal Representations
In this paper we propose a model to learn multimodal multilingual representations for matching images and sentences in different languages, with the aim of advancing multilingual versions of image search and image understanding.
An Analysis of Action Recognition Datasets for Language and Vision Tasks
A large amount of recent research has focused on tasks that combine language and vision, resulting in a proliferation of datasets and methods.
Training object class detectors with click supervision
Training object class detectors typically requires a large set of images with objects annotated by bounding boxes.
Cross-lingual Transfer of Correlations between Parts of Speech and Gaze Features
Several recent studies have shown that eye movements during reading provide information about grammatical and syntactic processing, which can assist the induction of NLP models.
Modeling Human Reading with Neural Attention
When humans read text, they fixate some words and skip others.
Unsupervised Visual Sense Disambiguation for Verbs using Multimodal Embeddings
We introduce a new task, visual sense disambiguation for verbs: given an image and a verb, assign the correct sense of the verb, i. e., the one that describes the action depicted in the image.
We don't need no bounding-boxes: Training object class detectors using only human verification
Training object class detectors typically requires a large set of images in which objects are annotated by bounding-boxes.
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities.
Incremental Tree Substitution Grammar for Parsing and Sentence Prediction
In this paper, we present the first incremental parser for Tree Substitution Grammar (TSG).
