Search Results for author:
Found 37 papers, 21 papers with code
Probing Cross-Modal Representations in Multi-Step Relational Reasoning
We investigate the representations learned by vision and language models in tasks that require relational reasoning.
Is Information Density Uniform in Task-Oriented Dialogues?
The Uniform Information Density principle states that speakers plan their utterances to reduce fluctuations in the density of the information transmitted.
Less Descriptive yet Discriminative: Quantifying the Properties of Multimodal Referring Utterances via CLIP
In this work, we use a transformer-based pre-trained multimodal model, CLIP, to shed light on the mechanisms employed by human speakers when referring to visual entities.
Analysing Human Strategies of Information Transmission as a Function of Discourse Context
Speakers are thought to use rational information transmission strategies for efficient communication (Genzel and Charniak, 2002; Aylett and Turk, 2004; Jaeger and Levy, 2007).
AnaLog: Testing Analytical and Deductive Logic Learnability in Language Models
We investigate the extent to which pre-trained language models acquire analytical and deductive logical reasoning capabilities as a side effect of learning word prediction.
Leveraging Speech for Gesture Detection in Multimodal Communication
Our findings indicate that expanding the speech buffer beyond visual time segments improves performance and that multimodal integration using cross-modal and early fusion techniques outperforms baseline methods using unimodal and late fusion methods.
Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?
With the rise of increasingly powerful and user-facing NLP systems, there is growing interest in assessing whether they have a good representation of uncertainty by evaluating the quality of their predictive distribution over outcomes.
Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions
Clarification questions are an essential dialogue tool to signal misunderstanding, ambiguities, and under-specification in language use.
Describing Images $\textit{Fast and Slow}$: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes
There is an intricate relation between the properties of an image and how humans behave while describing the image.
GROOViST: A Metric for Grounding Objects in Visual Storytelling
A proper evaluation of stories generated for a sequence of images -- the task commonly referred to as visual storytelling -- must consider multiple aspects, such as coherence, grammatical correctness, and visual grounding.
The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models
Despite the impressive performance achieved by pre-trained language-and-vision models in downstream tasks, it remains an open question whether this reflects a proper understanding of image-text interaction.
Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives
We present information value, a measure which quantifies the predictability of an utterance relative to a set of plausible alternatives.
Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
Finally, we conduct a case study on CLC when new factual associations are inserted in the PLMs via model editing.
Co-Speech Gesture Detection through Multi-Phase Sequence Labeling
Yet, the prevalent approach to automatic gesture detection treats the problem as binary classification, classifying a segment as either containing a gesture or not, thus failing to capture its inherently sequential and contextual nature.
Uncertainty in Natural Language Generation: From Theory to Applications
Recent advances of powerful Language Models have allowed Natural Language Generation (NLG) to emerge as an important technology that can not only perform traditional tasks like summarisation or translation, but also serve as a natural language interface to a variety of applications.
Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind
Inspired by psycholinguistic theories, we endow our speaker with the ability to adapt its referring expressions via a simulation module that monitors the effectiveness of planned utterances from the listener's perspective.
What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability
In Natural Language Generation (NLG) tasks, for any input, multiple communicative goals are plausible, and any goal can be put into words, or produced, in multiple ways.
Stop Measuring Calibration When Humans Disagree
Calibration is a popular framework to evaluate whether a classifier knows when it does not know - i. e., its predictive probabilities are a good indication of how likely a prediction is to be correct.
Construction Repetition Reduces Information Rate in Dialogue
We hypothesise that speakers use construction repetition to mitigate information rate, leading to an overall decrease in utterance information content over the course of a dialogue.
Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations
We investigate the extent to which modern, neural language models are susceptible to structural priming, the phenomenon whereby the structure of a sentence makes the same structure more probable in a follow-up sentence.
Words are the Window to the Soul: Language-based User Representations for Fake News Detection
Cognitive and social traits of individuals are reflected in language use.
Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts
We propose a generation model that produces referring utterances grounded in both the visual and the conversational context.
Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze
When speakers describe an image, they tend to look at objects before mentioning them.
Analysing Lexical Semantic Change with Contextualised Word Representations
This paper presents the first unsupervised approach to lexical semantic change that makes use of contextualised word representations.
You Shall Know a User by the Company It Keeps: Dynamic Representations for Social Media Users in NLP
Information about individuals can help to better understand what they say, particularly in social media where texts are short.
Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts
This work aims at modeling how the meaning of gradable adjectives of size (`big', `small') can be learned from visually-grounded contexts.
Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering
We study the issue of catastrophic forgetting in the context of neural multimodal approaches to Visual Question Answering (VQA).
The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue
This paper introduces the PhotoBook dataset, a large-scale collection of visually-grounded, task-oriented dialogues in English designed to investigate shared dialogue history accumulating during conversation.
Evaluating the Representational Hub of Language and Vision Models
The multimodal models used in the emerging field at the intersection of computational linguistics and computer vision implement the bottom-up processing of the `Hub and Spoke' architecture proposed in cognitive science to represent how the brain processes and combines multi-sensory inputs.
Automatic Evaluation of Neural Personality-based Chatbots
Stylistic variation is critical to render the utterances generated by conversational agents natural and engaging.
Short-Term Meaning Shift: A Distributional Exploration
We present the first exploration of meaning shift over short periods of time in online communities using distributional representations.
Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat
We compare our approach to an alternative system which extends the baseline with reinforcement learning.
Analysing the potential of seq-to-seq models for incremental interpretation in task-oriented dialogue
We investigate how encoder-decoder models trained on a synthetic dataset of task-oriented dialogues process disfluencies, such as hesitations and self-corrections.
The Road to Success: Assessing the Fate of Linguistic Innovations in Online Communities
We investigate the birth and diffusion of lexical innovations in a large dataset of online social communities.
Semantic Variation in Online Communities of Practice
We introduce a framework for quantifying semantic variation of common words in Communities of Practice and in sets of topic-related communities.
Examining a hate speech corpus for hate speech detection and popularity prediction
As research on hate speech becomes more and more relevant every day, most of it is still focused on hate speech detection.
The LAMBADA dataset: Word prediction requiring a broad discourse context
We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task.
