Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment

When a speaker, Mary, asks {``}Do you know that Florence is packed with visitors?{''}, we take her to believe that Florence is packed with visitors, but not if she asks {``}Do you think that Florence is packed with visitors?{''}. Inferring speaker commitment (aka event factuality) is crucial for information extraction and question answering. Here, we explore the hypothesis that linguistic deficits drive the error patterns of existing speaker commitment models by analyzing the linguistic correlates of model error on a challenging naturalistic dataset. We evaluate two state-of-the-art speaker commitment models on the CommitmentBank, an English dataset of naturally occurring discourses. The CommitmentBank is annotated with speaker commitment towards the content of the complement ({``}Florence is packed with visitors{''} in our example) of clause-embedding verbs ({``}know{''}, {``}think{''}) under four entailment-canceling environments (negation, modal, question, conditional). A breakdown of items by linguistic features reveals asymmetrical error patterns: while the models achieve good performance on some classes (e.g., negation), they fail to generalize to the diverse linguistic constructions (e.g., conditionals) in natural language, highlighting directions for improvement.