Search Results for author:
Found 92 papers, 6 papers with code
A Continuation Semantics for Abstract Meaning Representation
Abstract Meaning Representation (AMR) is a simple, expressive semantic framework whose emphasis on predicate-argument structure is effective for many tasks.
The CLAMS Platform at Work: Processing Audiovisual Data from the American Archive of Public Broadcasting
The Computational Linguistics Applications for Multimedia Services (CLAMS) platform provides access to computational content analysis tools for multimedia material.
Abstract Meaning Representation for Gesture
This paper presents Gesture AMR, an extension to Abstract Meaning Representation (AMR), that captures the meaning of gesture.
The VoxWorld Platform for Multimodal Embodied Agents
We present a five-year retrospective on the development of the VoxWorld platform, first introduced as a multimodal platform for modeling motion language, that has evolved into a platform for rapidly building and deploying embodied agents with contextual and situational awareness, capable of interacting with humans in multiple modalities, and exploring their environments.
Evaluating Retrieval for Multi-domain Scientific Publications
This paper provides an overview of the xDD/LAPPS Grid framework and provides results of evaluating the AskMe retrievalengine using the BEIR benchmark datasets.
Competence-based Question Generation
Models of natural language understanding often rely on question answering and logical inference benchmark challenges to evaluate the performance of a system.
AskMe: A LAPPS Grid-based NLP Query and Retrieval System for Covid-19 Literature
In a recent project, the Language Application Grid was augmented to support the mining of scientific publications.
SemEval-2022 Task 9: R2VQ – Competence-based Multimodal Question Answering
In this task, we identify a challenge that is reflective of linguistic and cognitive competencies that humans have when speaking and reasoning.
TRACE: Real-Time Multimodal Common Ground Tracking in Situated Collaborative Dialogues
We present TRACE, a novel system for live *common ground* tracking in situated collaborative tasks.
Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement
Our goal is to develop an AI Partner that can provide support for group problem solving and social dynamics.
Linguistically Conditioned Semantic Textual Similarity
We also propose a new method that largely improves the performance over baselines on the C-STS data by training the models with the answers.
Computational Thought Experiments for a More Rigorous Philosophy and Science of the Mind
We offer philosophical motivations for a method we call Virtual World Cognitive Science (VW CogSci), in which researchers use virtual embodied agents that are embedded in virtual worlds to explore questions in the field of Cognitive Science.
ChainNet: Structured Metaphor and Metonymy in WordNet
The senses of a word exhibit rich internal structure.
Common Ground Tracking in Multimodal Dialogue
Within Dialogue Modeling research in AI and NLP, considerable attention has been spent on ``dialogue state tracking'' (DST), which is the ability to update the representations of the speaker's needs at each turn in the dialogue by taking into account the past dialogue moves and history.
An Abstract Specification of VoxML as an Annotation Language
VoxML is a modeling language used to map natural language expressions into real-time visualizations using commonsense semantic knowledge of objects and events.
Causal schema induction for knowledge discovery
In this work, we investigate how to apply schema induction models to the task of knowledge discovery for enhanced search of English-language news texts.
Dense Paraphrasing for Textual Enrichment
Understanding inferences and answering questions from text requires more than merely recovering surface arguments, adjuncts, or strings associated with the query terms.
Representing Inferences and their Lexicalization
We have recently begun a project to develop a more effective and efficient way to marshal inferences from background knowledge to facilitate deep natural language understanding.
Neural Metaphor Detection with Visibility Embeddings
We present new results for the problem of sequence metaphor labeling, using the recently developed Visibility Embeddings.
Designing Multimodal Datasets for NLP Challenges
In this paper, we argue that the design and development of multimodal datasets for natural language processing (NLP) challenges should be enhanced in two significant respects: to more broadly represent commonsense semantic inferences; and to better reflect the dynamics of actions and events, through a substantive alignment of textual and visual information.
Neurosymbolic AI for Situated Language Understanding
In recent years, data-intensive AI, particularly the domain of natural language processing and understanding, has seen significant progress driven by the advent of large datasets and deep neural networks that have sidelined more classic AI approaches to the field.
A Two-Level Interpretation of Modality in Human-Robot Dialogue
We analyze the use and interpretation of modal expressions in a corpus of situated human-robot dialogue and ask how to effectively represent these expressions for automatic learning.
Representation, Learning and Reasoning on Spatial Language for Downstream NLP Tasks
Understating spatial semantics expressed in natural language can become highly complex in real-world applications.
Situated Multimodal Control of a Mobile Robot: Navigation through a Virtual Environment
We present a new interface for controlling a navigation robot in novel environments using coordinated gesture and language.
Exploration and Discovery of the COVID-19 Literature through Semantic Visualization
We are developing semantic visualization techniques in order to enhance exploration and enable discovery over large datasets of complex networks of relations.
COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation
To combat COVID-19, both clinicians and scientists need to digest vast amounts of relevant biomedical knowledge in scientific literature to understand the disease mechanism and related biological functions.
Interchange Formats for Visualization: LIF and MMIF
Promoting interoperrable computational linguistics (CL) and natural language processing (NLP) application platforms and interchange-able data formats have contributed improving discoverabilty and accessbility of the openly available NLP software.
Reproducing Neural Ensemble Classifier for Semantic Relation Extraction inScientific Papers
Within the natural language processing (NLP) community, shared tasks play an important role.
Improving Neural Metaphor Detection with Visual Datasets
We present new results on Metaphor Detection by using text from visual datasets.
A Formal Analysis of Multimodal Referring Strategies Under Common Ground
In this paper, we present an analysis of computationally generated mixed-modality definite referring expressions using combinations of gesture and linguistic descriptions.
Assessing the Efficacy of Clinical Sentiment Analysis and Topic Extraction in Psychiatric Readmission Risk Prediction
Predicting which patients are more likely to be readmitted to a hospital within 30 days after discharge is a valuable piece of information in clinical decision-making.
Multimodal Continuation-style Architectures for Human-Robot Interaction
We present an architecture for integrating real-time, multimodal input into a computational agent's contextual model.
VerbNet Representations: Subevent Semantics for Transfer Verbs
This paper announces the release of a new version of the English lexical resource VerbNet with substantially revised semantic representations designed to facilitate computer planning and reasoning based on human language.
Modeling Quantification and Scope in Abstract Meaning Representations
In this paper, we propose an extension to Abstract Meaning Representations (AMRs) to encode scope information of quantifiers and negation, in a way that overcomes the semantic gaps of the schema while maintaining its cognitive simplicity.
Computational Linguistics Applications for Multimedia Services
We present Computational Linguistics Applications for Multimedia Services (CLAMS), a platform that provides access to computational content analysis tools for archival multimedia material that appear in different media, such as text, audio, image, and video.
A Dynamic Semantics for Causal Counterfactuals
Under the standard approach to counterfactuals, to determine the meaning of a counterfactual sentence, we consider the {``}closest{''} possible world(s) where the antecedent is true, and evaluate the consequent.
Generating a Novel Dataset of Multimodal Referring Expressions
Referring expressions and definite descriptions of objects in space exploit information both about object characteristics and locations.
Distinguishing Clinical Sentiment: The Importance of Domain Adaptation in Psychiatric Patient Health Records
In this study, we undertook, to our knowledge, the first domain adaptation of sentiment analysis to psychiatric EHRs by defining psychiatric clinical sentiment, performing an annotation project, and evaluating multiple sentence-level sentiment machine learning (ML) models.
Situational Grounding within Multimodal Simulations
In this paper, we argue that simulation platforms enable a novel type of embodied spatial reasoning, one facilitated by a formal model of object and event semantics that renders the continuous quantitative search space of an open-world, real-time environment tractable.
Combining Deep Learning and Qualitative Spatial Reasoning to Learn Complex Structures from Sparse Examples with Noise
We present a novel approach to introducing new spatial structures to an AI agent, combining deep learning over qualitative spatial relations with various heuristic search algorithms.
Analysis of Risk Factor Domains in Psychosis Patient Health Records
Readmission after discharge from a hospital is disruptive and costly, regardless of the reason.
Every Object Tells a Story
Most work within the computational event modeling community has tended to focus on the interpretation and ordering of events that are associated with verbs and event nominals in linguistic expressions.
Integrating Vision and Language Datasets to Measure Word Concreteness
We present and take advantage of the inherent visualizability properties of words in visual corpora (the textual components of vision-language datasets) to compute concreteness scores for words.
Learning event representation: As sparse as possible, but not sparser
Selecting an optimal event representation is essential for event classification in real world contexts.
Fine-grained Event Learning of Human-Object Interaction with LSTM-CRF
Event learning is one of the most important problems in AI.
General Classification
Human-Object Interaction Detection
+1
SemEval-2017 Task 12: Clinical TempEval
Clinical TempEval 2017 aimed to answer the question: how well do systems trained on annotated timelines for one medical condition (colon cancer) perform in predicting timelines on another medical condition (brain cancer)?
Building Multimodal Simulations for Natural Language
Simulation and automatic visualization of events from natural language descriptions and supplementary modalities, such as gestures, allows humans to use their native capabilities as linguistic and visual interpreters to collaborate on tasks with an artificial agent or to put semantic intuitions to the test in an environment where user and agent share a common context. In previous work (Pustejovsky and Krishnaswamy, 2014; Pustejovsky, 2013a), we introduced a method for modeling natural language expressions within a 3D simulation environment built on top of the game development platform Unity (Goldstone, 2009).
The Development of Multimodal Lexical Resources
Human communication is a multimodal activity, involving not only speech and written expressions, but intonation, images, gestures, visual clues, and the interpretation of actions through perception.
VoxSim: A Visual Platform for Modeling Motion Language
Much existing work in text-to-scene generation focuses on generating static scenes.
LAPPS/Galaxy: Current State and Next Steps
The US National Science Foundation (NSF) SI2-funded LAPPS/Galaxy project has developed an open-source platform for enabling complex analyses while hiding complexities associated with underlying infrastructure, that can be accessed through a web interface, deployed on any Unix system, or run from the cloud.
Generating Simulations of Motion Events from Verbal Descriptions
The generated simulations act as a conceptual "debugger" for the semantics of different motion verbs: that is, by testing for consistency and informativeness in the model, simulations expose the presuppositions associated with linguistic expressions and their compositions.
VoxML: A Visualization Modeling Language
We present the specification for a modeling language, VoxML, which encodes semantic knowledge of real-world objects represented as three-dimensional models, and of events and attributes related to and enacted over these objects.
ECAT: Event Capture Annotation Tool
This paper introduces the Event Capture Annotation Tool (ECAT), a user-friendly, open-source interface tool for annotating events and their participants in video, capable of extracting the 3D positions and orientations of objects in video captured by Microsoft's Kinect(R) hardware.
Multimodal Semantic Simulations of Linguistically Underspecified Motion Events
In this paper, we describe a system for generating three-dimensional visual simulations of natural language motion expressions.
Annotation Methodologies for Vision and Language Dataset Creation
Annotated datasets are commonly used in the training and evaluation of tasks involving natural language and vision (image description generation, action recognition and visual question answering).
The Language Application Grid and Galaxy
The NSF-SI2-funded LAPPS Grid project is a collaborative effort among Brandeis University, Vassar College, Carnegie-Mellon University (CMU), and the Linguistic Data Consortium (LDC), which has developed an open, web-based infrastructure through which resources can be easily accessed and within which tailored language services can be efficiently composed, evaluated, disseminated and consumed by researchers, developers, and students across a wide variety of disciplines.
Identification of Technology Terms in Patents
Natural language analysis of patents holds promise for the development of tools designed to assist analysts in the monitoring of emerging technologies.
Image Annotation with ISO-Space: Distinguishing Content from Structure
This paper explores the use of ISO-Space, an annotation specification to capturing spatial information, for encoding spatial relations mentioned in descriptions of images.
The Language Application Grid
The Language Application (LAPPS) Grid project is establishing a framework that enables language service discovery, composition, and reuse and promotes sustainability, manageability, usability, and interoperability of natural language Processing (NLP) components.
Clinical TempEval
We describe the Clinical TempEval task which is currently in preparation for the SemEval-2015 evaluation exercise.
Temporal Annotation in the Clinical Domain
The corpus is available to the community and has been proposed for use in a SemEval 2015 task.
TempEval-3: Evaluating Events, Time Expressions, and Temporal Relations
We describe the TempEval-3 task which is currently in preparation for the SemEval-2013 evaluation exercise.
The TARSQI Toolkit
We present and demonstrate the updated version of the TARSQI Toolkit, a suite of temporal processing modules that extract temporal information from natural language texts.
ATLIS: Identifying Locational Information in Text Automatically
ATLIS (short for ATLIS Tags Locations in Strings) is a tool being developed using a maximum-entropy machine learning model for automatically identifying information relating to spatial and locational information in natural language text.
The Role of Model Testing in Standards Development: The Case of ISO-Space
In this paper, we describe the methodology being used to develop certain aspects of ISO-Space, an annotation language for encoding spatial and spatiotemporal information as expressed in natural language text.
Word Sense Inventories by Non-Experts.
In this paper, we explore different strategies for implementing a crowdsourcing methodology for a single-step construction of an empirically-derived sense inventory and the corresponding sense-annotated corpus.
SemEval-2010 Task 13: TempEval-2
Tempeval-2 comprises evaluation tasks for time expressions, events and temporal relations, the latter of which was split up in four sub tasks, motivated by the notion that smaller subtasks would make both data preparation and temporal relation extraction easier.
