1 code implementation • DMR (COLING) 2020 • Kenneth Lai, Lucia Donatelli, James Pustejovsky
Abstract Meaning Representation (AMR) is a simple, expressive semantic framework whose emphasis on predicate-argument structure is effective for many tasks.
no code implementations • LREC 2022 • Marc Verhagen, Kelley Lynch, Kyeongmin Rim, James Pustejovsky
The Computational Linguistics Applications for Multimedia Services (CLAMS) platform provides access to computational content analysis tools for multimedia material.
no code implementations • LREC 2022 • Richard Brutti, Lucia Donatelli, Kenneth Lai, James Pustejovsky
This paper presents Gesture AMR, an extension to Abstract Meaning Representation (AMR), that captures the meaning of gesture.
no code implementations • LREC 2022 • Nikhil Krishnaswamy, William Pickard, Brittany Cates, Nathaniel Blanchard, James Pustejovsky
We present a five-year retrospective on the development of the VoxWorld platform, first introduced as a multimodal platform for modeling motion language, that has evolved into a platform for rapidly building and deploying embodied agents with contextual and situational awareness, capable of interacting with humans in multiple modalities, and exploring their environments.
no code implementations • LREC 2022 • Nancy Ide, Keith Suderman, Jingxuan Tu, Marc Verhagen, Shanan Peters, Ian Ross, John Lawson, Andrew Borg, James Pustejovsky
This paper provides an overview of the xDD/LAPPS Grid framework and provides results of evaluating the AskMe retrievalengine using the BEIR benchmark datasets.
no code implementations • COLING 2022 • Jingxuan Tu, Kyeongmin Rim, James Pustejovsky
Models of natural language understanding often rely on question answering and logical inference benchmark challenges to evaluate the performance of a system.
no code implementations • EMNLP (NLP-COVID19) 2020 • Keith Suderman, Nancy Ide, Verhagen Marc, Brent Cochran, James Pustejovsky
In a recent project, the Language Application Grid was augmented to support the mining of scientific publications.
no code implementations • SemEval (NAACL) 2022 • Jingxuan Tu, Eben Holderness, Marco Maru, Simone Conia, Kyeongmin Rim, Kelley Lynch, Richard Brutti, Roberto Navigli, James Pustejovsky
In this task, we identify a challenge that is reflective of linguistic and cognitive competencies that humans have when speaking and reasoning.
no code implementations • 12 Mar 2025 • Hannah VanderHoeven, Brady Bhalla, Ibrahim Khebour, Austin Youngren, Videep Venkatesha, Mariah Bradford, Jack FitzGerald, Carlos Mabrey, Jingxuan Tu, Yifan Zhu, Kenneth Lai, Changsoo Jung, James Pustejovsky, Nikhil Krishnaswamy
We present TRACE, a novel system for live *common ground* tracking in situated collaborative tasks.
no code implementations • 8 Dec 2024 • Derek Palmer, Yifan Zhu, Kenneth Lai, Hannah VanderHoeven, Mariah Bradford, Ibrahim Khebour, Carlos Mabrey, Jack FitzGerald, Nikhil Krishnaswamy, Martha Palmer, James Pustejovsky
Our goal is to develop an AI Partner that can provide support for group problem solving and social dynamics.
1 code implementation • 6 Jun 2024 • Jingxuan Tu, Keer Xu, Liulu Yue, Bingyang Ye, Kyeongmin Rim, James Pustejovsky
We also propose a new method that largely improves the performance over baselines on the C-STS data by training the models with the answers.
no code implementations • 14 May 2024 • Iris Oved, Nikhil Krishnaswamy, James Pustejovsky, Joshua Hartshorne
We offer philosophical motivations for a method we call Virtual World Cognitive Science (VW CogSci), in which researchers use virtual embodied agents that are embedded in virtual worlds to explore questions in the field of Cognitive Science.
1 code implementation • 29 Mar 2024 • Rowan Hall Maudslay, Simone Teufel, Francis Bond, James Pustejovsky
The senses of a word exhibit rich internal structure.
1 code implementation • 26 Mar 2024 • Ibrahim Khebour, Kenneth Lai, Mariah Bradford, Yifan Zhu, Richard Brutti, Christopher Tam, Jingxuan Tu, Benjamin Ibarra, Nathaniel Blanchard, Nikhil Krishnaswamy, James Pustejovsky
Within Dialogue Modeling research in AI and NLP, considerable attention has been spent on ``dialogue state tracking'' (DST), which is the ability to update the representations of the speaker's needs at each turn in the dialogue by taking into account the past dialogue moves and history.
no code implementations • 22 May 2023 • Kiyong Lee, Nikhil Krishnaswamy, James Pustejovsky
VoxML is a modeling language used to map natural language expressions into real-time visualizations using commonsense semantic knowledge of objects and events.
1 code implementation • 27 Mar 2023 • Michael Regan, Jena D. Hwang, Keisuke Sakaguchi, James Pustejovsky
In this work, we investigate how to apply schema induction models to the task of knowledge discovery for enhanced search of English-language news texts.
no code implementations • 20 Oct 2022 • Jingxuan Tu, Kyeongmin Rim, Eben Holderness, James Pustejovsky
Understanding inferences and answering questions from text requires more than merely recovering surface arguments, adjuncts, or strings associated with the query terms.
no code implementations • 14 Dec 2021 • David McDonald, James Pustejovsky
We have recently begun a project to develop a more effective and efficient way to marshal inferences from background knowledge to facilitate deep natural language understanding.
no code implementations • Joint Conference on Lexical and Computational Semantics 2021 • Gitit Kehat, James Pustejovsky
We present new results for the problem of sequence metaphor labeling, using the recently developed Visibility Embeddings.
no code implementations • 12 May 2021 • James Pustejovsky, Eben Holderness, Jingxuan Tu, Parker Glenn, Kyeongmin Rim, Kelley Lynch, Richard Brutti
In this paper, we argue that the design and development of multimodal datasets for natural language processing (NLP) challenges should be enhanced in two significant respects: to more broadly represent commonsense semantic inferences; and to better reflect the dynamics of actions and events, through a substantive alignment of textual and visual information.
no code implementations • 5 Dec 2020 • Nikhil Krishnaswamy, James Pustejovsky
In recent years, data-intensive AI, particularly the domain of natural language processing and understanding, has seen significant progress driven by the advent of large datasets and deep neural networks that have sidelined more classic AI approaches to the field.
no code implementations • COLING 2020 • Lucia Donatelli, Kenneth Lai, James Pustejovsky
We analyze the use and interpretation of modal expressions in a corpus of situated human-robot dialogue and ask how to effectively represent these expressions for automatic learning.
no code implementations • EMNLP 2020 • Parisa Kordjamshidi, James Pustejovsky, Marie-Francine Moens
Understating spatial semantics expressed in natural language can become highly complex in real-world applications.
no code implementations • 13 Jul 2020 • Katherine Krajovic, Nikhil Krishnaswamy, Nathaniel J. Dimick, R. Pito Salas, James Pustejovsky
We present a new interface for controlling a navigation robot in novel environments using coordinated gesture and language.
no code implementations • NAACL 2021 • Jingxuan Tu, Marc Verhagen, Brent Cochran, James Pustejovsky
We are developing semantic visualization techniques in order to enhance exploration and enable discovery over large datasets of complex networks of relations.
no code implementations • NAACL 2021 • Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Haoran Zhang, Weili Liu, Aabhas Chauhan, Yingjun Guan, Bangzheng Li, Ruisong Li, Xiangchen Song, Yi R. Fung, Heng Ji, Jiawei Han, Shih-Fu Chang, James Pustejovsky, Jasmine Rah, David Liem, Ahmed Elsayed, Martha Palmer, Clare Voss, Cynthia Schneider, Boyan Onyshkevych
To combat COVID-19, both clinicians and scientists need to digest vast amounts of relevant biomedical knowledge in scientific literature to understand the disease mechanism and related biological functions.
no code implementations • LREC 2020 • Kyeongmin Rim, Kelley Lynch, Marc Verhagen, Nancy Ide, James Pustejovsky
Promoting interoperrable computational linguistics (CL) and natural language processing (NLP) application platforms and interchange-able data formats have contributed improving discoverabilty and accessbility of the openly available NLP software.
no code implementations • LREC 2020 • Kyeongmin Rim, Jingxuan Tu, Kelley Lynch, James Pustejovsky
Within the natural language processing (NLP) community, shared tasks play an important role.
no code implementations • LREC 2020 • Gitit Kehat, James Pustejovsky
We present new results on Metaphor Detection by using text from visual datasets.
no code implementations • LREC 2020 • Nikhil Krishnaswamy, James Pustejovsky
In this paper, we present an analysis of computationally generated mixed-modality definite referring expressions using combinations of gesture and linguistic descriptions.
no code implementations • WS 2019 • Elena Alvarez-Mellado, Eben Holderness, Nicholas Miller, Fyonn Dhang, Philip Cawkwell, Kirsten Bolton, James Pustejovsky, Mei-Hua Hall
Predicting which patients are more likely to be readmitted to a hospital within 30 days after discharge is a valuable piece of information in clinical decision-making.
no code implementations • 18 Sep 2019 • Nikhil Krishnaswamy, James Pustejovsky
We present an architecture for integrating real-time, multimodal input into a computational agent's contextual model.
no code implementations • WS 2019 • Susan Windisch Brown, Julia Bonn, James Gung, Annie Zaenen, James Pustejovsky, Martha Palmer
This paper announces the release of a new version of the English lexical resource VerbNet with substantially revised semantic representations designed to facilitate computer planning and reasoning based on human language.
no code implementations • WS 2019 • James Pustejovsky, Ken Lai, Nianwen Xue
In this paper, we propose an extension to Abstract Meaning Representations (AMRs) to encode scope information of quantifiers and negation, in a way that overcomes the semantic gaps of the schema while maintaining its cognitive simplicity.
no code implementations • WS 2019 • Kyeongmin Rim, Kelley Lynch, James Pustejovsky
We present Computational Linguistics Applications for Multimedia Services (CLAMS), a platform that provides access to computational content analysis tools for archival multimedia material that appear in different media, such as text, audio, image, and video.
no code implementations • WS 2019 • Kenneth Lai, James Pustejovsky
Under the standard approach to counterfactuals, to determine the meaning of a counterfactual sentence, we consider the {``}closest{''} possible world(s) where the antecedent is true, and evaluate the consequent.
no code implementations • WS 2019 • Nikhil Krishnaswamy, James Pustejovsky
Referring expressions and definite descriptions of objects in space exploit information both about object characteristics and locations.
no code implementations • WS 2019 • Eben Holderness, Philip Cawkwell, Kirsten Bolton, James Pustejovsky, Mei-Hua Hall
In this study, we undertook, to our knowledge, the first domain adaptation of sentiment analysis to psychiatric EHRs by defining psychiatric clinical sentiment, performing an annotation project, and evaluating multiple sentence-level sentiment machine learning (ML) models.
no code implementations • 5 Feb 2019 • James Pustejovsky, Nikhil Krishnaswamy
In this paper, we argue that simulation platforms enable a novel type of embodied spatial reasoning, one facilitated by a formal model of object and event semantics that renders the continuous quantitative search space of an open-world, real-time environment tractable.
no code implementations • 27 Nov 2018 • Nikhil Krishnaswamy, Scott Friedman, James Pustejovsky
We present a novel approach to introducing new spatial structures to an AI agent, combining deep learning over qualitative spatial relations with various heuristic search algorithms.
no code implementations • WS 2018 • Eben Holderness, Nicholas Miller, Philip Cawkwell, Kirsten Bolton, James Pustejovsky, Marie Meteer, Mei-Hua Hall
Readmission after discharge from a hospital is disruptive and costly, regardless of the reason.
no code implementations • COLING 2018 • James Pustejovsky, Nikhil Krishnaswamy
Most work within the computational event modeling community has tended to focus on the interpretation and ordering of events that are associated with verbs and event nominals in linguistic expressions.
no code implementations • IJCNLP 2017 • Gitit Kehat, James Pustejovsky
We present and take advantage of the inherent visualizability properties of words in visual corpora (the textual components of vision-language datasets) to compute concreteness scores for words.
no code implementations • 2 Oct 2017 • Tuan Do, James Pustejovsky
Selecting an optimal event representation is essential for event classification in real world contexts.
no code implementations • 30 Sep 2017 • Tuan Do, James Pustejovsky
Event learning is one of the most important problems in AI.
General Classification
Human-Object Interaction Detection
+1
no code implementations • SEMEVAL 2017 • Steven Bethard, Guergana Savova, Martha Palmer, James Pustejovsky
Clinical TempEval 2017 aimed to answer the question: how well do systems trained on annotated timelines for one medical condition (colon cancer) perform in predicting timelines on another medical condition (brain cancer)?
no code implementations • EACL 2017 • James Pustejovsky, Nikhil Krishnaswamy
Simulation and automatic visualization of events from natural language descriptions and supplementary modalities, such as gestures, allows humans to use their native capabilities as linguistic and visual interpreters to collaborate on tasks with an artificial agent or to put semantic intuitions to the test in an environment where user and agent share a common context. In previous work (Pustejovsky and Krishnaswamy, 2014; Pustejovsky, 2013a), we introduced a method for modeling natural language expressions within a 3D simulation environment built on top of the game development platform Unity (Goldstone, 2009).
no code implementations • WS 2016 • James Pustejovsky, Tuan Do, Gitit Kehat, Nikhil Krishnaswamy
Human communication is a multimodal activity, involving not only speech and written expressions, but intonation, images, gestures, visual clues, and the interpretation of actions through perception.
no code implementations • COLING 2016 • Nikhil Krishnaswamy, James Pustejovsky
Much existing work in text-to-scene generation focuses on generating static scenes.
no code implementations • WS 2016 • Nancy Ide, Keith Suderman, Eric Nyberg, James Pustejovsky, Marc Verhagen
The US National Science Foundation (NSF) SI2-funded LAPPS/Galaxy project has developed an open-source platform for enabling complex analyses while hiding complexities associated with underlying infrastructure, that can be accessed through a web interface, deployed on any Unix system, or run from the cloud.
no code implementations • SEMEVAL 2014 • James Pustejovsky, Nikhil Krishnaswamy
The generated simulations act as a conceptual "debugger" for the semantics of different motion verbs: that is, by testing for consistency and informativeness in the model, simulations expose the presuppositions associated with linguistic expressions and their compositions.
no code implementations • LREC 2016 • James Pustejovsky, Nikhil Krishnaswamy
We present the specification for a modeling language, VoxML, which encodes semantic knowledge of real-world objects represented as three-dimensional models, and of events and attributes related to and enacted over these objects.
no code implementations • 5 Oct 2016 • Tuan Do, Nikhil Krishnaswamy, James Pustejovsky
This paper introduces the Event Capture Annotation Tool (ECAT), a user-friendly, open-source interface tool for annotating events and their participants in video, capable of extracting the 3D positions and orientations of objects in video captured by Microsoft's Kinect(R) hardware.
no code implementations • 3 Oct 2016 • Nikhil Krishnaswamy, James Pustejovsky
In this paper, we describe a system for generating three-dimensional visual simulations of natural language motion expressions.
no code implementations • 10 Jul 2016 • Gitit Kehat, James Pustejovsky
Annotated datasets are commonly used in the training and evaluation of tasks involving natural language and vision (image description generation, action recognition and visual question answering).
no code implementations • LREC 2016 • Nancy Ide, Keith Suderman, James Pustejovsky, Marc Verhagen, Christopher Cieri
The NSF-SI2-funded LAPPS Grid project is a collaborative effort among Brandeis University, Vassar College, Carnegie-Mellon University (CMU), and the Linguistic Data Consortium (LDC), which has developed an open, web-based infrastructure through which resources can be easily accessed and within which tailored language services can be efficiently composed, evaluated, disseminated and consumed by researchers, developers, and students across a wide variety of disciplines.
no code implementations • LREC 2014 • Peter Anick, Marc Verhagen, James Pustejovsky
Natural language analysis of patents holds promise for the development of tools designed to assist analysts in the monitoring of emerging technologies.
no code implementations • LREC 2014 • James Pustejovsky, Zachary Yocum
This paper explores the use of ISO-Space, an annotation specification to capturing spatial information, for encoding spatial relations mentioned in descriptions of images.
no code implementations • LREC 2014 • Nancy Ide, James Pustejovsky, Christopher Cieri, Eric Nyberg, Di Wang, Keith Suderman, Marc Verhagen, Jonathan Wright
The Language Application (LAPPS) Grid project is establishing a framework that enables language service discovery, composition, and reuse and promotes sustainability, manageability, usability, and interoperability of natural language Processing (NLP) components.
no code implementations • 19 Mar 2014 • Steven Bethard, Leon Derczynski, James Pustejovsky, Marc Verhagen
We describe the Clinical TempEval task which is currently in preparation for the SemEval-2015 evaluation exercise.
no code implementations • TACL 2014 • William F. Styler IV, Steven Bethard, Sean Finan, Martha Palmer, Sameer Pradhan, Piet C de Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, James Pustejovsky
The corpus is available to the community and has been proposed for use in a SemEval 2015 task.
2 code implementations • 22 Jun 2012 • Naushad UzZaman, Hector Llorens, James Allen, Leon Derczynski, Marc Verhagen, James Pustejovsky
We describe the TempEval-3 task which is currently in preparation for the SemEval-2013 evaluation exercise.
no code implementations • LREC 2012 • Marc Verhagen, James Pustejovsky
We present and demonstrate the updated version of the TARSQI Toolkit, a suite of temporal processing modules that extract temporal information from natural language texts.
no code implementations • LREC 2012 • John Vogel, Marc Verhagen, James Pustejovsky
ATLIS (short for ATLIS Tags Locations in Strings) is a tool being developed using a maximum-entropy machine learning model for automatically identifying information relating to spatial and locational information in natural language text.
no code implementations • LREC 2012 • James Pustejovsky, Jessica Moszkowicz
In this paper, we describe the methodology being used to develop certain aspects of ISO-Space, an annotation language for encoding spatial and spatiotemporal information as expressed in natural language text.
no code implementations • LREC 2012 • Anna Rumshisky, Nick Botchan, Sophie Kushkuley, James Pustejovsky
In this paper, we explore different strategies for implementing a crowdsourcing methodology for a single-step construction of an empirically-derived sense inventory and the corresponding sense-annotated corpus.
no code implementations • Proceedings of the 5th International Workshop on Semantic Evaluation 2010 • Marc Verhagen, Roser Saurí, Tommaso Caselli, James Pustejovsky
Tempeval-2 comprises evaluation tasks for time expressions, events and temporal relations, the latter of which was split up in four sub tasks, motivated by the notion that smaller subtasks would make both data preparation and temporal relation extraction easier.