1 code implementation • Findings (NAACL) 2022 • Yutong Shao, Nikita Bhutani, Sajjadur Rahman, Estevam Hruschka
Entity set expansion (ESE) aims at obtaining a more complete set of entities given a textual corpus and a seed set of entities of a concept.
no code implementations • FEVER (ACL) 2022 • Chieh-Yang Huang, Jinfeng Li, Nikita Bhutani, Alexander Whedon, Estevam Hruschka, Yoshi Suhara
To alleviate this scarcity problem, we develop an unsupervised method, ZL-Distiller, which leverages contextual language representations of the reviews and their distributional patterns to identify salient sentences about entities.
1 code implementation • 30 Mar 2024 • Pouya Pezeshkpour, Estevam Hruschka
Our analysis of LLMs using MCRank indicates a significant decrease in performance as the number and complexity of items and conditions grow.
no code implementations • 2 Feb 2024 • Pouya Pezeshkpour, Eser Kandogan, Nikita Bhutani, Sajjadur Rahman, Tom Mitchell, Estevam Hruschka
We present a formal definition of reasoning capacity and illustrate its utility in identifying limitations within each component of the system.
1 code implementation • 10 Nov 2023 • Pouya Pezeshkpour, Hayate Iso, Thom Lake, Nikita Bhutani, Estevam Hruschka
We meticulously craft this benchmark to cater to a wide array of HR tasks, including matching and explaining resumes to job descriptions, extracting skills and experiences from resumes, and editing resumes.
no code implementations • 10 Nov 2023 • Nedelina Teneva, Estevam Hruschka
Despite the recent popularity of knowledge graph (KG) related tasks and benchmarks such as KG embeddings, link prediction, entity alignment and evaluation of the reasoning abilities of pretrained language models as KGs, the structure and properties of real KGs are not well studied.
no code implementations • 9 Nov 2023 • Aditi Mishra, Sajjadur Rahman, Hannah Kim, Kushan Mitra, Estevam Hruschka
We consider the task of generating knowledge-guided rationalization in natural language by using expert-written examples in a few-shot manner.
1 code implementation • 14 Sep 2023 • Yunshu Wu, Hayate Iso, Pouya Pezeshkpour, Nikita Bhutani, Estevam Hruschka
Large Language Models (LLMs) have shown promising performance in summary evaluation tasks, yet they face challenges such as high computational costs and the Lost-in-the-Middle problem where important information in the middle of long documents is often overlooked.
no code implementations • 25 Aug 2023 • Vishwas Mruthyunjaya, Pouya Pezeshkpour, Estevam Hruschka, Nikita Bhutani
Despite these advancements, there is a void in comprehensively evaluating whether LMs can encompass the intricate topological and semantic attributes of KGs, attributes crucial for reasoning processes.
no code implementations • 22 Aug 2023 • Pouya Pezeshkpour, Estevam Hruschka
Investigating the sensitivity of LLMs towards the order of options in multiple-choice questions, we demonstrate a considerable performance gap of approximately 13% to 75% in LLMs on different benchmarks, when answer options are reordered, even when using demonstrations in a few-shot setting.
no code implementations • 9 Jan 2023 • Sajjadur Rahman, Hannah Kim, Dan Zhang, Estevam Hruschka, Eser Kandogan
Human-centered AI workflows involve stakeholders with multiple roles interacting with each other and automated agents to accomplish diverse tasks.
no code implementations • 8 Jan 2023 • Dan Zhang, Hannah Kim, Rafael Li Chen, Eser Kandogan, Estevam Hruschka
We present MEGAnno, a novel exploratory annotation framework designed for NLP researchers and practitioners.
1 code implementation • 21 Dec 2022 • Bosung Kim, Hayate Iso, Nikita Bhutani, Estevam Hruschka, Ndapa Nakashole, Tom Mitchell
We propose a novel framework, ZETT (ZEro-shot Triplet extraction by Template infilling), that aligns the task objective to the pre-training objective of generative transformers to generalize to unseen relations.
Ranked #1 on Zero-shot Relation Triplet Extraction on FewRel
1 code implementation • 24 Jun 2021 • Dongjin Choi, Sara Evensen, Çağatay Demiralp, Estevam Hruschka
In this work, we extend the DPBD framework to span-level annotation tasks, arguably one of the most time-consuming NLP labeling tasks.