Search Results for author:
Found 119 papers, 70 papers with code
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings
Evaluating bias, fairness, and social impact in monolingual language models is a difficult task.
Improving Cross-lingual Text Classification with Zero-shot Instance-Weighting
In this paper, we propose zero-shot instance-weighting, a general model-agnostic zero-shot learning framework for improving CLTC by leveraging source instance weighting.
An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next
Dialogue summarization helps readers capture salient information from long conversations in meetings, interviews, and TV series.
Ascle: A Python Natural Language Processing Toolkit for Medical Text Generation
This study introduces Ascle, a pioneering natural language processing (NLP) toolkit designed for medical text generation.
Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization
Our study reveals that instruction controllable text summarization remains a challenging task for LLMs, since (1) all LLMs evaluated still make factual and other types of errors in their summaries; (2) all LLM-based evaluation methods cannot achieve a strong alignment with human annotators when judging the quality of candidate summaries; (3) different LLMs show large performance gaps in summary generation and evaluation.
Fair Abstractive Summarization of Diverse Perspectives
However, current work in summarization metrics and Large Language Models (LLMs) evaluation has not explored fair abstractive summarization.
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models
Recently, large language models (LLMs), especially those that are pretrained on code, have demonstrated strong capabilities in generating programs from natural language inputs in a few-shot or even zero-shot manner.
RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations
Despite significant progress having been made in question answering on tabular data (Table QA), it's unclear whether, and to what extent existing Table QA models are robust to task-specific perturbations, e. g., replacing key question entities or shuffling table columns.
bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark
We run the first systematic evaluation of pre-trained language models for Bulgarian, comparing and contrasting results across the nine tasks in the benchmark.
On Learning to Summarize with Large Language Models as References
Meanwhile, we perform a meta-analysis on this new learning setting that reveals a discrepancy between human and LLM-based evaluation, highlighting the benefits and risks of this LLM-as-reference setting we investigated.
QTSumm: Query-Focused Summarization over Tabular Data
Motivated by this, we define a new query-focused table summarization task, where text generation models have to perform human-like reasoning and analysis over the given table to generate a tailored summary.
Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies
In-context learning (ICL) has emerged as a new approach to various natural language processing tasks, utilizing large language models (LLMs) to make predictions based on context that has been supplemented with a few examples or task-specific instructions.
HPE:Answering Complex Questions over Text by Hybrid Question Parsing and Execution
It comprises two central pillars: (1) We parse the question of varying complexity into an intermediate representation, named H-expression, which is composed of simple questions as the primitives and symbolic operations representing the relationships among them; (2) To execute the resulting H-expressions, we design a hybrid executor, which integrates the deterministic rules to translate the symbolic operations with a drop-in neural reader network to answer each decomposed simple question.
HiPool: Modeling Long Documents Using Graph Neural Networks
Encoding long sequences in Natural Language Processing (NLP) is a challenging problem.
Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations
In this work, we evaluate LLM APIs (ChatGPT, GPT-3, and GPT-4) on the Japanese national medical licensing examinations from the past five years, including the current year.
Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation
Interpretability and efficiency are two important considerations for the adoption of neural automatic metrics.
ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics
We introduce ProofNet, a benchmark for autoformalization and formal proving of undergraduate-level mathematics.
LEVER: Learning to Verify Language-to-Code Generation with Execution
The advent of large language models trained on code (code LLMs) has led to significant progress in language-to-code generation.
Ranked #2 on
Semantic Parsing
on spider
LoFT: Enhancing Faithfulness and Diversity for Table-to-Text Generation via Logic Form Control
Logical Table-to-Text (LT2T) generation is tasked with generating logically faithful sentences from tables.
Rethinking Explaining Graph Neural Networks via Non-parametric Subgraph Matching
It takes advantage of MatchExplainer to fix the most informative portion of the graph and merely operates graph augmentations on the rest less informative part.
STRUDEL: Structured Dialogue Summarization for Dialogue Comprehension
Abstractive dialogue summarization has long been viewed as an important standalone task in natural language processing, but no previous work has explored the possibility of whether abstractive dialogue summarization can also be used as a means to boost an NLP system's performance on other important dialogue comprehension tasks.
On Improving Summarization Factual Consistency from Natural Language Feedback
We collect a high-quality dataset, DeFacto, containing human demonstrations and informational natural language feedback consisting of corrective instructions, edited summaries, and explanations with respect to the factual consistency of the summary.
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting
We find language adaptation to be effective at improving zero-shot performance in new languages.
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation
Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests.
Integration of Pre-trained Protein Language Models into Geometric Deep Learning Networks
Geometric deep learning has recently achieved great success in non-Euclidean domains, and learning on 3D structures of large biomolecules is emerging as a distinct research area.
Explicit Knowledge Transfer for Weakly-Supervised Code Generation
More specifically, we propose explicit knowledge transfer (EKT), which uses the few-shot capabilities of a teacher LLM to create NL-code pairs that we then filter for correctness and fine-tune the student on.
CREATIVESUMM: Shared Task on Automatic Summarization for Creative Writing
We detail the process of curating these datasets for the task, as well as the metrics used for the evaluation of the submissions.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
MACSum: Controllable Summarization with Mixed Attributes
We propose two simple and effective parameter-efficient approaches for the new task of mixed controllable summarization based on hard prompt tuning and soft prefix tuning.
Uni-Parser: Unified Semantic Parser for Question Answering on Knowledge Base and Database
Parsing natural language questions into executable logical forms is a useful and interpretable way to perform question answering on structured data such as knowledge bases (KB) or databases (DB).
Crosslingual Generalization through Multitask Finetuning
We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that appear only in the pretraining corpus.
Ranked #1 on
Question Answering
on StoryCloze
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
Reasoning over tabular data requires both table structure understanding and a broad set of table reasoning skills.
Ranked #3 on
Semantic Parsing
on WikiSQL
(Denotation accuracy (test) metric)
Binding Language Models in Symbolic Languages
We propose Binder, a training-free neural-symbolic framework that maps the task input to a program, which (1) allows binding a unified API of language model (LM) functionalities to a programming language (e. g., SQL, Python) to extend its grammar coverage and thus tackle more diverse questions, (2) adopts an LM as both the program parser and the underlying model called by the API during execution, and (3) requires only a few in-context exemplar annotations.
Ranked #4 on
Table-based Fact Verification
on TabFact
Look Ma, Only 400 Samples! Revisiting the Effectiveness of Automatic N-Gram Rule Generation for Spelling Normalization in Filipino
With 84. 75 million Filipinos online, the ability for models to process online text is crucial for developing Filipino NLP applications.
FOLIO: Natural Language Reasoning with First-Order Logic
We present FOLIO, a human-annotated, open-domain, and logically complex and diverse dataset for reasoning in natural language (NL), equipped with first order logic (FOL) annotations.
RealTime QA: What's the Answer Right Now?
We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version).
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims.
Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutions
We show that our use of self-sampled correct and partially-correct solutions can benefit learning and help guide the sampling process, leading to more efficient exploration of the solution space.
Ranked #138 on
Arithmetic Reasoning
on GSM8K
Leveraging Locality in Abstractive Text Summarization
Our experimental results show that our model has a better performance compared with strong baselines with efficient attention modules, and our analysis provides further insights into our locality-aware modeling strategy.
R2D2: Robust Data-to-Text with Replacement Detection
Unfaithful text generation is a common problem for text generation systems.
Twist Decoding: Diverse Generators Guide Each Other
Our extensive evaluations on machine translation and scientific paper summarization demonstrate that Twist decoding substantially outperforms each model decoded in isolation over various scenarios, including cases where domain-specific and general-purpose models are both available.
Discovering and Explaining the Representation Bottleneck of Graph Neural Networks from Multi-order Interactions
Graph neural networks (GNNs) mainly rely on the message-passing paradigm to propagate node features and build interactions, and different graph learning tasks require different ranges of node interactions.
Data Governance in the Age of Large-Scale Data-Driven Language Technology
The recent emergence and adoption of Machine Learning technology, and specifically of Large Language Models, has drawn attention to the need for systematic and transparent management of language data.
EHRKit: A Python Natural Language Processing Toolkit for Electronic Health Record Texts
The Electronic Health Record (EHR) is an essential part of the modern medical system and impacts healthcare delivery, operations, and research.
A Call for Clarity in Beam Search: How It Works and When It Stops
Based on this finding, we introduce a patience factor, a simple modification to this beam decoding implementation, that generalizes the stopping criterion and provides flexibility to the depth of search.
BRIO: Bringing Order to Abstractive Summarization
Abstractive summarization models are commonly trained using maximum likelihood estimation, which assumes a deterministic (one-point) target distribution in which an ideal model will assign all the probability mass to the reference summary.
Ranked #2 on
Text Summarization
on X-Sum
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
PromptSource is a system for creating, sharing, and using natural language prompts.
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models
Structured knowledge grounding (SKG) leverages structured knowledge to complete user requests, such as semantic parsing over databases and question answering over knowledge bases.
Ranked #1 on
Task-Oriented Dialogue Systems
on KVRET
A Transfer Learning Pipeline for Educational Resource Discovery with Application in Leading Paragraph Generation
In this paper, we propose the educational resource discovery (ERD) pipeline that automates web resource discovery for novel domains.
CLICKER: A Computational LInguistics Classification Scheme for Educational Resources
We propose a classification scheme -- CLICKER for CL/NLP based on the analysis of online lectures from 77 university courses on this subject.
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning
Using human evaluation and automatic faithfulness metrics, we show that our model significantly reduces all kinds of factual errors on the dialogue summarization, SAMSum corpus.
Surfer100: Generating Surveys From Web Resources, Wikipedia-style
Fast-developing fields such as Artificial Intelligence (AI) often outpace the efforts of encyclopedic sources such as Wikipedia, which either do not completely cover recently-introduced topics or lack such content entirely.
Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents
To the best of our knowledge, Summ$^N$ is the first multi-stage split-then-summarize framework for long input summarization.
DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization
Transformer-based models have achieved state-of-the-art performance on short-input summarization.
Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs
Then HMGs are constructed with both atom-level and motif-level nodes.
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries
Current pre-trained models applied to summarization are prone to factual inconsistencies which either misrepresent the source text or introduce extraneous information.
Efficient Variational Graph Autoencoders for Unsupervised Cross-domain Prerequisite Chains
Our novel model consists of a variational graph autoencoder (VGAE) and a domain discriminator.
An Exploratory Study on Long Dialogue Summarization: What Works and What's Next
Dialogue summarization helps readers capture salient information from long conversations in meetings, interviews, and TV series.
SummerTime: Text Summarization Toolkit for Non-experts
We also provide explanations for models and evaluation metrics to help users understand the model behaviors and select models that best suit their needs.
Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review
Electronic health records (EHRs), digital collections of patient healthcare events and observations, are ubiquitous in medicine and critical to healthcare delivery, operations, and research.
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference
It has been studied intensively in the past few years thanks to the availability of large-scale labeled datasets.
ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining
While online conversations can cover a vast amount of information in many different formats, abstractive text summarization has primarily focused on modeling solely news articles.
BookSum: A Collection of Datasets for Long-form Narrative Summarization
The majority of available text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies, and often contain strong layout and stylistic biases.
Unsupervised Cross-Domain Prerequisite Chain Learning using Variational Graph Autoencoders
For example, one may be an expert in the natural language processing (NLP) domain but want to determine the best order to learn new concepts in an unfamiliar Computer Vision domain (CV).
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization
As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed.
FeTaQA: Free-form Table Question Answering
Existing table question answering datasets contain abundant factual questions that primarily evaluate the query and schema comprehension capability of a system, but they fail to include questions that require complex reasoning and integration of information due to the constraint of the associated short-form answers.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Models pretrained with self-supervised objectives on large text corpora achieve state-of-the-art performance on English text summarization tasks.
Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start
We demonstrate that this framework enables a pretrained entailment model to work well on new entailment domains in a few-shot setting, and show its effectiveness as a unified solver for several downstream NLP tasks such as question answering and coreference resolution when the end-task annotations are limited.
GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
We present GraPPa, an effective pre-training approach for table semantic parsing that learns a compositional inductive bias in the joint representations of textual and tabular data.
Ranked #8 on
Semantic Parsing
on spider
SummEval: Re-evaluating Summarization Evaluation
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress.
DART: Open-Domain Structured Data Record to Text Generation
Data-to-Text annotations can be a costly process, especially when dealing with tables which are the major source of structured data and contain nontrivial structures.
CO-Search: COVID-19 Information Retrieval with Semantic Search, Question Answering, and Abstractive Summarization
The COVID-19 global pandemic has resulted in international efforts to understand, track, and mitigate the disease, yielding a significant corpus of COVID-19 and SARS-CoV-2-related publications across scientific disciplines.
ESPRIT: Explaining Solutions to Physical Reasoning Tasks
Our framework learns to generate explanations of how the physical simulation will causally evolve so that an agent or a human can easily reason about a solution using those interpretable descriptions.
R-VGAE: Relational-variational Graph Autoencoder for Unsupervised Prerequisite Chain Learning
The task of concept prerequisite chain learning is to automatically determine the existence of prerequisite relationships among concept pairs.
A Neural Topic-Attention Model for Medical Term Abbreviation Disambiguation
Specifically, a neural topic-attention model is applied to learn improved contextualized sentence representations for medical term abbreviation disambiguation.
CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases
We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems.
Ranked #8 on
Dialogue State Tracking
on CoSQL
The CL-SciSumm Shared Task 2018: Results and Key Insights
This overview describes the official results of the CL-SciSumm Shared Task 2018 -- the first medium-scale shared task on scientific document summarization in the computational linguistics (CL) domain.
Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
We focus on the cross-domain context-dependent text-to-SQL generation task.
Ranked #5 on
Text-To-SQL
on SParC
Overview and Results: CL-SciSumm Shared Task 2019
All papers are from the open access research papers in the CL domain.
Creating A Neural Pedagogical Agent by Jointly Learning to Review and Assess
Our model outperforms existing approaches over several metrics in predicting user response correctness, notably out-performing other methods on new users without large question-response histories.
Improving Low-Resource Cross-lingual Document Retrieval by Reranking with Deep Bilingual Representations
In this paper, we propose to boost low-resource cross-lingual document retrieval performance with deep bilingual query-document representations.
Cross-Lingual Information Retrieval
Cross-Lingual Word Embeddings
+3
SParC: Cross-Domain Semantic Parsing in Context
The best model obtains an exact match accuracy of 20. 2% over all questions and less than10% over all interaction sequences, indicating that the cross-domain setting and the con-textual phenomena of the dataset present significant challenges for future research.
Syntax-aware Neural Semantic Role Labeling with Supertags
We introduce a new syntax-aware model for dependency-based semantic role labeling that outperforms syntax-agnostic models for English and Spanish.
SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task
In this paper we propose SyntaxSQLNet, a syntax tree network to address the complex and cross-domain text-to-SQL generation task.
Ranked #7 on
Text-To-SQL
on SParC
SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-Domain Text-to-SQL Task
In this paper we propose SyntaxSQLNet, a syntax tree network to address the complex and cross-domain text-to-SQL generation task.
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
We define a new complex and cross-domain semantic parsing and text-to-SQL task where different complex SQL queries and databases appear in train and test sets.
Ranked #10 on
Semantic Parsing
on spider
Improving Text-to-SQL Evaluation Methodology
Second, we show that the current division of data into training and test sets measures robustness to variations in the way questions are asked, but only partially tests how well systems generalize to new queries; therefore, we propose a complementary dataset split for evaluation of future work.
Ranked #1 on
SQL Parsing
on IMDb
Neural Coreference Resolution with Deep Biaffine Attention by Joint Mention Detection and Mention Clustering
Coreference resolution aims to identify in a text all mentions that refer to the same real-world entity.
TypeSQL: Knowledge-based Type-Aware Neural Text-to-SQL Generation
Interacting with relational databases through natural language helps users of any background easily query and analyze a vast amount of data.
Ranked #2 on
Code Generation
on WikiSQL
Robust Multilingual Part-of-Speech Tagging via Adversarial Training
Adversarial training (AT) is a powerful regularization method for neural networks, aiming to achieve robustness to input perturbations.
Ranked #2 on
Part-Of-Speech Tagging
on UD
Addressee and Response Selection in Multi-Party Conversations with Speaker Interaction RNNs
In this paper, we study the problem of addressee and response selection in multi-party conversations.
Graph-based Neural Multi-Document Summarization
We propose a neural multi-document summarization (MDS) system that incorporates sentence relation graphs.
Ranked #1 on
Multi-Document Summarization
on DUC 2004
Cruciform: Solving Crosswords with Natural Language Processing
We describe the results of several of our experiments with the system.
Dependency Sensitive Convolutional Neural Networks for Modeling Sentences and Documents
Moreover, unlike other CNN-based models that analyze sentences locally by sliding windows, our system captures both the dependency information within each sentence and relationships across sentences in the same document.
Sentence Ordering and Coherence Modeling using Recurrent Neural Networks
Modeling the structure of coherent texts is a key NLP problem.
Extractive Summarization under Strict Length Constraints
In this paper we report a comparison of various techniques for single-document extractive summarization under strict length budgets, which is a common commercial use case (e. g. summarization of news articles by news aggregators).
Sentence Similarity based on Dependency Tree Kernels for Multi-document Summarization
We introduce an approach based on using the dependency grammar representations of sentences to compute sentence similarity for extractive multi-document summarization.
Classifying Syntactic Regularities for Hundreds of Languages
This paper presents a comparison of classification methods for linguistic typology for the purpose of expanding an extensive, but sparse language resource: the World Atlas of Language Structures (WALS) (Dryer and Haspelmath, 2013).
Humor in Collective Discourse: Unsupervised Funniness Detection in the New Yorker Cartoon Caption Contest
The New Yorker publishes a weekly captionless cartoon.
