Search Results for author:
Found 29 papers, 15 papers with code
Can Question Generation Debias Question Answering Models? A Case Study on Question–Context Lexical Overlap
Question answering (QA) models for reading comprehension have been demonstrated to exploit unintended dataset biases such as question–context lexical overlap.
LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs
This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs).
MoreHopQA: More Than Multi-hop Reasoning
Instead of relying solely on factual reasoning, we enhance the existing multi-hop questions by adding another layer of questioning that involves one, two, or all three of the following types of reasoning: commonsense, arithmetic, and symbolic.
What Makes Language Models Good-enough?
This result suggests that models with shallower depth and fewer heads can learn good-enough language processing.
PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments
The projectivity may vary depending on the combination of presupposition triggers and environments.
Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension
To precisely evaluate a language model's capability for logical reading comprehension, we present a dataset for testing the understanding of the rationale behind critical reasoning.
Probing Physical Reasoning with Counter-Commonsense Context
In this study, we create a CConS (Counter-commonsense Contextual Size comparison) dataset to investigate how physical commonsense affects the contextualized size comparison task; the proposed dataset consists of both contexts that fit physical commonsense and those that do not.
On Degrees of Freedom in Defining and Testing Natural Language Understanding
By demonstrating that current practices in NLU studies can be associated with those criteria and organizing them into a comprehensive checklist, we prove that the validity argument can serve as a coherent guideline for designing credible test sets and facilitating scientific communication.
Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering
To explain the predicted answers and evaluate the reasoning abilities of models, several studies have utilized underlying reasoning (UR) tasks in multi-hop question answering (QA) datasets.
Multi-hop Question Answering
Open-Ended Question Answering
+1
Cross-Modal Similarity-Based Curriculum Learning for Image Captioning
Moreover, the higher model performance on difficult examples and unseen data also demonstrates the generalization ability.
Which Shortcut Solution Do Question Answering Models Prefer to Learn?
We assume that the learnability of shortcuts, i. e., how easy it is to learn a shortcut, is useful to mitigate the problem.
Penalizing Confident Predictions on Largely Perturbed Inputs Does Not Improve Out-of-Distribution Generalization in Question Answering
Contrary to our expectations, although models become sensitive to the four types of perturbations, we find that the OOD generalization is not improved.
Debiasing Masks: A New Framework for Shortcut Mitigation in NLU
We propose a new debiasing method in which we identify debiased pruning masks that can be applied to a finetuned model.
Look to the Right: Mitigating Relative Position Bias in Extractive Question Answering
Specifically, we find that when the relative positions in a training set are biased, the performance on examples with relative positions unseen during training is significantly degraded.
How Well Do Multi-hop Reading Comprehension Models Understand Date Information?
Other results reveal that our probing questions can help to improve the performance of the models (e. g., by +10. 3 F1) on the main QA task and our dataset can be used for data augmentation to improve the robustness of the models.
Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios
The possible consequences for the same context may vary depending on the situation we refer to.
A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension
The issue of shortcut learning is widely known in NLP and has been an important research focus in recent years.
What Makes Reading Comprehension Questions Difficult?
For a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems.
Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap
Question answering (QA) models for reading comprehension have been demonstrated to exploit unintended dataset biases such as question-context lexical overlap.
Embracing Ambiguity: Shifting the Training Target of NLI Models
Natural Language Inference (NLI) datasets contain examples with highly ambiguous labels.
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?
However, we find that training crowdworkers, and then using an iterative process of collecting data, sending feedback, and qualifying workers based on expert judgments is an effective means of collecting challenging data.
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
The evidence information has two benefits: (i) providing a comprehensive explanation for predictions and (ii) evaluating the reasoning skills of a model.
Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation
While most existing QAG methods aim to improve the quality of synthetic examples, we conjecture that diversity-promoting QAG can mitigate the sparsity of training sets and lead to better robustness.
Benchmarking Machine Reading Comprehension: A Psychological Perspective
Machine reading comprehension (MRC) has received considerable attention as a benchmark for natural language understanding.
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
Existing analysis work in machine reading comprehension (MRC) is largely concerned with evaluating the capabilities of systems.
What Makes Reading Comprehension Questions Easier?
From this study, we observed that (i) the baseline performances for the hard subsets remarkably degrade compared to those of entire datasets, (ii) hard questions require knowledge inference and multiple-sentence reasoning in comparison with easy questions, and (iii) multiple-choice questions tend to require a broader range of reasoning skills than answer extraction and description questions.
Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability
Knowing the quality of reading comprehension (RC) datasets is important for the development of natural-language understanding systems.
Annotation and Analysis of Discourse Relations, Temporal Relations and Multi-Layered Situational Relations in Japanese Texts
This paper proposes a methodology for building a specialized Japanese data set for recognizing temporal relations and discourse relations.
