Search Results for author:
Found 94 papers, 52 papers with code
Q^{2}: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering
Neural knowledge-grounded generative models for dialogue often produce content that is factually inconsistent with the knowledge they rely on, making them unreliable and limiting their applicability.
Abstractive Text Summarization
Natural Language Inference
+3
Putting Words in BERT’s Mouth: Navigating Contextualized Vector Spaces with Pseudowords
We present a method for exploring regions around individual points in a contextualized vector space (particularly, BERT space), as a way to investigate how these regions correspond to word senses.
What do Language Model Probabilities Represent? From Distribution Estimation to Response Prediction
The notion of language modeling has gradually shifted in recent years from a distribution over finite-length strings to general-purpose prediction models for textual inputs and outputs, following appropriate alignment phases.
Control Illusion: The Failure of Instruction Hierarchies in Large Language Models
Large language models (LLMs) are increasingly deployed with hierarchical instruction schemes, where certain instructions (e. g., system-level directives) are expected to take precedence over others (e. g., user messages).
Improving Image Captioning by Mimicking Human Reformulation Feedback at Inference-time
We experiment with two types of reformulation feedback: first, we collect a dataset of human reformulations that correct errors in the generated captions.
Unsupervised Speech Segmentation: A General Approach Using Speech Language Models
Unlike traditional speech and audio segmentation, which mainly focuses on spectral changes in the input signal, e. g., phone segmentation, our approach tries to segment the spoken utterance into chunks with differing acoustic-semantic styles, focusing on acoustic-semantic information that does not translate well into text, e. g., emotion or speaker.
Computational Analysis of Character Development in Holocaust Testimonies
This work presents a computational approach to analyze character development along the narrative timeline.
Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning
Theory of Mind (ToM) capabilities in LLMs have recently become a central object of investigation.
Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective
NLP research on aligning lexical representation spaces to one another has so far focused on aligning language spaces in their entirety.
Exploring the Learning Capabilities of Language Models using LEVERWORLDS
Learning a model of a stochastic setting often involves learning both general structure rules and specific properties of the instance.
CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Models
The inconsistencies revealed by our analysis, as well their connection to prediction entropies and differences between model types, can serve as useful guides for future research on addressing these limitations.
Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
The model weights for both sizes are publicly available under the Jamba Open Model License and we release ExpertsInt8 as open source.
A Language-agnostic Model of Child Language Acquisition
This work reimplements a recent semantic bootstrapping child-language acquisition model, which was originally designed for English, and trains it to learn a new language: Hebrew.
Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMs
These findings highlight the need for improved multilingual knowledge representation in LLMs and suggest a path for the development of more robust and consistent multilingual LLMs.
The Future of Open Human Feedback
In this work, we bring together interdisciplinary experts to assess the opportunities and challenges to realizing an open ecosystem of human feedback for AI.
The ShareLM Collection and Plugin: Contributing Human-Model Chats for the Benefit of the Community
We introduce the ShareLM collection, a unified set of human conversations with large language models, and its accompanying plugin, a Web extension for voluntarily contributing user-model conversations.
Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulations
While cross-linguistic model transfer is effective in many settings, there is still limited understanding of the conditions under which it works.
Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy and Novel Ensemble Method
The task of image captioning has recently been gaining popularity, and with it the complex task of evaluating the quality of image captioning models.
CovScore: Evaluation of Multi-Document Abstractive Title Set Generation
This paper introduces CovScore, an automatic reference-less methodology for evaluating thematic title sets, extracted from a corpus of documents.
Learning from Naturally Occurring Feedback
Training with the extracted feedback shows significant performance improvements over baseline models, demonstrating the efficacy of our approach in enhancing model alignment to human preferences.
A Nurse is Blue and Elephant is Rugby: Cross Domain Alignment in Large Language Models Reveal Human-like Patterns
In this paper, we adapt this task from cognitive science to evaluate the conceptualization and reasoning abilities of large language models (LLMs) through a behavioral study.
Identifying Narrative Patterns and Outliers in Holocaust Testimonies Using Topic Modeling
We align testimony sections into fixed parts, revealing the evolution of topics across the corpus of testimonies.
Jamba: A Hybrid Transformer-Mamba Language Model
We present Jamba, a new base large language model based on a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture.
Human Learning by Model Feedback: The Dynamics of Iterative Prompting with Midjourney
Generating images with a Text-to-Image model often requires multiple trials, where human users iteratively update their prompt based on feedback, namely the output image.
Improving Cross-Lingual Transfer through Subtree-Aware Word Reordering
Despite the impressive growth of the abilities of multilingual language models, such as XLM-R and mT5, it has been shown that they still face difficulties when tackling typologically-distant languages, particularly in the low-resource setting.
Generating Benchmarks for Factuality Evaluation of Language Models
FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements.
MuLER: Detailed and Scalable Reference-based Evaluation
For example, nouns and verbs are among the most frequent POS tags.
Evaluating and Improving the Coreference Capabilities of Machine Translation Models
Machine translation (MT) requires a wide range of linguistic capabilities, which current end-to-end models are expected to learn implicitly by observing aligned sentences in bilingual corpora.
A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions
We study the relation between visual input and linguistic choices by training classifiers to predict the probability of expressing a property from raw images, and find evidence supporting the claim that linguistic properties are constrained by visual context across languages.
Parallel Context Windows for Large Language Models
We present Parallel Context Windows (PCW), a method that alleviates the context window restriction for any off-the-shelf LLM without further training.
Cognitive Simplification Operations Improve Text Simplification
Text Simplification (TS) is the task of converting a text into a form that is easier to read while maintaining the meaning of the original text.
DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering
Question answering models commonly have access to two sources of "knowledge" during inference time: (1) parametric knowledge - the factual knowledge encoded in the model weights, and (2) contextual knowledge - external knowledge (e. g., a Wikipedia passage) given to the model to generate a grounded answer.
Topical Segmentation of Spoken Narratives: A Test Case on Holocaust Survivor Testimonies
The task of topical segmentation is well studied, but previous work has mostly addressed it in the context of structured, well-defined segments, such as segmentation into paragraphs, chapters, or segmenting text that originated from multiple sources.
Reinforcement Learning with Large Action Spaces for Neural Machine Translation
Applying Reinforcement learning (RL) following maximum likelihood estimation (MLE) pre-training is a versatile method for enhancing neural machine translation (NMT) performance.
PreQuEL: Quality Estimation of Machine Translation Outputs in Advance
We show that this augmentation method can improve the performance of the Quality-Estimation task as well.
A Computational Acquisition Model for Multimodal Word Categorization
Recent advances in self-supervised modeling of text and images open new opportunities for computational models of child language acquisition, which is believed to rely heavily on cross-modal signals.
Some Grammatical Errors are Frequent, Others are Important
In Grammatical Error Correction, systems are evaluated by the number of errors they correct.
MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning
Huge language models (LMs) have ushered in a new era for AI, serving as a gateway to natural-language-based knowledge tasks.
Semantics-aware Attention Improves Neural Machine Translation
We further show an additional gain when using both semantic and syntactic structures in some language pairs.
On the Relation between Syntactic Divergence and Zero-Shot Performance
We explore the link between the extent to which syntactic relations are preserved in translation and the ease of correctly constructing a parse tree in a zero-shot setting.
On Neurons Invariant to Sentence Structural Changes in Neural Machine Translation
We present a methodology that explores how sentence structure is reflected in neural representations of machine translation systems.
Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowords
We present a method for exploring regions around individual points in a contextualized vector space (particularly, BERT space), as a way to investigate how these regions correspond to word senses.
Cross-linguistically Consistent Semantic and Syntactic Annotation of Child-directed Speech
We then demonstrate the utility of the compiled corpora through (1) a longitudinal corpus study of the prevalence of different syntactic and semantic phenomena in the CDS, and (2) applying an existing computational model of language acquisition to the two corpora and briefly comparing the results across languages.
The Grammar-Learning Trajectories of Neural Language Models
These findings suggest that there is some mutual inductive bias that underlies these models' learning of linguistic phenomena.
Part of Speech and Universal Dependency effects on English Arabic Machine Translation
In this research paper, I will elaborate on a method to evaluate machine translation models based on their performance on underlying syntactical phenomena between English and Arabic languages.
$Q^{2}$: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering
Neural knowledge-grounded generative models for dialogue often produce content that is factually inconsistent with the knowledge they rely on, making them unreliable and limiting their applicability.
Mediators in Determining what Processing BERT Performs First
Probing neural models for the ability to perform downstream tasks using their activation patterns is often used to localize what parts of the network specialize in performing what tasks.
SERRANT: a syntactic classifier for English Grammatical Error Types
SERRANT is a system and code for automatic classification of English grammatical errors that combines SErCl and ERRANT.
Enhancing the Transformer Decoder with Transition-based Syntax
Notwithstanding recent advances, syntactic generalization remains a challenge for text decoders.
UCCA's Foundational Layer: Annotation Guidelines v2.1
This is the annotation manual for Universal Conceptual Cognitive Annotation (UCCA; Abend and Rappoport, 2013), specifically the Foundational Layer.
Cross-lingual Semantic Representation for NLP with UCCA
This is an introductory tutorial to UCCA (Universal Conceptual Cognitive Annotation), a cross-linguistically applicable framework for semantic representation, with corpora annotated in English, German and French, and ongoing annotation in Russian and Hebrew.
Semantic Structural Decomposition for Neural Machine Translation
Building on recent advances in semantic parsing and text simplification, we investigate the use of semantic splitting of the source sentence as preprocessing for machine translation.
Comparison by Conversion: Reverse-Engineering UCCA from Syntax and Lexical Semantics
Building robust natural language understanding systems will require a clear characterization of whether and how various linguistic meaning representations complement each other.
MRP 2020: The Second Shared Task on Cross-Framework and Cross-Lingual Meaning Representation Parsing
Extending a similar setup from the previous year, five distinct approaches to the representation of sentence meaning in the form of directed graphs were represented in the English training and evaluation data for the task, packaged in a uniform graph abstraction and serialization; for four of these representation frameworks, additional training and evaluation data was provided for one additional language per framework.
Classifying Syntactic Errors in Learner Language
We present a method for classifying syntactic errors in learner language, namely errors whose correction alters the morphosyntactic structure of a sentence.
PMI-Masking: Principled masking of correlated spans
Specifically, we show experimentally that PMI-Masking reaches the performance of prior masking approaches in half the training time, and consistently improves performance at the end of training.
Machine Reading of Historical Events
Machine reading is an ambitious goal in NLP that subsumes a wide range of text understanding capabilities.
Fine-Grained Analysis of Cross-Linguistic Syntactic Divergences
The patterns in which the syntax of different languages converges and diverges are often used to inform work on cross-lingual transfer.
Language (Re)modelling: Towards Embodied Language Understanding
While natural language understanding (NLU) is advancing rapidly, today's technology differs from human-like language understanding in fundamental ways, notably in its inferior efficiency, interpretability, and generalization.
Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation
We show that the state-of-the-art Transformer MT model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, long-distance dependencies remain a challenge for the model.
MRP 2019: Cross-Framework Meaning Representation Parsing
The 2019 Shared Task at the Conference for Computational Language Learning (CoNLL) was devoted to Meaning Representation Parsing (MRP) across frameworks.
Made for Each Other: Broad-coverage Semantic Structures Meet Preposition Supersenses
Universal Conceptual Cognitive Annotation (UCCA; Abend and Rappoport, 2013) is a typologically-informed, broad-coverage semantic annotation scheme that describes coarse-grained predicate-argument structure but currently lacks semantic roles.
Automatically Extracting Challenge Sets for Non local Phenomena in Neural Machine Translation
We show that the state of the art Transformer Machine Translation (MT) model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, long-distance dependencies remain a challenge for the model.
Preparing SNACS for Subjects and Objects
Research on adpositions and possessives in multiple languages has led to a small inventory of general-purpose meaning classes that disambiguate tokens.
On the Weaknesses of Reinforcement Learning for Neural Machine Translation
Reinforcement learning (RL) is frequently used to increase performance in text generation tasks, including machine translation (MT), notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN).
Semantically Constrained Multilayer Annotation: The Case of Coreference
We propose a coreference annotation scheme as a layer on top of the Universal Conceptual Cognitive Annotation foundational layer, treating units in predicate-argument structure as a basis for entity and event mentions.
Content Differences in Syntactic and Semantic Representation
Syntactic analysis plays an important role in semantic parsing, but the nature of this role remains a topic of ongoing debate.
The Language of Legal and Illegal Activity on the Darknet
The non-indexed parts of the Internet (the Darknet) have become a haven for both legal and illegal anonymous activity.
Content Differences in Syntactic and Semantic Representations
Syntactic analysis plays an important role in semantic parsing, but the nature of this role remains a topic of ongoing debate.
SemEval-2019 Task 1: Cross-lingual Semantic Parsing with UCCA
We present the SemEval 2019 shared task on UCCA parsing in English, German and French, and discuss the participating systems and results.
BLEU is Not Suitable for the Evaluation of Text Simplification
BLEU is widely considered to be an informative metric for text-to-text generation, including Text Simplification (TS).
Semantic Structural Evaluation for Text Simplification
Current measures for evaluating text simplification systems focus on evaluating lexical text aspects, neglecting its structural aspects.
Simple and Effective Text Simplification Using Semantic and Neural Methods
Here we present a simple and efficient splitting algorithm based on an automatic semantic parser.
Ranked #19 on
Text Simplification
on TurkCorpus
Universal Dependency Parsing with a General Transition-Based DAG Parser
This paper presents our experiments with applying TUPA to the CoNLL 2018 UD shared task.
Inherent Biases in Reference-based Evaluation for Grammatical Error Correction
The prevalent use of too few references for evaluating text-to-text generation is known to bias estimates of their quality (henceforth, low coverage bias or LCB).
SemEval 2019 Shared Task: Cross-lingual Semantic Parsing with UCCA - Call for Participation
Given the success of recent semantic parsing shared tasks (on SDP and AMR), we expect the task to have a significant contribution to the advancement of UCCA parsing in particular, and semantic parsing in general.
Comprehensive Supersense Disambiguation of English Prepositions and Possessives
Semantic relations are often signaled with prepositional or possessive marking--but extreme polysemy bedevils their analysis and automatic interpretation.
Ranked #4 on
Natural Language Understanding
on STREUSLE
(Role F1 (Preps) metric)
Multitask Parsing Across Semantic Representations
The ability to consolidate information of different types is at the core of intelligence, and has tremendous practical value in allowing learning for one task to benefit from generalizations learned for others.
Ranked #3 on
UCCA Parsing
on SemEval 2019 Task 1
Automatic Metric Validation for Grammatical Error Correction
Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings.
Inherent Biases in Reference based Evaluation for Grammatical Error Correction and Text Simplification
The prevalent use of too few references for evaluating text-to-text generation is known to bias estimates of their quality ({\it low coverage bias} or LCB).
Reference-less Measure of Faithfulness for Grammatical Error Correction
We propose USim, a semantic measure for Grammatical Error Correction (GEC) that measures the semantic faithfulness of the output to the source, thereby complementing existing reference-less measures (RLMs) for measuring the output's grammaticality.
The State of the Art in Semantic Representation
Semantic representation is receiving growing attention in NLP in the past few years, and many proposals for semantic schemes (e. g., AMR, UCCA, GMB, UDS) have been put forth.
Adposition and Case Supersenses v2.6: Guidelines for English
This document offers a detailed linguistic description of SNACS (Semantic Network of Adposition and Case Supersenses; Schneider et al., 2018), an inventory of 52 semantic labels ("supersenses") that characterize the use of adpositions and case markers at a somewhat coarse level of granularity, as demonstrated in the STREUSLE corpus (https://github. com/nert-nlp/streusle/ ; version 4. 5 tracks guidelines version 2. 6).
A Transition-Based Directed Acyclic Graph Parser for UCCA
We present the first parser for UCCA, a cross-linguistically applicable framework for semantic representation, which builds on extensive typological work and supports rapid annotation.
Ranked #4 on
UCCA Parsing
on SemEval 2019 Task 1
HUME: Human UCCA-Based Evaluation of Machine Translation
Human evaluation of machine translation normally uses sentence-level measures such as relative ranking or adequacy scales.
