Search Results for author:
Found 73 papers, 41 papers with code
Active Learning for BERT: An Empirical Study
Here, we present a large-scale empirical study on active learning techniques for BERT-based classification, addressing a diverse set of AL strategies and datasets.
Q^{2}: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering
Neural knowledge-grounded generative models for dialogue often produce content that is factually inconsistent with the knowledge they rely on, making them unreliable and limiting their applicability.
Abstractive Text Summarization
Natural Language Inference
+3
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
Cultural biases in multilingual datasets pose significant challenges for their effectiveness as global benchmarks.
ZipNN: Lossless Compression for AI Models
With the growth of model sizes and the scale of their deployment, their sheer size burdens the infrastructure requiring more network and more storage to accommodate these.
Model merging with SVD to tie the Knots
We study this phenomenon and observe that the weights of LoRA finetuned models showcase a lower degree of alignment compared to their fully-finetuned counterparts.
A Hitchhiker's Guide to Scaling Law Estimation
Moreover, while different model families differ scaling behavior, they are often similar enough that a target model's behavior can be predicted from a single model with the same architecture, along with scaling parameter estimates derived from other model families.
Can You Trust Your Metric? Automatic Concatenation-Based Tests for Metric Validity
We applied these tests in a model safety scenario to assess the reliability of harmfulness detection metrics, uncovering a number of inconsistencies.
Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMs
These findings highlight the need for improved multilingual knowledge representation in LLMs and suggest a path for the development of more robust and consistent multilingual LLMs.
The Future of Open Human Feedback
In this work, we bring together interdisciplinary experts to assess the opportunities and challenges to realizing an open ecosystem of human feedback for AI.
The ShareLM Collection and Plugin: Contributing Human-Model Chats for the Benefit of the Community
We introduce the ShareLM collection, a unified set of human conversations with large language models, and its accompanying plugin, a Web extension for voluntarily contributing user-model conversations.
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to a particular domain or task.
Data Contamination Report from the 2024 CONDA Shared Task
The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models.
Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench
Despite the crucial role of BAT for benchmark builders and consumers, there are no standardized procedures for such agreement testing.
Learning from Naturally Occurring Feedback
Training with the extracted feedback shows significant performance improvements over baseline models, demonstrating the efficacy of our approach in enhancing model alignment to human preferences.
Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Fine-tuning large language models (LLMs) with low-rank adaptations (LoRAs) has become common practice, often yielding numerous copies of the same LLM differing only in their LoRA updates.
Efficient multi-prompt evaluation of LLMs
Most popular benchmarks for comparing LLMs rely on a limited set of prompt templates, which may not fully capture the LLMs' abilities and can affect the reproducibility of results on leaderboards.
Holmes: A Benchmark to Assess the Linguistic Competence of Language Models
We introduce Holmes, a new benchmark designed to assess language models (LMs) linguistic competence - their unconscious understanding of linguistic phenomena.
[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
The big changes for this year's competition are as follows: First, we replace the loose track with a paper track, which allows (for example) non-model-based submissions, novel cognitively-inspired benchmarks, or analysis techniques.
Lossless and Near-Lossless Compression for Foundation Models
With the growth of model sizes and scale of their deployment, their sheer size burdens the infrastructure requiring more network and more storage to accommodate these.
NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning
Language models struggle with handling numerical data and performing arithmetic operations.
Asymmetry in Low-Rank Adapters of Foundation Models
Specifically, when updating the parameter matrices of a neural network by adding a product $BA$, we observe that the $B$ and $A$ matrices have distinct functions: $A$ extracts features from the input, while $B$ uses these features to create the desired output.
tinyBenchmarks: evaluating LLMs with fewer examples
The versatility of large language models (LLMs) led to the creation of diverse benchmarks that thoroughly test a variety of language models' abilities.
Label-Efficient Model Selection for Text Generation
Model selection for a given target task can be costly, as it may entail extensive annotation of the quality of outputs of different models.
Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI
In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, and model combinations.
Genie: Achieving Human Parity in Content-Grounded Datasets Generation
Furthermore, we compare models trained on our data with models trained on human-written data -- ELI5 and ASQA for LFQA and CNN-DailyMail for Summarization.
Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability
Given a collection of seed documents, DCT prompts LMs to generate additional text implied by these documents, reason globally about the correctness of this generated text, and finally fine-tune on text inferred to be correct.
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
Despite the efficiency of PEFT methods, the size of expert models can make it onerous to retrieve expert models per query over high-latency networks like the Internet or serve multiple experts on a single GPU.
Human Learning by Model Feedback: The Dynamics of Iterative Prompting with Midjourney
Generating images with a Text-to-Image model often requires multiple trials, where human users iteratively update their prompt based on feedback, namely the output image.
Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion
Model fusion research aims to aggregate the knowledge of multiple individual models to enhance performance by combining their weights.
Efficient Benchmarking of Language Models
The increasing versatility of language models (LMs) has given rise to a new class of benchmarks that comprehensively assess a broad range of capabilities.
TIES-Merging: Resolving Interference When Merging Models
To address this, we propose our method, TRIM, ELECT SIGN & MERGE (TIES-Merging), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign.
MuLER: Detailed and Scalable Reference-based Evaluation
For example, nouns and verbs are among the most frequent POS tags.
Jump to Conclusions: Short-Cutting Transformers With Linear Transformations
This approximation far exceeds the prevailing practice of inspecting hidden representations from all layers, in the space of the final layer.
Knowledge is a Region in Weight Space for Fine-tuned Language Models
Notably, we show that language models that have been finetuned on the same dataset form a tight cluster in the weight space, while models finetuned on different datasets from the same underlying task form a looser cluster.
Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
In partnership with CoNLL and CMCL, we provide a platform for approaches to pretraining with a limited-size corpus sourced from data inspired by the input to children.
ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning
We propose a new paradigm to continually evolve pretrained models, denoted ColD Fusion.
DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering
Question answering models commonly have access to two sources of "knowledge" during inference time: (1) parametric knowledge - the factual knowledge encoded in the model weights, and (2) contextual knowledge - external knowledge (e. g., a Wikipedia passage) given to the model to generate a grounded answer.
Where to start? Analyzing the potential value of intermediate models
Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset.
Reinforcement Learning with Large Action Spaces for Neural Machine Translation
Applying Reinforcement learning (RL) following maximum likelihood estimation (MLE) pre-training is a versatile method for enhancing neural machine translation (NMT) performance.
Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours
Text classification can be useful in many real-world scenarios, saving a lot of time for end users.
PreQuEL: Quality Estimation of Machine Translation Outputs in Advance
We show that this augmentation method can improve the performance of the Quality-Estimation task as well.
Some Grammatical Errors are Frequent, Others are Important
In Grammatical Error Correction, systems are evaluated by the number of errors they correct.
Fusing finetuned models for better pretraining
We also show that fusing is often better than intertraining.
Cluster & Tune: Boost Cold Start Performance in Text Classification
In real-world scenarios, a text classification task often begins with a cold start, when labeled data is scarce.
Semantics-aware Attention Improves Neural Machine Translation
We further show an additional gain when using both semantic and syntactic structures in some language pairs.
On Neurons Invariant to Sentence Structural Changes in Neural Machine Translation
We present a methodology that explores how sentence structure is reflected in neural representations of machine translation systems.
The Grammar-Learning Trajectories of Neural Language Models
These findings suggest that there is some mutual inductive bias that underlies these models' learning of linguistic phenomena.
ComSum: Commit Messages Summarization and Meaning Preservation
We present ComSum, a data set of 7 million commit messages for text summarization.
Part of Speech and Universal Dependency effects on English Arabic Machine Translation
In this research paper, I will elaborate on a method to evaluate machine translation models based on their performance on underlying syntactical phenomena between English and Arabic languages.
$Q^{2}$: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering
Neural knowledge-grounded generative models for dialogue often produce content that is factually inconsistent with the knowledge they rely on, making them unreliable and limiting their applicability.
Mediators in Determining what Processing BERT Performs First
Probing neural models for the ability to perform downstream tasks using their activation patterns is often used to localize what parts of the network specialize in performing what tasks.
GrASP: A Library for Extracting and Exploring Human-Interpretable Textual Patterns
Data exploration is an important step of every data science and machine learning project, including those involving textual data.
SERRANT: a syntactic classifier for English Grammatical Error Types
SERRANT is a system and code for automatic classification of English grammatical errors that combines SErCl and ERRANT.
Enhancing the Transformer Decoder with Transition-based Syntax
Notwithstanding recent advances, syntactic generalization remains a challenge for text decoders.
Cluster & Tune: Enhance BERT Performance in Low Resource Text Classification
In such low resources scenarios, we suggest performing an unsupervised classification task prior to fine-tuning on the target task.
Classifying Syntactic Errors in Learner Language
We present a method for classifying syntactic errors in learner language, namely errors whose correction alters the morphosyntactic structure of a sentence.
Unsupervised Expressive Rules Provide Explainability and Assist Human Experts Grasping New Domains
Approaching new data can be quite deterrent; you do not know how your categories of interest are realized in it, commonly, there is no labeled data at hand, and the performance of domain adaptation methods is unsatisfactory.
Corpus Wide Argument Mining -- a Working Solution
One of the main tasks in argument mining is the retrieval of argumentative content pertaining to a given topic.
Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation
We show that the state-of-the-art Transformer MT model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, long-distance dependencies remain a challenge for the model.
Automatically Extracting Challenge Sets for Non local Phenomena in Neural Machine Translation
We show that the state of the art Transformer Machine Translation (MT) model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, long-distance dependencies remain a challenge for the model.
Are You Convinced? Choosing the More Convincing Evidence with a Siamese Network
With the advancement in argument detection, we suggest to pay more attention to the challenging task of identifying the more convincing arguments.
On the Weaknesses of Reinforcement Learning for Neural Machine Translation
Reinforcement learning (RL) is frequently used to increase performance in text generation tasks, including machine translation (MT), notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN).
Learning to combine Grammatical Error Corrections
We also present a spellchecker created for this task which outperforms standard spellcheckers tested on the task of spellchecking.
Ranked #9 on
Grammatical Error Correction
on BEA-2019 (test)
Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets
We further show that this pattern of results reflects the interplay between the way neural networks learn benchmark datasets.
The Language of Legal and Illegal Activity on the Darknet
The non-indexed parts of the Internet (the Darknet) have become a haven for both legal and illegal anonymous activity.
SemEval-2019 Task 1: Cross-lingual Semantic Parsing with UCCA
We present the SemEval 2019 shared task on UCCA parsing in English, German and French, and discuss the participating systems and results.
Inherent Biases in Reference-based Evaluation for Grammatical Error Correction
The prevalent use of too few references for evaluating text-to-text generation is known to bias estimates of their quality (henceforth, low coverage bias or LCB).
Will it Blend? Blending Weak and Strong Labeled Data in a Neural Network for Argumentation Mining
We propose a methodology to blend high quality but scarce strong labeled data with noisy but abundant weak labeled data during the training of neural networks.
SemEval 2019 Shared Task: Cross-lingual Semantic Parsing with UCCA - Call for Participation
Given the success of recent semantic parsing shared tasks (on SDP and AMR), we expect the task to have a significant contribution to the advancement of UCCA parsing in particular, and semantic parsing in general.
Automatic Metric Validation for Grammatical Error Correction
Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings.
Inherent Biases in Reference based Evaluation for Grammatical Error Correction and Text Simplification
The prevalent use of too few references for evaluating text-to-text generation is known to bias estimates of their quality ({\it low coverage bias} or LCB).
Reference-less Measure of Faithfulness for Grammatical Error Correction
We propose USim, a semantic measure for Grammatical Error Correction (GEC) that measures the semantic faithfulness of the output to the source, thereby complementing existing reference-less measures (RLMs) for measuring the output's grammaticality.
DORA The Explorer: Directed Outreaching Reinforcement Action-Selection
We compare our approach to commonly used RL techniques, and show that using $E$-values improves learning and performance over traditional counters.
