Search Results for author:
Found 12 papers, 8 papers with code
The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models
Instead of finding competing subnetworks, we find that all subnetworks -- whether they generalize or not -- share a set of attention heads, which we refer to as the heuristic core.
Interpretability Illusions in the Generalization of Simplified Models
A common method to study deep learning systems is to use simplified model representations -- for example, using singular value decomposition to visualize the model's hidden states in a lower dimensional space.
Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
This approach - which we call the teleological approach - leads us to identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input.
Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations
We investigate the inductive biases of ICL from the perspective of feature bias: which feature ICL is more likely to use given a set of underspecified demonstrations in which two features are equally predictive of the labels.
Finding Dataset Shortcuts with Grammar Induction
Many NLP datasets have been found to contain shortcuts: simple decision rules that achieve surprisingly high accuracy.
The Vendi Score: A Diversity Evaluation Metric for Machine Learning
Importantly, unlike many existing metrics in ML, the Vendi Score does not require a reference dataset or distribution over samples or labels, it is therefore general and applicable to any generative model, decoding algorithm, and dataset from any domain where similarity can be defined.
Single-dataset Experts for Multi-dataset Question Answering
Many datasets have been created for training reading comprehension models, and a natural question is whether we can combine them to build models that (1) perform better on all of the training datasets and (2) generalize and transfer better to new datasets.
Factual Probing Is [MASK]: Learning vs. Learning to Recall
Petroni et al. (2019) demonstrated that it is possible to retrieve world facts from a pre-trained language model by expressing them as cloze-style prompts and interpret the model's prediction accuracy as a lower bound on the amount of factual information it encodes.
ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation Networks
Scientific article summarization is challenging: large, annotated corpora are not available, and the summary should ideally include the article's impacts on research community.
Ranked #1 on
Scientific Document Summarization
on CL-SciSumm
Syntax-aware Neural Semantic Role Labeling with Supertags
We introduce a new syntax-aware model for dependency-based semantic role labeling that outperforms syntax-agnostic models for English and Spanish.
Deep density networks and uncertainty in recommender systems
Building robust online content recommendation systems requires learning complex interactions between user preferences and content features.
