Search Results for author:
Found 17 papers, 4 papers with code
How Human is Human Evaluation? Improving the Gold Standard for NLG with Utility Theory
Human ratings are treated as the gold standard in NLG evaluation.
Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words
Cosine similarity of contextual embeddings is used in many NLP tasks (e. g., QA, IR, MT) and metrics (e. g., BERTScore).
Richer Countries and Richer Representations
We examine whether some countries are more richly represented in embedding space than others.
Information-Theoretic Measures of Dataset Difficulty
Estimating the difficulty of a dataset typically involves comparing state-of-the-art models to humans; the bigger the performance gap, the harder the dataset is said to be.
Conditional probing: measuring usable information beyond a baseline
Probing experiments investigate the extent to which neural representations make properties -- like part-of-speech -- predictable.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Attention Flows are Shapley Value Explanations
Shapley Values, a solution to the credit assignment problem in cooperative game theory, are a popular type of explanation in machine learning, having been used to explain the importance of features, embeddings, and even neurons.
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform.
Frequency-based Distortions in Contextualized Word Embeddings
How does word frequency in pre-training data affect the behavior of similarity metrics in contextualized BERT embeddings?
Utility is in the Eye of the User: A Critique of NLP Leaderboards
Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models.
BLEU Neighbors: A Reference-less Approach to Automatic Evaluation
To this end, we propose BLEU Neighbors, a nearest neighbors model for estimating language quality by using the BLEU score as a kernel function.
Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Bounds
However, manually annotating a large dataset with a protected attribute is slow and expensive.
Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space
A notable property of word embeddings is that word relationships can exist as linear substructures in the embedding space.
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.
Understanding Undesirable Word Embedding Associations
Word embeddings are often criticized for capturing undesirable word associations such as gender stereotypes.
Towards Understanding Linear Word Analogies
A surprising property of word vectors is that word analogies can often be solved with vector arithmetic.
Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline
We propose a random walk model that is robust to this confound, where the probability of word generation is inversely related to the angular distance between the word and sentence embeddings.
