Search Results for author:
Found 39 papers, 19 papers with code
A Benchmark for Interpretability Methods in Deep Neural Networks
We propose an empirical measure of the approximate accuracy of feature importance estimates in deep neural networks.
The State of Sparsity in Deep Neural Networks
We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet.
What Do Compressed Deep Neural Networks Forget?
However, this measure of performance conceals significant differences in how different classes and images are impacted by model compression techniques.
The (Un)reliability of saliency methods
Saliency methods aim to explain the predictions of deep neural networks.
Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
The Mixture of Experts (MoE) is a widely known neural architecture where an ensemble of specialized sub-models optimizes overall performance with a constant computational cost.
The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI
The race to train language models on vast, diverse, and inconsistently documented datasets has raised pressing concerns about the legal and ethical risks for practitioners.
Estimating Example Difficulty Using Variance of Gradients
In this work, we propose Variance of Gradients (VoG) as a valuable and efficient metric to rank data by difficulty and to surface a tractable subset of the most challenging examples for human-in-the-loop auditing.
Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models
Considerable effort has been dedicated to mitigating toxicity, but existing methods often require drastic modifications to model parameters or the use of computationally intensive auxiliary models.
From One to Many: Expanding the Scope of Toxicity Mitigation in Language Models
To date, toxicity mitigation in language models has almost entirely been focused on single-language settings.
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics
Modern machine learning research relies on relatively few carefully curated datasets.
The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
We present our findings as the starting point for further research into evaluating how LLMs interpret language in context and to drive the development of more pragmatic and useful models of human discourse.
Randomness In Neural Network Training: Characterizing The Impact of Tooling
However, we also find that the cost of ensuring determinism varies dramatically between neural network architectures and hardware types, e. g., with overhead up to $746\%$, $241\%$, and $196\%$ on a spectrum of widely used GPU accelerator architectures, relative to non-deterministic training.
The Grand Illusion: The Myth of Software Portability and Implications for ML Progress
Collectively, our results reveal how costly straying from a narrow set of hardware-software combinations can be - and suggest that specialization of hardware impedes innovation in machine learning research.
A Tale Of Two Long Tails
As machine learning models are increasingly employed to assist human decision-makers, it becomes critical to communicate the uncertainty associated with these model predictions.
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research
We evaluate the implications of these changes on the reproducibility of findings that compare the relative merits of models and methods that aim to curb toxicity.
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation
However, evaluation of the trade-offs incurred by popular compression techniques has been centered on high-resource datasets.
The Hardware Lottery
Hardware, systems and algorithms research communities have historically had different incentive structures and fluctuating motivation to engage with each other explicitly.
Studying the impact of magnitude pruning on contrastive learning methods
We study the impact of different pruning techniques on the representation learned by deep neural networks trained with contrastive loss functions.
Locally Differentially Private Document Generation Using Zero Shot Prompting
Numerous studies have highlighted the privacy risks associated with pretrained large language models.
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
With the recent wave of progress in artificial intelligence (AI) has come a growing awareness of the large-scale impacts of AI systems, and recognition that existing regulations and norms in industry and academia are insufficient to ensure responsible AI development.
Computers and Society
Characterising Bias in Compressed Models
However, overall accuracy hides disproportionately high errors on a small subset of examples; we call this subset Compression Identified Exemplars (CIE).
Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
Based upon these findings, we show that gradient flow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime.
When does loss-based prioritization fail?
Each example is propagated forward and backward through the network the same amount of times, independent of how much the example contributes to the learning protocol.
Selective Brain Damage: Measuring the Disparate Impact of Model Pruning
Neural network pruning techniques have demonstrated it is possible to remove the majority of weights in a network with surprisingly little degradation to top-1 test set accuracy.
When less is more: Simplifying inputs aids neural network understanding
To answer these questions, we need a clear measure of input simplicity (or inversely, complexity), an optimization objective that correlates with simplification, and a framework to incorporate such objective into training and inference.
Robust Distillation for Worst-class Performance
We show empirically that our robust distillation techniques not only achieve better worst-class performance, but also lead to Pareto improvement in the tradeoff between overall performance and worst-class performance compared to other baseline methods.
Efficient Methods for Natural Language Processing: A Survey
Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows.
Intriguing Properties of Compression on Multilingual Models
Multilingual models are often particularly dependent on scaling to generalize to a growing number of languages.
FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensembling
Ensembling multiple Deep Neural Networks (DNNs) is a simple and effective way to improve top-line metrics and to outperform a larger single model.
Evaluating the Social Impact of Generative AI Systems in Systems and Society
We move toward a standard approach in evaluating a generative AI system for any modality, in two overarching categories: what is able to be evaluated in a base system that has no predetermined application and what is able to be evaluated in society.
Frontier AI Regulation: Managing Emerging Risks to Public Safety
To address these challenges, at least three building blocks for the regulation of frontier models are needed: (1) standard-setting processes to identify appropriate requirements for frontier AI developers, (2) registration and reporting requirements to provide regulators with visibility into frontier AI development processes, and (3) mechanisms to ensure compliance with safety standards for the development and deployment of frontier AI models.
When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale
In this work, we take a wider view and explore scalable estimates of data quality that can be used to systematically measure the quality of pretraining data.
Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluation
Human evaluation is increasingly critical for assessing large language models, capturing linguistic nuances, and reflecting user preferences more accurately than traditional automated metrics.
Elo Uncovered: Robustness and Best Practices in Language Model Evaluation
In Natural Language Processing (NLP), the Elo rating system, originally designed for ranking players in dynamic games such as chess, is increasingly being used to evaluate Large Language Models (LLMs) through "A vs B" paired comparisons.
Generalisable Agents for Neural Network Optimisation
Optimising deep neural networks is a challenging task due to complex training dynamics, high computational requirements, and long training times.
On The Fairness Impacts of Hardware Selection in Machine Learning
In the machine learning ecosystem, hardware selection is often regarded as a mere utility, overshadowed by the spotlight on algorithms and data.
Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
The Aya initiative also serves as a valuable case study in participatory research, involving collaborators from 119 countries.
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
Recent breakthroughs in large language models (LLMs) have centered around a handful of data-rich languages.
Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
AI alignment in the shape of Reinforcement Learning from Human Feedback (RLHF) is increasingly treated as a crucial ingredient for high performance large language models.
