Search Results for author:
Found 27 papers, 12 papers with code
BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance on a wide variety of biomedical NLP tasks.
The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning
We study two natural scaling techniques -- weight pruning and simply training a smaller or larger model, which we refer to as dense scaling -- and their effects on two core capabilities of LLMs: (a) recalling facts presented during pre-training and (b) processing information presented in-context during inference.
Turaco: Complexity-Guided Data Sampling for Training Neural Surrogates of Programs
We first characterize the proportion of data to sample from each region of a program's input space (corresponding to different execution paths of the program) based on the complexity of learning a surrogate of the corresponding execution path.
Computably Continuous Reinforcement-Learning Objectives are PAC-learnable
In particular, for the analysis that considers only sample complexity, we prove that if an objective given as an oracle is uniformly continuous, then it is PAC-learnable.
Acela: Predictable Datacenter-level Maintenance Job Scheduling
While it is tempting to use prior machine learning techniques for predicting job duration, we find that the structure of the maintenance job scheduling problem creates a unique challenge.
Pruning's Effect on Generalization Through the Lens of Training and Regularization
Pruning models in this over-parameterized regime leads to a contradiction -- while theory predicts that reducing model size harms generalization, pruning to a range of sparsities nonetheless improves it.
SCOPE: Safe Exploration for Dynamic Computer Systems Optimization
Such solutions monitor past system executions to learn the system's behavior under different hardware resource allocations before dynamically tuning resources to optimize the application execution.
Cello: Efficient Computer Systems Optimization with Predictive Early Termination and Censored Regression
Our evaluation shows that compared to the state-of-the-art SEML approach in computer systems optimization, Cello improves latency by 1. 19X for minimizing latency under a power constraint, and improves energy by 1. 18X for minimizing energy under a latency constraint.
Programming with Neural Surrogates of Programs
With surrogate adaptation, programmers develop a surrogate of a program then retrain that surrogate on a different task.
On the (In)Tractability of Reinforcement Learning for LTL Objectives
In recent years, researchers have made significant progress in devising reinforcement-learning algorithms for optimizing linear temporal logic (LTL) objectives and LTL-like objectives.
Reinforcement Learning with General LTL Objectives is Intractable
In recent years, researchers have made significant progress in devising reinforcement-learning algorithms for optimizing linear temporal logic (LTL) objectives and LTL-like objectives.
Studying the Consistency and Composability of Lottery Ticket Pruning Masks
Magnitude pruning is a common, effective technique to identify sparse subnetworks at little cost to accuracy.
The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
We extend the scope of LTH and question whether matching subnetworks still exist in pre-trained computer vision models, that enjoy the same downstream transfer performance.
DiffTune: Optimizing CPU Simulator Parameters with Learned Differentiable Surrogates
In this paper we present DiffTune, a system for learning the parameters of x86 basic block CPU simulators from coarse-grained end-to-end measurements.
Pruning Neural Networks at Initialization: Why are We Missing the Mark?
Recent work has explored the possibility of pruning neural networks at initialization.
The Lottery Ticket Hypothesis for Pre-trained BERT Networks
For a range of downstream tasks, we indeed find matching subnetworks at 40% to 90% sparsity.
$λ_S$: Computable Semantics for Differentiable Programming with Higher-Order Functions and Datatypes
Deep learning is moving towards increasingly sophisticated optimization objectives that employ higher-order functions, such as integration, continuous optimization, and root-finding.
Programming Languages Logic in Computer Science D.3.1; F.3.2
On the Predictability of Pruning Across Scales
We show that the error of iteratively magnitude-pruned networks empirically follows a scaling law with interpretable coefficients that depend on the architecture and task.
TIRAMISU: A Polyhedral Compiler for Dense and Sparse Deep Learning
In this paper, we demonstrate a compiler that can optimize sparse and recurrent neural networks, both of which are currently outside of the scope of existing neural network compilers (sparse neural networks here stand for networks that can be accelerated with sparse tensor algebra techniques).
Comparing Rewinding and Fine-tuning in Neural Network Pruning
Learning rate rewinding (which we propose) trains the unpruned weights from their final values using the same learning rate schedule as weight rewinding.
Linear Mode Connectivity and the Lottery Ticket Hypothesis
We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e. g., random data order and augmentation).
Compiler Auto-Vectorization with Imitation Learning
We show that the learnt policy produces a vectorization scheme which is better than industry standard compiler heuristics both in terms of static measures and runtime performance.
Mode Connectivity and Sparse Neural Networks
We observe that these subnetworks match the accuracy of the full network only when two SGD runs for the same subnetwork are connected by linear paths with the no change in test error.
Stabilizing the Lottery Ticket Hypothesis
With this change, it finds small subnetworks of deeper networks (e. g., 80% sparsity on Resnet-50) that can complete the training process to match the accuracy of the original network on more challenging tasks (e. g., ImageNet).
Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks
Predicting the number of clock cycles a processor takes to execute a block of assembly instructions in steady state (the throughput) is important for both compiler designers and performance engineers.
The Three Pillars of Machine Programming
In this position paper, we describe our vision of the future of machine programming through a categorical examination of three pillars of research.
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations.
