Search Results for author:
Found 71 papers, 35 papers with code
‘Tecnologica cosa’: Modeling Storyteller Personalities in Boccaccio’s ‘Decameron’
We explore Boccaccio’s Decameron to see how digital humanities tools can be used for tasks that have limited data in a language no longer in contemporary use: medieval Italian.
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice
We articulate fundamental mismatches between technical methods for machine unlearning in Generative AI, and documented aspirations for broader impact that these methods could have for law and policy.
Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices
To analyze the framework, we develop a taxonomy of all such operators based on their computational and algebraic properties and show that differences in the compute-optimal scaling laws are mostly governed by a small number of variables that we introduce.
QTIP: Quantization with Trellises and Incoherence Processing
Here, we introduce QTIP, which instead uses trellis coded quantization (TCQ) to achieve ultra-high-dimensional quantization.
Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes
In one dimension, we show that a step size less than $1/\lambda$ suffices for global convergence.
Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity
Zeroth-order optimization (ZO) is a memory-efficient strategy for fine-tuning Large Language Models using only forward passes.
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing their weights to low-precision.
Inference for Probabilistic Dependency Graphs
Probabilistic dependency graphs (PDGs) are a flexible class of probabilistic graphical models, subsuming Bayesian Networks and Factor Graphs.
ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers
We propose a memory-efficient finetuning algorithm for large language models (LLMs) that supports finetuning LLMs with 65B parameters in 2/3/4-bit precision on as little as one 24GB GPU.
QuIP: 2-Bit Quantization of Large Language Models With Guarantees
This work studies post-training parameter quantization in large language models (LLMs).
InfoDiffusion: Representation Learning Using Information Maximizing Diffusion Models
While diffusion models excel at generating high-quality samples, their latent variables typically lack semantic meaning and are not suitable for representation learning.
Coneheads: Hierarchy Aware Attention
These networks rely heavily on the dot product attention operator, which computes the similarity between two points by taking their inner product.
Shadow Cones: A Generalized Framework for Partial Order Embeddings
Specifically, we model partial orders as subset relations between shadows formed by a light source and opaque objects in hyperbolic space.
Coordinating Distributed Example Orders for Provably Accelerated Training
Recent research on online Gradient Balancing (GraB) has revealed that there exist permutation-based example orderings for SGD that are guaranteed to outperform random reshuffling (RR).
STEP: Learning N:M Structured Sparsity Masks from Scratch with Precondition
Recent innovations on hardware (e. g. Nvidia A100) have motivated learning N:M structured sparsity masks from scratch for fast model inference.
Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification
Variance in predictions across different trained models is a significant, under-explored source of error in fair binary classification.
MCTensor: A High-Precision Deep Learning Library with Multi-Component Floating-Point
In this paper, we introduce MCTensor, a library based on PyTorch for providing general-purpose and high-precision arithmetic for DL training.
Non-Determinism and the Lawlessness of Machine Learning Code
In this paper, we clarify the overlap and differences between these two concepts, and show that the effects of non-determinism, and consequently its implications for the law, become clearer from the perspective of reasoning about ML outputs as distributions over possible outcomes.
Low-Precision Stochastic Gradient Langevin Dynamics
While low-precision optimization has been widely used to accelerate deep learning, low-precision sampling remains largely unexplored.
GraB: Finding Provably Better Data Permutations than Random Reshuffling
To reduce the memory overhead, we leverage discrepancy minimization theory to propose an online Gradient Balancing algorithm (GraB) that enjoys the same rate as herding, while reducing the memory usage from $O(nd)$ to just $O(d)$ and computation from $O(n^2)$ to $O(n)$, where $d$ denotes the model dimension.
Structured Pruning is All You Need for Pruning CNNs at Initialization
In addition, since PreCropping compresses CNNs at initialization, the computational and memory costs of CNNs are reduced for both training and inference on commodity hardware.
Random Laplacian Features for Learning with Hyperbolic Space
Due to its geometric properties, hyperbolic space can support high-fidelity embeddings of tree- and graph-structured data, upon which various hyperbolic networks have been developed.
Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam
1-bit gradient compression and local steps are two representative techniques that enable drastic communication reduction in distributed SGD.
Understanding Hyperdimensional Computing for Parallel Single-Pass Learning
Using representation theory, we characterize which similarity matrices can be "expressed" by finite group VSA hypervectors, and we show how these VSAs can be constructed.
A General Analysis of Example-Selection for Stochastic Gradient Descent
In this paper, we develop a broad condition on the sequence of examples used by SGD that is sufficient to prove tight convergence rates in both strongly convex and non-convex settings.
Tecnologica cosa: Modeling Storyteller Personalities in Boccaccio's Decameron
We explore Boccaccio's Decameron to see how digital humanities tools can be used for tasks that have limited data in a language no longer in contemporary use: medieval Italian.
Model Preserving Compression for Neural Networks
After training complex deep learning models, a common task is to compress the model to reduce compute and storage demands.
Equivariant Manifold Flows
Tractably modelling distributions over manifolds has long been an important goal in the natural sciences.
How Low Can We Go: Trading Memory for Error in Low-Precision Training
Low-precision arithmetic trains deep learning models using less energy, less memory and less time.
Variance Reduced Training with Stratified Sampling for Forecasting Models
In large-scale time series forecasting, one often encounters the situation where the temporal patterns of time series, while drifting over time, differ from one another in the same dataset.
Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
Low-precision training has become a popular approach to reduce compute requirements, memory footprint, and energy consumption in supervised learning.
Hyperparameter Optimization Is Deceiving Us, and How to Stop It
Recent empirical work shows that inconsistent results based on choice of hyperparameter optimization (HPO) configuration are a widespread problem in ML research.
Revisiting BFfloat16 Training
We ask can we do pure 16-bit training which requires only 16-bit compute units, while still matching the model accuracy attained by 32-bit training.
Revisiting BFloat16 Training
State-of-the-art generic low-precision training algorithms use a mix of 16-bit and 32-bit precision, creating the folklore that 16-bit hardware compute units alone are not enough to maximize model accuracy.
Meta-Learning Divergences of Variational Inference
Variational inference (VI) plays an essential role in approximate Bayesian inference due to its computational efficiency and broad applicability.
Accuracy-Efficiency Trade-Offs and Accountability in Distributed ML Systems
Trade-offs between accuracy and efficiency pervade law, public health, and other non-computing domains, which have developed policies to guide how to balance the two in conditions of uncertainty.
Asymptotically Optimal Exact Minibatch Metropolis-Hastings
Metropolis-Hastings (MH) is a commonly-used MCMC algorithm, but it can be intractable on large datasets due to requiring computations over the whole dataset.
Neural Manifold Ordinary Differential Equations
To better conform to data geometry, recent deep generative modelling techniques adapt Euclidean constructions to non-Euclidean spaces.
Optimal Complexity in Decentralized Training
Decentralization is a promising method of scaling up parallel machine learning systems.
MixML: A Unified Analysis of Weakly Consistent Parallel Learning
Parallelism is a ubiquitous method for accelerating machine learning algorithms.
Optimizing JPEG Quantization for Classification Networks
While a long history of work has sought better Q-tables, existing work either seeks to minimize image distortion or to optimize for models of the human visual system.
AMAGOLD: Amortized Metropolis Adjustment for Efficient Stochastic Gradient MCMC
This improves performance, but introduces bias that can cause SGHMC to converge to the wrong distribution.
Differentiating through the Fréchet Mean
Recent advances in deep representation learning on Riemannian manifolds extend classical deep learning operations to better capture the geometry of the manifold.
Moniqua: Modulo Quantized Communication in Decentralized SGD
Running Stochastic Gradient Descent (SGD) in a decentralized fashion has shown promising results.
Poisson-Minibatching for Gibbs Sampling with Convergence Rate Guarantees
Gibbs sampling is a Markov chain Monte Carlo method that is often used for learning and inference on graphical models.
OverQ: Opportunistic Outlier Quantization for Neural Network Accelerators
Specialized hardware for handling activation outliers can enable low-precision neural networks, but at the cost of nontrivial area overhead.
QPyTorch: A Low-Precision Arithmetic Simulation Framework
Low-precision training reduces computational cost and produces efficient models.
PipeMare: Asynchronous Pipeline Parallel DNN Training
Pipeline parallelism (PP) when training neural networks enables larger models to be partitioned spatially, leading to both lower network communication and overall higher hardware utilization.
Dimension-Free Bounds for Low-Precision Training
Low-precision training is a promising way of decreasing the time and energy cost of training machine learning models.
SWALP : Stochastic Weight Averaging in Low-Precision Training
Low precision operations can provide scalability, memory savings, portability, and energy efficiency.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Distributed Learning with Sublinear Communication
In distributed statistical learning, $N$ samples are split across $m$ machines and a learner wishes to use minimal communication to learn as well as if the examples were on a single machine.
Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training.
Building Efficient Deep Neural Networks with Unitary Group Convolutions
UGConvs generalize two disparate ideas in CNN architecture, channel shuffling (i. e. ShuffleNet) and block-circulant networks (i. e. CirCNN), and provide unifying insights that lead to a deeper understanding of each technique.
Minibatch Gibbs Sampling on Large Graphical Models
Gibbs sampling is the de facto Markov chain Monte Carlo method used for inference and learning on large scale graphical models.
Channel Gating Neural Networks
Combining our method with knowledge distillation reduces the compute cost of ResNet-18 by 2. 6$\times$ without accuracy drop on ImageNet.
Representation Tradeoffs for Hyperbolic Embeddings
Given a tree, we give a combinatorial construction that embeds the tree in hyperbolic space with arbitrarily low distortion without using optimization.
The Convergence of Stochastic Gradient Descent in Asynchronous Shared Memory
Stochastic Gradient Descent (SGD) is a fundamental algorithm in machine learning, representing the optimization backbone for training several classic models, from regression to neural networks.
A Kernel Theory of Modern Data Augmentation
Data augmentation, a technique in which a training set is expanded with class-preserving transformations, is ubiquitous in modern machine learning pipelines.
High-Accuracy Low-Precision Training
Low-precision computation is often used to lower the time and energy cost of machine learning, and recently hardware accelerators have been developed to support it.
Gaussian Quadrature for Kernel Features
We show that deterministic feature maps can be constructed, for any $\gamma > 0$, to achieve error $\epsilon$ with $O(e^{e^\gamma} + \epsilon^{-1/\gamma})$ samples as $\epsilon$ goes to 0.
Accelerated Stochastic Power Iteration
We propose a simple variant of the power iteration with an added momentum term, that achieves both the optimal sample and iteration complexity.
Socratic Learning: Augmenting Generative Models to Incorporate Latent Subsets in Training Data
Prior work has explored learning accuracies for these sources even without ground truth labels, but they assume that a single accuracy parameter is sufficient to model the behavior of these sources over the entire training set.
Parallel SGD: When does averaging help?
Consider a number of workers running SGD independently on the same pool of data and averaging the models every once in a while -- a common but not well understood practice.
Scan Order in Gibbs Sampling: Models in Which it Matters and Bounds on How Much
Gibbs sampling is a Markov Chain Monte Carlo sampling technique that iteratively samples variables from their conditional distributions.
Data Programming: Creating Large Training Sets, Quickly
Additionally, in initial user studies we observed that data programming may be an easier way for non-experts to create machine learning models when training data is limited or unavailable.
Ensuring Rapid Mixing and Low Bias for Asynchronous Gibbs Sampling
Gibbs sampling is a Markov chain Monte Carlo technique commonly used for estimating marginal distributions.
Rapidly Mixing Gibbs Sampling for a Class of Factor Graphs Using Hierarchy Width
Gibbs sampling on factor graphs is a widely used inference technique, which often produces good empirical results.
Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
with relaxed assumptions on the sparsity of the problem; (2) we analyze asynchronous SGD algorithms for non-convex matrix problems including matrix completion; and (3) we design and analyze an asynchronous SGD algorithm, called Buckwild!, that uses lower-precision arithmetic.
Incremental Knowledge Base Construction Using DeepDive
Populating a database with unstructured information is a long-standing problem in industry and research that encompasses problems of extraction, cleaning, and integration.
Global Convergence of Stochastic Gradient Descent for Some Non-convex Matrix Problems
Stochastic gradient descent (SGD) on a low-rank factorization is commonly employed to speed up matrix problems including matrix completion, subspace tracking, and SDP relaxation.
