Search Results for author:
Found 60 papers, 35 papers with code
NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQA
The Privacy Preserving Federated Learning Document VQA (PFL-DocVQA) competition challenged the community to develop provably private and communication-efficient solutions in a federated setting for a real-life use case: invoice processing.
Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models
Understanding and mitigating the potential risks associated with foundation models (FMs) hinges on developing effective interpretability methods.
Position: LLM Unlearning Benchmarks are Weak Measures of Progress
Unlearning methods have the potential to improve the privacy and safety of large language models (LLMs) by removing sensitive or harmful information post hoc.
Agreement-Based Cascading for Efficient Inference
Adaptive inference schemes reduce the cost of machine learning inference by assigning smaller models to easier examples, attempting to avoid invocation of larger models when possible.
Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients
Large language model (LLM) training and finetuning are often bottlenecked by limited GPU memory.
RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
With this per-step scheme, we are able to attain consistent gains over only positive data, attaining performance similar to amplifying the amount of synthetic data by $\mathbf{8 \times}$.
Jogging the Memory of Unlearned LLMs Through Targeted Relearning Attacks
Machine unlearning is a promising approach to mitigate undesirable memorization of training data in LLMs.
Federated LoRA with Sparse Communication
Prior work that has studied LoRA in the context of federated learning has focused on improving LoRA's robustness to heterogeneity and privacy.
Privacy Amplification for the Gaussian Mechanism via Bounded Support
Data-dependent privacy accounting frameworks such as per-instance differential privacy (pDP) and Fisher information loss (FIL) confer fine-grained privacy guarantees for individuals in a fixed training dataset.
Many-Objective Multi-Solution Transport
Optimizing the performance of many objectives (instantiated by tasks or clients) jointly with a few Pareto stationary solutions (models) is critical in machine learning.
Guardrail Baselines for Unlearning in LLMs
Recent work has demonstrated that finetuning is a promising approach to 'unlearn' concepts from large language models.
No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices
Advances in generative models have made it possible for AI-generated text, code, and images to mirror human-generated content in many applications.
Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes
Given the generational gap in available hardware between lay practitioners and the most endowed institutions, LLMs are becoming increasingly inaccessible as they grow in size.
On the Benefits of Public Representations for Private Transfer Learning under Distribution Shift
Public pretraining is a promising approach to improve differentially private model training.
Complementary Benefits of Contrastive Learning and Self-Training Under Distribution Shift
Self-training and contrastive learning have emerged as leading techniques for incorporating unlabeled data, both under distribution shift (unsupervised domain adaptation) and when it is absent (semi-supervised learning).
Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies
Unrolled computation graphs are prevalent throughout machine learning but present challenges to automatic differentiation (AD) gradient estimation methods when their loss functions exhibit extreme local sensitivtiy, discontinuity, or blackbox characteristics.
Federated Learning as a Network Effects Game
Federated Learning (FL) aims to foster collaboration among a population of clients to improve the accuracy of machine learning without directly sharing local data.
Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shifts
Some robust training algorithms (e. g., Group DRO) specialize to group shifts and require group information on all training points.
On Noisy Evaluation in Federated Hyperparameter Tuning
In this work, we perform the first systematic study on the effect of noisy evaluation in federated hyperparameter tuning.
Differentially Private Adaptive Optimization with Delayed Preconditioners
Privacy noise may negate the benefits of using adaptive optimizers in differentially private model training.
Validating Large Language Models with ReLM
Although large language models (LLMs) have been touted for their ability to generate natural-sounding text, there are growing concerns around possible negative effects of LLMs such as data memorization, bias, and inappropriate language.
Motley: Benchmarking Heterogeneity and Personalization in Federated Learning
To better answer these questions, we propose Motley, a benchmark for personalized federated learning.
On Privacy and Personalization in Cross-Silo Federated Learning
While the application of differential privacy (DP) has been well-studied in cross-device federated learning (FL), there is a lack of work considering DP and its implications for cross-silo FL, a setting characterized by a limited number of clients each containing many data subjects.
Adversarial Unlearning: Reducing Confidence Along Adversarial Directions
Supervised learning methods trained with maximum likelihood objectives often overfit on training data.
Maximizing Global Model Appeal in Federated Learning
We provide convergence guarantees for MaxFL and show that MaxFL achieves a $22$-$40\%$ and $18$-$50\%$ test accuracy improvement for the training clients and unseen clients respectively, compared to a wide range of FL modeling approaches, including those that tackle data heterogeneity, aim to incentivize clients, and learn personalized or fair models.
Fair Federated Learning via Bounded Group Loss
In particular, we explore and extend the notion of Bounded Group Loss as a theoretically-grounded approach for group fairness.
Private Adaptive Optimization with Side Information
Adaptive optimization methods have become the default solvers for many machine learning tasks.
Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines
Our analysis of over two million ML jobs in Google datacenters reveals that a significant fraction of model training jobs could benefit from faster input data pipelines.
Diverse Client Selection for Federated Learning via Submodular Maximization
In every communication round of federated learning, a random subset of clients communicate their model updates back to the server which then aggregates them all.
On Tilted Losses in Machine Learning: Theory and Applications
Finally, we demonstrate that TERM can be used for a multitude of applications in machine learning, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance.
Private Multi-Task Learning: Formulation and Applications to Federated Learning
Many problems in machine learning rely on multi-task learning (MTL), in which the goal is to solve multiple related machine learning tasks simultaneously.
A Field Guide to Federated Optimization
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection.
On Large-Cohort Training for Federated Learning
Our work highlights a number of challenges stemming from the use of larger cohorts.
Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
Tuning hyperparameters is a crucial but arduous part of the machine learning pipeline.
Heterogeneity for the Win: One-Shot Federated Clustering
In this work, we explore the unique challenges -- and opportunities -- of unsupervised federated learning (FL).
Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
We categorize meta-learning evaluation into two settings: $\textit{in-distribution}$ [ID], in which the train and test tasks are sampled $\textit{iid}$ from the same underlying task distribution, and $\textit{out-of-distribution}$ [OOD], in which they are not.
Label Leakage and Protection in Two-party Split Learning
Two-party split learning is a popular technique for learning a model across feature-partitioned data.
Ditto: Fair and Robust Federated Learning Through Personalization
Fairness and robustness are two important concerns for federated learning systems.
Is Support Set Diversity Necessary for Meta-Learning?
Meta-learning is a popular framework for learning with limited data in which an algorithm is produced by training over multiple few-shot learning tasks.
Tilted Empirical Risk Minimization
Empirical risk minimization (ERM) is typically designed to perform well on the average loss, which can result in estimators that are sensitive to outliers, generalize poorly, or treat subgroups unfairly.
FedDANE: A Federated Newton-Type Method
Federated learning aims to jointly learn statistical models over massively distributed remote devices.
Enhancing the Privacy of Federated Learning with Sketching
Federated learning methods run training tasks directly on user devices and do not share the raw user data with third parties.
Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketches
Many existing works treat these concerns separately.
Progressive Compressed Records: Taking a Byte out of Deep Learning Data
Deep learning accelerators efficiently train over vast and growing amounts of data, placing a newfound burden on commodity networks and storage devices.
Federated Learning: Challenges, Methods, and Future Directions
Federated learning involves training statistical models over remote devices or siloed data centers, such as mobile phones or hospitals, while keeping data localized.
Fair Resource Allocation in Federated Learning
Federated learning involves training statistical models in massive, heterogeneous networks.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
One-Shot Federated Learning
We present one-shot federated learning, where a central server learns a global model over a network of federated devices in a single round of communication.
Federated Optimization in Heterogeneous Networks
Theoretically, we provide convergence guarantees for our framework when learning over data from non-identical distributions (statistical heterogeneity), and while adhering to device-level systems constraints by allowing each participating device to perform a variable amount of work (systems heterogeneity).
LEAF: A Benchmark for Federated Settings
Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts of data each day.
Efficient Augmentation via Data Subsampling
Data augmentation is commonly used to encode invariances in learning methods.
Model Aggregation via Good-Enough Model Spaces
In this work, we present Good-Enough Model Spaces (GEMS), a novel framework for learning a global model by carefully intersecting the sets of "good-enough" models across each node.
A Kernel Theory of Modern Data Augmentation
Data augmentation, a technique in which a training set is expanded with class-preserving transformations, is ubiquitous in modern machine learning pipelines.
Federated Multi-Task Learning
Federated learning poses new statistical and systems challenges in training machine learning models over distributed networks of devices.
CoCoA: A General Framework for Communication-Efficient Distributed Optimization
The scale of modern datasets necessitates the development of efficient distributed optimization methods for machine learning.
Distributed Optimization with Arbitrary Local Solvers
To this end, we present a framework for distributed optimization that both allows the flexibility of arbitrary solvers to be used on each (single) machine locally, and yet maintains competitive performance against other state-of-the-art special-purpose distributed methods.
L1-Regularized Distributed Optimization: A Communication-Efficient Primal-Dual Framework
Despite the importance of sparsity in many large-scale applications, there are few methods for distributed optimization of sparsity-inducing objectives.
Adding vs. Averaging in Distributed Primal-Dual Optimization
Distributed optimization methods for large-scale machine learning suffer from a communication bottleneck.
Communication-Efficient Distributed Dual Coordinate Ascent
Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning.
MLI: An API for Distributed Machine Learning
MLI is an Application Programming Interface designed to address the challenges of building Machine Learn- ing algorithms in a distributed setting based on data-centric computing.
