Search Results for author: Virginia Smith

Found 48 papers, 29 papers with code

Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes

1 code implementation8 Feb 2024 Lucio Dery, Steven Kolawole, Jean-François Kagy, Virginia Smith, Graham Neubig, Ameet Talwalkar

Given the generational gap in available hardware between lay practitioners and the most endowed institutions, LLMs are becoming increasingly inaccessible as they grow in size.

Leveraging Public Representations for Private Transfer Learning

no code implementations24 Dec 2023 Pratiksha Thaker, Amrith Setlur, Zhiwei Steven Wu, Virginia Smith

Motivated by the recent empirical success of incorporating public data into differentially private learning, we theoretically investigate how a shared representation learned from public data can improve private learning.

regression Transfer Learning

Complementary Benefits of Contrastive Learning and Self-Training Under Distribution Shift

no code implementations NeurIPS 2023 Saurabh Garg, Amrith Setlur, Zachary Chase Lipton, Sivaraman Balakrishnan, Virginia Smith, aditi raghunathan

Self-training and contrastive learning have emerged as leading techniques for incorporating unlabeled data, both under distribution shift (unsupervised domain adaptation) and when it is absent (semi-supervised learning).

Contrastive Learning Unsupervised Domain Adaptation

Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies

1 code implementation NeurIPS 2023 Oscar Li, James Harrison, Jascha Sohl-Dickstein, Virginia Smith, Luke Metz

Unrolled computation graphs are prevalent throughout machine learning but present challenges to automatic differentiation (AD) gradient estimation methods when their loss functions exhibit extreme local sensitivtiy, discontinuity, or blackbox characteristics.

Federated Learning as a Network Effects Game

no code implementations16 Feb 2023 Shengyuan Hu, Dung Daniel Ngo, Shuran Zheng, Virginia Smith, Zhiwei Steven Wu

Federated Learning (FL) aims to foster collaboration among a population of clients to improve the accuracy of machine learning without directly sharing local data.

Federated Learning

Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shifts

1 code implementation6 Feb 2023 Amrith Setlur, Don Dennis, Benjamin Eysenbach, aditi raghunathan, Chelsea Finn, Virginia Smith, Sergey Levine

Some robust training algorithms (e. g., Group DRO) specialize to group shifts and require group information on all training points.

On Noisy Evaluation in Federated Hyperparameter Tuning

1 code implementation17 Dec 2022 Kevin Kuo, Pratiksha Thaker, Mikhail Khodak, John Nguyen, Daniel Jiang, Ameet Talwalkar, Virginia Smith

In this work, we perform the first systematic study on the effect of noisy evaluation in federated hyperparameter tuning.

Federated Learning

Differentially Private Adaptive Optimization with Delayed Preconditioners

1 code implementation1 Dec 2022 Tian Li, Manzil Zaheer, Ken Ziyu Liu, Sashank J. Reddi, H. Brendan McMahan, Virginia Smith

Privacy noise may negate the benefits of using adaptive optimizers in differentially private model training.

Validating Large Language Models with ReLM

1 code implementation21 Nov 2022 Michael Kuchnik, Virginia Smith, George Amvrosiadis

Although large language models (LLMs) have been touted for their ability to generate natural-sounding text, there are growing concerns around possible negative effects of LLMs such as data memorization, bias, and inappropriate language.

Language Modelling Memorization

On Privacy and Personalization in Cross-Silo Federated Learning

1 code implementation16 Jun 2022 Ziyu Liu, Shengyuan Hu, Zhiwei Steven Wu, Virginia Smith

While the application of differential privacy (DP) has been well-studied in cross-device federated learning (FL), there is a lack of work considering DP and its implications for cross-silo FL, a setting characterized by a limited number of clients each containing many data subjects.

Federated Learning Multi-Task Learning

Adversarial Unlearning: Reducing Confidence Along Adversarial Directions

no code implementations3 Jun 2022 Amrith Setlur, Benjamin Eysenbach, Virginia Smith, Sergey Levine

Supervised learning methods trained with maximum likelihood objectives often overfit on training data.

Data Augmentation

Maximizing Global Model Appeal in Federated Learning

no code implementations30 May 2022 Yae Jee Cho, Divyansh Jhunjhunwala, Tian Li, Virginia Smith, Gauri Joshi

We provide convergence guarantees for MaxFL and show that MaxFL achieves a $22$-$40\%$ and $18$-$50\%$ test accuracy improvement for the training clients and unseen clients respectively, compared to a wide range of FL modeling approaches, including those that tackle data heterogeneity, aim to incentivize clients, and learn personalized or fair models.

Federated Learning

Fair Federated Learning via Bounded Group Loss

no code implementations18 Mar 2022 Shengyuan Hu, Zhiwei Steven Wu, Virginia Smith

In particular, we explore and extend the notion of Bounded Group Loss as a theoretically-grounded approach for group fairness.

Fairness Federated Learning

Private Adaptive Optimization with Side Information

1 code implementation12 Feb 2022 Tian Li, Manzil Zaheer, Sashank J. Reddi, Virginia Smith

Adaptive optimization methods have become the default solvers for many machine learning tasks.

Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines

2 code implementations7 Nov 2021 Michael Kuchnik, Ana Klimovic, Jiri Simsa, Virginia Smith, George Amvrosiadis

Our analysis of over two million ML jobs in Google datacenters reveals that a significant fraction of model training jobs could benefit from faster input data pipelines.

BIG-bench Machine Learning

Diverse Client Selection for Federated Learning via Submodular Maximization

no code implementations ICLR 2022 Ravikumar Balakrishnan, Tian Li, Tianyi Zhou, Nageen Himayat, Virginia Smith, Jeff Bilmes

In every communication round of federated learning, a random subset of clients communicate their model updates back to the server which then aggregates them all.

Fairness Federated Learning

On Tilted Losses in Machine Learning: Theory and Applications

1 code implementation13 Sep 2021 Tian Li, Ahmad Beirami, Maziar Sanjabi, Virginia Smith

Finally, we demonstrate that TERM can be used for a multitude of applications in machine learning, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance.

BIG-bench Machine Learning Fairness +1

Private Multi-Task Learning: Formulation and Applications to Federated Learning

1 code implementation30 Aug 2021 Shengyuan Hu, Zhiwei Steven Wu, Virginia Smith

Many problems in machine learning rely on multi-task learning (MTL), in which the goal is to solve multiple related machine learning tasks simultaneously.

BIG-bench Machine Learning Distributed Optimization +2

Heterogeneity for the Win: One-Shot Federated Clustering

2 code implementations1 Mar 2021 Don Kurian Dennis, Tian Li, Virginia Smith

In this work, we explore the unique challenges -- and opportunities -- of unsupervised federated learning (FL).

Clustering Federated Learning

Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution

1 code implementation NeurIPS 2021 Amrith Setlur, Oscar Li, Virginia Smith

We categorize meta-learning evaluation into two settings: $\textit{in-distribution}$ [ID], in which the train and test tasks are sampled $\textit{iid}$ from the same underlying task distribution, and $\textit{out-of-distribution}$ [OOD], in which they are not.

Few-Shot Learning Learning Theory +2

Is Support Set Diversity Necessary for Meta-Learning?

no code implementations28 Nov 2020 Amrith Setlur, Oscar Li, Virginia Smith

Meta-learning is a popular framework for learning with limited data in which an algorithm is produced by training over multiple few-shot learning tasks.

Few-Shot Learning

Tilted Empirical Risk Minimization

2 code implementations ICLR 2021 Tian Li, Ahmad Beirami, Maziar Sanjabi, Virginia Smith

Empirical risk minimization (ERM) is typically designed to perform well on the average loss, which can result in estimators that are sensitive to outliers, generalize poorly, or treat subgroups unfairly.


Enhancing the Privacy of Federated Learning with Sketching

no code implementations5 Nov 2019 Zaoxing Liu, Tian Li, Virginia Smith, Vyas Sekar

Federated learning methods run training tasks directly on user devices and do not share the raw user data with third parties.

Federated Learning

Progressive Compressed Records: Taking a Byte out of Deep Learning Data

1 code implementation1 Nov 2019 Michael Kuchnik, George Amvrosiadis, Virginia Smith

Deep learning accelerators efficiently train over vast and growing amounts of data, placing a newfound burden on commodity networks and storage devices.

Federated Learning: Challenges, Methods, and Future Directions

1 code implementation21 Aug 2019 Tian Li, Anit Kumar Sahu, Ameet Talwalkar, Virginia Smith

Federated learning involves training statistical models over remote devices or siloed data centers, such as mobile phones or hospitals, while keeping data localized.

BIG-bench Machine Learning Distributed Optimization +2

One-Shot Federated Learning

no code implementations28 Feb 2019 Neel Guha, Ameet Talwalkar, Virginia Smith

We present one-shot federated learning, where a central server learns a global model over a network of federated devices in a single round of communication.

Ensemble Learning Federated Learning

Federated Optimization in Heterogeneous Networks

18 code implementations14 Dec 2018 Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith

Theoretically, we provide convergence guarantees for our framework when learning over data from non-identical distributions (statistical heterogeneity), and while adhering to device-level systems constraints by allowing each participating device to perform a variable amount of work (systems heterogeneity).

Distributed Optimization Federated Learning +1

LEAF: A Benchmark for Federated Settings

7 code implementations3 Dec 2018 Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMahan, Virginia Smith, Ameet Talwalkar

Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts of data each day.

Autonomous Vehicles Benchmarking +3

Model Aggregation via Good-Enough Model Spaces

no code implementations20 May 2018 Neel Guha, Virginia Smith

In this work, we present Good-Enough Model Spaces (GEMS), a novel framework for learning a global model by carefully intersecting the sets of "good-enough" models across each node.

Distributed Optimization Sentiment Analysis

A Kernel Theory of Modern Data Augmentation

no code implementations16 Mar 2018 Tri Dao, Albert Gu, Alexander J. Ratner, Virginia Smith, Christopher De Sa, Christopher Ré

Data augmentation, a technique in which a training set is expanded with class-preserving transformations, is ubiquitous in modern machine learning pipelines.

BIG-bench Machine Learning Data Augmentation

Federated Multi-Task Learning

2 code implementations NeurIPS 2017 Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, Ameet Talwalkar

Federated learning poses new statistical and systems challenges in training machine learning models over distributed networks of devices.

BIG-bench Machine Learning Federated Learning +1

L1-Regularized Distributed Optimization: A Communication-Efficient Primal-Dual Framework

2 code implementations13 Dec 2015 Virginia Smith, Simone Forte, Michael. I. Jordan, Martin Jaggi

Despite the importance of sparsity in many large-scale applications, there are few methods for distributed optimization of sparsity-inducing objectives.

Distributed Optimization

Distributed Optimization with Arbitrary Local Solvers

1 code implementation13 Dec 2015 Chenxin Ma, Jakub Konečný, Martin Jaggi, Virginia Smith, Michael. I. Jordan, Peter Richtárik, Martin Takáč

To this end, we present a framework for distributed optimization that both allows the flexibility of arbitrary solvers to be used on each (single) machine locally, and yet maintains competitive performance against other state-of-the-art special-purpose distributed methods.

Distributed Optimization

Adding vs. Averaging in Distributed Primal-Dual Optimization

1 code implementation12 Feb 2015 Chenxin Ma, Virginia Smith, Martin Jaggi, Michael. I. Jordan, Peter Richtárik, Martin Takáč

Distributed optimization methods for large-scale machine learning suffer from a communication bottleneck.

Distributed Optimization

MLI: An API for Distributed Machine Learning

no code implementations21 Oct 2013 Evan R. Sparks, Ameet Talwalkar, Virginia Smith, Jey Kottalam, Xinghao Pan, Joseph Gonzalez, Michael J. Franklin, Michael. I. Jordan, Tim Kraska

MLI is an Application Programming Interface designed to address the challenges of building Machine Learn- ing algorithms in a distributed setting based on data-centric computing.

BIG-bench Machine Learning

Cannot find the paper you are looking for? You can Submit a new open access paper.