Search Results for author:
Found 46 papers, 17 papers with code
Tackling Byzantine Clients in Federated Learning
The natural approach to robustify FL against adversarial clients is to replace the simple averaging operation at the server in the standard $\mathsf{FedAvg}$ algorithm by a \emph{robust averaging rule}.
Robustness, Efficiency, or Privacy: Pick Two in Machine Learning
The success of machine learning (ML) applications relies on vast datasets and distributed architectures which, as they grow, present major challenges.
Epidemic Learning: Boosting Decentralized Learning with Randomized Communication
We present Epidemic Learning (EL), a simple yet powerful decentralized learning (DL) algorithm that leverages changing communication topologies to achieve faster model convergence compared to conventional DL approaches.
SABLE: Secure And Byzantine robust LEarning
SABLE leverages HTS, a novel and efficient homomorphic operator implementing the prominent coordinate-wise trimmed mean robust aggregator.
Evolutionary Algorithms in the Light of SGD: Limit Equivalence, Minima Flatness, and Transfer Learning
In this paper, we show that a class of evolutionary algorithms (EAs) inspired by the Gillespie-Orr Mutational Landscapes model for natural evolution is formally equivalent to SGD in certain settings and, in practice, is well adapted to large ANNs.
Byzantine-Resilient Learning Beyond Gradients: Distributing Evolutionary Search
We then leverage this definition to show that a general class of gradient-free ML algorithms - ($1,\lambda$)-Evolutionary Search - can be combined with classical distributed consensus algorithms to generate gradient-free byzantine-resilient distributed learning algorithms.
Stochastic Parrots Looking for Stochastic Parrots: LLMs are Easy to Fine-Tune and Hard to Detect with other LLMs
This prominence amplified prior concerns regarding the misuse of LLMs and led to the emergence of numerous tools to detect LLMs in the wild.
On the Privacy-Robustness-Utility Trilemma in Distributed Learning
The latter amortizes the dependence on the dimension in the error (caused by adversarial workers and DP), while being agnostic to the statistical properties of the data.
Fixing by Mixing: A Recipe for Optimal Byzantine ML under Heterogeneity
Byzantine machine learning (ML) aims to ensure the resilience of distributed learning algorithms to misbehaving (or Byzantine) machines.
Accelerating Transfer Learning with Near-Data Computation on Cloud Object Stores
This facilitates our second technique, storage-side batch adaptation, which enables increased concurrency in the storage tier while avoiding out-of-memory errors.
On the Impossible Safety of Large AI Models
Large AI Models (LAIMs), of which large language models are the most prominent recent example, showcase some impressive performance.
Robust Collaborative Learning with Linear Gradient Overhead
We present MoNNA, a new algorithm that (a) is provably robust under standard assumptions and (b) has a gradient computation overhead that is linear in the fraction of faulty machines, which is conjectured to be tight.
Byzantine Machine Learning Made Easy by Resilient Averaging of Momentums
We present \emph{RESAM (RESilient Averaging of Momentums)}, a unified framework that makes it simple to establish optimal Byzantine resilience, relying only on standard machine learning assumptions.
An Equivalence Between Data Poisoning and Byzantine Gradient Attacks
More specifically, we prove that every gradient attack can be reduced to data poisoning, in any personalized federated learning system with PAC guarantees (which we show are both desirable and realistic).
Combining Differential Privacy and Byzantine Resilience in Distributed SGD
Privacy and Byzantine resilience (BR) are two crucial requirements of modern-day distributed machine learning.
Evolutionary perspective on model fine-tuning
Be it in natural language generation or in the image generation, massive performances gains have been achieved in the last years.
Strategyproof Learning: Building Trustworthy User-Generated Datasets
We prove in this paper that, perhaps surprisingly, incentivizing data misreporting is not a fatality.
Differential Privacy and Byzantine Resilience in SGD: Do They Add Up?
This paper addresses the problem of combining Byzantine resilience with privacy in machine learning (ML).
Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent
We propose a practical method which, despite increasing the variance, reduces the variance-norm ratio, mitigating the identified weakness.
Microsecond Consensus for Microsecond Applications
We propose Mu, a system that takes less than 1. 3 microseconds to replicate a (small) request in memory, and less than a millisecond to fail-over the system - this cuts the replication and fail-over latencies of the prior systems by at least 61% and 90%.
Distributed, Parallel, and Cluster Computing
Garfield: System Support for Byzantine Machine Learning
We present Garfield, a library to transparently make machine learning (ML) applications, initially built with popular (but fragile) frameworks, e. g., TensorFlow and PyTorch, Byzantine-resilient.
Efficient Multi-word Compare and Swap
Then we present the first algorithm that requires k+1 CASes per call to k-CAS in the common uncontended case.
Distributed, Parallel, and Cluster Computing
Collaborative Learning in the Jungle (Decentralized, Byzantine, Heterogeneous, Asynchronous and Nonconvex Learning)
We study Byzantine collaborative learning, where $n$ nodes seek to collectively learn from each others' local data.
FLeet: Online Federated Learning via Staleness Awareness and Performance Prediction
Federated Learning (FL) is very appealing for its privacy benefits: essentially, a global model is trained with updates computed on mobile devices while keeping the data of users local.
Differentially Private Stochastic Coordinate Descent
In this paper we tackle the challenge of making the stochastic coordinate descent algorithm differentially private.
Host-Pathongen Co-evolution Inspired Algorithm Enables Robust GAN Training
Generative adversarial networks (GANs) are pairs of artificial neural networks that are trained one against each other.
Distributed Momentum for Byzantine-resilient Learning
Momentum is a variant of gradient descent that has been proposed for its benefits on convergence.
Fast Machine Learning with Byzantine Workers and Servers
We moreover show that the throughput gain of LiuBei compared to another state-of-the-art Byzantine-resilient ML algorithm (that assumes network asynchrony) is 70%.
The Consensus Number of a Cryptocurrency (Extended Version)
As stated in the original paper by Nakamoto, at the heart of these systems lies the problem of preventing double-spending; this is usually solved by achieving consensus on the order of transfers among the participants.
Distributed, Parallel, and Cluster Computing
The Impact of RDMA on Agreement
This technology allows a process to directly read and write the memory of a remote host, with a mechanism to control access permissions.
Distributed, Parallel, and Cluster Computing Data Structures and Algorithms
Genuinely Distributed Byzantine Machine Learning
The third, Minimum-Diameter Averaging (MDA), is a statistically-robust gradient aggregation rule whose goal is to tolerate Byzantine workers.
Fast and Robust Distributed Learning in High Dimension
Given $n$ workers, $f$ of which are arbitrary malicious (Byzantine) and $m=n-f$ are not, we prove that multi-Bulyan can ensure a strong form of Byzantine resilience, as well as an ${\frac{m}{n}}$ slowdown, compared to averaging, the fastest (but non Byzantine resilient) rule for distributed machine learning.
BIG-bench Machine Learning
Vocal Bursts Intensity Prediction
The Probabilistic Fault Tolerance of Neural Networks in the Continuous Limit
We study fault tolerance of neural networks subject to small random neuron/weight crash failures in a probabilistic setting.
Removing Algorithmic Discrimination (With Minimal Individual Error)
We first solve the problem analytically in the case of two populations, with a uniform bonus-malus on the zones where each population is a majority.
Virtuously Safe Reinforcement Learning
We show that when a third party, the adversary, steps into the two-party setting (agent and operator) of safely interruptible reinforcement learning, a trade-off has to be made between the probability of following the optimal policy in the limit, and the probability of escaping a dangerous situation created by the adversary.
Asynchronous Byzantine Machine Learning (the case of SGD)
The dampening component bounds the convergence rate by adjusting to stale information through a generic gradient weighting scheme.
The Hidden Vulnerability of Distributed Learning in Byzantium
Based on this leeway, we build a simple attack, and experimentally show its strong to utmost effectivity on CIFAR-10 and MNIST.
Learning to Gather without Communication
A standard belief on emerging collective behavior is that it emerges from simple individual rules.
Deep Learning Works in Practice. But Does it Work in Theory?
Deep learning relies on a very specific kind of neural networks: those superposing several neural layers.
Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent
We propose \emph{Krum}, an aggregation rule that satisfies our resilience property, which we argue is the first provably Byzantine-resilient algorithm for distributed SGD.
Sequences, Items And Latent Links: Recommendation With Consumed Item Packs
In this paper, we introduce the notion of consumed item pack (CIP) which enables to link users (or items) based on their implicit analogous consumption behavior.
On The Robustness of a Neural Network
This bound involves dependencies on the network parameters that can be seen as being too pessimistic in the average case.
When Neurons Fail
We view a neural network as a distributed system of which neurons can fail independently, and we evaluate its robustness in the absence of any (recovery) learning phase.
Personalized and Private Peer-to-Peer Machine Learning
The rise of connected personal devices together with privacy concerns call for machine learning algorithms capable of leveraging the data of a large number of agents to learn personalized models under strong privacy requirements.
Dynamic Safe Interruptibility for Decentralized Multi-Agent Reinforcement Learning
We give realistic sufficient conditions on the learning algorithm to enable dynamic safe interruptibility in the case of joint action learners, yet show that these conditions are not sufficient for independent learners.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Byzantine-Tolerant Machine Learning
The growth of data, the need for scalability and the complexity of models used in modern machine learning calls for distributed implementations.
