Search Results for author:
Found 121 papers, 62 papers with code
Self-Supervised Neural Topic Modeling
Topic models are useful tools for analyzing and interpreting the main underlying themes of large corpora of text.
Layerwise Linear Mode Connectivity
In the federated setup one performs an aggregation of separate local models multiple times during training in order to obtain a stronger global model; most often aggregation is a simple averaging of the parameters.
Provably Personalized and Robust Federated Learning
Clustering clients with similar objectives and learning a model per cluster is an intuitive and interpretable approach to personalization in federated learning.
Faster Causal Attention Over Large Sequences Through Sparse Flash Attention
While many works have proposed schemes to sparsify the attention patterns and reduce the computational overhead of self-attention, those are often limited by implementations concerns and end up imposing a simple and static structure over the attention matrix.
Shuffle SGD is Always Better than SGD: Improved Analysis of SGD with Arbitrary Data Orders
Stochastic Gradient Descent (SGD) algorithms are widely used in optimizing neural networks, with Random Reshuffling (RR) and Single Shuffle (SS) being popular choices for cycling through random or single permutations of the training data.
Collaborative Learning via Prediction Consensus
To facilitate the exchange of expertise among agents, we propose a distillation-based method leveraging unlabeled auxiliary data, which is pseudo-labeled by the collective.
Rotational Optimizers: Simple & Robust DNN Training
These interactions can give rise to Spherical Motion Dynamics in scale-invariant layers (e. g., normalized layers), which converge to an equilibrium state, where the weight norm and the expected rotational update size are fixed.
Hardware-Efficient Transformer Training via Piecewise Affine Operations
Finally, we show that we can eliminate all multiplications in the entire training process, including operations in the forward pass, backward pass and optimizer update, demonstrating the first successful training of modern neural network architectures in a fully multiplication-free fashion.
Ghost Noise for Regularizing Deep Neural Networks
Batch Normalization (BN) is widely used to stabilize the optimization process and improve the test performance of deep neural networks.
Landmark Attention: Random-Access Infinite Context Length for Transformers
While transformers have shown remarkable success in natural language processing, their attention mechanism's large memory requirements have limited their ability to handle longer contexts.
Linearization Algorithms for Fully Composite Optimization
This paper studies first-order algorithms for solving fully composite optimization problems over convex and compact sets.
Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methods
Our helper framework offers the algorithm designer high flexibility for constructing and analyzing the stochastic Cubic Newton methods, allowing arbitrary size batches, and the use of noisy and possibly biased estimates of the gradients and Hessians, incorporating both the variance reduction and the lazy Hessian updates.
Beyond spectral gap (extended): The role of the topology in decentralized learning
This paper aims to paint an accurate picture of sparsely-connected distributed optimization.
Second-order optimization with lazy Hessians
This provably improves the total arithmetical complexity of second-order algorithms by a factor $\sqrt{d}$.
Scalable Collaborative Learning via Representation Sharing
To do so, each client releases averaged last hidden layer activations of similar labels to a central server that only acts as a relay (i. e., is not involved in the training or aggregation of the models).
Accuracy Boosters: Epoch-Driven Mixed-Mantissa Block Floating-Point for DNN Training
The unprecedented growth in DNN model complexity, size, and amount of training data has led to a commensurate increase in demand for computing and a search for minimal encoding.
Modular Clinical Decision Support Networks (MoDN) -- Updatable, Interpretable, and Portable Predictions for Evolving Clinical Environments
Data-driven Clinical Decision Support Systems (CDSS) have the potential to improve and standardise care with personalised probabilistic guidance.
FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings
In this work, we propose a novel cross-silo dataset suite focused on healthcare, FLamby (Federated Learning AMple Benchmark of Your cross-silo strategies), to bridge the gap between theory and practice of cross-silo FL.
Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning
In this work (i) we obtain a tighter convergence rate of $\mathcal{O}\!\left(\sigma^2\epsilon^{-2}+ \sqrt{\tau_{\max}\tau_{avg}}\epsilon^{-1}\right)$ without any change in the algorithm where $\tau_{avg}$ is the average delay, which can be significantly smaller than $\tau_{\max}$.
Beyond spectral gap: The role of the topology in decentralized learning
In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model: more accurate gradients allow them to use larger learning rates and optimize faster.
Special Properties of Gradient Descent with Large Learning Rates
However, we show through a novel set of experiments that the stochastic noise is not sufficient to explain good non-convex training, and that instead the effect of a large learning rate itself is essential for obtaining best performance. We demonstrate the same effects also in the noise-less case, i. e. for full-batch GD.
SKILL: Structured Knowledge Infusion for Large Language Models
The models pre-trained on factual triples compare competitively with the ones on natural language sentences that contain the same knowledge.
Data-heterogeneity-aware Mixing for Decentralized Learning
Decentralized learning provides an effective framework to train machine learning models with data distributed over arbitrary communication graphs.
Improving Generalization via Uncertainty Driven Perturbations
We show that UDP is guaranteed to achieve the maximum margin decision boundary on linear models and that it notably increases it on challenging simulated datasets.
Agree to Disagree: Diversity through Disagreement for Better Transferability
This behavior can hinder the transferability of trained models by (i) favoring the learning of simpler but spurious features -- present in the training data but absent from the test data -- and (ii) by only leveraging a small subset of predictive features.
Byzantine-Robust Decentralized Learning via ClippedGossip
In this paper, we study the challenging task of Byzantine-robust decentralized training on arbitrary communication graphs.
Breaking the centralized barrier for cross-device federated learning
Federated learning (FL) is a challenging setting for optimization due to the heterogeneity of the data across different clients which gives rise to the client drift phenomenon.
Interpreting Language Models Through Knowledge Graph Extraction
In this paper, we compare BERT-based language models through snapshots of acquired knowledge at sequential stages of the training process.
Linear Speedup in Personalized Collaborative Learning
Collaborative training can improve the accuracy of a model for a user by trading off the model's bias (introduced by using data from other users who are potentially different) against its variance (due to the limited amount of data on any single user).
Optimal Model Averaging: Towards Personalized Collaborative Learning
In federated learning, differences in the data or objectives between the participating nodes motivate approaches to train a personalized machine learning model for each node.
WAFFLE: Weighted Averaging for Personalized Federated Learning
Through a series of experiments, we compare our new approach to two recent personalized federated learning methods--Weight Erosion and APFL--as well as two general FL methods--Federated Averaging and SCAFFOLD.
RelaySum for Decentralized Deep Learning on Heterogeneous Data
A key challenge, primarily in decentralized deep learning, remains the handling of differences between the workers' local data distributions.
Improved Generalization-Robustness Trade-off via Uncertainty Targeted Attacks
The deep learning models' sensitivity to small input perturbations raises security concerns and limits their use for applications where reliability is critical.
On Second-order Optimization Methods for Federated Learning
We consider federated learning (FL), where the training data is distributed across a large number of clients.
Semantic Perturbations with Normalizing Flows for Improved Generalization
We find that our latent adversarial perturbations adaptive to the classifier throughout its training are most effective, yielding the first test accuracy improvement results on real-world datasets -- CIFAR-10/100 -- via latent-space perturbations.
A Field Guide to Federated Optimization
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection.
IFedAvg: Interpretable Data-Interoperability for Federated Learning
Thus, enabling the detection of outlier datasets in the federation and also learning the compensation for local data distribution shifts without sharing any original data.
Implicit Gradient Alignment in Distributed and Federated Learning
A major obstacle to achieving global convergence in distributed and federated learning is the misalignment of gradients across clients, or mini-batches due to heterogeneity and stochasticity of the distributed data.
Masked Training of Neural Networks with Partial Gradients
State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD).
Obtaining Better Static Word Embeddings Using Contextual Embedding Models
The advent of contextual word embeddings -- representations of words which incorporate semantic and syntactic information from their context -- has led to tremendous improvements on a wide variety of NLP tasks.
Lightweight Cross-Lingual Sentence Representation Learning
Large-scale models for learning fixed-dimensional cross-lingual sentence representations like LASER (Artetxe and Schwenk, 2019b) lead to significant improvement in performance on downstream tasks.
Federated Learning for Malware Detection in IoT Devices
In this context, a framework that uses federated learning to detect malware affecting IoT devices is presented.
Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates
It has been experimentally observed that the efficiency of distributed training with stochastic gradient (SGD) depends decisively on the batch size and -- in asynchronous implementations -- on the gradient staleness.
Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
In this paper, we investigate and identify the limitation of several decentralized optimization algorithms for different degrees of data heterogeneity.
Consensus Control for Decentralized Deep Learning
Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters.
Exact Optimization of Conformal Predictors via Incremental and Decremental Learning
We evaluate our findings empirically, and discuss when methods are suitable for CP optimization.
On the Effect of Consensus in Decentralized Deep Learning
Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters.
Faster Training of Word Embeddings
Word embeddings have gained increasing popularity in the recent years due to the Word2vec library and its extension fastText that uses subword information.
Learning from History for Byzantine Robust Optimization
Secondly, we prove that even if the aggregation rules may succeed in limiting the influence of the attackers in a single round, the attackers can couple their attacks across time eventually leading to divergence.
Practical Low-Rank Communication Compression in Decentralized Deep Learning
Lossy gradient compression has become a practical tool to overcome the communication bottleneck in centrally coordinated distributed training of machine learning models.
A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!
Decentralized optimization methods enable on-device training of machine learning models without a central coordinator.
Byzantine-Robust Learning on Heterogeneous Datasets via Resampling
In Byzantine-robust distributed optimization, a central server wants to train a machine learning model over data distributed across multiple workers.
Sparse Communication for Training Deep Networks
Although distributed training reduces the computation time, the communication overhead associated with the gradient exchange forms a scalability bottleneck for the algorithm.
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Federated learning (FL) is a challenging setting for optimization due to the heterogeneity of the data across different clients which gives rise to the client drift phenomenon.
PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
Lossy gradient compression has become a practical tool to overcome the communication bottleneck in centrally coordinated distributed training of machine learning models.
Multi-Head Attention: Collaborate Instead of Concatenate
We also show that it is possible to re-parametrize a pre-trained multi-head attention layer into our collaborative attention layer.
Taming GANs with Lookahead-Minmax
Generative Adversarial Networks are notoriously challenging to train.
Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing
In Byzantine robust distributed or federated learning, a central server wants to train a machine learning model over data distributed across multiple workers.
Ensemble Distillation for Robust Model Fusion in Federated Learning
In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side.
Extrapolation for Large-batch Training in Deep Learning
Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fraction of the training data.
Secure Byzantine-Robust Machine Learning
Increasingly machine learning systems are being deployed to edge servers and devices (e. g. mobile phones) and trained in a collaborative manner.
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models
We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning.
Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
As a result, we find across various workloads of data set, network model, and optimization algorithm that there exists a general scaling trend between batch size and number of training steps to convergence for the effect of data parallelism, and further, difficulty of training under sparsity.
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
Decentralized stochastic optimization methods have gained a lot of attention recently, mainly because of their cheap per iteration cost, data locality, and their communication-efficiency.
Robust Cross-lingual Embeddings from Parallel Sentences
Recent advances in cross-lingual word embeddings have primarily relied on mapping-based methods, which project pretrained word embeddings from different languages into a shared space through a linear transformation.
Cross-Lingual Document Classification
Cross-Lingual Word Embeddings
+6
Advances and Open Problems in Federated Learning
FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches.
On the Relationship between Self-Attention and Convolutional Layers
This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice.
Ranked #154 on
Image Classification
on CIFAR-10
Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
The performance of optimizers, particularly in deep learning, depends considerably on their chosen hyperparameter configuration.
Model Fusion via Optimal Transport
Finally, our approach also provides a principled way to combine the parameters of neural networks with different widths, and we explore its application for model compression.
On the Tunability of Optimizers in Deep Learning
There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD.
Decentralized Deep Learning with Arbitrary Communication Compression
Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters.
Correlating Twitter Language with Community-Level Health Outcomes
We study how language on social media is linked to diseases such as atherosclerotic heart disease (AHD), diabetes and various types of cancer.
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization.
On Linear Learning with Manycore Processors
A new generation of manycore processors is on the rise that offers dozens and more cores on a chip and, in a sense, fuses host processor and accelerator.
Better Word Embeddings by Disentangling Contextual n-Gram Information
Pre-trained word vectors are ubiquitous in Natural Language Processing applications.
Crosslingual Document Embedding as Reduced-Rank Ridge Regression
Finally, although not trained for embedding sentences and words, it also achieves competitive performance on crosslingual sentence and word retrieval tasks.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Interpretable Structure-aware Document Encoders with Hierarchical Attention
We show our model's interpretability by visualizing how our model distributes attention inside a document.
Forecasting intracranial hypertension using multi-scale waveform metrics
Approach: We developed a prediction framework that forecasts onsets of acute intracranial hypertension in the next 8 hours.
Evaluating the Search Phase of Neural Architecture Search
Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks.
Overcoming Multi-Model Forgetting
We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, due to the overwriting of shared parameters.
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
We (i) propose a novel gossip-based stochastic gradient descent algorithm, CHOCO-SGD, that converges at rate $\mathcal{O}\left(1/(nT) + 1/(T \delta^2 \omega)^2\right)$ for strongly convex objectives, where $T$ denotes the number of iterations and $\delta$ the eigengap of the connectivity matrix.
Unsupervised Scalable Representation Learning for Multivariate Time Series
Time series constitute a challenging data type for machine learning algorithms, due to their highly variable lengths and sparse labeling in practice.
Error Feedback Fixes SignSGD and other Gradient Compression Schemes
These issues arise because of the biased nature of the sign compression operator.
Efficient Greedy Coordinate Descent for Composite Problems
For these problems we provide (i) the first linear rates of convergence independent of $n$, and show that our greedy update rule provides speedups similar to those obtained in the smooth case.
Sparsified SGD with Memory
Huge scale machine learning problems are nowadays tackled by distributed optimization algorithms, i. e. algorithms that leverage the compute power of many devices for training.
Context Mover's Distance & Barycenters: Optimal Transport of Contexts for Building Representations
We present a framework for building unsupervised representations of entities and their compositions, where each entity is viewed as a probability distribution rather than a vector embedding.
Don't Use Large Mini-Batches, Use Local SGD
Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks.
COLA: Decentralized Linear Learning
Decentralized machine learning is a promising emerging paradigm in view of global challenges of data ownership and privacy.
A Distributed Second-Order Algorithm You Can Trust
Due to the rapid growth of data and computational resources, distributed optimization has become an active research area in recent years.
Wasserstein is all you need
We propose a unified framework for building unsupervised representations of individual objects or entities (and their compositions), by associating with each object both a distributional as well as a point estimate (vector embedding).
Global linear convergence of Newton's method without strong-convexity or Lipschitz gradients
We show that Newton's method converges globally at a linear rate for objective functions whose Hessians are stable.
Training DNNs with Hybrid Block Floating Point
We identify block floating point (BFP) as a promising alternative representation since it exhibits wide dynamic range and enables the majority of DNN operations to be performed with fixed-point logic.
On Matching Pursuit and Coordinate Descent
Exploiting the connection between the two algorithms, we present a unified analysis of both, providing affine invariant sublinear $\mathcal{O}(1/t)$ rates on smooth objectives and linear convergence on strongly convex objectives.
Simple Unsupervised Keyphrase Extraction using Sentence Embeddings
EmbedRank achieves higher F-scores than graph-based state of the art systems on standard datasets and is suitable for real-time processing of large amounts of Web data.
An Accelerated Communication-Efficient Primal-Dual Optimization Framework for Structured Machine Learning
In this paper, an accelerated variant of CoCoA+ is proposed and shown to possess a convergence rate of $\mathcal{O}(1/t^2)$ in terms of reducing suboptimality.
Safe Adaptive Importance Sampling
Importance sampling has become an indispensable strategy to speed up optimization algorithms for large-scale applications.
Efficient Use of Limited-Memory Accelerators for Linear Learning on Heterogeneous Systems
We propose a generic algorithmic building block to accelerate training of machine learning models on heterogeneous compute systems.
Learning Aerial Image Segmentation from Online Maps
We adapt a state-of-the-art CNN architecture for semantic segmentation of buildings and roads in aerial images, and compare its performance when using different training data sets, ranging from manually labeled, pixel-accurate ground truth of the same city to automatic training data derived from OpenStreetMap data from distant locations.
Unsupervised robust nonparametric learning of hidden community properties
We consider learning of fundamental properties of communities in large noisy networks, in the prototypical situation where the nodes or users are split into two classes according to a binary property, e. g., according to their opinions or preferences on a topic.
Approximate Steepest Coordinate Descent
We propose a new selection rule for the coordinate selection in coordinate descent methods for huge-scale optimization.
Greedy Algorithms for Cone Constrained Optimization with Convergence Guarantees
Greedy optimization methods such as Matching Pursuit (MP) and Frank-Wolfe (FW) algorithms regained popularity in recent years due to their simplicity, effectiveness and theoretical guarantees.
Generating Steganographic Text with LSTMs
Motivated by concerns for user privacy, we design a steganographic system ("stegosystem") that enables two users to exchange encrypted messages without an adversary detecting that such an exchange is taking place.
Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
The recent tremendous success of unsupervised word embeddings in a multitude of applications raises the obvious question if similar methods could be derived to improve embeddings (i. e. semantic representations) of word sequences as well.
Faster Coordinate Descent via Adaptive Importance Sampling
Coordinate descent methods employ random partial updates of decision variables in order to solve huge-scale convex optimization problems.
Leveraging Large Amounts of Weakly Supervised Data for Multi-Language Sentiment Classification
This paper presents a novel approach for multi-lingual sentiment classification in short texts.
A Unified Optimization View on Generalized Matching Pursuit and Frank-Wolfe
Two of the most fundamental prototypes of greedy optimization are the matching pursuit and Frank-Wolfe algorithms.
CoCoA: A General Framework for Communication-Efficient Distributed Optimization
The scale of modern datasets necessitates the development of efficient distributed optimization methods for machine learning.
Screening Rules for Convex Problems
We propose a new framework for deriving screening rules for convex optimization problems.
Primal-Dual Rates and Certificates
We propose an algorithm-independent framework to equip existing optimization methods with primal-dual certificates.
Pursuits in Structured Non-Convex Matrix Factorizations
Efficiently representing real world data in a succinct and parsimonious manner is of central importance in many fields.
Distributed Optimization with Arbitrary Local Solvers
To this end, we present a framework for distributed optimization that both allows the flexibility of arbitrary solvers to be used on each (single) machine locally, and yet maintains competitive performance against other state-of-the-art special-purpose distributed methods.
L1-Regularized Distributed Optimization: A Communication-Efficient Primal-Dual Framework
Despite the importance of sparsity in many large-scale applications, there are few methods for distributed optimization of sparsity-inducing objectives.
On the Global Linear Convergence of Frank-Wolfe Optimization Variants
In this paper, we highlight and clarify several variants of the Frank-Wolfe optimization algorithm that have been successfully applied in practice: away-steps FW, pairwise FW, fully-corrective FW and Wolfe's minimum norm point algorithm, and prove for the first time that they all enjoy global linear convergence, under a weaker condition than strong convexity of the objective.
Adding vs. Averaging in Distributed Primal-Dual Optimization
Distributed optimization methods for large-scale machine learning suffer from a communication bottleneck.
Communication-Efficient Distributed Dual Coordinate Ascent
Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning.
An Equivalence between the Lasso and Support Vector Machines
As a consequence, many existing optimization algorithms for both SVMs and Lasso can also be applied to the respective other problem instances.
