Search Results for author:
Found 33 papers, 15 papers with code
A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale
It constructs a block-diagonal preconditioner where each block consists of a coarse Kronecker product approximation to full-matrix AdaGrad for each parameter of the neural network.
DINOv2: Learning Robust Visual Features without Supervision
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision.
Ranked #1 on
Image Retrieval
on AmsterTime
(using extra training data)
Green Federated Learning
The rapid progress of AI is fueled by increasingly large and computationally intensive machine learning models and datasets.
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations.
lo-fi: distributed fine-tuning without communication
When fine-tuning DeiT-base and DeiT-large on ImageNet, this procedure matches accuracy in-distribution and improves accuracy under distribution shift compared to the baseline, which observes the same amount of data but communicates gradients at each step.
Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity.
The Hidden Uniform Cluster Prior in Self-Supervised Learning
A successful paradigm in representation learning is to perform self-supervised pretraining using tasks based on mini-batch statistics (e. g., SimCLR, VICReg, SwAV, MSN).
Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity.
Positive Unlabeled Contrastive Learning
We extend this paradigm to the classical positive unlabeled (PU) setting, where the task is to learn a binary classifier given only a few labeled positive samples, and (often) a large amount of unlabeled samples (which could be positive or negative).
FedShuffle: Recipes for Better Use of Local Work in Federated Learning
The practice of applying several local updates before aggregation across clients has been empirically shown to be a successful approach to overcoming the communication bottleneck in Federated Learning (FL).
Masked Siamese Networks for Label-Efficient Learning
We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations.
Self-Supervised Image Classification
Self-Supervised Learning
+1
Federated Learning with Partial Model Personalization
We consider two federated learning algorithms for training partially personalized models, where the shared and personal parameters are updated either simultaneously or alternately on the devices.
Stochastic Polyak Stepsize with a Moving Target
MOTAPS can be seen as a variant of the Stochastic Polyak (SP) which is also a method that also uses loss values to adjust the stepsize.
Federated Learning with Buffered Asynchronous Aggregation
On the other hand, asynchronous aggregation of client updates in FL (i. e., asynchronous FL) alleviates the scalability issue.
Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples
This paper proposes a novel method of learning by predicting view assignments with support samples (PAWS).
A Closer Look at Codistillation for Distributed Training
Surprisingly, we find that even at moderate batch sizes, models trained with codistillation can perform as well as models trained with synchronous data-parallel methods, despite using a much weaker synchronization mechanism.
Advances in Asynchronous Parallel and Distributed Optimization
Motivated by large-scale optimization problems arising in the context of machine learning, there have been several advances in the study of asynchronous parallel and distributed optimization methods during the past decade.
Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
We investigate a strategy for improving the efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training.
On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings
We study Nesterov's accelerated gradient method with constant step-size and momentum parameters in the stochastic approximation setting (unbiased gradients with bounded variance) and the finite-sum setting (where randomness is due to sampling mini-batches).
Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge
Conclusion: The challenge led to new developments in machine learning for image reconstruction, provided insight into the current state of the art in the field, and highlighted remaining hurdles for clinical adoption.
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
We provide theoretical convergence guarantees showing that SlowMo converges to a stationary point of smooth non-convex losses.
Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning
We show that we can run several loosely coupled GALA agents in parallel on a single GPU and achieve significantly higher hardware utilization and frame-rates than vanilla A2C at comparable power draws.
A Graph-CNN for 3D Point Cloud Classification
Graph convolutional neural networks (Graph-CNNs) extend traditional CNNs to handle data that is supported on a graph.
Stochastic Gradient Push for Distributed Deep Learning
Distributed data-parallel algorithms aim to accelerate the training of deep neural networks by parallelizing the computation of large mini-batch gradient updates across multiple nodes.
fastMRI: An Open Dataset and Benchmarks for Accelerated MRI
Accelerating Magnetic Resonance Imaging (MRI) by taking fewer measurements has the potential to reduce medical costs, minimize stress to patients and make MRI possible in applications where it is currently prohibitively slow or expensive.
Provably Accelerated Randomized Gossip Algorithms
In this work we present novel provably accelerated gossip algorithms for solving the average consensus problem.
TarMAC: Targeted Multi-Agent Communication
We propose a targeted communication architecture for multi-agent reinforcement learning, where agents learn both what messages to send and whom to address them to while performing cooperative tasks in partially-observable environments.
Learning graphs from data: A signal representation perspective
The construction of a meaningful graph topology plays a crucial role in the effective representation, processing, analysis and visualization of structured data.
Efficient Large-Scale Similarity Search Using Matrix Factorization
Experiments with standard image search benchmarks, including the Yahoo100M dataset comprising 100 million images, show that our method gives comparable (and sometimes superior) accuracy compared to exhaustive search while requiring only 10% of the vector operations and memory.
Memory vectors for similarity search in high-dimensional spaces
We study an indexing architecture to store and search in a database of high-dimensional vectors from the perspective of statistical signal processing and decision theory.
Combating Corrupt Messages in Sparse Clustered Associative Memories
In this paper we analyze and extend the neural network based associative memory proposed by Gripon and Berrou.
Storing sequences in binary tournament-based neural networks
An extension to a recently introduced architecture of clique-based neural networks is presented.
Improving Sparse Associative Memories by Escaping from Bogus Fixed Points
The latter outperforms the former in terms of retrieval rate by a huge margin.
