Search Results for author:
Found 14 papers, 7 papers with code
RITA: a Study on Scaling Up Generative Protein Sequence Models
In this work we introduce RITA: a suite of autoregressive generative models for protein sequences, with up to 1. 2 billion parameters, trained on over 280 million protein sequences belonging to the UniRef-100 database.
PAGnol: An Extra-Large French Generative Model
We fit a scaling law for compute for the French language, and compare it with its English counterpart.
Is the Number of Trainable Parameters All That Actually Matters?
Recent work has identified simple empirical scaling laws for language models, linking compute budget, dataset size, model size, and autoregressive modeling loss.
ROPUST: Improving Robustness through Fine-tuning with Photonic Processors and Synthetic Gradients
Robustness to adversarial attacks is typically obtained through expensive adversarial training with Projected Gradient Descent.
Photonic Differential Privacy with Direct Feedback Alignment
Optical Processing Units (OPUs) -- low-power photonic chips dedicated to large scale random projections -- have been used in previous work to train deep neural networks using Direct Feedback Alignment (DFA), an effective alternative to backpropagation.
Photonic co-processors in HPC: using LightOn OPUs for Randomized Numerical Linear Algebra
Randomized Numerical Linear Algebra (RandNLA) is a powerful class of methods, widely used in High Performance Computing (HPC).
Contrastive Embeddings for Neural Architectures
The performance of algorithms for neural architecture search strongly depends on the parametrization of the search space.
Adversarial Robustness by Design through Analog Computing and Synthetic Gradients
In the white-box setting, our defense works by obfuscating the parameters of the random projection.
Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment
We present a photonic accelerator for Direct Feedback Alignment, able to compute random projections with trillions of parameters.
Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
Despite being the workhorse of deep learning, the backpropagation algorithm is no panacea.
Online Change Point Detection in Molecular Dynamics With Optical Random Features
In this work, we focus on the identification of such events given many noisy observables.
Light-in-the-loop: using a photonics co-processor for scalable training of neural networks
As neural networks grow larger and more complex and data-hungry, training costs are skyrocketing.
Principled Training of Neural Networks with Direct Feedback Alignment
In this work, we focus on direct feedback alignment and present a set of best practices justified by observations of the alignment angles.
NEWMA: a new method for scalable model-free online change-point detection
We consider the problem of detecting abrupt changes in the distribution of a multi-dimensional time series, with limited computing power and memory.
