Search Results for author:
Found 8 papers, 3 papers with code
Simple and Scalable Strategies to Continually Pre-train Large Language Models
In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks.
Can We Learn Communication-Efficient Optimizers?
Although many variants of these approaches have been proposed, they can sometimes lag behind state-of-the-art adaptive optimizers for deep learning.
Continual Pre-Training of Large Language Models: How to (re)warm your model?
We study the warmup phase of models pre-trained on the Pile (upstream data, 300B tokens) as we continue to pre-train on SlimPajama (downstream data, 297B tokens), following a linear warmup and cosine decay schedule.
Object Re-Identification from Point Clouds
To our knowledge, we are the first to study object re-identification from real point cloud observations.
A Closer Look at Robustness to L-infinity and Spatial Perturbations and their Composition
In adversarial machine learning, the popular $\ell_\infty$ threat model has been the focus of much previous work.
Interpretable Deep Tracking
By enumerating different tracking decisions and associated reasoning procedures, we can train individual networks to reason about the possible decisions via IIT.
Exploring the Optimality of Tight-Frame Scattering Networks
The wavelet filters used in the scattering transform are typically selected to create a tight frame via a parameterized mother wavelet.
Parametric Scattering Networks
The wavelet scattering transform creates geometric invariants and deformation stability.
