Search Results for author:
Found 12 papers, 7 papers with code
Bridging Offline and Online Reinforcement Learning for LLMs
We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online to fully online regimes for both verifiable and non-verifiable tasks.
LLM Pretraining with Continuous Concepts
We propose Continuous Concept Mixing (CoCoMix), a novel pretraining framework that combines discrete next token prediction with continuous concepts.
Thinking LLMs: General Instruction Following with Thought Generation
LLMs are typically trained to answer user questions or follow instructions similarly to how human experts respond.
Improved Uncertainty Estimation of Graph Neural Network Potentials Using Engineered Latent Space Distances
However, one limitation of GNNs in this context is the lack of useful uncertainty prediction methods, as this is critical to the material discovery pipeline.
AdsorbML: A Leap in Efficiency for Adsorption Energy Calculations using Generalizable Machine Learning Potentials
Computational catalysis is playing an increasingly significant role in the design of catalysts across a wide range of applications.
Spherical Channels for Modeling Atomic Interactions
We propose the Spherical Channel Network (SCN) to model atomic energies and forces.
The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts
The development of machine learning models for electrocatalysts requires a broad set of training data to enable their use across a wide variety of materials.
Uncovering the impact of learning rate for global magnitude pruning
However, prior work has implicitly assumed that the best training configuration for model performance was also the best configuration for mask discovery.
Plug and Play Language Models: A Simple Approach to Controlled Text Generation
Large transformer-based language models (LMs) trained on huge text corpora have shown unparalleled generation capabilities.
First-Order Preconditioning via Hypergradient Descent
Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space. These difficulties can be addressed by second-order approaches that apply a pre-conditioning matrix to the gradient to improve convergence.
LCA: Loss Change Allocation for Neural Network Training
We propose a new window into training called Loss Change Allocation (LCA), in which credit for changes to the network loss is conservatively partitioned to the parameters.
Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keeping the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights.
