Search Results for author:
Found 9 papers, 3 papers with code
HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimation
The family of hyperpower methods are well-known for their rigorous convergence guarantees on matrix inverse approximation, while the matrix multiplication operation can involve intractable memory and computation costs on large-scale models.
Dynamic Gradient Alignment for Online Data Mixing
DGA dynamically estimates the pre-training data mixture on which the models' gradients align as well as possible with those of the model on the specific task.
Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
In this work, we build specialist models from large generalist training sets instead.
Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants
We investigate the potential scale of this vulnerability by measuring the degree to which AI assistants can complete assessment questions in standard university-level STEM courses.
Deep Grokking: Would Deep Neural Networks Generalize Better?
Grokking refers to a sharp rise of the network's generalization accuracy on the test set, which occurs long after an extended overfitting phase, during which the network perfectly fits the training set.
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Large language models (LLMs) can potentially democratize access to medical knowledge.
Ranked #1 on
Multiple Choice Question Answering (MCQA)
on MedMCQA
(Dev Set (Acc-%) metric)
DoGE: Domain Reweighting with Generalization Estimation
Moreover, aiming to generalize to out-of-domain target tasks, which is unseen in the pretraining corpus (OOD domain), DoGE can effectively identify inter-domain dependencies, and consistently achieves better test perplexity on the target domain.
Irreducible Curriculum for Language Model Pretraining
Automatic data selection and curriculum design for training large language models is challenging, with only a few existing methods showing improvements over standard training.
Towards Process-Oriented, Modular, and Versatile Question Generation that Meets Educational Needs
NLP-powered automatic question generation (QG) techniques carry great pedagogical potential of saving educators' time and benefiting student learning.
