Search Results for author:
Found 26 papers, 19 papers with code
Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping
Our insight is that in addition to systems optimization, one can also redesign the model architecture to decouple communication from computation.
Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models
The results show that standard SFT can lead to an average performance drop of up to $16. 7\%$ on multiple benchmarks, such as MMLU and TruthfulQA.
Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler
This is not only because there is a complicated correlation between learning rate, batch size, number of training tokens, model size, and other hyperparameters but also because it is prohibitively expensive to perform a hyperparameter search for large language models with Billions or Trillions of parameters.
Enhancing Training Efficiency Using Packing with Flash Attention
Padding is often used in tuning LLM models by adding special tokens to shorter training examples to match the length of the longest sequence in each batch.
The infrastructure powering IBM's Gen AI model development
This infrastructure includes (1) Vela: an AI-optimized supercomputing capability directly integrated into the IBM Cloud, delivering scalable, dynamic, multi-tenant and geographically distributed infrastructure for large-scale model training and other AI workflow steps and (2) Blue Vela: a large-scale, purpose-built, on-premises hosting environment that is optimized to support our largest and most ambitious AI model training tasks.
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
MQA and GQA both modify the design of the attention block so that multiple query heads can share a single key/value head, reducing the number of distinct key/value heads by a large factor while only minimally degrading accuracy.
Granite Code Models: A Family of Open Foundation Models for Code Intelligence
Increasingly, code LLMs are being integrated into software development environments to improve the productivity of human programmers, and LLM-based agents are beginning to show promise for handling complex tasks autonomously.
Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
Mixture-of-Experts (MoE) language models can reduce computational costs by 2-4$\times$ compared to dense models without sacrificing performance, making them more efficient in computation-bounded scenarios.
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
We show that regularizing both the inputs and outputs is crucial for preventing a model's "migrating" the difficulty in input quantization to the weights, which makes post-training quantization (PTQ) of weights more difficult.
DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets
In DeiT-LT, we introduce an efficient and effective way of distillation from CNN via distillation DIST token by using out-of-distribution images and re-weighting the distillation loss to enhance focus on tail classes.
Ranked #1 on
Image Classification
on iNaturalist
(Overall metric)
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Despite these efforts, such models encounter challenges such as limited multilingual capabilities, risks of catastrophic forgetting during continual pretraining, and the high costs of training models from scratch, alongside the need to align with AI safety standards and regulatory frameworks.
StarCoder 2 and The Stack v2: The Next Generation
Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size.
Ranked #35 on
Code Generation
on MBPP
BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback
Distribution matching methods for language model alignment such as Generation with Distributional Control (GDC) and Distributional Policy Gradient (DPG) have not received the same level of attention in reinforcement learning from human feedback (RLHF) as contrastive methods such as Sequence Likelihood Calibration (SLiC), Direct Preference Optimization (DPO) and its variants.
DeiT-LT: Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets
In DeiT-LT we introduce an efficient and effective way of distillation from CNN via distillation \texttt DIST token by using out-of-distribution images and re-weighting the distillation loss to enhance focus on tail classes.
Prompting with Pseudo-Code Instructions
Prompting with natural language instructions has recently emerged as a popular method of harnessing the capabilities of large language models.
StarCoder: may the source be with you!
The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15. 5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention.
Ranked #53 on
Code Generation
on MBPP
SantaCoder: don't reach for the stars!
The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code.
Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data
Real-world datasets exhibit imbalances of varying types and degrees.
Ranked #1 on
Long-tail Learning
on CIFAR-10-LT (ρ=50)
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Joint Reasoning on Hybrid-knowledge sources for Task-Oriented Dialog
Traditional systems designed for task oriented dialog utilize knowledge present only in structured knowledge sources to generate responses.
A Closer Look at Smoothness in Domain Adversarial Training
Based on the analysis, we introduce the Smooth Domain Adversarial Training (SDAT) procedure, which effectively enhances the performance of existing domain adversarial methods for both classification and object detection tasks.
Ranked #7 on
Domain Adaptation
on VisDA2017
Variational Learning for Unsupervised Knowledge Grounded Dialogs
Recent methods for knowledge grounded dialogs generate responses by incorporating information from an external textual document.
Accelerating Gradient-based Meta Learner
Meta Learning has been in focus in recent years due to the meta-learner model's ability to adapt well and generalize to new tasks, thus, reducing both the time and data requirements for learning.
Adversarial Approximate Inference for Speech to Electroglottograph Conversion
A popular alternative is to indirectly obtain the source information through the Electroglottographic (EGG) signal that measures the electrical admittance around the vocal folds using dedicated hardware.
Variational Inference with Latent Space Quantization for Adversarial Resilience
Specifically, we simultaneously auto-encode the data manifold and its perturbations implicitly through the perturbations of the regularized and quantized generative latent space, realized using variational inference.
