Search Results for author:
Found 19 papers, 9 papers with code
Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers
Training and fine-tuning large language models (LLMs) with hundreds of billions to trillions of parameters requires tens of thousands of GPUs, and a highly scalable software stack.
Gemstones: A Model Suite for Multi-Faceted Scaling Laws
Scaling laws are typically fit using a family of models with a narrow range of frozen hyper-parameter choices.
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
We scale a proof-of-concept model to 3. 5 billion parameters and 800 billion tokens.
HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages
Large Language Model (LLM) based coding tools have been tremendously successful as software development assistants, yet they are often designed for general purpose programming tasks and perform poorly for more specialized domains such as high performance computing.
Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs
Large language models can memorize and repeat their training data, causing privacy and copyright risks.
From Pixels to Prose: A Large Dataset of Dense Image Captions
Training large vision-language models requires extensive, high-quality image-text pairs.
Loki: Low-rank Keys for Efficient Sparse Attention
Inference on large language models (LLMs) can be expensive in terms of the compute and memory costs involved, especially when long sequence lengths are used.
Transformers Can Do Arithmetic with the Right Embeddings
The poor performance of transformers on arithmetic tasks seems to stem in large part from their inability to keep track of the exact position of each digit inside of a large span of digits.
Performance-Aligned LLMs for Generating Fast Code
Optimizing scientific software is a difficult task because codebases are often large and complex, and performance can depend upon several factors including the algorithm, its implementation, and hardware among others.
Can Large Language Models Write Parallel Code?
Large language models are increasingly becoming a popular tool for software development.
Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimization
The primary efficiency bottleneck in such optimizers is matrix inverse calculations in the preconditioning step, which are expensive to compute on GPUs.
HPC-Coder: Modeling Parallel Programs using Large Language Models
In this paper, we show how large language models (LLMs) can be applied to tasks specific to high performance and scientific codes.
A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs
This 4D approach is a hybrid of 3D tensor and data parallelism, and is implemented in the AxoNN framework.
A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training
Mixture-of-Experts (MoE) is a neural network architecture that adds sparsely activated expert blocks to a base model, increasing the number of parameters without impacting computational costs.
Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Training
Parallel training of neural networks at scale is challenging due to significant overheads arising from communication.
A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks
This phenomenon has spurred the development of algorithms for distributed training of neural networks over a larger number of hardware accelerators.
AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning
This has necessitated the development of efficient algorithms to train these neural networks in parallel on large-scale GPU-based clusters.
Analytics of Longitudinal System Monitoring Data for Performance Prediction
In recent years, several HPC facilities have started continuous monitoring of their systems and jobs to collect performance-related data for understanding performance and operational efficiency.
Scalable Comparative Visualization of Ensembles of Call Graphs
Optimizing the performance of large-scale parallel codes is critical for efficient utilization of computing resources.
Distributed, Parallel, and Cluster Computing Performance
