Search Results for author:
Found 10 papers, 2 papers with code
Can Large Language Models Write Parallel Code?
Large language models are increasingly becoming a popular tool for software development.
Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimization
The primary efficiency bottleneck in such optimizers is matrix inverse calculations in the preconditioning step, which are expensive to compute on GPUs.
Modeling Parallel Programs using Large Language Models
In this paper, we show how large language models (LLMs) can be applied to tasks specific to high performance and scientific codes.
A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs
Large communication costs are a critical bottleneck in training state-of-the-art neural networks on distributed systems.
A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training
Mixture-of-Experts (MoE) is a neural network architecture that adds sparsely activated expert blocks to a base model, increasing the number of parameters without impacting computational costs.
Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Training
Parallel training of neural networks at scale is challenging due to significant overheads arising from communication.
A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks
This phenomenon has spurred the development of algorithms for distributed training of neural networks over a larger number of hardware accelerators.
AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning
This has necessitated the development of efficient algorithms to train these neural networks in parallel on large-scale GPU-based clusters.
Analytics of Longitudinal System Monitoring Data for Performance Prediction
In recent years, several HPC facilities have started continuous monitoring of their systems and jobs to collect performance-related data for understanding performance and operational efficiency.
Scalable Comparative Visualization of Ensembles of Call Graphs
Optimizing the performance of large-scale parallel codes is critical for efficient utilization of computing resources.
Distributed, Parallel, and Cluster Computing Performance
