Search Results for author:
Found 15 papers, 7 papers with code
Microscaling Data Formats for Deep Learning
Narrow bit-width data formats are key to reducing the computational and storage costs of modern deep learning applications.
Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures
During the past decade, Deep Learning (DL) algorithms, programming systems and hardware have converged with the High Performance Computing (HPC) counterparts.
FP8 Formats for Deep Learning
FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors.
FPGA-based AI Smart NICs for Scalable Distributed AI Training Systems
The overhead of these collective communication operations in a distributed AI training system can bottleneck its performance, with more pronounced effects as the number of nodes increases.
Efficient and Generic 1D Dilated Convolution Layer for Deep Learning
Finally, we demonstrate the performance of our optimized 1D convolution layer by utilizing it in the end-to-end neural network training with real genomics datasets and achieve up to 6. 86x speedup over the oneDNN library-based implementation on Cascade Lake CPUs.
DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks
Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible.
Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloads
The TPP specification is platform-agnostic, thus code expressed via TPPs is portable, whereas the TPP implementation is highly-optimized and platform-specific.
PolyDL: Polyhedral Optimizations for Creation of High Performance DL primitives
However, given the constant emergence of new DNN architectures, creating hand optimized code is expensive, slow and is not scalable.
Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architectures
During the last two years, the goal of many researchers has been to squeeze the last bit of performance out of HPC system for AI tasks.
PolyScientist: Automatic Loop Transformations Combined with Microkernels for Optimization of Deep Learning Primitives
In this paper, we develop a hybrid solution to the development of deep learning kernels that achieves the best of both worlds: the expert coded microkernels are utilized for the innermost loops of kernels and we use the advanced polyhedral technology to automatically tune the outer loops for performance.
Training Neural Machine Translation (NMT) Models using Tensor Train Decomposition on TensorFlow (T3F)
For the IWSLT English-Vietnamese training, we obtain BLEU test/dev scores of 24. 0/21. 9 and 24. 2/21. 9 using core dimensions $(2, 2, 256) \times (2, 2, 512)$ with learning rate 0. 0012 and rank distributions $(1, 4, 4, 1)$ and $(1, 4, 16, 1)$ respectively.
High-Performance Deep Learning via a Single Building Block
Deep learning (DL) is one of the most prominent branches of machine learning.
A Study of BFLOAT16 for Deep Learning Training
In this paper, we discuss the flow of tensors and various key operations in mixed precision training, and delve into details of operations, such as the rounding modes for converting FP32 tensors to BFLOAT16.
Anatomy Of High-Performance Deep Learning Convolutions On SIMD Architectures
Convolution layers are prevalent in many classes of deep neural networks, including Convolutional Neural Networks (CNNs) which provide state-of-the-art results for tasks like image recognition, neural machine translation and speech recognition.
Distributed, Parallel, and Cluster Computing
Mixed Precision Training of Convolutional Neural Networks using Integer Operations
The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017).
