Search Results for author:
Found 37 papers, 6 papers with code
Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization
Large Language Models (LLMs) are proficient in natural language processing tasks, but their deployment is often restricted by extensive parameter sizes and computational demands.
Token-Scaled Logit Distillation for Ternary Weight Generative Language Models
Generative Language Models (GLMs) have shown impressive performance in tasks such as text generation, understanding, and reasoning.
Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers
Pre-trained Transformer models such as BERT have shown great success in a wide range of applications, but at the cost of substantial increases in model complexity.
Sleep Model -- A Sequence Model for Predicting the Next Sleep Stage
As sleep disorders are becoming more prevalent there is an urgent need to classify sleep stages in a less disturbing way. In particular, sleep-stage classification using simple sensors, such as single-channel electroencephalography (EEG), electrooculography (EOG), electromyography (EMG), or electrocardiography (ECG) has gained substantial interest.
Exploring Attention Map Reuse for Efficient Transformer Neural Networks
In this paper, we provide a comprehensive study on attention map reuse focusing on its ability to accelerate inference.
Macro-block dropout for improved regularization in training end-to-end speech recognition models
In our work, we define a macro-block that contains a large number of units from the input to a Recurrent Neural Network (RNN).
A Comparison of Transformer, Convolutional, and Recurrent Neural Networks on Phoneme Recognition
Our analyses show that Transformer and Conformer models benefit from the long-range accessibility of self-attention through input frames.
Similarity and Content-based Phonetic Self Attention for Speech Recognition
Especially, SA heads in lower layers capture various phonetic characteristics by the query-key dot product, which is designed to compute the pairwise relationship between frames.
Korean Tokenization for Beam Search Rescoring in Speech Recognition
Although the common approach is to use the same tokenization method for external LM as the ASR model, we show that it may not be the best choice for Korean.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
TernGEMM: GEneral Matrix Multiply Library with Ternary Weights for Fast DNN Inference
We propose TernGEMM, a special GEMM library using SIMD instructions for Deep Neural Network (DNN) inference with ternary weights and activations under 8-bit.
Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling
While Transformer-based models have shown impressive language modeling performance, the large computation cost is often prohibitive for practical use.
Understanding the Role of Self Attention for Efficient Speech Recognition
Self-attention (SA) is a critical component of Transformer neural networks that have succeeded in automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks
In this study, we propose stochastic precision ensemble training for QDNNs (SPEQ).
S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
The proposed method is compared with the conventional SGD method and previous weight-noise injection algorithms using convolutional neural networks for image classification.
Quantized Neural Networks: Characterization and Holistic Optimization
This study proposes a holistic approach for the optimization of QDNNs, which contains QDNN training methods as well as quantization-friendly architecture design.
SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
Model averaging is a promising approach for achieving the good generalization capability of DNNs, especially when the loss surface for training contains many sharp minima.
Knowledge distillation for optimization of quantized deep neural networks
Knowledge distillation (KD) is a very popular method for model size reduction.
Fully Neural Network Based Speech Recognition on Mobile and Embedded Devices
The RNN implementation on embedded devices can suffer from excessive DRAM accesses because the parameter size of a neural network usually exceeds that of the cache memory and the parameters are used only once for each time step.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
EXPLORATION OF EFFICIENT ON-DEVICE ACOUSTIC MODELING WITH NEURAL NETWORKS
In addition, we explore neural networks that equip one-dimensional (1-D) convolution at each layer of these algorithms, and by which can obtain a very large performance increase in the QRNNs and Gated ConvNets.
Single Stream Parallelization of Recurrent Neural Networks for Low Power and Fast Inference
As neural network algorithms show high performance in many applications, their efficient inference on mobile and embedded systems are of great interests.
SVD-Softmax: Fast Softmax Approximation on Large Vocabulary Neural Networks
The approximate probability of each word can be estimated with only a small part of the weight matrix by using a few large singular values and the corresponding elements for most of the words.
Structured Sparse Ternary Weight Coding of Deep Neural Networks for Efficient Hardware Implementations
Deep neural networks (DNNs) usually demand a large amount of operations for real-time inference.
Fixed-point optimization of deep neural networks with adaptive step size retraining
Fixed-point optimization of deep neural networks plays an important role in hardware based design and low-power implementations.
Quantized neural network design under weight capacity constraint
The complexity of deep neural network algorithms for hardware implementation can be lowered either by scaling the number of units or reducing the word-length of weights.
Compact Deep Convolutional Neural Networks With Coarse Pruning
We propose feature map and kernel level pruning for reducing the computational complexity of a deep convolutional neural network.
FPGA-Based Low-Power Speech Recognition with Recurrent Neural Networks
The weights are quantized to 6 bits to store all of them in the on-chip memory of an FPGA.
Character-Level Language Modeling with Hierarchical Recurrent Neural Networks
Recurrent neural network (RNN) based character-level language models (CLMs) are extremely useful for modeling out-of-vocabulary words by nature.
Dynamic Hand Gesture Recognition for Wearable Devices with Low Complexity Recurrent Neural Networks
Gesture recognition is a very essential technology for many wearable devices.
Generative Knowledge Transfer for Neural Language Models
In this paper, we propose a generative knowledge transfer technique that trains an RNN based language model (student network) using text and output probabilities generated from a previously trained RNN (teacher network).
FPGA Based Implementation of Deep Neural Networks Using On-chip Memory Only
In this work, we have developed an FPGA based fixed-point DNN system using only on-chip memory not to access external DRAM.
Character-Level Incremental Speech Recognition with Recurrent Neural Networks
The output values of the CTC-trained RNN are character-level probabilities, which are processed by beam search decoding.
Online Keyword Spotting with a Character-Level Recurrent Neural Network
In this paper, we propose a context-aware keyword spotting model employing a character-level recurrent neural network (RNN) for spoken term detection in continuous speech.
Structured Pruning of Deep Convolutional Neural Networks
To decide the importance of network connections and paths, the proposed method uses a particle filtering approach.
Fixed-Point Performance Analysis of Recurrent Neural Networks
Recurrent neural networks have shown excellent performance in many applications, however they require increased complexity in hardware or software based implementations.
Online Sequence Training of Recurrent Neural Networks with Connectionist Temporal Classification
Our online model achieves 20. 7% phoneme error rate (PER) on the very long input sequence that is generated by concatenating all 192 utterances in the TIMIT core test set.
Resiliency of Deep Neural Networks under Quantization
In this work, the effects of retraining are analyzed for a feedforward deep neural network (FFDNN) and a convolutional neural network (CNN).
Single stream parallelization of generalized LSTM-like RNNs on a GPU
Recurrent neural networks (RNNs) have shown outstanding performance on processing sequence data.
