Search Results for author:
Found 7 papers, 1 papers with code
Why "classic" Transformers are shallow and how to make them go deep
Since its introduction in 2017, Transformer has emerged as the leading neural network architecture, catalyzing revolutionary advancements in many AI disciplines.
POViT: Vision Transformer for Multi-objective Design and Characterization of Nanophotonic Devices
In this work, we propose the first-ever Transformer model (POViT) to efficiently design and simulate semiconductor photonic devices with multiple objectives.
Variability of Neural Networks and Han-Layer: A Variability-Inspired Model
We introduce a notion of variability to view such issues under the setting of a fixed number of parameters which is, in general, a dominant cost-factor.
A Lightweight and Gradient-Stable Neural Layer
To enhance resource efficiency and model deployability of neural networks, we propose a neural-layer architecture based on Householder weighting and absolute-value activating, called Householder-absolute neural layer or simply Han-layer.
Multi-layer Perceptron Trainability Explained via Variability
Despite the tremendous successes of deep neural networks (DNNs) in various applications, many fundamental aspects of deep learning remain incompletely understood, including DNN trainability.
The simpler the better: vanilla sgd revisited
The stochastic gradient descent (SGD) method, first proposed in 1950's, has been the foundation for deep-neural-network (DNN) training with numerous enhancements including adding a momentum or adaptively selecting learning rates, or using both strategies and more.
AuxBlocks: Defense Adversarial Example via Auxiliary Blocks
Deep learning models are vulnerable to adversarial examples, which poses an indisputable threat to their applications.
