Search Results for author:
Found 59 papers, 12 papers with code
Demystifying Lazy Training of Neural Networks from a Macroscopic Viewpoint
While NTK typically assumes that $\lim_{m\to\infty}\frac{\log \kappa}{\log m}=\frac{1}{2}$, and imposes each weight parameters to scale by the factor $\frac{1}{\sqrt{m}}$, in our theta-lazy regime, we discard the factor and relax the conditions to $\lim_{m\to\infty}\frac{\log \kappa}{\log m}>0$.
On the dynamics of three-layer neural networks: initial condensation
Empirical and theoretical works show that the input weights of two-layer neural networks, when initialized with small values, converge towards isolated orientations.
A priori Estimates for Deep Residual Network in Continuous-time Reinforcement Learning
However, existing performance analyses ignores the unique characteristics of continuous-time control problems, is unable to directly estimate the generalization error of the Bellman optimal loss and require a boundedness assumption.
Importance-Aware Image Segmentation-based Semantic Communication for Autonomous Driving
Therefore, we propose a vehicular image segmentation-oriented semantic communication system, termed VIS-SemCom, where image segmentation features of important objects are transmitted to reduce transmission redundancy.
VQA4CIR: Boosting Composed Image Retrieval with Visual Question Answering
By feeding the retrieved image and question to the VQA model, one can find the images inconsistent with relative caption when the answer by VQA is inconsistent with the answer in the QA pair.
Ultra-Long Sequence Distributed Transformer
This paper presents a novel and efficient distributed training method, the Long Short-Sequence Transformer (LSS Transformer), for training transformer with long sequences.
Structure and Gradient Dynamics Near Global Minima of Two-layer Neural Networks
Under mild assumptions, we investigate the structure of loss landscape of two-layer neural networks near global minima, determine the set of parameters which give perfect generalization, and fully characterize the gradient flows around it.
Exploring Winograd Convolution for Cost-effective Neural Network Fault Tolerance
When it is applied on fault-tolerant neural networks enhanced with fault-aware retraining and constrained activation functions, the resulting model accuracy generally shows significant improvement in presence of various faults.
Optimistic Estimate Uncovers the Potential of Nonlinear Models
We propose an optimistic estimate to evaluate the best possible fitting performance of nonlinear models.
Stochastic Modified Equations and Dynamics of Dropout Algorithm
In order to investigate the underlying mechanism by which dropout facilitates the identification of flatter minima, we study the noise structure of the derived stochastic modified equation for dropout.
DeepFire2: A Convolutional Spiking Neural Network Accelerator on FPGAs
Brain-inspired spiking neural networks (SNNs) replace the multiply-accumulate operations of traditional neural networks by integrate-and-fire neurons, with the goal of achieving greater energy efficiency.
Hierarchical Weight Averaging for Deep Neural Networks
Despite the simplicity, stochastic gradient descent (SGD)-like algorithms are successful in training deep neural networks (DNNs).
Phase Diagram of Initial Condensation for Two-layer Neural Networks
The phenomenon of distinct behaviors exhibited by neural networks under varying scales of initialization remains an enigma in deep learning research.
Optimization of Image Transmission in a Cooperative Semantic Communication Networks
In this paper, a semantic communication framework for image transmission is developed.
Multi-agent Reinforcement Learning
Reinforcement Learning (RL)
Linear Stability Hypothesis and Rank Stratification for Nonlinear Models
By these results, model rank of a target function predicts a minimal training data size for its successful recovery.
Desire Backpropagation: A Lightweight Training Algorithm for Multi-Layer Spiking Neural Networks based on Spike-Timing-Dependent Plasticity
Spiking neural networks (SNNs) are a viable alternative to conventional artificial neural networks when resource efficiency and computational complexity are of importance.
Beyond Codebook-Based Analog Beamforming at mmWave: Compressed Sensing and Machine Learning Methods
Analog beamforming is the predominant approach for millimeter wave (mmWave) communication given its favorable characteristics for limited-resource devices.
Statistical Modeling of Soft Error Influence on Neural Networks
Prior work mainly rely on fault simulation to analyze the influence of soft errors on NN processing.
URGLQ: An Efficient Covariance Matrix Reconstruction Method for Robust Adaptive Beamforming
The computational complexity of the conventional adaptive beamformer is relatively large, and the performance degrades significantly due to the model mismatch errors and the unwanted signals in received data.
Performance Optimization for Semantic Communications: An Attention-based Reinforcement Learning Approach
Hence, the BS must select an appropriate resource block for each user as well as determine and transmit part of the semantic information to the users.
A Resource-efficient Spiking Neural Network Accelerator Supporting Emerging Neural Encoding
However, current accelerators for SNN cannot well support the emerging encoding schemes.
Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks
Regarding the easy training of deep networks, we show that local minimum of an NN can be lifted to strict saddle points of a deeper NN.
An Experimental Comparison Between Temporal Difference and Residual Gradient with Neural Network Approximation
In this work, we perform extensive experiments to show that TD outperforms RG, that is, when the training leads to a small Bellman residual error, the solution found by TD has a better policy and is more robust against the perturbation of neural network parameters.
Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width
Through experiments under three-layer condition, our phase diagram suggests a complicated dynamical regimes consisting of three possible regimes, together with their mixture, for deep NNs and provides a guidance for studying deep NNs in different initialization regimes, which reveals the possibility of completely different dynamics emerging within a deep NN for its different layers.
Degree aware based adversarial graph convolutional networks for entity alignment in heterogeneous knowledge graph
This degree difference between equivalent entities poses a great challenge for entity alignment.
Winograd Convolution: A Perspective from Fault Tolerance
Winograd convolution is originally proposed to reduce the computing overhead by converting multiplication in neural network (NN) with addition via linear transformation.
Limitation of Characterizing Implicit Regularization by Data-independent Functions
In recent years, understanding the implicit regularization of neural networks (NNs) has become a central task in deep learning theory.
Overview frequency principle/spectral bias in deep learning
This low-frequency implicit bias reveals the strength of neural network in learning low-frequency functions as well as its deficiency in learning high-frequency functions.
Optimizing for In-memory Deep Learning with Emerging Memory Technology
In-memory deep learning has already demonstrated orders of magnitude higher performance density and energy efficiency.
Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
We prove a general Embedding Principle of loss landscape of deep neural networks (NNs) that unravels a hierarchical structure of the loss landscape of NNs, i. e., loss landscape of an NN contains all critical points of all the narrower NNs.
E3NE: An End-to-End Framework for Accelerating Spiking Neural Networks with Emerging Neural Encoding on FPGAs
Compiler frameworks are crucial for the widespread use of FPGA-based deep learning accelerators.
BLUnet: Arithmetic-free Inference with Bit-serialised Table Lookup Operation for Efficient Deep Neural Networks
In this paper, we propose BLUnet, a table lookup-based DNN model with bit-serialized input to overcome this challenge.
MOD-Net: A Machine Learning Approach via Model-Operator-Data Network for Solving PDEs
In this paper, we propose a a machine learning approach via model-operator-data network (MOD-Net) for solving PDEs.
Privacy Budget Scheduling
We describe PrivateKube, an extension to the popular Kubernetes datacenter orchestrator that adds privacy as a new type of resource to be managed alongside other traditional compute resources, such as CPU, GPU, and memory.
Embedding Principle of Loss Landscape of Deep Neural Networks
Understanding the structure of loss landscape of deep neural networks (DNNs)is obviously important.
An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
frequency in DNN training.
Towards Understanding the Condensation of Neural Networks at Initial Training
Our theoretical analysis confirms experiments for two cases, one is for the activation function of multiplicity one with arbitrary dimension input, which contains many common activation functions, and the other is for the layer with one-dimensional input and arbitrary multiplicity.
DTNN: Energy-efficient Inference with Dendrite Tree Inspired Neural Networks for Edge Vision Applications
DTNN achieved significant energy saving (19. 4X and 64. 9X improvement on ResNet-18 and VGG-11 with ImageNet, respectively) with negligible loss of accuracy.
Efficient Spiking Neural Networks with Radix Encoding
Traditionally, a spike train needs around one thousand time steps to approach similar accuracy as its ANN counterpart.
Nonlinear Weighted Directed Acyclic Graph and A Priori Estimates for Neural Networks
In an attempt to better understand structural benefits and generalization power of deep neural networks, we firstly present a novel graph theoretical formulation of neural network models, including fully connected, residual network (ResNet) and densely connected networks (DenseNet).
RCT: Resource Constrained Training for Edge AI
For example, quantisation-aware training (QAT) method involves two copies of model parameters, which is usually beyond the capacity of on-chip memory in edge devices.
QROSS: QUBO Relaxation Parameter Optimisation via Learning Solver Surrogates
In this way, we are able capture the common structure of the instances and their interactions with the solver, and produce good choices of penalty parameters with fewer number of calls to the QUBO solver.
Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks
Why heavily parameterized neural networks (NNs) do not overfit the data is an important long standing open question.
Meta-Reinforcement Learning for Reliable Communication in THz/VLC Wireless VR Networks
The problem is formulated as an optimization problem whose goal is to maximize the reliability of the VR network by selecting the appropriate VAPs to be turned on and controlling the user association with SBSs.
Adaptive Precision Training for Resource Constrained Devices
We use model of the same precision for both forward and backward pass in order to reduce memory usage for training.
A comprehensive study on the semileptonic decay of heavy flavor mesons
The semileptonic decay of heavy flavor mesons offers a clean environment for extraction of the Cabibbo-Kobayashi-Maskawa (CKM) matrix elements, which describes the CP-violating and flavor changing process in the Standard Model.
High Energy Physics - Phenomenology High Energy Physics - Experiment
Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning
A supervised learning problem is to find a function in a hypothesis function space given values on isolated data points.
On the exact computation of linear frequency principle dynamics and its generalization
Recent works show an intriguing phenomenon of Frequency Principle (F-Principle) that deep neural networks (DNNs) fit the target function from low to high frequency during the training, which provides insight into the training and generalization behavior of DNNs in complex tasks.
A regularized deep matrix factorized model of matrix completion for image restoration
In this work, we propose a Regularized Deep Matrix Factorized (RDMF) model for image restoration, which utilizes the implicit bias of the low rank of deep neural networks and the explicit bias of total variation.
Phase diagram for two-layer ReLU neural networks at infinite-width limit
In this work, inspired by the phase diagram in statistical mechanics, we draw the phase diagram for the two-layer ReLU neural network at the infinite-width limit for a complete characterization of its dynamical regimes and their dependence on hyperparameters related to initialization.
Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)
Gradient descent yields zero training loss in polynomial time for deep neural networks despite non-convex nature of the objective function.
Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
The problem of solving partial differential equations (PDEs) can be formulated into a least-squares minimization problem, where neural networks are used to parametrize PDE solutions.
EDCompress: Energy-Aware Model Compression for Dataflows
EDCompress could also find the optimal dataflow type for specific neural networks in terms of energy consumption, which can guide the deployment of CNN models on hardware systems.
Deep Learning for Optimal Deployment of UAVs with Visible Light Communications
Using GRUs and CNNs, the UAVs can model the long-term historical illumination distribution and predict the future illumination distribution.
Gated Recurrent Units Learning for Optimal Deployment of Visible Light Communications Enabled UAVs
This problem is formulated as an optimization problem whose goal is to minimize the total transmit power while meeting the illumination and communication requirements of users.
Theory of the Frequency Principle for General Deep Neural Networks
Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Principle (F-Principle): a DNN tends to learn a target function from low to high frequencies during the training.
Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
It remains a puzzle that why deep neural networks (DNNs), with more parameters than samples, often generalize well.
A type of generalization error induced by initialization in deep neural networks
Overall, our work serves as a baseline for the further investigation of the impact of initialization and loss function on the generalization of DNNs, which can potentially guide and improve the training of DNNs in practice.
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective.
