Search Results for author:
Found 9 papers, 2 papers with code
Edge Private Graph Neural Networks with Singular Value Perturbation
Eclipse adds noise to the low-rank singular values instead of the entire graph, thereby preserving the graph privacy while still maintaining enough of the graph structure to maintain model utility.
Ethos: Rectifying Language Models in Orthogonal Parameter Space
Evaluations show Ethos is more effective in removing undesired knowledge and maintaining the overall model performance compared to current task arithmetic methods.
ATP: Enabling Fast LLM Serving via Attention on Top Principal Keys
In particular, ATP barely loses accuracy with only $1/2$ principal keys, and only incurs around $2\%$ accuracy drops with $1/4$ principal keys.
All Rivers Run to the Sea: Private Learning with Asymmetric Flows
The main part flows into a small model while the residuals are offloaded to a large model.
mL-BFGS: A Momentum-based L-BFGS for Distributed Large-Scale Neural Network Optimization
Quasi-Newton methods still face significant challenges in training large-scale neural networks due to additional compute costs in the Hessian related computations and instability issues in stochastic training.
Federated Learning of Large Models at the Edge via Principal Sub-Model Training
However, the heterogeneous-client setting requires some clients to train full model, which is not aligned with the resource-constrained setting; while the latter ones break privacy promises in FL when sharing intermediate representations or labels with the server.
Lottery Aware Sparsity Hunting: Enabling Federated Learning on Resource-Limited Edge
A possible solution to this problem is to utilize off-the-shelf sparse learning algorithms at the clients to meet their resource budget.
SLIM-QN: A Stochastic, Light, Momentumized Quasi-Newton Optimizer for Deep Neural Networks
SLIM-QN addresses two key barriers in existing second-order methods for large-scale DNNs: 1) the high computational cost of obtaining the Hessian matrix and its inverse in every iteration (e. g. KFAC); 2) convergence instability due to stochastic training (e. g. L-BFGS).
SPEC2: SPECtral SParsE CNN Accelerator on FPGAs
On the other hand, weight pruning techniques address the redundancy in model parameters by converting dense convolutional kernels into sparse ones.
