Search Results for author:
Found 72 papers, 17 papers with code
Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation
Intermediate layer matching is shown as an effective approach for improving knowledge distillation (KD).
How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge in a large neural network into a smaller one.
RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation
Knowledge Distillation (KD) is extensively used in Natural Language Processing to compress the pre-training and task-specific fine-tuning phases of large neural language models.
KroneckerBERT: Significant Compression of Pre-trained Language Models Through Kronecker Decomposition and Knowledge Distillation
We push the limits of state-of-the-art Transformer-based pre-trained language model compression using Kronecker decomposition.
Learning Chemotherapy Drug Action via Universal Physics-Informed Neural Networks
Quantitative systems pharmacology (QSP) is widely used to assess drug effects and toxicity before the drug goes to clinical trial.
Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling
In the rapidly evolving landscape of deep learning, the quest for models that balance expressivity with computational efficiency has never been more critical.
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
Finetuning large language models requires huge GPU memory, restricting the choice to acquire Larger models.
WERank: Towards Rank Degradation Prevention for Self-Supervised Learning Using Weight Regularization
A common phenomena confining the representation quality in Self-Supervised Learning (SSL) is dimensional collapse (also known as rank degeneration), where the learned representations are mapped to a low dimensional subspace of the representation space.
Scalable Graph Self-Supervised Learning
Our experiments demonstrate that the cost associated with the loss computation can be reduced via node or dimension sampling without lowering the downstream performance.
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference
We extend SortedNet to generative NLP tasks, making large language models dynamic without any Pre-Training and by only replacing Standard Fine-Tuning (SFT) with Sorted Fine-Tuning (SoFT).
SortedNet, a Place for Every Network and Every Network in its Place: Towards a Generalized Solution for Training Many-in-One Neural Networks
Deep neural networks (DNNs) must cater to a variety of users with different performance needs and budgets, leading to the costly practice of training, storing, and maintaining numerous specific models.
Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey
Then, we introduce LSTM gates and cells, history and variants of LSTM, and Gated Recurrent Units (GRU).
Improved knowledge distillation by utilizing backward pass knowledge in neural networks
Augmenting the training set by adding this auxiliary improves the performance of KD significantly and leads to a closer match between the student and the teacher.
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization
Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowledge from a larger model (a teacher).
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
Moreover, we observe that this simple optimization technique is able to outperform the state-of-the-art KD methods for compact models.
DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation
Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training.
Do we need Label Regularization to Fine-tune Pre-trained Language Models?
Knowledge Distillation (KD) is a prominent neural model compression technique that heavily relies on teacher network predictions to guide the training of a student model.
Theoretical Connection between Locally Linear Embedding, Factor Analysis, and Probabilistic PCA
Locally Linear Embedding (LLE) is a nonlinear spectral dimensionality reduction and manifold learning method.
When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation
From a pre-generated pool of augmented samples, Glitter adaptively selects a subset of worst-case samples with maximal loss, analogous to adversarial DA.
Spectral, Probabilistic, and Deep Metric Learning: Tutorial and Survey
In deep learning methods, we first introduce reconstruction autoencoders and supervised loss functions for metric learning.
Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Survey
Finally, we explain the autoencoders based on adversarial learning including adversarial autoencoder, PixelGAN, and implicit autoencoder.
Sufficient Dimension Reduction for High-Dimensional Regression and Low-Dimensional Embedding: Tutorial and Survey
Finally, we explain Kernel Dimension Reduction (KDR) both for supervised and unsupervised learning.
Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher
A case in point is that the best performing checkpoint of the teacher might not necessarily be the best teacher for training the student in KD.
KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey
Then, we explain second-order methods including Newton's method for unconstrained, equality constrained, and inequality constrained problems....
Knowledge Distillation with Noisy Labels for Natural Language Understanding
Knowledge Distillation (KD) is extensively used to compress and deploy large pre-trained language models on edge devices for real-world applications.
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one.
KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation
We present our KroneckerBERT, a compressed version of the BERT_BASE model obtained using this framework.
Uniform Manifold Approximation and Projection (UMAP) and its Variants: Tutorial and Survey
We start with UMAP algorithm where we explain probabilities of neighborhood in the input and embedding spaces, optimization of cost function, training algorithm, derivation of gradients, and supervised and semi-supervised embedding by UMAP.
Johnson-Lindenstrauss Lemma, Linear and Nonlinear Random Projections, Random Fourier Features, and Random Kitchen Sinks: Tutorial and Survey
This is a tutorial and survey paper on the Johnson-Lindenstrauss (JL) lemma and linear and nonlinear random projections.
Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey
Then, we introduce the structures of BM and RBM.
Generative locally linear embedding: A module for manifold unfolding and visualization
One can unfold the nonlinear manifold of a dataset for low-dimensional visualization and feature extraction.
Unified Framework for Spectral Dimensionality Reduction, Maximum Variance Unfolding, and Kernel Learning By Semidefinite Programming: Tutorial and Survey
This is a tutorial and survey paper on unification of spectral dimensionality reduction methods, kernel learning by Semidefinite Programming (SDP), Maximum Variance Unfolding (MVU) or Semidefinite Embedding (SDE), and its variants.
SymbolicGPT: A Generative Transformer Model for Symbolic Regression
Symbolic regression is the task of identifying a mathematical expression that best fits a provided dataset of input and output values.
Legendre Deep Neural Network (LDNN) and its application for approximation of nonlinear Volterra Fredholm Hammerstein integral equations
In this paper, we propose Legendre Deep Neural Network (LDNN) for solving nonlinear Volterra Fredholm Hammerstein integral equations (VFHIEs).
Reproducing Kernel Hilbert Space, Mercer's Theorem, Eigenfunctions, Nyström Method, and Use of Kernels in Machine Learning: Tutorial and Survey
We start with reviewing the history of kernels in functional analysis and machine learning.
Laplacian-Based Dimensionality Reduction Including Spectral Clustering, Laplacian Eigenmap, Locality Preserving Projection, Graph Embedding, and Diffusion Map: Tutorial and Survey
Versions of graph embedding are then explained which are generalized versions of Laplacian eigenmap and locality preserving projection.
Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMax
We exploit a semi-supervised approach based on KD to train a model on augmented data.
Annealing Knowledge Distillation
Knowledge distillation (KD) is a prominent model compression technique for deep neural networks in which the knowledge of a trained large teacher model is transferred to a smaller student model.
Generative Locally Linear Embedding
In this work, we propose two novel generative versions of LLE, named Generative LLE (GLLE), whose linear reconstruction steps are stochastic rather than deterministic.
Computationally instrument-resolution-independent de novo peptide sequencing for high-resolution devices
De novo peptide sequencing is the key technology for finding novel peptides from mass spectra.
Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Survey
Finally, VAE is explained where the encoder, decoder and sampling from the latent space are introduced.
Legendre Deep Neural Network (LDNN) and its application for approximation of nonlinear Volterra–Fredholm–Hammerstein integral equations
In particular, Volterra–Fredholm–Hammerstein integral equations are the main type of these integral equations and researchers are interested in investigating and solving these equations.
Locally Linear Embedding and its Variants: Tutorial and Survey
In this paper, we first cover LLE, kernel LLE, inverse LLE, and feature fusion with LLE.
Attention Mechanism, Transformers, BERT, and GPT: Tutorial and Survey
Thereafter, we introduce the Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT) as the stacks of encoders and decoders of transformer, respectively.
Symbolically Solving Partial Differential Equations using Deep Learning
We describe a neural-based method for generating exact or approximate solutions to differential equations in the form of mathematical expressions.
A Neuro-Symbolic Method for Solving Differential and Functional Equations
When neural networks are used to solve differential equations, they usually produce solutions in the form of black-box functions that are not directly mathematically interpretable.
Stochastic Neighbor Embedding with Gaussian and Student-t Distributions: Tutorial and Survey
Stochastic Neighbor Embedding (SNE) is a manifold learning and dimensionality reduction method with a probabilistic approach.
Multidimensional Scaling, Sammon Mapping, and Isomap: Tutorial and Survey
Then, Sammon mapping, Isomap, and kernel Isomap are explained.
Segmentation Approach for Coreference Resolution Task
This has been accomplished by defining an embedding method for the position of all members of a coreference cluster in a document and resolving all of them for a given mention.
DeepNovoV2: Better de novo peptide sequencing with deep learning
Personalized cancer vaccines are envisioned as the next generation rational cancer immunotherapy.
Deep learning enables de novo peptide sequencing from data-independent-acquisition mass spectrometry
We present DeepNovo-DIA, a de novo peptide-sequencing method for data-independent acquisition (DIA) mass spectrometry data.
Deep Variational Sufficient Dimensionality Reduction
We consider the problem of sufficient dimensionality reduction (SDR), where the high-dimensional observation is transformed to a low-dimensional sub-space in which the information of the observations regarding the label variable is preserved.
SRP: Efficient class-aware embedding learning for large-scale data via supervised random projections
While stochastic approximation strategies have been explored for unsupervised dimensionality reduction to tackle this challenge, such approaches are not well-suited for accelerating computational speed for supervised dimensionality reduction.
Text Classification based on Multiple Block Convolutional Highways
In the Text Classification areas of Sentiment Analysis, Subjectivity/Objectivity Analysis, and Opinion Polarity, Convolutional Neural Networks have gained special attention because of their performance and accuracy.
A Berkeley View of Systems Challenges for AI
With the increasing commoditization of computer vision, speech recognition and machine translation systems and the widespread deployment of learning-based back-end technologies such as digital advertising and intelligent infrastructures, AI (Artificial Intelligence) has moved from research labs to production.
Disentangling Dynamics and Content for Control and Planning
In this paper, We study the problem of learning a controllable representation for high-dimensional observations of dynamical systems.
JADE: Joint Autoencoders for Dis-Entanglement
The problem of feature disentanglement has been explored in the literature, for the purpose of image and video processing and text analysis.
Improving the Accuracy of Pre-trained Word Embeddings for Sentiment Analysis
In this paper we propose a novel method, Improved Word Vectors (IWV), which increases the accuracy of pre-trained word embeddings in sentiment analysis.
Robust Locally-Linear Controllable Embedding
We also propose a principled variational approximation of the embedding posterior that takes the future observation into account, and thus, makes the variational approximation more robust against the noise.
Deep Structure for end-to-end inverse rendering
Inverse rendering in a 3D format denoted to recovering the 3D properties of a scene given 2D input image(s) and is typically done using 3D Morphable Model (3DMM) based methods from single view images.
Fast Spectral Clustering Using Autoencoders and Landmarks
Spectral clustering is a powerful clustering algorithm that suffers from high computational complexity, due to eigen decomposition.
Generative Mixture of Networks
The model consists of K networks that are trained together to learn the underlying distribution of a given data set.
Semi-Supervised Representation Learning based on Probabilistic Labeling
Then, we map the data to lower-dimensional space using a linear transformation such that the dependency between the transformed data and the assigned labels is maximized.
Semi-supervised Dictionary Learning Based on Hilbert-Schmidt Independence Criterion
The proposed method benefits from the supervisory information by learning the dictionary in a space where the dependency between the data and class labels is maximized.
On the Invariance of Dictionary Learning and Sparse Representation to Projecting Data to a Discriminative Space
In this paper, it is proved that dictionary learning and sparse representation is invariant to a linear transformation.
Supervised Dictionary Learning and Sparse Representation-A Review
This review provides a broad, yet deep, view of the state-of-the-art methods for S-DLSR and allows for the advancement of research and development in this emerging area of research.
The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox
In this work, we present Velox, a new component of the Berkeley Data Analytics Stack.
Databases
Greedy Column Subset Selection for Large-scale Data Sets
The algorithm first learns a concise representation of all columns using random projection, and it then solves a generalized column subset selection problem at each machine in which a subset of columns are selected from the sub-matrix on that machine such that the reconstruction error of the concise representation is minimized.
A Fast Greedy Algorithm for Generalized Column Subset Selection
This paper defines a generalized column subset selection problem which is concerned with the selection of a few columns from a source matrix A that best approximate the span of a target matrix B.
Highly Available Transactions: Virtues and Limitations (Extended Version)
To minimize network latency and remain online during server failures and network partitions, many modern distributed data storage systems eschew transactional functionality, which provides strong semantic guarantees for groups of multiple operations over multiple data items.
Databases
Supervised Texture Classification Using a Novel Compression-Based Similarity Measure
To this end, by design, it solely uses P-frame coding to find the (dis)similarity among patches/images.
Kernelized Supervised Dictionary Learning
In this paper, we propose supervised dictionary learning (SDL) by incorporating information on class labels into the learning of the dictionary.
