Search Results for author:
Found 19 papers, 4 papers with code
Resonance RoPE: Improving Context Length Generalization of Large Language Models
This paper addresses the challenge of train-short-test-long (TSTL) scenarios in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE), where models pre-trained on shorter sequences face difficulty with out-of-distribution (OOD) token positions in longer sequences.
Hyperparameter Optimization for Large Language Model Instruction-Tuning
The performance on downstream tasks of models fine-tuned with LoRA heavily relies on a set of hyperparameters including the rank of the decomposition.
Attribute Controlled Dialogue Prompting
Prompt-tuning has become an increasingly popular parameter-efficient method for adapting large pretrained language models to downstream tasks.
LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization
Label Smoothing (LS) is another simple, versatile and efficient regularization which can be applied to various supervised classification tasks.
Mathematical Challenges in Deep Learning
Deep models are dominating the artificial intelligence (AI) industry since the ImageNet challenge in 2012.
KronA: Parameter Efficient Tuning with Kronecker Adapter
We apply the proposed methods for fine-tuning T5 on the GLUE benchmark to show that incorporating the Kronecker-based modules can outperform state-of-the-art PET methods.
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization
Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowledge from a larger model (a teacher).
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
Moreover, we observe that this simple optimization technique is able to outperform the state-of-the-art KD methods for compact models.
DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation
Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training.
Learning Functions on Multiple Sets using Multi-Set Transformers
We propose a general deep architecture for learning functions on multiple permutation-invariant sets.
Do we need Label Regularization to Fine-tune Pre-trained Language Models?
Knowledge Distillation (KD) is a prominent neural model compression technique that heavily relies on teacher network predictions to guide the training of a student model.
A Short Study on Compressing Decoder-Based Language Models
Pre-trained Language Models (PLMs) have been successful for a wide range of natural language processing (NLP) tasks.
Equivariant Finite Normalizing Flows
Generative modeling seeks to uncover the underlying factors that give rise to observed data that can often be modeled as the natural symmetries that manifest themselves through invariances and equivariances to certain transformation laws.
Pseudo Knowledge Distillation: Towards Learning Optimal Instance-specific Label Smoothing Regularization
Knowledge Distillation (KD) is an algorithm that transfers the knowledge of a trained, typically larger, neural network into another model under training.
Polarized-VAE: Proximity Based Disentangled Representation Learning for Text Generation
Learning disentangled representations of real-world data is a challenging open problem.
Generating Emotionally Aligned Responses in Dialogues using Affect Control Theory
State-of-the-art neural dialogue systems excel at syntactic and semantic modelling of language, but often have a hard time establishing emotional alignment with the human interactant during a conversation.
Normalizing Flows: An Introduction and Review of Current Methods
Normalizing Flows are generative models which produce tractable distributions where both sampling and density evaluation can be efficient and exact.
Tails of Lipschitz Triangular Flows
We investigate the ability of popular flow based methods to capture tail-properties of a target density by studying the increasing triangular maps used in these flow methods acting on a tractable source density.
Representation Learning for Dynamic Graphs: A Survey
Graphs arise naturally in many real-world applications including social networks, recommender systems, ontologies, biology, and computational finance.
