Search Results for author:
Found 12 papers, 5 papers with code
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
This suggests that a flexible positional embedding design and attention alignment can go a long way toward Transformer length extrapolation.
Advancing Regular Language Reasoning in Linear Recurrent Neural Networks
In recent studies, linear recurrent neural networks (LRNNs) have achieved Transformer-level performance in natural language and long-range modeling, while offering rapid parallel training and constant inference cost.
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
The use of positional embeddings in transformer language models is widely accepted.
Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolation
Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages.
Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis
Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences.
Training Discrete Deep Generative Models via Gapped Straight-Through Estimator
While deep generative models have succeeded in image processing, natural language processing, and reinforcement learning, training that involves discrete random variables remains challenging due to the high variance of its gradient estimation process.
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation
Relative positional embeddings (RPE) have received considerable attention since RPEs effectively model the relative distance among tokens and enable length extrapolation.
Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective
Off-policy Actor-Critic algorithms have demonstrated phenomenal experimental performance but still require better explanations.
Soft Actor-Critic With Integer Actions
Reinforcement learning is well-studied under discrete actions.
PowerGym: A Reinforcement Learning Environment for Volt-Var Control in Power Distribution Systems
We introduce PowerGym, an open-source reinforcement learning environment for Volt-Var control in power distribution systems.
A Contraction Approach to Model-based Reinforcement Learning
Despite its experimental success, Model-based Reinforcement Learning still lacks a complete theoretical understanding.
Model Imitation for Model-Based Reinforcement Learning
Based on the claim, we propose to learn the transition model by matching the distributions of multi-step rollouts sampled from the transition model and the real ones via WGAN.
Model-based Reinforcement Learning
reinforcement-learning
+1
