Search Results for author:
Found 11 papers, 5 papers with code
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
This suggests that a flexible positional embedding design and attention alignment can go a long way toward Transformer length extrapolation.
Advancing Regular Language Reasoning in Linear Recurrent Neural Networks
In recent studies, linear recurrent neural networks (LRNNs) have achieved Transformer-level performance in natural language and long-range modeling, while offering rapid parallel training and constant inference cost.
Structured Dialogue Discourse Parsing
In addition, unlike in previous work, we do not rely on hand-crafted features; this improves the model's robustness.
Ranked #1 on
Discourse Parsing
on STAC
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
The use of positional embeddings in transformer language models is widely accepted.
Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolation
Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages.
Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis
Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences.
Training Discrete Deep Generative Models via Gapped Straight-Through Estimator
While deep generative models have succeeded in image processing, natural language processing, and reinforcement learning, training that involves discrete random variables remains challenging due to the high variance of its gradient estimation process.
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation
Relative positional embeddings (RPE) have received considerable attention since RPEs effectively model the relative distance among tokens and enable length extrapolation.
Zero-Shot Dialogue Disentanglement by Self-Supervised Entangled Response Selection
In this paper, we are the first to propose a~\textbf{zero-shot} dialogue disentanglement solution.
Tartan: A retrieval-based socialbot powered by a dynamic finite-state machine architecture
Tartan is a non-goal-oriented socialbot focused around providing users with an engaging and fluent casual conversation.
Learning Conversational Systems that Interleave Task and Non-Task Content
These systems work well when users have clear and explicit intentions that are well-aligned to the systems' capabilities.
