Search Results for author:
Found 18 papers, 12 papers with code
Unsupervised Dependency Graph Network
We introduce a new model, the Unsupervised Dependency Graph Network (UDGN), that can induce dependency structures from raw corpora and the masked language modeling task.
Stick-breaking Attention
We propose an alternative attention mechanism based on the stick-breaking process: For each token before the current, we determine a break point $\beta_{i, j}$, which represents the proportion of the remaining stick to allocate to the current token.
Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler
This is not only because there is a complicated correlation between learning rate, batch size, number of training tokens, model size, and other hyperparameters but also because it is prohibitively expensive to perform a hyperparameter search for large language models with Billions or Trillions of parameters.
Scattered Mixture-of-Experts Implementation
We present ScatterMoE, an implementation of Sparse Mixture-of-Experts (SMoE) on GPUs.
CattleEyeView: A Multi-task Top-down View Cattle Dataset for Smarter Precision Livestock Farming
Cattle farming is one of the important and profitable agricultural industries.
Sparse Universal Transformer
The Universal Transformer (UT) is a variant of the Transformer that shares parameters across its layers.
ModuleFormer: Modularity Emerges from Mixture-of-Experts
In our experiment, we found that the modular architecture enables three important abilities for large pre-trained language models: 1) Efficiency, since ModuleFormer only activates a subset of its modules for each input token, thus it could achieve the same performance as dense LLMs with more than two times throughput; 2) Extendability, ModuleFormer is more immune to catastrophic forgetting than dense LLMs and can be easily extended with new modules to learn new knowledge that is not included in the training data; 3) Specialisation, finetuning ModuleFormer could specialize a subset of modules to the finetuning task and the task-unrelated modules could be easily pruned for a lightweight deployment.
Learning to Dequantise with Truncated Flows
Addtionally, since the support of the marginal $q(z)$ is bounded and the support of prior $p(z)$ is not, we propose renormalising the prior distribution over the support of $q(z)$.
Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle
In the present work, we propose a new syntax-aware language model: Syntactic Ordered Memory (SOM).
Recursive Top-Down Production for Sentence Generation with Latent Trees
We model the recursive production property of context-free grammars for natural and synthetic languages.
Ordered Memory
Inspired by Ordered Neurons (Shen et al., 2018), we introduce a new attention-based mechanism and use its cumulative probability to control the writing and erasing operation of the memory.
Icentia11K: An Unsupervised Representation Learning Dataset for Arrhythmia Subtype Discovery
We release the largest public ECG dataset of continuous raw signals for representation learning containing 11 thousand patients and 2 billion labelled beats.
{COMPANYNAME}11K: An Unsupervised Representation Learning Dataset for Arrhythmia Subtype Discovery
We release the largest public ECG dataset of continuous raw signals for representation learning containing over 11k patients and 2 billion labelled beats.
Investigating Biases in Textual Entailment Datasets
The ability to understand logical relationships between sentences is an important task in language understanding.
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed.
Improving Explorability in Variational Inference with Annealed Variational Objectives
Despite the advances in the representational capacity of approximate distributions for variational inference, the optimization process can still limit the density that is ultimately learned.
Generating Contradictory, Neutral, and Entailing Sentences
Learning distributed sentence representations remains an interesting problem in the field of Natural Language Processing (NLP).
Self-organized Hierarchical Softmax
We propose a new self-organizing hierarchical softmax formulation for neural-network-based language models over large vocabularies.
