Search Results for author:
Found 13 papers, 12 papers with code
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models.
Ranked #2 on
Bug fixing
on SWE-bench
How Language Model Hallucinations Can Snowball
A major risk of using language models in practical applications is their tendency to hallucinate incorrect statements.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
What Language Model to Train if You Have One Million GPU Hours?
All our models and code are open-sourced at https://huggingface. co/bigscience .
Measuring and Narrowing the Compositionality Gap in Language Models
We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems.
Ranked #4 on
Question Answering
on Bamboogle
Transformer Language Models without Positional Encodings Still Learn Positional Information
Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings.
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training?
Shortformer: Better Language Modeling using Shorter Inputs
Increasing the input length has been a driver of progress in language modeling with transformers.
Ranked #26 on
Language Modelling
on WikiText-103
Improving Transformer Models by Reordering their Sublayers
Multilayer transformer networks consist of interleaved self-attention and feedforward sublayers.
Ranked #7 on
Language Modelling
on enwik8
Partially Shuffling the Training Data to Improve Language Models
Although SGD requires shuffling the training data between epochs, currently none of the word-level language modeling systems do this.
Ranked #15 on
Language Modelling
on Penn Treebank (Word Level)
You May Not Need Attention
In NMT, how far can we get without attention and without separate encoding and decoding?
Language Generation with Recurrent Generative Adversarial Networks without Pre-training
Generative Adversarial Networks (GANs) have shown great promise recently in image generation.
Using the Output Embedding to Improve Language Models
We study the topmost weight matrix of neural network language models.
