Search Results for author:
Found 32 papers, 19 papers with code
RWKV-7 "Goose" with Expressive Dynamic State Evolution
We present RWKV-7 "Goose", a new sequence modeling architecture, along with pre-trained language models that establish a new state-of-the-art in downstream performance at the 3 billion parameter scale on multilingual tasks, and match current SoTA English language performance despite being trained on dramatically fewer tokens than other top 3B models.
A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers
We address these questions by analyzing the expressive power of transformers whose depth can grow minimally with context length $n$.
Humanity's Last Exam
However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities.
Ranked #4 on
Humanity's Last Exam
on Humanity's Last Exam
2 OLMo 2 Furious
Our modified model architecture and training recipe achieve both better training stability and improved per-token efficiency.
Evaluating $n$-Gram Novelty of Language Models Using Rusty-DAWG
In this work, we investigate the extent to which modern LMs generate $n$-grams from their training data, evaluating both (i) the probability LMs assign to complete training $n$-grams and (ii) $n$-novelty, the proportion of $n$-grams generated by an LM that did not appear in the training data (for arbitrarily large $n$).
Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
We also provide a theoretical characterization of the class of problems where filler tokens are useful in terms of the quantifier depth of a first-order formula.
The Illusion of State in State-Space Models
Our analysis reveals that the expressive power of SSMs is limited very similarly to transformers: SSMs cannot express computation outside the complexity class $\mathsf{TC}^0$.
Can You Learn Semantics Through Next-Word Prediction? The Case of Entailment
Do LMs infer the semantics of text from co-occurrence patterns in their training data?
OLMo: Accelerating the Science of Language Models
Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs.
What Formal Languages Can Transformers Express? A Survey
As transformers have gained prominence in natural language processing, some researchers have investigated theoretically what problems they can and cannot solve, by treating problems as formal languages.
The Expressive Power of Transformers with Chain of Thought
Motivated by this, we ask: Does such intermediate generation fundamentally extend the computational power of a decoder-only transformer?
How Language Model Hallucinations Can Snowball
A major risk of using language models in practical applications is their tendency to hallucinate incorrect statements.
A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks
Grokking is a phenomenon where a model trained on an algorithmic task first overfits but, then, after a large amount of additional training, undergoes a phase transition to generalize perfectly.
Transparency Helps Reveal When Language Models Learn Meaning
Many current NLP systems are built from language models trained to optimize unsupervised objectives on large amounts of raw text.
Entailment Semantics Can Be Extracted from an Ideal Language Model
Language models are often trained on text alone, without additional grounding.
The Parallelism Tradeoff: Limitations of Log-Precision Transformers
Despite their omnipresence in modern NLP, characterizing the computational power of transformer neural nets remains an interesting open question.
ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension
Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain.
Extracting Finite Automata from RNNs Using State Merging
One way to interpret the behavior of a blackbox recurrent neural network (RNN) is to extract from it a more interpretable discrete computational model, like a finite state machine, that captures its behavior.
Saturated Transformers are Constant-Depth Threshold Circuits
Transformers have become a standard neural network architecture for many NLP problems, motivating theoretical analysis of their power in terms of formal languages.
Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?
We study whether assertions enable a system to emulate representations preserving semantic relations like equivalence.
Competency Problems: On Finding and Removing Artifacts in Language Data
In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems.
Formal Language Theory Meets Modern NLP
NLP is deeply intertwined with the formal study of language, both conceptually and historically.
Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent
To better understand this bias, we study the tendency for transformer parameters to grow in magnitude ($\ell_2$ norm) during training, and its implications for the emergent representations within self attention layers.
CORD-19: The COVID-19 Open Research Dataset
The COVID-19 Open Research Dataset (CORD-19) is a growing resource of scientific papers on COVID-19 and related historical coronavirus research.
A Formal Hierarchy of RNN Architectures
While formally extending these findings to unsaturated RNNs is left to future work, we hypothesize that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy.
On the Linguistic Capacity of Real-Time Counter Automata
This work makes general contributions to the theory of formal languages that are of potential interest for understanding recurrent neural networks.
Finding Hierarchical Structure in Neural Stacks Using Unsupervised Parsing
Neural network architectures have been augmented with differentiable stacks in order to introduce a bias toward learning hierarchy-sensitive regularities.
Detecting Syntactic Change Using a Neural Part-of-Speech Tagger
Thus, our results suggest that our tagger is implicitly learning to model syntactic change in American English over the course of the 19th, 20th, and early 21st centuries.
Finding Syntactic Representations in Neural Stacks
Neural network architectures have been augmented with differentiable stacks in order to introduce a bias toward learning hierarchy-sensitive regularities.
Sequential Neural Networks as Automata
This work attempts to explain the types of computation that neural networks can perform by relating them to automata.
Context-Free Transductions with Neural Stacks
This paper analyzes the behavior of stack-augmented recurrent neural network (RNN) models.
End-to-end Graph-based TAG Parsing with Neural Networks
We present a graph-based Tree Adjoining Grammar (TAG) parser that uses BiLSTMs, highway connections, and character-level CNNs.
