Search Results for author:
Found 16 papers, 1 papers with code
Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs?
Bridging Associative Memory and Probabilistic Modeling
Based on the observation that associative memory's energy functions can be seen as probabilistic modeling's negative log likelihoods, we build a bridge between the two that enables useful flow of ideas in both directions.
Investigating Data Contamination for Pre-training Language Models
Language models pre-trained on web-scale corpora demonstrate impressive capabilities on diverse downstream tasks.
Disentangling Fact from Grid Cell Fiction in Trained Deep Path Integrators
Work on deep learning-based models of grid cells suggests that grid cells generically and robustly arise from optimizing networks to path integrate, i. e., track one's spatial position by integrating self-velocity signals.
What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes
Using a combination of theory and experiments, we show that incidental polysemanticity can arise due to multiple reasons including regularization and neural noise; this incidental polysemanticity occurs because random initialization can, by chance alone, initially assign multiple features to the same neuron, and the training dynamics then strengthen such overlap.
Testing Assumptions Underlying a Unified Theory for the Origin of Grid Cells
At both the population and single-cell levels, we find evidence suggesting that neither of the assumptions are likely true in biological neural representations.
Pretraining on the Test Set Is All You Need
Inspired by recent work demonstrating the promise of smaller Transformer-based language models pretrained on carefully curated data, we supercharge such approaches by investing heavily in curating a novel, high quality, non-synthetic data mixture based solely on evaluation benchmarks.
FACADE: A Framework for Adversarial Circuit Anomaly Detection and Evaluation
We present FACADE, a novel probabilistic and geometric framework designed for unsupervised mechanistic anomaly detection in deep neural networks.
Deceptive Alignment Monitoring
As the capabilities of large machine learning models continue to grow, and as the autonomy afforded to such models continues to expand, the spectre of a new adversary looms: the models themselves.
Invalid Logic, Equivalent Gains: The Bizarreness of Reasoning in Language Model Prompting
We find that the logically \textit{invalid} reasoning prompts do indeed achieve similar performance gains on BBH tasks as logically valid reasoning prompts.
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
Yet, while the literature on the trustworthiness of GPT models remains limited, practitioners have proposed employing capable GPT models for sensitive applications such as healthcare and finance -- where mistakes can be costly.
Are Emergent Abilities of Large Language Models a Mirage?
Recent work claims that large language models display emergent abilities, abilities not present in smaller-scale models that are present in larger-scale models.
Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle
Double descent is a surprising phenomenon in machine learning, in which as the number of model parameters grows relative to the number of data, test error drops as models grow ever larger into the highly overparameterized (data undersampled) regime.
Streaming Inference for Infinite Non-Stationary Clustering
Learning from a continuous stream of non-stationary data in an unsupervised manner is arguably one of the most common and most challenging settings facing intelligent agents.
An Algorithmic Theory of Metacognition in Minds and Machines
To the machine learning community, our proposed theory creates a novel interaction between the Actor and Critic in Actor-Critic agents and notes a novel connection between RL and Bayesian Optimization.
Reverse-engineering recurrent neural network solutions to a hierarchical inference task for mice
Third, the geometry of RNN dynamics reflects an induced coupling between the two separate inference processes necessary to solve the task.
