Search Results for author:
Found 14 papers, 3 papers with code
Sparks of Science: Hypothesis Generation Using Structured Paper Data
We demonstrate that framing hypothesis generation as conditional language modelling, with the model fine-tuned on Bit-Flip-Spark and the Chain-of-Reasoning (and where, at inference, we only provide the Bit), leads to improvements in the overall quality of the hypotheses.
From superposition to sparse codes: interpretable representations in neural networks
By bridging insights from theoretical neuroscience, representation learning, and interpretability research, we propose an emerging perspective on understanding neural representations in both artificial and biological systems.
Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures
Our results build on and extend recent work on category-theoretic foundations for deep learning, offering deeper insights into the algebraic structure of attention mechanisms.
Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders
We demonstrate this generalises to SAEs applied to large language models, where more expressive encoders achieve greater interpretability.
pathfinder: A Semantic Framework for Literature Review and Knowledge Discovery in Astronomy
The exponential growth of astronomical literature poses significant challenges for researchers navigating and synthesizing general insights or even domain-specific knowledge.
Disentangling Dense Embeddings with Sparse Autoencoders
Sparse autoencoders (SAEs) have shown promise in extracting interpretable features from complex neural networks.
Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models
We propose training sparse autoencoders on carefully designed positive and negative examples, where the model can only correctly predict the next token for the positive examples.
Measuring Sharpness in Grokking
Neural networks sometimes exhibit grokking, a phenomenon where perfect or near-perfect performance is achieved on a validation set well after the same performance has been obtained on the corresponding training set.
Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity
In some settings neural networks exhibit a phenomenon known as \textit{grokking}, where they achieve perfect or near-perfect accuracy on the validation set long after the same performance has been achieved on the training set.
Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Content
The performance of our approach is evaluated through classification accuracy on a dataset consisting of problematic prompts not detected by GPT-4, as well as a selection of contentious but unproblematic prompts.
Steering Language Generation: Harnessing Contrastive Expert Guidance and Negative Prompting for Coherent and Diverse Synthetic Data Generation
Large Language Models (LLMs) hold immense potential to generate synthetic data of high quality and utility, which has numerous applications from downstream model training to practical data utilisation.
Rice paddy disease classifications using CNNs
Rice is a staple food in the world's diet, and yet huge percentages of crop yields are lost each year to disease.
Eigenvalue initialisation and regularisation for Koopman autoencoders
In addition, we suggest the "eigenloss" penalty scheme that penalises the eigenvalues of the Koopman operator during training.
