Search Results for author:
Found 21 papers, 10 papers with code
Breaking Neural Network Scaling Laws with Modularity
Using recent theoretical progress in explaining neural network generalization, we investigate how the amount of training data required to generalize on a task varies with the intrinsic dimensionality of a task's input.
Unified Neural Network Scaling Laws and Scale-time Equivalence
Here, we present a novel theoretical characterization of how three factors -- model size, training time, and data volume -- interact to determine the performance of deep neural networks.
How Diffusion Models Learn to Factorize and Compose
Diffusion models are capable of generating photo-realistic images that combine elements which likely do not appear together in the training set, demonstrating the ability to compositionally generalize.
Towards Exact Computation of Inductive Bias
Our approach involves modeling the loss distribution of random hypotheses drawn from a hypothesis space to estimate the required inductive bias for a task relative to these hypotheses.
Delay Embedding Theory of Neural Sequence Models
To test whether language models are effectively constructing delay embeddings, we measure the capacities of sequence models to reconstruct unobserved dynamics.
Resampling-free Particle Filters in High-dimensions
State estimation is crucial for the performance and safety of numerous robotic applications.
Do Diffusion Models Learn Semantically Meaningful and Efficient Representations?
Corresponding to each of these phases, we identify qualitatively different generation behaviors: 1) multiple bumps are generated, 2) one bump is generated but at inaccurate $x$ and $y$ locations, 3) a bump is generated at the correct $x$ and y location.
Neuro-Inspired Fragmentation and Recall to Overcome Catastrophic Forgetting in Curiosity
Deep reinforcement learning methods exhibit impressive performance on a range of tasks but still struggle on hard exploration tasks in large environments with sparse rewards.
Grid Cell-Inspired Fragmentation and Recall for Efficient Map Building
If observations at a fracture point match observations in one of the stored local maps, that map is recalled (and thus reused) from LTM.
Improving Protein Optimization with Smoothed Fitness Landscapes
The ability to engineer novel proteins with higher fitness for a desired property would be revolutionary for biotechnology and medicine.
Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysis
To bridge this gap, we introduce a novel similarity metric that compares two systems at the level of their dynamics, called Dynamical Similarity Analysis (DSA).
Model-agnostic Measure of Generalization Difficulty
The measure of a machine learning algorithm is the difficulty of the tasks it can perform, and sufficiently difficult tasks are critical drivers of strong machine learning models.
See and Copy: Generation of complex compositional movements from modular and geometric RNN representations
A hallmark of biological intelligence and control is combinatorial generalization: animals are able to learn various things, then piece them together in new combinations to produce appropriate outputs for new tasks.
Winning the lottery with neural connectivity constraints: faster learning across cognitive tasks with spatially constrained sparse RNNs
We study LM-RNNs in a multitask learning setting relevant to cognitive systems neuroscience with a commonly used set of tasks, 20-Cog-tasks [Yang et al., 2019].
How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspective
Recent works have examined theoretical and empirical properties of wide neural networks trained in the Neural Tangent Kernel (NTK) regime.
Reverse-engineering recurrent neural network solutions to a hierarchical inference task for mice
Third, the geometry of RNN dynamics reflects an induced coupling between the two separate inference processes necessary to solve the task.
Bipartite expander Hopfield networks as self-decoding high-capacity error correcting codes
Neural network models of memory and error correction famously include the Hopfield network, which can directly store---and error-correct through its dynamics---arbitrary N-bit patterns, but only for ~N such patterns.
Kernel RNN Learning (KeRNL)
We describe Kernel RNN Learning (KeRNL), a reduced-rank, temporal eligibility trace-based approximation to backpropagation through time (BPTT) for training recurrent neural networks (RNNs) that gives competitive performance to BPTT on long time-dependence tasks.
Associative content-addressable networks with exponentially many robust stable states
The brain must robustly store a large number of memories, corresponding to the many events encountered over a lifetime.
Training recurrent networks to generate hypotheses about how the brain solves hard navigation problems
Self-localization during navigation with noisy sensors in an ambiguous world is computationally challenging, yet animals and humans excel at it.
Neurons and Cognition
