Search Results for author:
Found 12 papers, 4 papers with code
Looking for Complexity at Phase Boundaries in Continuous Cellular Automata
One key challenge in Artificial Life is designing systems that display an emergence of complex behaviors.
Arrows of Time for Large Language Models
We study the probabilistic modeling performed by Autoregressive Large Language Models (LLMs) through the angle of time directionality, addressing a question first raised in (Shannon, 1951).
Feature Learning in $L_{2}$-regularized DNNs: Attraction/Repulsion and Sparsity
This second reformulation allows us to prove a sparsity result for homogeneous DNNs: any local minimum of the $L_{2}$-regularized loss can be achieved with at most $N(N+1)$ neurons in each hidden layer (where $N$ is the size of the training set).
Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity
The dynamics of Deep Linear Networks (DLNs) is dramatically affected by the variance $\sigma^2$ of the parameters at initialization $\theta_0$.
Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
For a two-layer overparameterized network of width $ r^*+ h =: m $ we explicitly describe the manifold of global minima: it consists of $ T(r^*, m) $ affine subspaces of dimension at least $ h $ that are connected to one another.
Smart Proofs via Smart Contracts: Succinct and Informative Mathematical Derivations via Decentralized Markets
We propose a game-theoretic discussion of SPRIG, showing how agents with various types of information interact, leading to a proof tree with an appropriate level of detail and to the invalidation of wrong proofs, and we discuss resilience against various attacks.
Kernel Alignment Risk Estimator: Risk Prediction from Training Data
Under a natural universality assumption (that the KRR moments depend asymptotically on the first two moments of the observations) we capture the mean and variance of the KRR predictor.
Implicit Regularization of Random Feature Models
We investigate, by means of random matrix theory, the connection between Gaussian RF models and Kernel Ridge Regression (KRR).
The asymptotic spectrum of the Hessian of DNN throughout training
The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK).
Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
Moving the network into the chaotic regime prevents checkerboard patterns; we propose a graph-based parametrization which eliminates border artifacts; finally, we introduce a new layer-dependent learning rate to improve the convergence of DC-NNs.
Scaling description of generalization with number of parameters in deep learning
At this threshold, we argue that $\|f_{N}\|$ diverges.
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training.
