Search Results for author:
Found 59 papers, 20 papers with code
A Database of Multimodal Data to Construct a Simulated Dialogue Partner with Varying Degrees of Cognitive Health
An assistive robot Pepper has been designed to administer Referential Communication Tasks (RCTs) to human subjects without dementia as a step towards an agent to administer RCTs to dementia patients, potentially for earlier diagnosis.
FOCUS: First Order Concentrated Updating Scheme
Large language models (LLMs) demonstrate remarkable performance, and improving their pre-training process appears to be key to enhancing their capabilities further.
Physics of Skill Learning
We aim to understand physics of skill learning, i. e., how skills are learned in neural networks during training.
Fokker-Planck to Callan-Symanzik: evolution of weight matrices under training
The dynamical evolution of a neural network during training has been an incredibly fascinating subject of study.
Financial Risk Assessment via Long-term Payment Behavior Sequence Folding
We evaluate LBSF on the financial risk assessment task using a large-scale real-world dataset.
On the expressiveness and spectral bias of KANs
This shows that the approximation and representation capabilities of KANs are at least as good as MLPs.
How Diffusion Models Learn to Factorize and Compose
Diffusion models are capable of generating photo-realistic images that combine elements which likely do not appear together in the training set, demonstrating the ability to \textit{compositionally generalize}.
Epsilon: Exploring Comprehensive Visual-Semantic Projection for Multi-Label Zero-Shot Learning
This paper investigates a challenging problem of zero-shot learning in the multi-label scenario (MLZSL), wherein the model is trained to recognize multiple unseen classes within a sample (e. g., an image) based on seen classes and auxiliary knowledge, e. g., semantic information.
KAN 2.0: Kolmogorov-Arnold Networks Meet Science
The synergy is bidirectional: science to KAN (incorporating scientific knowledge into KANs), and KAN to science (extracting scientific insights from KANs).
Survival of the Fittest Representation: A Case Study with Modular Addition
To investigate this Survival of the Fittest hypothesis, we conduct a case study on neural networks performing modular addition, and find that these networks' multiple circular representations at different Fourier frequencies undergo such competitive dynamics, with only a few circles surviving at the end.
Using a Convolutional Neural Network and Explainable AI to Diagnose Dementia Based on MRI Scans
Altogether, this combination of the convolution neural network and explainable AI algorithm creates a system that can be used in the medical field to not only aid in the proper classification of dementia but also allow everyone involved to visualize and understand the results.
How Do Transformers "Do" Physics? Investigating the Simple Harmonic Oscillator
We develop four criteria for the use of a method within the simple testbed of linear regression, where our method is $y = wx$ and our intermediate is $w$: (1) Can the intermediate be predicted from hidden states?
OptPDE: Discovering Novel Integrable Systems via AI-Human Collaboration
Integrable partial differential equation (PDE) systems are of great interest in natural science, but are exceedingly rare and difficult to discover.
KAN: Kolmogorov-Arnold Networks
Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs).
DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
Scaling multi-dimensional transformers to long sequences is indispensable across various domains.
GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theory
We present GenEFT: an effective theory framework for shedding light on the statics and dynamics of neural network generalization, and illustrate it with graph learning examples.
A Resource Model For Neural Scaling Law
A task is usually composite hence can be decomposed into many subtasks, which compete for resources (measured by the number of neurons allocated to subtasks).
Opening the AI black box: program synthesis via mechanistic interpretability
We present MIPS, a novel method for program synthesis based on automated mechanistic interpretability of neural networks trained to perform the desired task, auto-distilling the learned algorithm into Python code.
Do Diffusion Models Learn Semantically Meaningful and Efficient Representations?
Corresponding to each of these phases, we identify qualitatively different generation behaviors: 1) multiple bumps are generated, 2) one bump is generated but at inaccurate $x$ and $y$ locations, 3) a bump is generated at the correct $x$ and y location.
AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference
The experiments demonstrate that AutoChunk can reduce over 80\% of activation memory while maintaining speed loss within 10%, extend max sequence length by 3. 2x to 11. 7x, and outperform state-of-the-art methods by a large margin.
ParsNets: A Parsimonious Orthogonal and Low-Rank Linear Networks for Zero-Shot Learning
Then, to facilitate the generalization of local linearities, we construct a maximal margin geometry on the learned features by enforcing low-rank constraints on intra-class samples and high-rank constraints on inter-class samples, resulting in orthogonal subspaces for different classes and each subspace lies on a compact manifold.
Generating Interpretable Networks using Hypernetworks
The hypernetwork is carefully designed such that it can control network complexity, leading to a diverse family of interpretable algorithms ranked by their complexity.
Growing Brains: Co-emergence of Anatomical and Functional Modularity in Recurrent Neural Networks
Recurrent neural networks (RNNs) trained on compositional tasks can exhibit functional modularity, in which neurons can be clustered by activity similarity and participation in shared computational subtasks.
Grokking as Compression: A Nonlinear Complexity Perspective
To do so, we define linear mapping number (LMN) to measure network complexity, which is a generalized version of linear region number for ReLU networks.
A Neural Scaling Law from Lottery Ticket Ensembling
Neural scaling laws (NSL) refer to the phenomenon where model performance improves with scale.
GBE-MLZSL: A Group Bi-Enhancement Framework for Multi-Label Zero-Shot Learning
That is, in the process of inferring unseen classes, global features represent the principal direction of the image in the feature space, while local features should maintain uniqueness within a certain range.
The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks
Do neural networks, trained on well-understood algorithmic tasks, reliably rediscover known algorithms for solving those tasks?
Restart Sampling for Improving Generative Processes
Restart not only outperforms the previous best SDE results, but also accelerates the sampling speed by 10-fold / 2-fold on CIFAR-10 / ImageNet $64 \times 64$.
Discovering New Interpretable Conservation Laws as Sparse Invariants
Discovering conservation laws for a given dynamical system is important but challenging.
Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability
We introduce Brain-Inspired Modular Training (BIMT), a method for making neural networks more modular and interpretable.
DRPT: Disentangled and Recurrent Prompt Tuning for Compositional Zero-Shot Learning
Compositional Zero-shot Learning (CZSL) aims to recognize novel concepts composed of known knowledge without training samples.
GenPhys: From Physical Processes to Generative Models
We introduce a general family, Generative Models from Physical Processes (GenPhys), where we translate partial differential equations (PDEs) describing physical processes to generative models.
The Quantization Model of Neural Scaling
We tentatively find that the frequency at which these quanta are used in the training distribution roughly follows a power law corresponding with the empirical scaling exponent for language models, a prediction of our theory.
PFGM++: Unlocking the Potential of Physics-Inspired Generative Models
The new models reduce to PFGM when $D{=}1$ and to diffusion models when $D{\to}\infty$.
Ranked #1 on
Image Generation
on FFHQ 64x64 - 4x upscaling
(ML)$^2$P-Encoder: On Exploration of Channel-Class Correlation for Multi-Label Zero-Shot Learning
Recent studies usually approach multi-label zero-shot learning (MLZSL) with visual-semantic mapping on spatial-class correlation, which can be computationally costly, and worse still, fails to capture fine-grained class-specific semantics.
Decomposed Soft Prompt Guided Fusion Enhancing for Compositional Zero-Shot Learning
Existing methods either learn the combined state-object representation, challenging the generalization of unseen compositions, or design two classifiers to identify state and object separately from image features, ignoring the intrinsic relationship between them.
ProCC: Progressive Cross-primitive Compatibility for Open-World Compositional Zero-Shot Learning
Open-World Compositional Zero-shot Learning (OW-CZSL) aims to recognize novel compositions of state and object primitives in images with no priors on the compositional space, which induces a tremendously large output space containing all possible state-object compositions.
Precision Machine Learning
We explore unique considerations involved in fitting ML models to data with very high precision, as is often required for science applications.
Omnigrok: Grokking Beyond Algorithmic Data
Grokking, the unusual phenomenon for algorithmic datasets where generalization happens long after overfitting the training data, has remained elusive.
Poisson Flow Generative Models
We interpret the data points as electrical charges on the $z=0$ hyperplane in a space augmented with an additional dimension $z$, generating a high-dimensional electric field (the gradient of the solution to Poisson equation).
EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models
Although the AI community has expanded the model scale to the trillion parameter level, the practical deployment of 10-100 billion parameter models is still uncertain due to the latency, throughput, and memory constraints.
Fed-FSNet: Mitigating Non-I.I.D. Federated Learning via Fuzzy Synthesizing Network
Concretely, we maintain an edge-agnostic hidden model in the cloud server to estimate a less-accurate while direction-aware inversion of the global model.
Second Order Ensemble Langevin Method for Sampling and Inverse Problems
We propose a sampling method based on an ensemble approximation of second order Langevin dynamics.
Anchor Sampling for Federated Learning with Partial Client Participation
The lack of inactive clients' updates in partial client participation makes it more likely for the model aggregation to deviate from the aggregation based on full client participation.
Towards Understanding Grokking: An Effective Theory of Representation Learning
We aim to understand grokking, a phenomenon where models generalize long after overfitting their training set.
AI Poincaré 2.0: Machine Learning Conservation Laws from Differential Equations
We present a machine learning algorithm that discovers conservation laws from differential equations, both numerically (parametrized as neural networks) and symbolically, ensuring their functional independence (a non-linear generalization of linear independence).
Towards Unbiased Multi-label Zero-Shot Learning with Pyramid and Semantic Attention
We argue that disregarding the connection between major and minor classes, i. e., correspond to the global and local information, respectively, is the cause of the problem.
From Deterioration to Acceleration: A Calibration Approach to Rehabilitating Step Asynchronism in Federated Optimization
In the setting of federated optimization, where a global model is aggregated periodically, step asynchronism occurs when participants conduct model training by efficiently utilizing their computational resources.
Physics-Augmented Learning: A New Paradigm Beyond Physics-Informed Learning
Integrating physical inductive biases into machine learning can improve model generalizability.
Machine-learning hidden symmetries
We present an automated method for finding hidden symmetries, defined as symmetries that become manifest only in a new coordinate system that must be discovered.
Machine-Learning Non-Conservative Dynamics for New-Physics Detection
Energy conservation is a basic physics principle, the breakdown of which often implies new physics.
AI Poincaré: Machine Learning Conservation Laws from Trajectories
We present AI Poincar\'e, a machine learning algorithm for auto-discovering conserved quantities using trajectory data from unknown dynamical systems.
HRDNet: High-resolution Detection Network for Small Objects
By extracting various features from high to low resolutions, the MD-IPN is able to improve the performance of small object detection as well as maintaining the performance of middle and large objects.
DTG-Net: Differentiated Teachers Guided Self-Supervised Video Action Recognition
Finally, the DTG-Net is evaluated in two ways: (i) the self-supervised DTG-Net to pre-train the supervised action recognition models with only unlabeled videos; (ii) the supervised DTG-Net to be jointly trained with the supervised action networks in an end-to-end way.
Schrödinger PCA: On the Duality between Principal Component Analysis and Schrödinger Equation
Counterintuitively, by drawing the connection between PCA and Schr\"odinger equation, we can not only attack the undersampling challenge but also compute in an efficient and decoupled way with the proposed algorithm called Schr\"odinger PCA.
Influenza Modeling Based on Massive Feature Engineering and International Flow Deconvolution
Finally, both the original model and the perturbed model are tested on regional examples, as validations of our models.
Quantum-Inspired Hamiltonian Monte Carlo for Bayesian Sampling
Hamiltonian Monte Carlo (HMC) is an efficient Bayesian sampling method that can make distant proposals in the parameter space by simulating a Hamiltonian dynamical system.
IPG-Net: Image Pyramid Guidance Network for Small Object Detection
In this paper, except for top-down combining of information for shallow layers, we propose a novel network called Image Pyramid Guidance Network (IPG-Net) to make sure both the spatial information and semantic information are abundant for each layer.
Multi-PCA based Fault Detection Model Combined with Prior knowledge of HVAC
The traditional PCA fault detection methods completely depend on the training data.
