Search Results for author:
Found 63 papers, 22 papers with code
Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking
Recent work by Power et al. (2022) highlighted a surprising "grokking" phenomenon in learning arithmetic tasks: a neural net first "memorizes" the training set, resulting in perfect training accuracy but near-random test accuracy, and after training for sufficiently longer, it suddenly transitions to perfect test accuracy.
A Coefficient Makes SVRG Effective
Stochastic Variance Reduced Gradient (SVRG), introduced by Johnson & Zhang (2013), is a theoretically compelling optimization method.
Complex Organ Mask Guided Radiology Report Generation
The goal of automatic report generation is to generate a clinically accurate and coherent phrase from a single given X-ray image, which could alleviate the workload of traditional radiology reporting.
Optimistic Multi-Agent Policy Gradient for Cooperative Tasks
However, with function approximation optimism can amplify overestimation and thus fail on complex tasks.
Distributionally Robust Optimization and Invariant Representation Learning for Addressing Subgroup Underrepresentation: Mechanisms and Limitations
Spurious correlation caused by subgroup underrepresentation has received increasing attention as a source of bias that can be perpetuated by deep neural networks (DNNs).
Exploring Annotation-free Image Captioning with Retrieval-augmented Pseudo Sentence Generation
We further show that with a simple extension, the generated pseudo sentences can be deployed as weak supervision to boost the 1% semi-supervised image caption benchmark up to 93. 4 CIDEr score (+8. 9) which showcases the versatility and effectiveness of our approach.
The Marginal Value of Momentum for Small Learning Rate SGD
Momentum is known to accelerate the convergence of gradient descent in strongly convex settings without stochastic gradient noise.
Prototype-Driven and Multi-Expert Integrated Multi-Modal MR Brain Tumor Image Segmentation
To this end, a multi-modal MR brain tumor segmentation method with tumor prototype-driven and multi-expert integration is proposed.
The Inductive Bias of Flatness Regularization for Deep Matrix Factorization
Recent works on over-parameterized neural networks have shown that the stochasticity in optimizers has the implicit regularization effect of minimizing the sharpness of the loss function (in particular, the trace of its Hessian) over the family zero-loss solutions.
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
Given the massive cost of language model pre-training, a non-trivial improvement of the optimization algorithm would lead to a material reduction on the time and cost of training.
Sequential Latent Variable Models for Few-Shot High-Dimensional Time-Series Forecasting
We compared the presented framework with a comprehensive set of baseline models trained 1) globally on the large meta-training set with diverse dynamics, and 2) individually on single dynamics, both with and without fine-tuning to k-shot support series used by the meta-models.
Learning Empirical Bregman Divergence for Uncertain Distance Representation
Deep metric learning techniques have been used for visual representation in various supervised and unsupervised learning tasks through learning embeddings of samples with deep networks.
A Novel Collaborative Self-Supervised Learning Method for Radiomic Data
We compared our proposed method with other state-of-the-art self-supervised learning methods on a simulation study and two independent datasets.
Learning Generalized Hybrid Proximity Representation for Image Recognition
Recently, deep metric learning techniques received attention, as the learned distance representations are useful to capture the similarity relationship among samples and further improve the performance of various of supervised or unsupervised learning tasks.
Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing
It is believed that Gradient Descent (GD) induces an implicit bias towards good generalization in training machine learning models.
How Does Sharpness-Aware Minimization Minimize Sharpness?
SAM intends to penalize a notion of sharpness of the model but implements a computationally efficient variant; moreover, a third notion of sharpness was used for proving generalization guarantees.
Interpretable Modeling and Reduction of Unknown Errors in Mechanistic Operators
Prior knowledge about the imaging physics provides a mechanistic forward operator that plays an important role in image reconstruction, although myriad sources of possible errors in the operator could negatively impact the reconstruction solutions.
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models
Toward understanding this implicit bias, we prove that SGD with standard mini-batch noise implicitly prefers flatter minima in language models, and empirically observe a strong correlation between flatness and downstream performance among models with the same minimal pre-training loss.
Few-shot Generation of Personalized Neural Surrogates for Cardiac Simulation via Bayesian Meta-Learning
As test time, metaPNS delivers a personalized neural surrogate by fast feed-forward embedding of a small and flexible number of data available from an individual, achieving -- for the first time -- personalization and surrogate construction for expensive simulations in one end-to-end learning framework.
Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent
Conversely, continuous mirror descent with any Legendre function can be viewed as gradient flow with a related commuting parametrization.
Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction
Normalization layers (e. g., Batch Normalization, Layer Normalization) were introduced to help with optimization difficulties in very deep nets, but they clearly also help generalization, even in not-so-deep nets.
Understanding Gradient Descent on Edge of Stability in Deep Learning
The current paper mathematically analyzes a new mechanism of implicit regularization in the EoS phase, whereby GD updates due to non-smooth loss landscape turn out to evolve along some deterministic flow on the manifold of minimum loss.
Interpretability of Neural Network With Physiological Mechanisms
Deep learning continues to play as a powerful state-of-art technique that has achieved extraordinary accuracy levels in various domains of regression and classification tasks, including images, video, signal, and natural language data.
A Novel Ontology-guided Attribute Partitioning Ensemble Learning Model for Early Prediction of Cognitive Deficits using Quantitative Structural MRI in Very Preterm Infants
We applied the OAP-EL to predict cognitive deficits at 2 years of age using quantitative brain maturation and geometric features obtained at term equivalent age in very preterm infants.
Robust Training of Neural Networks Using Scale Invariant Architectures
In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models.
Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias
The current paper is able to establish this global optimality for two-layer Leaky ReLU nets trained with gradient flow on linearly separable and symmetric data, regardless of the width.
What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
Understanding the implicit bias of Stochastic Gradient Descent (SGD) is one of the key challenges in deep learning, especially for overparametrized models, where the local minimizers of the loss function $L$ can form a manifold.
DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection
Few-shot object detection, which aims at detecting novel objects rapidly from extremely few annotated examples of previously unseen classes, has attracted significant research interest in the community.
Ranked #4 on
Few-Shot Object Detection
on MS-COCO (1-shot)
Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning
Concretely, we view the Bellman error as a surrogate loss for the optimality gap, and prove the followings: (1) In double sampling regime, the excess risk of Empirical Risk Minimizer (ERM) is bounded by the Rademacher complexity of the function class.
A Two-Stage Variable Selection Approach for Correlated High Dimensional Predictors
To solve the challenge, we propose a two-stage approach that combines a variable clustering stage and a group variable stage for the group variable selection problem.
On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
It is generally recognized that finite learning rate (LR), in contrast to infinitesimal LR, is important for good generalization in real-life deep nets.
A Supernova-driven, Magnetically-collimated Outflow as the Origin of the Galactic Center Radio Bubbles
Our simulations are run with different combinations of two main parameters, the supernova birth rate and the strength of a global magnetic field being vertically oriented with respect to the disk.
Astrophysics of Galaxies
Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
Matrix factorization is a simple and natural test-bed to investigate the implicit regularization of gradient descent.
Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?
However, this has not been made mathematically rigorous, and the hurdle is that the fully connected net can always simulate the convolutional net (for a fixed task).
Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
Recent works (e. g., (Li and Arora, 2020)) suggest that the use of popular normalization schemes (including Batch Normalization) in today's deep learning can move it far from a traditional optimization viewpoint, e. g., use of exponentially increasing learning rates.
Learning Geometry-Dependent and Physics-Based Inverse Image Reconstruction
However, many reconstruction problems involve imaging physics that are dependent on the underlying non-Euclidean geometry.
Chemical abundances in Sgr A East: evidence for a type Iax supernova remnant
We report evidence that SNR Sgr A East in the Galactic center resulted from a pure turbulent deflagration of a Chandrasekhar-mass carbon-oxygen WD, an explosion mechanism used for type Iax SNe.
High Energy Astrophysical Phenomena
When is Particle Filtering Efficient for Planning in Partially Observed Linear Dynamical Systems?
Though errors in past actions may affect the future, we are able to bound the number of particles needed so that the long-run reward of the policy based on particle filtering is close to that based on exact inference.
Semi-supervised Medical Image Classification with Global Latent Mixing
In this work, we argue that regularizing the global smoothness of neural functions by filling the void in between data points can further improve SSL.
Progressive Learning and Disentanglement of Hierarchical Representations
By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the first attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representations.
Implicit Regularization and Convergence for Weight Normalization
For certain stepsizes of g and w , we show that they can converge close to the minimum norm solution.
Enhanced Convolutional Neural Tangent Kernels
An exact algorithm to compute CNTK (Arora et al., 2019) yielded the finding that classification accuracy of CNTK on CIFAR-10 is within 6-7% of that of that of the corresponding CNN architecture (best figure being around 78%) which is interesting performance for a fixed kernel.
An Exponential Learning Rate Schedule for Deep Learning
This paper suggests that the phenomenon may be due to Batch Normalization or BN, which is ubiquitous and provides benefits in optimization and generalization across all standard architectures.
Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks
On VOC07 testbed for few-shot image classification tasks on ImageNet with transfer learning (Goyal et al., 2019), replacing the linear SVM currently used with a Convolutional NTK SVM consistently improves performance.
Improving Disentangled Representation Learning with the Beta Bernoulli Process
We note that the independence within and the complexity of the latent density are two different properties we constrain when regularizing the posterior density: while the former promotes the disentangling ability of VAE, the latter -- if overly limited -- creates an unnecessary competition with the data reconstruction objective in VAE.
Semi-Supervised Learning by Disentangling and Self-Ensembling Over Stochastic Latent Space
In this work, we hypothesize -- from the generalization perspective -- that self-ensembling can be improved by exploiting the stochasticity of a disentangled latent space.
Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
Mode connectivity is a surprising phenomenon in the loss landscape of deep nets.
Feature-level and Model-level Audiovisual Fusion for Emotion Recognition in the Wild
In this paper, we proposed two strategies to fuse information extracted from different modalities, i. e., audio and visual.
Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data.
The role of over-parametrization in generalization of neural networks
Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization.
On Exact Computation with an Infinitely Wide Neural Net
An attraction of such ideas is that a pure kernel-based method is used to capture the power of a fully-trained deep net of infinite width.
Identity-Free Facial Expression Recognition using conditional Generative Adversarial Network
A novel Identity-Free conditional Generative Adversarial Network (IF-GAN) was proposed for Facial Expression Recognition (FER) to explicitly reduce high inter-subject variations caused by identity-related facial attributes, e. g., age, race, and gender.
Facial Expression Recognition
Facial Expression Recognition (FER)
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
This paper analyzes training and generalization for a simple 2-layer ReLU net with random initialization, and provides the following improvements over recent works: (i) Using a tighter characterization of training speed than recent papers, an explanation for why training a neural net with random labels leads to slower training, as originally observed in [Zhang et al. ICLR'17].
Probabilistic Attribute Tree in Convolutional Neural Networks for Facial Expression Recognition
In this paper, we proposed a novel Probabilistic Attribute Tree-CNN (PAT-CNN) to explicitly deal with the large intra-class variations caused by identity-related attributes, e. g., age, race, and gender.
Facial Expression Recognition
Facial Expression Recognition (FER)
Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization.
Deep Template Matching for Offline Handwritten Chinese Character Recognition
Just like its remarkable achievements in many computer vision tasks, the convolutional neural networks (CNN) provide an end-to-end solution in handwritten Chinese character recognition (HCCR) with great success.
Binary Classification
Offline Handwritten Chinese Character Recognition
+1
Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization.
Online Improper Learning with an Approximation Oracle
We revisit the question of reducing online learning to approximate optimization of the offline problem.
Building Efficient CNN Architecture for Offline Handwritten Chinese Character Recognition
Deep convolutional networks based methods have brought great breakthrough in images classification, which provides an end-to-end solution for handwritten Chinese character recognition(HCCR) problem through learning discriminative features automatically.
Island Loss for Learning Discriminative Features in Facial Expression Recognition
Over the past few years, Convolutional Neural Networks (CNNs) have shown promise on facial expression recognition.
Ranked #3 on
Facial Expression Recognition (FER)
on SFEW
Facial Expression Recognition
Facial Expression Recognition (FER)
Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition
Most recently, Convolutional Neural Networks (CNNs) have shown promise for facial AU recognition, where predefined and fixed convolution filter sizes are employed.
Solving Marginal MAP Problems with NP Oracles and Parity Constraints
Arising from many applications at the intersection of decision making and machine learning, Marginal Maximum A Posteriori (Marginal MAP) Problems unify the two main classes of inference, namely maximization (optimization) and marginal inference (counting), and are believed to have higher complexity than both of them.
Learning in Games: Robustness of Fast Convergence
We show that learning algorithms satisfying a $\textit{low approximate regret}$ property experience fast convergence to approximate optimality in a large class of repeated games.
