Search Results for author:
Found 64 papers, 22 papers with code
VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format
We construct MMDuetIT, a video-text training dataset designed to adapt VideoLLMs to video-text duet interaction format.
AIDBench: A benchmark for evaluating the authorship identification capability of large language models
As large language models (LLMs) rapidly advance and integrate into daily life, the privacy risks they pose are attracting increasing attention.
Understanding Multimodal Hallucination with Parameter-Free Representation Alignment
To analyze image representations while completely avoiding the influence of all other factors other than the image representation itself, we propose a parametric-free representation alignment metric (Pfram) that can measure the similarities between any two representation systems without requiring additional training parameters.
ReMamba: Equip Mamba with Effective Long-Sequence Modeling
While the Mamba architecture demonstrates superior inference efficiency and competitive performance on short-context natural language processing (NLP) tasks, empirical evidence suggests its capacity to comprehend long contexts is limited compared to transformer-based models.
Evidence-Enhanced Triplet Generation Framework for Hallucination Alleviation in Generative Question Answering
To address the hallucination in generative question answering (GQA) where the answer can not be derived from the document, we propose a novel evidence-enhanced triplet generation framework, EATQA, encouraging the model to predict all the combinations of (Question, Evidence, Answer) triplet by flipping the source pair and the target label to understand their logical relationships, i. e., predict Answer(A), Question(Q), and Evidence(E) given a QE, EA, and QA pairs, respectively.
Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules
More interestingly, with a fixed parameter budget, MoM-large enables an over 38% increase in depth for computation graphs compared to GPT-2-large, resulting in absolute gains of 1. 4 on GLUE and 1 on XSUM.
Efficient Continual Pre-training by Mitigating the Stability Gap
This process involves updating the pre-trained LLM with a corpus from a new domain, resulting in a shift in the training distribution.
Automatic Jailbreaking of the Text-to-Image Generative AI Systems
Recent AI systems have shown extremely powerful performance, even surpassing human performance, on various tasks such as information retrieval, language generation, and image generation based on large language models (LLMs).
xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
This paper introduces xRAG, an innovative context compression method tailored for retrieval-augmented generation.
©Plug-in Authorization for Human Content Copyright Protection in Text-to-Image Model
To mitigate this, we propose the \copyright Plug-in Authorization framework, introducing three operations: addition, extraction, and combination.
On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
This paper aims to clearly distinguish between Stochastic Gradient Descent with Momentum (SGDM) and Adam in terms of their convergence rates.
Differentially Private Synthetic Data via Foundation Model APIs 2: Text
Lin et al. (2024) recently introduced the Private Evolution (PE) algorithm to generate DP synthetic images with only API access to diffusion models.
Exploring Transferability for Randomized Smoothing
However, the model robustness, which is a critical aspect for safety, is often optimized for each specific task rather than at the pretraining stage.
Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapults
Although gradient descent with Polyak's momentum is widely used in modern machine and deep learning, a concrete understanding of its effects on the training trajectory remains elusive.
On the Generalization Properties of Diffusion Models
This precisely elucidates the adverse effect of "modes shift" in ground truths on the model generalization.
Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity
Recently, Arjevani et al. [1] established a lower bound of iteration complexity for the first-order optimization under an $L$-smooth condition and a bounded noise variance assumption.
FILM: How can Few-Shot Image Classification Benefit from Pre-Trained Language Models?
Few-shot learning aims to train models that can be generalized to novel classes with only a few samples.
When and Why Momentum Accelerates SGD:An Empirical Study
In the comparison of SGDM and SGD with the same effective learning rate and the same batch size, we observe a consistent pattern: when $\eta_{ef}$ is small, SGDM and SGD experience almost the same empirical training losses; when $\eta_{ef}$ surpasses a certain threshold, SGDM begins to perform better than SGD.
UADB: Unsupervised Anomaly Detection Booster
This is a challenging task given the heterogeneous model structures and assumptions adopted by existing UAD methods.
Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions
We provide a simple convergence proof for AdaGrad optimizing non-convex objectives under only affine noise variance and bounded smoothness assumptions.
Selective Pre-training for Private Fine-tuning
In this work, we show that a careful pre-training on a \emph{subset} of the public dataset that is guided by the private dataset is crucial to train small language models with differential privacy.
ResiDual: Transformer with Dual Residual Connections
In this paper, we propose ResiDual, a novel Transformer architecture with Pre-Post-LN (PPLN), which fuses the connections in Post-LN and Pre-LN together and inherits their advantages while avoids their limitations.
Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping
To reduce the compute time overhead of private learning, we show that \emph{per-layer clipping}, where the gradient of each neural network layer is clipped separately, allows clipping to be performed in conjunction with backpropagation in differentially private optimization.
Denoising Masked AutoEncoders Help Robust Classification
In this paper, we propose a new self-supervised method, which is called Denoising Masked AutoEncoders (DMAE), for learning certified robust classifiers of images.
Provable Adaptivity of Adam under Non-uniform Smoothness
We present the first convergence analysis of RR Adam without the bounded smoothness assumption.
Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization
By ensuring differential privacy in the learning algorithms, one can rigorously mitigate the risk of large models memorizing sensitive training data.
Adversarial Noises Are Linearly Separable for (Nearly) Random Neural Networks
Adversarial examples, which are usually generated for specific inputs with a specific model, are ubiquitous for neural networks.
Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent
Differentially private stochastic gradient descent (DP-SGD) is the workhorse algorithm for recent advances in private deep learning.
Robust Quantity-Aware Aggregation for Federated Learning
Federated learning (FL) enables multiple clients to collaboratively train models without sharing their local data, and becomes an important privacy-preserving machine learning framework.
Availability Attacks Create Shortcuts
We are the first to unveil an important population property of the perturbations of these attacks: they are almost \textbf{linearly separable} when assigned with the target labels of the corresponding samples, which hence can work as \emph{shortcuts} for the learning objective.
Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD
We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized.
Differentially Private Fine-tuning of Language Models
For example, on the MNLI dataset we achieve an accuracy of $87. 8\%$ using RoBERTa-Large and $83. 5\%$ using RoBERTa-Base with a privacy budget of $\epsilon = 6. 7$.
Does Momentum Change the Implicit Regularization on Separable Data?
The momentum acceleration technique is widely adopted in many optimization algorithms.
Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum
Specifically, we disentangle the effects of Adaptive Learning Rate and Momentum of the Adam dynamics on saddle-point escaping and flat minima selection.
Regularized-OFU: an efficient algorithm for general contextual bandit with optimization oracles
In contextual bandit, one major challenge is to develop theoretically solid and empirically efficient algorithms for general function classes.
Regularized OFU: an Efficient UCB Estimator forNon-linear Contextual Bandit
However, it is in general unknown how to deriveefficient and effective EE trade-off methods for non-linearcomplex tasks, suchas contextual bandit with deep neural network as the reward function.
Large Scale Private Learning via Low-rank Reparametrization
We propose a reparametrization scheme to address the challenges of applying differentially private SGD on large neural networks, which are 1) the huge memory cost of storing individual gradients, 2) the added noise suffering notorious dimensional dependence.
Two Coupled Rejection Metrics Can Tell Adversarial Examples Apart
Along with this routine, we find that confidence and a rectified confidence (R-Con) can form two coupled rejection metrics, which could provably distinguish wrongly classified inputs from correctly classified ones.
Optimizing Information-theoretical Generalization Bound via Anisotropic Noise of SGLD
We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized.
Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning
The privacy leakage of the model about the training data can be bounded in the differential privacy mechanism.
BN-invariant sharpness regularizes the training model to better generalization
However, it has been pointed out that the usual definitions of sharpness, which consider either the maxima or the integral of loss over a $\delta$ ball of parameters around minima, cannot give consistent measurement for scale invariant neural networks, e. g., networks with batch normalization layer.
On the Stability of Multi-branch Network
More importantly, we propose a new design ``STAM aggregation" that can guarantee to STAbilize the forward/backward process of Multi-branch networks irrespective of the number of branches.
How Does Data Augmentation Affect Privacy in Machine Learning?
Even further, we show that the proposed approach can achieve higher MI attack success rates on models trained with some data augmentation than the existing methods on models trained without data augmentation.
Adai: Separating the Effects of Adaptive Learning Rate and Momentum Inertia
Specifically, we disentangle the effects of Adaptive Learning Rate and Momentum of the Adam dynamics on saddle-point escaping and minima selection.
On Layer Normalization in the Transformer Architecture
This motivates us to remove the warm-up stage for the training of Pre-LN Transformers.
Gradient Perturbation is Underrated for Differentially Private Convex Optimization
By using the \emph{expected curvature}, we show that gradient perturbation can achieve a significantly improved utility guarantee that can theoretically justify the advantage of gradient perturbation over other perturbation methods.
THE EFFECT OF ADVERSARIAL TRAINING: A THEORETICAL CHARACTERIZATION
It has widely shown that adversarial training (Madry et al., 2018) is effective in defending adversarial attack empirically.
STABILITY AND CONVERGENCE THEORY FOR LEARNING RESNET: A FULL CHARACTERIZATION
We show that for standard initialization used in practice, $\tau =1/\Omega(\sqrt{L})$ is a sharp value in characterizing the stability of forward/backward process of ResNet, where $L$ is the number of residual blocks.
Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data
Our theory captures how the convergence of distributed algorithms behaves as the number of machines and the size of local data vary.
G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
Optimization on Multiple Manifolds
Optimization on manifold has been widely used in machine learning, to handle optimization problems with constraint.
Stabilize Deep ResNet with A Sharp Scaling Factor $τ$
Moreover, for ResNets with normalization layer, adding such a factor $\tau$ also stabilizes the training and obtains significant performance gain for deep ResNet.
SGD Converges to Global Minimum in Deep Learning via Star-convex Path
Stochastic gradient descent (SGD) has been found to be surprisingly effective in training a variety of deep neural networks.
On the Local Hessian in Back-propagation
We study the Hessian of the local back-matching loss (local Hessian) and connect it to the efficiency of BP.
Capacity Control of ReLU Neural Networks by Basis-path Norm
Motivated by this, we propose a new norm \emph{Basis-path Norm} based on a group of linearly independent paths to measure the capacity of neural networks more accurately.
Train Feedfoward Neural Network with Layer-wise Adaptive Rate via Approximating Back-matching Propagation
This inconsistence of gradient magnitude across different layers renders optimization of deep neural network with a single learning rate problematic.
Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimization
With strongly convex regularizers, we further establish the generalization error bounds for nonconvex loss functions under proximal SGD with high-probability guarantee, i. e., exponential concentration in probability.
$\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
Block-diagonal Hessian-free Optimization for Training Neural Networks
Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence.
Nonconvex Low-Rank Matrix Recovery with Arbitrary Outliers via Median-Truncated Gradient Descent
Recent work has demonstrated the effectiveness of gradient descent for directly recovering the factors of low-rank matrices from random linear measurements in a globally convergent manner when initialized properly.
Reshaped Wirtinger Flow for Solving Quadratic System of Equations
In contrast to the smooth loss function used in WF, we adopt a nonsmooth but lower-order loss function, and design a gradient-like algorithm (referred to as reshaped-WF).
Reshaped Wirtinger Flow and Incremental Algorithm for Solving Quadratic System of Equations
We further develop the incremental (stochastic) reshaped Wirtinger flow (IRWF) and show that IRWF converges linearly to the true signal.
Median-Truncated Nonconvex Approach for Phase Retrieval with Outliers
This paper investigates the phase retrieval problem, which aims to recover a signal from the magnitudes of its linear measurements.
Analysis of Robust PCA via Local Incoherence
We investigate the robust PCA problem of decomposing an observed matrix into the sum of a low-rank and a sparse error matrices via convex programming Principal Component Pursuit (PCP).
