Search Results for author: Qi Lei

Found 48 papers, 9 papers with code

Bridging Domains with Approximately Shared Features

1 code implementation11 Mar 2024 Ziliang Samuel Zhong, Xiang Pan, Qi Lei

Under our framework, we design and analyze a learning procedure consisting of learning approximately shared feature representation from source tasks and fine-tuning it on the target task.

Domain Adaptation feature selection

Controllable Prompt Tuning For Balancing Group Distributional Robustness

no code implementations5 Mar 2024 Hoang Phan, Andrew Gordon Wilson, Qi Lei

Models trained on data composed of different groups or domains can suffer from severe performance degradation under distribution shifts.

Data Reconstruction Attacks and Defenses: A Systematic Evaluation

no code implementations13 Feb 2024 Sheng Liu, Zihan Wang, Qi Lei

In this work, we propose a strong reconstruction attack in the setting of federated learning.

Federated Learning Reconstruction Attack

An Information-Theoretic Analysis of In-Context Learning

no code implementations28 Jan 2024 Hong Jun Jeon, Jason D. Lee, Qi Lei, Benjamin Van Roy

Previous theoretical results pertaining to meta-learning on sequences build on contrived assumptions and are somewhat convoluted.

In-Context Learning Meta-Learning

Beyond Gradient and Priors in Privacy Attacks: Leveraging Pooler Layer Inputs of Language Models in Federated Learning

no code implementations10 Dec 2023 Jianwei Li, Sheng Liu, Qi Lei

Language models trained via federated learning (FL) demonstrate impressive capabilities in handling complex tasks while protecting user privacy.

CoLA Federated Learning +3

Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models

no code implementations19 Oct 2023 Jianwei Li, Qi Lei, Wei Cheng, Dongkuan Xu

The pruning objective has recently extended beyond accuracy and sparsity to robustness in language models.

Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection

no code implementations19 Oct 2023 Jianwei Li, Weizhi Gao, Qi Lei, Dongkuan Xu

It is widely acknowledged that large and sparse models have higher accuracy than small and dense models under the same model size constraints.

Sample Complexity for Quadratic Bandits: Hessian Dependent Bounds and Optimal Algorithms

no code implementations NeurIPS 2023 Qian Yu, Yining Wang, Baihe Huang, Qi Lei, Jason D. Lee

We consider a fundamental setting in which the objective function is quadratic, and provide the first tight characterization of the optimal Hessian-dependent sample complexity.

valid

Reconstructing Training Data from Model Gradient, Provably

no code implementations7 Dec 2022 Zihan Wang, Jason D. Lee, Qi Lei

Understanding when and how much a model gradient leaks information about the training sample is an important question in privacy.

Federated Learning Tensor Decomposition

Optimization for Amortized Inverse Problems

no code implementations25 Oct 2022 Tianci Liu, Tong Yang, Quan Zhang, Qi Lei

Incorporating a deep generative model as the prior distribution in inverse problems has established substantial success in reconstructing images from corrupted observations.

Denoising

Efficient Medical Image Assessment via Self-supervised Learning

no code implementations28 Sep 2022 Chun-Yin Huang, Qi Lei, Xiaoxiao Li

Existing data assessment methods commonly require knowing the labels in advance, which are not feasible to achieve our goal of 'knowing which data to label.'

Self-Supervised Learning

Nearly Minimax Algorithms for Linear Bandits with Shared Representation

no code implementations29 Mar 2022 Jiaqi Yang, Qi Lei, Jason D. Lee, Simon S. Du

We give novel algorithms for multi-task and lifelong linear bandits with shared representation.

Sample Efficiency of Data Augmentation Consistency Regularization

no code implementations24 Feb 2022 Shuo Yang, Yijun Dong, Rachel Ward, Inderjit S. Dhillon, Sujay Sanghavi, Qi Lei

Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to use augmented data.

Data Augmentation Generalization Bounds

Bi-CLKT: Bi-Graph Contrastive Learning based Knowledge Tracing

no code implementations22 Jan 2022 XiangYu Song, JianXin Li, Qi Lei, Wei Zhao, Yunliang Chen, Ajmal Mian

The goal of Knowledge Tracing (KT) is to estimate how well students have mastered a concept based on their historical learning of related exercises.

Contrastive Learning Knowledge Tracing +1

PANOM: Automatic Hyper-parameter Tuning for Inverse Problems

no code implementations NeurIPS Workshop Deep_Invers 2021 Tianci Liu, Quan Zhang, Qi Lei

Automated hyper-parameter tuning for unsupervised learning, including inverse problems, remains a long-standing open problem due to the lack of validation data.

Bilevel Optimization

Provable Hierarchy-Based Meta-Reinforcement Learning

no code implementations18 Oct 2021 Kurtland Chua, Qi Lei, Jason D. Lee

To address this gap, we analyze HRL in the meta-RL setting, where a learner learns latent hierarchical structure during meta-training for use in a downstream task.

Hierarchical Reinforcement Learning Learning Theory +4

Theoretical Analysis of Consistency Regularization with Limited Augmented Data

no code implementations29 Sep 2021 Shuo Yang, Yijun Dong, Rachel Ward, Inderjit S Dhillon, Sujay Sanghavi, Qi Lei

Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to use augmented data.

Data Augmentation Generalization Bounds +1

Going Beyond Linear RL: Sample Efficient Neural Function Approximation

no code implementations NeurIPS 2021 Baihe Huang, Kaixuan Huang, Sham M. Kakade, Jason D. Lee, Qi Lei, Runzhe Wang, Jiaqi Yang

While the theory of RL has traditionally focused on linear function approximation (or eluder dimension) approaches, little is known about nonlinear RL with neural net approximations of the Q functions.

Reinforcement Learning (RL)

Optimal Gradient-based Algorithms for Non-concave Bandit Optimization

no code implementations NeurIPS 2021 Baihe Huang, Kaixuan Huang, Sham M. Kakade, Jason D. Lee, Qi Lei, Runzhe Wang, Jiaqi Yang

This work considers a large family of bandit problems where the unknown underlying reward function is non-concave, including the low-rank generalized linear bandit problems and two-layer neural network with polynomial activation bandit problem.

A Short Note on the Relationship of Information Gain and Eluder Dimension

no code implementations6 Jul 2021 Kaixuan Huang, Sham M. Kakade, Jason D. Lee, Qi Lei

Eluder dimension and information gain are two widely used methods of complexity measures in bandit and reinforcement learning.

LEMMA reinforcement-learning +1

Near-Optimal Linear Regression under Distribution Shift

no code implementations23 Jun 2021 Qi Lei, Wei Hu, Jason D. Lee

Transfer learning is essential when sufficient data comes from the source domain, with scarce labeled data from the target domain.

regression Transfer Learning

How Fine-Tuning Allows for Effective Meta-Learning

no code implementations NeurIPS 2021 Kurtland Chua, Qi Lei, Jason D. Lee

Representation learning has been widely studied in the context of meta-learning, enabling rapid learning of new tasks through shared representations.

Few-Shot Learning Representation Learning

A Theory of Label Propagation for Subpopulation Shift

no code implementations22 Feb 2021 Tianle Cai, Ruiqi Gao, Jason D. Lee, Qi Lei

In this work, we propose a provably effective framework for domain adaptation based on label propagation.

Domain Adaptation Generalization Bounds

Compressed Sensing with Invertible Generative Models and Dependent Noise

no code implementations23 Oct 2020 Jay Whang, Qi Lei, Alex Dimakis

We study image inverse problems with invertible generative priors, specifically normalizing flow models.

Denoising

Predicting What You Already Know Helps: Provable Self-Supervised Learning

no code implementations NeurIPS 2021 Jason D. Lee, Qi Lei, Nikunj Saunshi, Jiacheng Zhuo

Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data to learn useful semantic representations.

Representation Learning Self-Supervised Learning

Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting

no code implementations23 Mar 2020 Lemeng Wu, Mao Ye, Qi Lei, Jason D. Lee, Qiang Liu

Recently, Liu et al.[19] proposed a splitting steepest descent (S2D) method that jointly optimizes the neural parameters and architectures based on progressively growing network structures by splitting neurons into multiple copies in a steepest descent fashion.

Few-Shot Learning via Learning the Representation, Provably

no code implementations ICLR 2021 Simon S. Du, Wei Hu, Sham M. Kakade, Jason D. Lee, Qi Lei

First, we study the setting where this common representation is low-dimensional and provide a fast rate of $O\left(\frac{\mathcal{C}\left(\Phi\right)}{n_1T} + \frac{k}{n_2}\right)$; here, $\Phi$ is the representation function class, $\mathcal{C}\left(\Phi\right)$ is its complexity measure, and $k$ is the dimension of the representation.

Few-Shot Learning Representation Learning

Last iterate convergence in no-regret learning: constrained min-max optimization for convex-concave landscapes

no code implementations17 Feb 2020 Qi Lei, Sai Ganesh Nagarajan, Ioannis Panageas, Xiao Wang

In a recent series of papers it has been established that variants of Gradient Descent/Ascent and Mirror Descent exhibit last iterate convergence in convex-concave zero-sum games.

CAT: Customized Adversarial Training for Improved Robustness

no code implementations17 Feb 2020 Minhao Cheng, Qi Lei, Pin-Yu Chen, Inderjit Dhillon, Cho-Jui Hsieh

Adversarial training has become one of the most effective methods for improving robustness of neural networks.

Communication-Efficient Asynchronous Stochastic Frank-Wolfe over Nuclear-norm Balls

no code implementations17 Oct 2019 Jiacheng Zhuo, Qi Lei, Alexandros G. Dimakis, Constantine Caramanis

Large-scale machine learning training suffers from two prior challenges, specifically for nuclear-norm constrained problems with distributed systems: the synchronization slowdown due to the straggling workers, and high communication costs.

BIG-bench Machine Learning

SGD Learns One-Layer Networks in WGANs

no code implementations ICML 2020 Qi Lei, Jason D. Lee, Alexandros G. Dimakis, Constantinos Daskalakis

Generative adversarial networks (GANs) are a widely used framework for learning generative models.

Inverting Deep Generative models, One layer at a time

1 code implementation NeurIPS 2019 Qi Lei, Ajil Jalal, Inderjit S. Dhillon, Alexandros G. Dimakis

For generative models of arbitrary depth, we show that exact recovery is possible in polynomial time with high probability, if the layers are expanding and the weights are randomly selected.

Primal-Dual Block Frank-Wolfe

1 code implementation6 Jun 2019 Qi Lei, Jiacheng Zhuo, Constantine Caramanis, Inderjit S. Dhillon, Alexandros G. Dimakis

We propose a variant of the Frank-Wolfe algorithm for solving a class of sparse/low-rank optimization problems.

General Classification Multi-class Classification +1

Random Warping Series: A Random Features Method for Time-Series Embedding

1 code implementation14 Sep 2018 Lingfei Wu, Ian En-Hsu Yen, Jin-Feng Yi, Fangli Xu, Qi Lei, Michael Witbrock

The proposed kernel does not suffer from the issue of diagonal dominance while naturally enjoys a \emph{Random Features} (RF) approximation, which reduces the computational complexity of existing DTW-based techniques from quadratic to linear in terms of both the number and the length of time-series.

Clustering Dynamic Time Warping +2

Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization

1 code implementation ICML 2018 Jiong Zhang, Qi Lei, Inderjit S. Dhillon

Theoretically, we demonstrate that our parameterization does not lose any expressive power, and show how it controls generalization of RNN for the classification task.

Hessian-based Analysis of Large Batch Training and Robustness to Adversaries

6 code implementations NeurIPS 2018 Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, Michael W. Mahoney

Extensive experiments on multiple networks show that saddle-points are not the cause for generalization gap of large batch size training, and the results consistently show that large batch converges to points with noticeably higher Hessian spectrum.

Doubly Greedy Primal-Dual Coordinate Descent for Sparse Empirical Risk Minimization

no code implementations ICML 2017 Qi Lei, Ian En-Hsu Yen, Chao-yuan Wu, Inderjit S. Dhillon, Pradeep Ravikumar

We consider the popular problem of sparse empirical risk minimization with linear predictors and a large number of both features and observations.

Negative-Unlabeled Tensor Factorization for Location Category Inference from Highly Inaccurate Mobility Data

no code implementations21 Feb 2017 Jinfeng Yi, Qi Lei, Wesley Gifford, Ji Liu, Junchi Yan

In order to efficiently solve the proposed framework, we propose a parameter-free and scalable optimization algorithm by effectively exploring the sparse and low-rank structure of the tensor.

Gradient Coding

2 code implementations10 Dec 2016 Rashish Tandon, Qi Lei, Alexandros G. Dimakis, Nikos Karampatziakis

We propose a novel coding theoretic framework for mitigating stragglers in distributed learning.

Coordinate-wise Power Method

no code implementations NeurIPS 2016 Qi Lei, Kai Zhong, Inderjit S. Dhillon

The vanilla power method simultaneously updates all the coordinates of the iterate, which is essential for its convergence analysis.

A Greedy Approach for Budgeted Maximum Inner Product Search

no code implementations NeurIPS 2017 Hsiang-Fu Yu, Cho-Jui Hsieh, Qi Lei, Inderjit S. Dhillon

Maximum Inner Product Search (MIPS) is an important task in many machine learning applications such as the prediction phase of a low-rank matrix factorization model for a recommender system.

Recommendation Systems

Coordinate Descent Methods for Symmetric Nonnegative Matrix Factorization

no code implementations4 Sep 2015 Arnaud Vandaele, Nicolas Gillis, Qi Lei, Kai Zhong, Inderjit Dhillon

Given a symmetric nonnegative matrix $A$, symmetric nonnegative matrix factorization (symNMF) is the problem of finding a nonnegative matrix $H$, usually with much fewer columns than $A$, such that $A \approx HH^T$.

Clustering

Cannot find the paper you are looking for? You can Submit a new open access paper.