no code implementations • 8 Mar 2024 • Huiying Zhong, Zhun Deng, Weijie J. Su, Zhiwei Steven Wu, Linjun Zhang
Our work \textit{initiates} the theoretical study of multi-party RLHF that explicitly models the diverse preferences of multiple individuals.
no code implementations • 6 Jul 2023 • Xinming Tu, James Zou, Weijie J. Su, Linjun Zhang
LLMs can also play a significant role in the classroom as interactive teaching and learning tools, contributing to personalized education.
no code implementations • 9 Jun 2023 • Hua Wang, Sheng Gao, Huanyu Zhang, Weijie J. Su, Milan Shen
In our paper, we introduce DP-HyPO, a pioneering framework for ``adaptive'' private hyperparameter optimization, aiming to bridge the gap between private and non-private hyperparameter optimization.
1 code implementation • 28 May 2023 • Ziang Song, Tianle Cai, Jason D. Lee, Weijie J. Su
This insight allows us to derive closed-form expressions for the reward distribution associated with a set of utility functions in an asymptotic regime.
no code implementations • 27 May 2023 • Lei Wu, Weijie J. Su
By contrast, for gradient descent (GD), the stability imposes a similar constraint but only on the largest eigenvalue of Hessian.
no code implementations • 21 Apr 2023 • Yuling Yan, Weijie J. Su, Jianqing Fan
We demonstrate that an author is incentivized to provide accurate rankings when her utility takes the form of a convex additive function of the adjusted review scores.
2 code implementations • 31 Oct 2022 • Hangfeng He, Weijie J. Su
While deep learning has enabled significant advances in many areas of science, its black-box nature hinders architecture design for future artificial intelligence applications and interpretation for high-stakes decision makings.
1 code implementation • 29 Sep 2022 • Yizhou Liu, Weijie J. Su, Tongyang Li
Classical algorithms are often not effective for solving nonconvex optimization problems where local minima are separated by high barriers.
no code implementations • 14 Jun 2022 • Weijie J. Su
Given the noisy grades provided by an independent party, can Bob (appraiser) obtain accurate estimates of the ground-truth grades of the items by asking Alice a question about the grades?
1 code implementation • 9 Jun 2022 • Hua Wang, Sheng Gao, Huanyu Zhang, Milan Shen, Weijie J. Su
Many modern machine learning algorithms are composed of simple private algorithms; thus, an increasingly important problem is to efficiently compute the overall privacy loss under composition.
no code implementations • 6 Jun 2022 • Zhun Deng, Jiayao Zhang, Linjun Zhang, Ting Ye, Yates Coley, Weijie J. Su, James Zou
Specifically, FIFA encourages both classification and fairness generalization and can be flexibly combined with many existing fair learning methods with logits-based losses.
1 code implementation • 31 Jan 2022 • Jiayao Zhang, Hongming Zhang, Weijie J. Su, Dan Roth
Commonsense causality reasoning (CCR) aims at identifying plausible causes and effects in natural language descriptions that are deemed reasonable by an average person.
no code implementations • 17 Dec 2021 • Weijie J. Su
To advance deep learning methodologies in the next decade, a theoretical framework for reasoning about modern neural networks is needed.
no code implementations • 27 Oct 2021 • Weijie J. Su
To address this withholding of information, in this paper, I introduce the Isotonic Mechanism, a simple and efficient approach to improving imprecise raw scores by leveraging certain information that the owner is incentivized to provide.
1 code implementation • NeurIPS 2021 • Jiayao Zhang, Hua Wang, Weijie J. Su
Our main finding uncovers a sharp phase transition phenomenon regarding the {intra-class impact: if the SDEs are locally elastic in the sense that the impact is more significant on samples from the same class as the input, the features of the training data become linearly separable, meaning vanishing training loss; otherwise, the features are not separable, regardless of how long the training time is.
no code implementations • ICLR 2022 • Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, Weijie J. Su
We prove that gradient flow on this model converges to critical points of a minimum-norm separation problem exhibiting neural collapse in its global minimizer.
1 code implementation • ICLR 2022 • Shuxiao Chen, Koby Crammer, Hangfeng He, Dan Roth, Weijie J. Su
In this paper, we introduce Target-Aware Weighted Training (TAWT), a weighted training algorithm for cross-task learning based on minimizing a representation-based task distance between the source and target tasks.
1 code implementation • 27 May 2021 • Zhiqi Bu, Jason Klusowski, Cynthia Rush, Weijie J. Su
Sorted l1 regularization has been incorporated into many methods for solving high-dimensional statistical estimation problems, including the SLOPE estimator in linear regression.
no code implementations • 18 May 2021 • Gang Qiao, Weijie J. Su, Li Zhang
Being able to efficiently and accurately select the top-$k$ elements with differential privacy is an integral component of various private data analysis tasks.
no code implementations • 5 Apr 2021 • Jinshuo Dong, Aaron Roth, Weijie J. Su
In this rejoinder, we aim to address two broad issues that cover most comments made in the discussion.
no code implementations • NeurIPS 2021 • Jinshuo Dong, Weijie J. Su, Linjun Zhang
The central question, therefore, is to understand which noise distribution optimizes the privacy-accuracy trade-off, especially when the dimension of the answer vector is high.
no code implementations • 2 Mar 2021 • Shuxiao Chen, Qinqing Zheng, Qi Long, Weijie J. Su
A widely recognized difficulty in federated learning arises from the statistical heterogeneity among clients: local datasets often come from different but not entirely unrelated distributions, and personalization is, therefore, necessary to achieve optimal results from each individual's perspective.
1 code implementation • 22 Feb 2021 • Qinqing Zheng, Shuxiao Chen, Qi Long, Weijie J. Su
Federated learning (FL) is a training paradigm where the clients collaboratively learn models by repeatedly sharing information without compromising much on the privacy of their local sensitive data.
1 code implementation • 29 Jan 2021 • Cong Fang, Hangfeng He, Qi Long, Weijie J. Su
More importantly, when moving to the imbalanced case, our analysis of the Layer-Peeled Model reveals a hitherto unknown phenomenon that we term \textit{Minority Collapse}, which fundamentally limits the performance of deep learning models on the minority classes.
no code implementations • 27 Oct 2020 • Zhun Deng, Hangfeng He, Weijie J. Su
Given that, we propose \emph{locally elastic stability} as a weaker and distribution-dependent stability notion, which still yields exponential generalization bounds.
no code implementations • 22 Oct 2020 • Fan Yang, Hongyang R. Zhang, Sen Wu, Christopher Ré, Weijie J. Su
Intuitively, the transfer effect from one task to another task depends on dataset shifts such as sample sizes and covariance matrices.
1 code implementation • NeurIPS 2020 • Shuxiao Chen, Hangfeng He, Weijie J. Su
As a popular approach to modeling the dynamics of training overparametrized neural networks (NNs), the neural tangent kernels (NTK) are known to fall behind real-world NNs in generalization ability.
no code implementations • ICML 2020 • Zhun Deng, Hangfeng He, Jiaoyang Huang, Weijie J. Su
An acknowledged weakness of neural networks is their vulnerability to adversarial perturbations to the inputs.
2 code implementations • NeurIPS 2020 • Hua Wang, Yachong Yang, Zhiqi Bu, Weijie J. Su
A fundamental problem in the high-dimensional regression is to understand the tradeoff between type I and type II errors or, equivalently, false discovery rate (FDR) and power in variable selection.
Statistics Theory Information Theory Information Theory Statistics Theory
no code implementations • 15 Apr 2020 • Bin Shi, Weijie J. Su, Michael. I. Jordan
In this paper, we present a general theoretical analysis of the effect of the learning rate in stochastic gradient descent (SGD).
1 code implementation • ICML 2020 • Qinqing Zheng, Jinshuo Dong, Qi Long, Weijie J. Su
To address this question, we introduce a family of analytical and sharp privacy bounds under composition using the Edgeworth expansion in the framework of the recently proposed f-differential privacy.
3 code implementations • 26 Nov 2019 • Zhiqi Bu, Jinshuo Dong, Qi Long, Weijie J. Su
Leveraging the appealing properties of $f$-differential privacy in handling composition and subsampling, this paper derives analytically tractable expressions for the privacy guarantees of both stochastic gradient descent and Adam used in training deep neural networks, without the need of developing sophisticated techniques as [3] did.
1 code implementation • ICLR 2020 • Hangfeng He, Weijie J. Su
This phenomenon is shown to persist for neural networks with nonlinear activation functions through extensive simulations on real-life and synthetic datasets, whereas this is not observed in linear classifiers.
3 code implementations • 7 May 2019 • Jinshuo Dong, Aaron Roth, Weijie J. Su
More precisely, the privacy guarantees of \emph{any} hypothesis testing based definition of privacy (including original DP) converges to GDP in the limit under composition.
no code implementations • NeurIPS 2019 • Bin Shi, Simon S. Du, Weijie J. Su, Michael. I. Jordan
We study first-order optimization methods obtained by discretizing ordinary differential equations (ODEs) corresponding to Nesterov's accelerated gradient methods (NAGs) and Polyak's heavy-ball method.
no code implementations • 21 Oct 2018 • Bin Shi, Simon S. Du, Michael. I. Jordan, Weijie J. Su
We also show that these ODEs are more accurate surrogates for the underlying algorithms; in particular, they not only distinguish between NAG-SC and Polyak's heavy-ball method, but they allow the identification of a term that we refer to as "gradient correction" that is present in NAG-SC but not in the heavy-ball method and is responsible for the qualitative difference in convergence of the two methods.
no code implementations • 11 Jul 2018 • Cynthia Dwork, Weijie J. Su, Li Zhang
Differential privacy provides a rigorous framework for privacy-preserving data analysis.
1 code implementation • 1 Jul 2018 • Edgar Dobriban, Weijie J. Su
In this paper, we propose methods that are robust to large and unequal noise in different observational units (i. e., heteroskedasticity) for statistical inference in linear regression.
Statistics Theory Methodology Statistics Theory
no code implementations • 13 Feb 2018 • Weijie J. Su, Yuancheng Zhu
Stochastic gradient descent (SGD) is an immensely popular approach for online learning in settings where data arrives in a stream or data sizes are very large.
no code implementations • 10 Aug 2017 • Weijie J. Su
In a regime of certain sparsity levels, however, three examples of sequential procedures--forward stepwise, the lasso, and least angle regression--are shown to include the first spurious variable unexpectedly early.
1 code implementation • 11 Oct 2016 • Jingshu Wang, Lin Gui, Weijie J. Su, Chiara Sabatti, Art B. Owen
Replicability is a fundamental quality of scientific discoveries: we are interested in those signals that are detectable in different laboratories, study populations, across time etc.
Methodology