Search Results for author:
Found 57 papers, 23 papers with code
Discovery of the Hidden World with Large Language Models
The rise of large language models (LLMs) that are trained to learn rich knowledge from the massive observations of the world, provides a new opportunity to assist with discovering high-level hidden variables from the raw observational data.
Enhancing Neural Subset Selection: Integrating Background Information into Set Representations
Learning neural subset selection tasks, such as compound selection in AI-aided drug discovery, have become increasingly pivotal across diverse applications.
Enhancing Evolving Domain Generalization through Dynamic Latent Representations
However, in non-stationary tasks where new domains evolve in an underlying continuous structure, such as time, merely extracting the invariant features is insufficient for generalization to the evolving new domains.
SPT: Fine-Tuning Transformer-based Language Models Efficiently with Sparsification
Thus, we design the sparse MHA module, which computes and stores only large attention weights to reduce memory consumption, and the routed FFN module, which dynamically activates a subset of model parameters for each token to reduce computation cost.
Positional Information Matters for Invariant In-Context Learning: A Case Study of Simple Function Classes
Surprisingly, DeepSet outperforms transformers across a variety of distribution shifts, implying that preserving permutation invariance symmetry to input demonstrations is crucial for OOD ICL.
Towards out-of-distribution generalizable predictions of chemical kinetics properties
Under this framework, we create comprehensive datasets to benchmark (1) the state-of-the-art ML approaches for reaction prediction in the OOD setting and (2) the state-of-the-art graph OOD methods in kinetics property prediction problems.
Understanding and Improving Feature Learning for Out-of-Distribution Generalization
Moreover, when fed the ERM learned features to the OOD objectives, the invariant feature learning quality significantly affects the final OOD performance, as OOD objectives rarely learn new features.
Follower Agnostic Methods for Stackelberg Games
In this paper, we present an efficient algorithm to solve online Stackelberg games, featuring multiple followers, in a follower-agnostic manner.
DGI: Easy and Efficient Inference for GNNs
In this paper, we develop Deep Graph Inference (DGI) -- a system for easy and efficient GNN model inference, which automatically translates the training code of a GNN model for layer-wise execution.
Measuring and Improving the Use of Graph Information in Graph Neural Networks
Graph neural networks (GNNs) have been widely used for representation learning on graph data.
A Representation Learning Framework for Property Graphs
Representation learning on graphs, also called graph embedding, has demonstrated its significant impact on a series of machine learning applications such as classification, prediction and recommendation.
Efficient Private SCO for Heavy-Tailed Data via Clipping
In this paper, we resolve this issue and derive the first high-probability bounds for the private stochastic method with clipping.
Fast and Reliable Evaluation of Adversarial Robustness with Minimum-Margin Attack
The AutoAttack (AA) has been the most reliable method to evaluate adversarial robustness when considerable computational resources are available.
Pareto Invariant Risk Minimization: Towards Mitigating the Optimization Dilemma in Out-of-Distribution Generalization
Recently, there has been a growing surge of interest in enabling machine learning systems to generalize well to Out-of-Distribution (OOD) data.
An Adaptive Incremental Gradient Method With Support for Non-Euclidean Norms
Stochastic variance reduced methods have shown strong performance in solving finite-sum problems.
Understanding and Improving Graph Injection Attack by Promoting Unnoticeability
Recently Graph Injection Attack (GIA) emerges as a practical attack scenario on Graph Neural Networks (GNNs), where the adversary can merely inject few malicious nodes instead of modifying existing nodes or edges, i. e., Graph Modification Attack (GMA).
Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs
Despite recent success in using the invariance principle for out-of-distribution (OOD) generalization on Euclidean data (e. g., images), studies on graph data are still limited.
Accelerating Perturbed Stochastic Iterates in Asynchronous Lock-Free Optimization
We show that stochastic acceleration can be achieved under the perturbed iterate framework (Mania et al., 2017) in asynchronous lock-free optimization, which leads to the optimal incremental gradient complexity for finite-sum objectives.
Local Reweighting for Adversarial Training
However, when tested on attacks different from the given attack simulated in training, the robustness may drop significantly (e. g., even worse than no reweighting).
Vertex-Centric Visual Programming for Graph Neural Networks
Graph neural networks (GNNs) have achieved remarkable performance in many graph analytics tasks such as node classification, link prediction and graph clustering.
Practical Schemes for Finding Near-Stationary Points of Convex Finite-Sums
In convex optimization, the problem of finding near-stationary points has not been adequately studied yet, unlike other optimality measures such as the function value.
DGCL: an efficient communication library for distributed GNN training
Graph neural networks (GNNs) have gained increasing popularity in many areas such as e-commerce, social networks and bio-informatics.
Seastar: vertex-centric programming for graph neural networks
Graph neural networks (GNNs) have achieved breakthrough performance in graph analytics such as node classification, link prediction and graph clustering.
Elastic Deep Learning in Multi-Tenant GPU Clusters
We study how to support elasticity, that is, the ability to dynamically adjust the parallelism (i. e., the number of GPUs), for deep neural network (DNN) training in a GPU cluster.
Calibrating and Improving Graph Contrastive Learning
To assess the discrepancy between the prediction and the ground-truth in the downstream tasks for these contrastive pairs, we adapt the expected calibration error (ECE) to graph contrastive learning.
The item selection problem for user cold-start recommendation
When a new user just signs up on a website, we usually have no information about him/her, i. e. no interaction with items, no user profile and no social links with other users.
Rethinking Graph Regularization for Graph Neural Networks
The graph Laplacian regularization term is usually used in semi-supervised representation learning to provide graph structure information for a model $f(X)$.
Understanding Graph Neural Networks from Graph Signal Denoising Perspectives
This paper aims to provide a theoretical framework to understand GNNs, specifically, spectral graph convolutional networks and graph attention networks, from graph signal denoising perspectives.
Boosting First-Order Methods by Shifting Objective: New Schemes with Faster Worst-Case Rates
Specifically, instead of tackling the original objective directly, we construct a shifted objective function that has the same minimizer as the original objective and encodes both the smoothness and strong convexity of the original objective in an interpolation condition.
TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism
A good parallelization strategy can significantly improve the efficiency or reduce the cost for the distributed training of deep neural networks (DNNs).
Self-Enhanced GNN: Improving Graph Neural Networks Using Model Outputs
In this paper, we propose self-enhanced GNN (SEG), which improves the quality of the input data using the outputs of existing GNN models for better performance on semi-supervised node classification.
Convolutional Embedding for Edit Distance
Edit-distance-based string similarity search has many applications such as spell correction, data de-duplication, and sequence alignment.
Norm-Explicit Quantization: Improving Vector Quantization for Maximum Inner Product Search
In this paper, we present a new angle to analyze the quantization error, which decomposes the quantization error into norm error and direction error.
Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning
In particular, at the high compression ratio end, HSQ provides a low per-iteration communication cost of $O(\log d)$, which is favorable for federated learning.
Understanding and Improving Proximity Graph based Maximum Inner Product Search
Then we explain the good performance of ip-NSW as matching the norm bias of the MIPS problem - large norm items have big in-degrees in the ip-NSW proximity graph and a walk on the graph spends the majority of computation on these items, thus effectively avoids unnecessary computation on small norm items.
Amortized Nesterov's Momentum: Robust and Lightweight Momentum for Deep Learning
Stochastic Gradient Descent (SGD) with Nesterov's momentum is a widely used optimizer in deep learning, which is observed to have excellent generalization performance.
PMD: An Optimal Transportation-based User Distance for Recommender Systems
Collaborative filtering, a widely-used recommendation technique, predicts a user's preference by aggregating the ratings from similar users.
Norm-Range Partition: A Universal Catalyst for LSH based Maximum Inner Product Search (MIPS)
Recently, locality sensitive hashing (LSH) was shown to be effective for MIPS and several algorithms including $L_2$-ALSH, Sign-ALSH and Simple-LSH have been proposed.
Bilinear Factor Matrix Norm Minimization for Robust PCA: Algorithms and Applications
The heavy-tailed distributions of corrupted outliers and singular values of all channels in low-level vision have proven effective priors for many applications such as background modeling, photometric stereo and image alignment.
ASVRG: Accelerated Proximal SVRG
This paper proposes an accelerated proximal stochastic variance reduced gradient (ASVRG) method, in which we design a simple and effective momentum acceleration trick.
Norm-Ranging LSH for Maximum Inner Product Search
Neyshabur and Srebro proposed Simple-LSH, which is the state-of-the-art hashing method for maximum inner product search (MIPS) with performance guarantee.
A Simple Stochastic Variance Reduced Algorithm with Fast Convergence Rates
Recent years have witnessed exciting progress in the study of stochastic variance reduced gradient methods (e. g., SVRG, SAGA), their accelerated variants (e. g, Katyusha) and their extensions in many different settings (e. g., online, sparse, asynchronous, distributed).
Tractable and Scalable Schatten Quasi-Norm Approximations for Rank Minimization
The Schatten quasi-norm was introduced to bridge the gap between the trace norm and rank function.
Guaranteed Sufficient Decrease for Stochastic Variance Reduced Gradient Optimization
In order to make sufficient decrease for stochastic optimization, we design a new sufficient decrease criterion, which yields sufficient decrease versions of stochastic variance reduction algorithms such as SVRG-SD and SAGA-SD as a byproduct.
VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning
In this paper, we propose a simple variant of the original SVRG, called variance reduced stochastic gradient descent (VR-SGD).
Accelerated First-order Methods for Geodesically Convex Optimization on Riemannian Manifolds
In this paper, we propose an accelerated first-order method for geodesically convex optimization, which is the generalization of the standard Nesterov's accelerated method from Euclidean space to nonlinear Riemannian space.
Accelerated Variance Reduced Stochastic ADMM
Besides having a low per-iteration complexity as existing stochastic ADMM methods, ASVRG-ADMM improves the convergence rate on general convex problems from O(1/T) to O(1/T^2).
Fast Stochastic Variance Reduced Gradient Method with Momentum Acceleration for Machine Learning
Recently, research on accelerated stochastic gradient descent methods (e. g., SVRG) has made exciting progress (e. g., linear convergence for strongly convex problems).
Guaranteed Sufficient Decrease for Variance Reduced Stochastic Gradient Descent
In order to make sufficient decrease for stochastic optimization, we design a new sufficient decrease criterion, which yields sufficient decrease versions of variance reduction algorithms such as SVRG-SD and SAGA-SD as a byproduct.
Scalable Algorithms for Tractable Schatten Quasi-Norm Minimization
In this paper, we first define two tractable Schatten quasi-norms, i. e., the Frobenius/nuclear hybrid and bi-nuclear quasi-norms, and then prove that they are in essence the Schatten-2/3 and 1/2 quasi-norms, respectively, which lead to the design of very efficient algorithms that only need to update two much smaller factor matrices.
Unified Scalable Equivalent Formulations for Schatten Quasi-Norms
In this paper, we rigorously prove that for any p, p1, p2>0 satisfying 1/p=1/p1+1/p2, the Schatten-p quasi-norm of any matrix is equivalent to minimizing the product of the Schatten-p1 norm (or quasi-norm) and Schatten-p2 norm (or quasi-norm) of its two factor matrices.
Regularized Orthogonal Tensor Decompositions for Multi-Relational Learning
We first induce the equivalence relation of the Schatten p-norm (0<p<\infty) of a low multi-linear rank tensor and its core tensor.
Generalized Higher-Order Orthogonal Iteration for Tensor Decomposition and Completion
Then the Schatten 1-norm of the core tensor is used to replace that of the whole tensor, which leads to a much smaller-scale matrix SNM problem.
Structured Low-Rank Matrix Factorization with Missing and Grossly Corrupted Observations
In this paper, we propose a scalable, provable structured low-rank matrix factorization method to recover low-rank and sparse matrices from missing and grossly corrupted data, i. e., robust matrix completion (RMC) problems, or incomplete and grossly corrupted measurements, i. e., compressive principal component pursuit (CPCP) problems.
Generalized Higher-Order Tensor Decomposition via Parallel ADMM
To address these problems, we first propose a parallel trace norm regularized tensor decomposition method, and formulate it as a convex optimization problem.
Tripartite Graph Clustering for Dynamic Sentiment Analysis on Social Media
We further investigate the evolution of user-level sentiments and latent feature vectors in an online framework and devise an efficient online algorithm to sequentially update the clustering of tweets, users and features with newly arrived data.
Truss Decomposition in Massive Networks
We first improve the existing in-memory algorithm for computing k-truss in networks of moderate size.
Databases
