Search Results for author:
Found 26 papers, 9 papers with code
LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.
Large Language Models are Interpretable Learners
In the proposed LLM-based Symbolic Programs (LSPs), the pretrained LLM with natural language prompts provides a massive set of interpretable modules that can transform raw input into natural language concepts.
Automatic Engineering of Long Prompts
Large language models (LLMs) have demonstrated remarkable capabilities in solving complex open-domain tasks, guided by comprehensive instructions and demonstrations provided in the form of prompts.
Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory
The resulting algorithm sets new SOTA on ImageNet-1K: we can scale up to 50 IPCs (Image Per Class) on ImageNet-1K on a single GPU (all previous methods can only scale to 2 IPCs on ImageNet-1K), leading to the best accuracy (only 5. 9% accuracy drop against full dataset training) while utilizing only 4. 2% of the number of data points - an 18. 2% absolute gain over prior SOTA.
Ranked #5 on
Dataset Distillation - 1IPC
on CIFAR-10
Two-stage LLM Fine-tuning with Less Specialization and More Generalization
Pretrained large language models (LLMs) are general purpose problem solvers applicable to a diverse set of tasks with prompts.
DC-BENCH: Dataset Condensation Benchmark
Dataset Condensation is a newly emerging technique aiming at learning a tiny dataset that captures the rich information encoded in the original dataset.
Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding
Attentional mechanisms are order-invariant.
How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
For end-to-end efficiency, unlike previous work that assumes random hyperparameter tuning, which over-emphasizes the tuning time, we propose to evaluate with a bandit hyperparameter tuning strategy.
Multi-Stage Influence Function
With this score, we can identify the pretraining examples in the pretraining task that contribute most to a prediction in the finetuning task.
Auto Completion of User Interface Layout Design Using Transformer-Based Tree Decoders
While we focus on interface layout prediction, our model can be generally applicable for other layout prediction problems that involve tree structures and 2-dimensional placements.
A Unified Framework for Data Poisoning Attack to Graph-based Semi-supervised Learning
In this paper, we proposed a general framework for data poisoning attacks to graph-based semi-supervised learning (G-SSL).
Robustness Verification of Tree-based Models
We show that there is a simple linear time algorithm for verifying a single tree, and for tree ensembles, the verification problem can be cast as a max-clique problem on a multi-partite graph with bounded boxicity.
Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection.
Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
Furthermore, Cluster-GCN allows us to train much deeper GCN without much time and memory overhead, which leads to improved prediction accuracy---using a 5-layer Cluster-GCN, we achieve state-of-the-art test F1 score 99. 36 on the PPI dataset, while the previous best result was 98. 71 by [16].
Ranked #1 on
Node Classification
on Amazon2M
You Look Twice: GaterNet for Dynamic Filter Selection in CNNs
The concept of conditional computation for deep nets has been proposed previously to improve model performance by selectively using only parts of the model conditioned on the sample it is processing.
Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks
The algorithm achieves an order of magnitude faster inference than the original softmax layer for predicting top-$k$ words in various tasks such as beam search in machine translation or next words prediction.
Area Attention
We propose area attention: a way to attend to areas in the memory, where each area contains a group of items that are structurally adjacent, e. g., spatially for a 2D memory such as images, or temporally for a 1D memory such as natural language sentences.
GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking
Model compression is essential for serving large deep neural nets on devices with limited resources or applications that require real-time responses.
Nonlinear Online Learning with Adaptive Nyström Approximation
Use of nonlinear feature maps via kernel approximation has led to success in many online learning tasks.
Gradient Boosted Decision Trees for High Dimensional Sparse Output
In this paper, we study the gradient boosted decision trees (GBDT) when the output space is high dimensional and sparse.
GPU-acceleration for Large-scale Tree Boosting
In this paper, we present a novel massively parallel algorithm for accelerating the decision tree building procedure on GPUs (Graphics Processing Units), which is a crucial step in Gradient Boosted Decision Tree (GBDT) and random forests training.
Kernel Ridge Regression via Partitioning
In this paper, we investigate a divide and conquer approach to Kernel Ridge Regression (KRR).
Communication-Efficient Parallel Block Minimization for Kernel Machines
Kernel machines often yield superior predictive performance on various tasks; however, they suffer from severe computational challenges.
Multi-Scale Spectral Decomposition of Massive Graphs
Thus, eigenvectors of the clusters serve as good initializations to a block Lanczos algorithm that is used to compute spectral decomposition of the original graph.
Fast Prediction for Large-Scale Kernel Machines
Second, we provide a new theoretical analysis on bounding the error of the solution computed by using Nystr¨om kernel approximation method, and show that the error is related to the weighted kmeans objective function where the weights are given by the model computed from the original kernel.
A Divide-and-Conquer Solver for Kernel Support Vector Machines
We show theoretically that the support vectors identified by the subproblem solution are likely to be support vectors of the entire kernel SVM problem, provided that the problem is partitioned appropriately by kernel clustering.
