Search Results for author:
Found 31 papers, 11 papers with code
BG-HGNN: Toward Scalable and Efficient Heterogeneous Graph Neural Network
Many computer vision and machine learning problems are modelled as learning tasks on heterogeneous graphs, featuring a wide array of relations from diverse types of nodes and edges.
Improving Implicit Regularization of SGD with Preconditioning for Least Square Problems
In this paper, we study the generalization performance of SGD with preconditioning for the least squared problem.
On the Topology Awareness and Generalization Performance of Graph Neural Networks
Using this framework, we investigate the effects of topology awareness on GNN generalization performance.
LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
The immense sizes of LLMs have led to very high resource demand and cost for running the models.
LoRA Meets Dropout under a Unified Framework
Hence, a possible contradiction arises from negligible trainable parameters of LoRA and the effectiveness of previous dropout methods, which has been largely overlooked.
PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA
Hopefully, the conspicuously higher parameter efficiency can establish PRoLoRA as a resource-friendly alternative to LoRA.
MSPipe: Efficient Temporal GNN Training via Staleness-aware Pipeline
However, the iterative reading and updating process of the memory module in MTGNNs to obtain up-to-date information needs to follow the temporal dependencies.
Towards Robust Graph Incremental Learning on Evolving Graphs
We provide a formal formulation and analysis of the problem, and propose a novel regularization-based technique called Structural-Shift-Risk-Mitigation (SSRM) to mitigate the impact of the structural shift on catastrophic forgetting of the inductive NGIL problem.
Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models
This work explores the integration of diffusion models and Chain-of-Thought (CoT), a well-established technique to improve the reasoning ability in autoregressive language models.
PRES: Toward Scalable Memory-Based Dynamic Graph Neural Networks
Memory-based Dynamic Graph Neural Networks (MDGNNs) are a family of dynamic graph neural networks that leverage a memory module to extract, distill, and memorize long-term temporal dependencies, leading to superior performance compared to memory-less counterparts.
GNNFlow: A Distributed Framework for Continuous Temporal GNN Learning on Dynamic Graphs
We introduce GNNFlow, a distributed framework that enables efficient continuous temporal graph representation learning on dynamic graphs on multi-GPU machines.
DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines
This paper proposes a dynamic micro-batching approach to tackle sequence length variation and enable efficient multi-task model training.
CDMPP: A Device-Model Agnostic Framework for Latency Prediction of Tensor Programs
Considering the large space of DNN models and devices that impede direct profiling of all combinations, recent efforts focus on building a predictor to model the performance of DNN models on different devices.
Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Training
Distributed full-graph training of Graph Neural Networks (GNNs) over large graphs is bandwidth-demanding and time-consuming.
A Cognitive Stimulation Dialogue System with Multi-source Knowledge Fusion for Elders with Cognitive Impairment
In this paper, we propose a multi-source knowledge fusion method for CS dialogue (CSD), to generate open-ended responses guided by the CS principle and emotional support strategy.
Auto-Parallelizing Large Models with Rhino: A Systematic Approach on Production AI Platform
We present Rhino, a system for accelerating tensor programs with automatic parallelization on AI platform for real production environment.
Expediting Distributed DNN Training with Device Topology-Aware Graph Deployment
This paper presents TAG, an automatic system to derive optimized DNN training graph and its deployment onto any device topology, for expedited training in device- and topology- heterogeneous ML clusters.
Towards Robust Inductive Graph Incremental Learning via Experience Replay
Inductive node-wise graph incremental learning is a challenging task due to the dynamic nature of evolving graphs and the dependencies between nodes.
MixSynthFormer: A Transformer Encoder-like Structure with Mixed Synthetic Self-attention for Efficient Human Pose Estimation
Human pose estimation in videos has wide-ranging practical applications across various fields, many of which require fast inference on resource-scarce devices, necessitating the development of efficient and accurate algorithms.
dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training
Distributed training using multiple devices (e. g., GPUs) has been widely adopted for learning DNN models over large datasets.
BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing
Extensive experiments on various GNN models and large graph datasets show that BGL significantly outperforms existing GNN training systems by 20. 68x on average.
Adversarial Deep Learning for Online Resource Allocation
Inspired by adversarial training from Generative Adversarial Net (GAN) and the fact that competitive ratio of an online algorithm is based on worst-case input, we adopt deep neural networks to learn an online algorithm for a resource allocation and pricing problem from scratch, with the goal that the performance gap between offline optimum and the learned online algorithm can be minimized for worst-case input.
OneFlow: Redesign the Distributed Deep Learning Framework from Scratch
Aiming at a simple, neat redesign of distributed deep learning frameworks for various parallelism paradigms, we present OneFlow, a novel distributed training framework based on an SBP (split, broadcast and partial-value) abstraction and the actor model.
On Locality in Graph Learning via Graph Neural Network
In this paper, we study how the training set in the input graph effects the performance of GNN.
WN-Salience: A Corpus of News Articles with Entity Salience Annotations
To support research on entity salience, we present a new dataset, the WikiNews Salience dataset (WN-Salience), which can be used to benchmark tasks such as entity salience detection and salient entity linking.
Distributed Machine Learning through Heterogeneous Edge Systems
Many emerging AI applications request distributed machine learning (ML) among edge systems (e. g., IoT devices and PCs at the edge of the Internet), where data cannot be uploaded to a central venue for model training, due to their large volumes and/or security/privacy concerns.
Characterizing Deep Learning Training Workloads on Alibaba-PAI
One critical issue for efficiently operating practical AI clouds, is to characterize the computing and data transfer demands of these workloads, and more importantly, the training performance given the underlying software framework and hardware configurations.
DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters
DL2 is a DL-driven scheduler for DL clusters, targeting global training job expedition by dynamically resizing resources allocated to jobs.
Online Job Scheduling in Distributed Machine Learning Clusters
In a shared cluster handling multiple training jobs, a fundamental issue is how to efficiently schedule jobs and set the number of concurrent workers to run for each job, such that server resources are maximally utilized and model training can be completed in time.
Distributed, Parallel, and Cluster Computing
Normalized Direction-preserving Adam
Adaptive optimization algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in some scenarios.
Online Influence Maximization in Non-Stationary Social Networks
Nevertheless, the existing studies mostly investigate the problem on a one-off basis, assuming fixed known influence probabilities among users, or the knowledge of the exact social network topology.
