Search Results for author:
Found 69 papers, 32 papers with code
Accelerating AIGC Services with Latent Action Diffusion Scheduling in Edge Networks
The LAD-TS generates a near-optimal offloading decision by leveraging the diffusion model's conditional generation capability and the reinforcement learning's environment interaction ability, thereby minimizing the service delays under multiple resource constraints.
FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion
In this work, we provide a causal view to find that this performance drop of OFL methods comes from the isolation problem, which means that local isolatedly trained models in OFL may easily fit to spurious correlations due to the data heterogeneity.
Should We Really Edit Language Models? On the Evaluation of Edited Language Models
Our findings indicate that current editing methods are only suitable for small-scale knowledge updates within language models, which motivates further research on more practical and reliable editing methods.
ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference
To tackle these inference-specific challenges, we introduce ExpertFlow, a comprehensive system specifically designed to enhance inference efficiency by accommodating flexible routing and enabling efficient expert scheduling between CPU and GPU.
FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression
To alleviate hardware scarcity in training large deep neural networks (DNNs), particularly large language models (LLMs), we present FusionLLM, a decentralized training system designed and implemented for training DNNs using geo-distributed GPUs across different computing clusters or individual devices.
LPZero: Language Model Zero-cost Proxy Search from Zero
Specifically, we model the ZC proxy as a symbolic equation and incorporate a unified proxy search space that encompasses existing ZC proxies, which are composed of a predefined set of mathematical symbols.
LongGenBench: Long-context Generation Benchmark
Current long-context benchmarks primarily focus on retrieval-based tests, requiring Large Language Models (LLMs) to locate specific information within extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark.
Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL).
Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devices
We implement Asteroid on heterogeneous edge devices with both vision and language models, demonstrating up to 12. 2x faster training than conventional parallelism methods and 2. 1x faster than state-of-the-art hybrid parallelism methods through evaluations.
STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs
In this paper, we present the first structural binarization method for LLM compression to less than 1-bit precision.
3D Question Answering for City Scene Understanding
3D multimodal question answering (MQA) plays a crucial role in scene understanding by enabling intelligent agents to comprehend their surroundings in 3D environments.
Multi-Task Domain Adaptation for Language Grounding with 3D Objects
However, they have failed to consider exploring the cross-modal representation of language-vision alignment in the cross-domain field.
Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
Sparsely-activated Mixture-of-Expert (MoE) layers have found practical applications in enlarging the model size of large-scale foundation models, with only a sub-linear increase in computation demands.
EditFollower: Tunable Car Following Models for Customizable Adaptive Cruise Control Systems
To overcome these limitations, we propose the Editable Behavior Generation (EBG) model, a data-driven car-following model that allows for adjusting driving discourtesy levels.
Generating and Evolving Reward Functions for Highway Driving with Large Language Models
Reinforcement Learning (RL) plays a crucial role in advancing autonomous driving technologies by maximizing reward functions to achieve the optimal policy.
Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models
In particular, we devise an elaborate search space encompassing the existing pruning metrics to discover the potential symbolic pruning metric.
Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inference
Transformer-based models have unlocked a plethora of powerful intelligent applications at the edge, such as voice assistant in smart home.
Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
Model quantization, by converting high-precision numbers to lower-precision, reduces the computational demands and memory needs of ViTs, allowing the creation of hardware specifically optimized for these quantized algorithms, boosting efficiency.
VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting
Combining CNNs or ViTs, with RNNs for spatiotemporal forecasting, has yielded unparalleled results in predicting temporal and spatial dynamics.
BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation
The upscaling of Large Language Models (LLMs) has yielded impressive advances in natural language processing, yet it also poses significant deployment challenges.
FedImpro: Measuring and Improving Client Update in Federated Learning
First, we analyze the generalization contribution of local training and conclude that this generalization contribution is bounded by the conditional Wasserstein distance between the data distribution of different clients.
ParZC: Parametric Zero-Cost Proxies for Efficient NAS
Recent advancements in Zero-shot Neural Architecture Search (NAS) highlight the efficacy of zero-cost proxies in various NAS benchmarks.
CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning
Neural Radiance Fields (NeRF) have demonstrated impressive performance in novel view synthesis.
Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models
For end users, our benchmark and findings help better understand different optimization techniques, training and inference frameworks, together with hardware platforms in choosing configurations for deploying LLMs.
FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs
The rapid growth of memory and computation requirements of large language models (LLMs) has outpaced the development of hardware, hindering people who lack large-scale high-end GPUs from training or deploying LLMs.
EnsembleFollower: A Hybrid Car-Following Framework Based On Reinforcement Learning and Hierarchical Planning
The experimental results illustrate that EnsembleFollower yields improved accuracy of human-like behavior and achieves effectiveness in combining hybrid models, demonstrating that our proposed framework can handle diverse car-following conditions by leveraging the strengths of various low-level models.
LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning
The low-rank adaptation (LoRA) method can largely reduce the amount of trainable parameters for fine-tuning large language models (LLMs), however, it still requires expensive activation memory to update low-rank weights.
Evaluation and Optimization of Gradient Compression for Distributed Deep Learning
To accelerate distributed training, many gradient compression methods have been proposed to alleviate the communication bottleneck in synchronous stochastic gradient descent (S-SGD), but their efficacy in real-world applications still remains unclear.
FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training
It improves the training efficiency, remarkably relaxes the requirements on the hardware, and supports efficient large-scale FL experiments with stateful clients by: (1) sequential training clients on devices; (2) decomposing original aggregation into local and global aggregation on devices and server respectively; (3) scheduling tasks to mitigate straggler problems and enhance computing utility; (4) distributed client state manager to support various FL algorithms.
DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining
Communication scheduling has been shown to be effective in accelerating distributed training, which enables all-reduce communications to be overlapped with backpropagation computations.
Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Grounding
From the perspective of robotic cognition, we design a two-stage fine-grained robotic grasping framework (named LangPartGPD), including a novel 3D part language grounding model and a part-aware grasp pose detection model, in which explicit language input from human or large language models (LLMs) could guide a robot to generate part-level 6-DoF grasping pose with textual explanation.
Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on Disparity
Existing learning-based multi-view stereo (MVS) methods rely on the depth range to build the 3D cost volume and may fail when the range is too large or unreliable.
NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension
One-shot neural architecture search (NAS) substantially improves the search efficiency by training one supernet to estimate the performance of every possible child architecture (i. e., subnet).
Ranked #26 on
Neural Architecture Search
on NAS-Bench-201, CIFAR-10
SphereDepth: Panorama Depth Estimation from Spherical Domain
The panorama image can simultaneously demonstrate complete information of the surrounding environment and has many advantages in virtual tourism, games, robotics, etc.
Ranked #17 on
Depth Estimation
on Stanford2D3D Panoramic
EASNet: Searching Elastic and Accurate Network Architecture for Stereo Matching
However, existing NAS studies on the dense prediction task, especially stereo matching, still cannot be efficiently and effectively deployed on devices of different computing capabilities.
Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning
In federated learning (FL), model performance typically suffers from client drift induced by data heterogeneity, and mainstream works focus on correcting client drift.
AdaProp: Learning Adaptive Propagation for Graph Neural Network based Knowledge Graph Reasoning
An important design component of GNN-based KG reasoning methods is called the propagation path, which contains a set of involved entities in each propagation step.
EAGAN: Efficient Two-stage Evolutionary Architecture Search for GANs
Some recent works try to search both generator (G) and discriminator (D), but they suffer from the instability of GAN training.
Ranked #16 on
Image Generation
on STL-10
FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks
To bridge the gap and facilitate the development of FL for computer vision tasks, in this work, we propose a federated learning library and benchmarking framework, named FedCV, to evaluate FL on the three most representative computer vision tasks: image classification, image segmentation, and object detection.
Embracing Structure in Data for Billion-Scale Semantic Product Search
When training a dyadic model, one seeks to embed two different types of entities (e. g., queries and documents or users and movies) in a common vector space such that pairs with high relevance are positioned nearby.
FADNet++: Real-Time and Accurate Disparity Estimation with Configurable Networks
The disparity estimation problem tends to be addressed by DNNs which achieve much better prediction accuracy than traditional hand-crafted feature-based methods.
A Comprehensive Survey of Incentive Mechanism for Federated Learning
Federated learning utilizes various resources provided by participants to collaboratively train a global model, which potentially address the data privacy issue of machine learning.
Evolutionary Multi-objective Architecture Search Framework: Application to COVID-19 3D CT Classification
We propose a new objective, namely potential, which can help exploit promising models to indirectly reduce the number of models involved in weights training, thus alleviating search instability.
Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scans
The experimental results show that our automatically searched models (CovidNet3D) outperform the baseline human-designed models on the three datasets with tens of times smaller model size and higher accuracy.
EDNet: Efficient Disparity Estimation with Cost Volume Combination and Attention-based Spatial Residual
Existing state-of-the-art disparity estimation works mostly leverage the 4D concatenation volume and construct a very deep 3D convolution neural network (CNN) for disparity regression, which is inefficient due to the high memory consumption and slow inference speed.
Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
Distributed training techniques have been widely deployed in large-scale deep neural networks (DNNs) training on dense-GPU clusters.
A Quantitative Survey of Communication Optimizations in Distributed Deep Learning
In this article, we present a quantitative survey of communication optimization techniques for data parallel distributed DL.
FADNet: A Fast and Accurate Network for Disparity Estimation
Deep neural networks (DNNs) have achieved great success in the area of computer vision.
Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
In this paper, we provide a comprehensive survey of the communication-efficient distributed training algorithms, focusing on both system-level and algorithmic-level optimizations.
Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs
We thus propose a provable algorithm, AdaDUAL, to efficiently schedule those communication tasks.
FMore: An Incentive Scheme of Multi-dimensional Auction for Federated Learning in MEC
In MEC, edge nodes would not like to voluntarily participate in learning, and they differ in the provision of multi-dimensional resources, both of which might deteriorate the performance of federated learning.
Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection
2) Each worker only needs to communicate with a single peer at each communication round with a highly compressed model, which can significantly reduce the communication traffic on the worker.
A Survey of Deep Learning Techniques for Neural Machine Translation
In recent years, natural language processing (NLP) has got great development with deep learning techniques.
IRS: A Large Naturalistic Indoor Robotics Stereo Dataset to Train Deep Models for Disparity and Surface Normal Estimation
Besides, we present DTN-Net, a two-stage deep model for surface normal estimation.
MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning
Distributed synchronous stochastic gradient descent has been widely used to train deep neural networks (DNNs) on computer clusters.
Understanding Top-k Sparsification in Distributed Deep Learning
Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck.
Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees
To reduce the long training time of large deep neural network (DNN) models, distributed synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers.
Computer-Aided Clinical Skin Disease Diagnosis Using CNN and Object Detection Models
Our results show that object detection can help improve the accuracy of some skin disease classes.
Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training
Different from the existing end-to-end benchmarks which only present the training time, We try to investigate the impact of hardware, vendor's software library, and deep learning framework on the performance and energy consumption of AI training.
AutoML: A Survey of the State-of-the-Art
Deep learning (DL) techniques have penetrated all aspects of our lives and brought us great convenience.
A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks
Current methods that use AllGather to accumulate the sparse gradients have a communication complexity of $O(kP)$, where $P$ is the number of workers, which is inefficient on low bandwidth networks with a large number of workers.
MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms
Distributed synchronous stochastic gradient descent has been widely used to train deep neural networks on computer clusters.
Distributed, Parallel, and Cluster Computing
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
(3) We propose highly optimized all-reduce algorithms that achieve up to 3x and 11x speedup on AlexNet and ResNet-50 respectively than NCCL-based training on a cluster with 1024 Tesla P40 GPUs.
Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
Deep learning frameworks have been widely deployed on GPU servers for deep learning applications in both academia and industry.
Distributed, Parallel, and Cluster Computing
Performance Evaluation of Deep Learning Tools in Docker Containers
We first benchmark the performance of system components (IO, CPU and GPU) in a docker container and the host system and compare the results to see if there's any difference.
Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
Rectifier neuron units (ReLUs) have been widely used in deep convolutional networks.
Supervised Learning Based Algorithm Selection for Deep Neural Networks
In this paper, we target at optimizing the operations of multiplying a matrix with the transpose of another matrix (referred to as NT operation hereafter), which contribute about half of the training time of fully connected deep neural networks.
Benchmarking State-of-the-Art Deep Learning Software Tools
We first benchmark the running performance of these tools with three popular types of neural networks on two CPU platforms and three GPU platforms.
Dissecting GPU Memory Hierarchy through Microbenchmarking
Memory access efficiency is a key factor in fully utilizing the computational power of graphics processing units (GPUs).
Hardware Architecture Distributed, Parallel, and Cluster Computing
