Search Results for author:
Found 40 papers, 27 papers with code
BAMBOO: a predictive and transferable machine learning force field framework for liquid electrolyte development
Despite the widespread applications of machine learning force field (MLFF) on solids and small molecules, there is a notable gap in applying MLFF to complex liquid electrolytes.
M2ORT: Many-To-One Regression Transformer for Spatial Transcriptomics Prediction from Histopathology Images
To address this limit, we propose M2ORT, a many-to-one regression Transformer that can accommodate the hierarchical structure of the pathology images through a decoupled multi-scale feature extractor.
TrustLLM: Trustworthiness in Large Language Models
This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions.
FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs
However, existing GPU and transformer-based accelerators cannot efficiently process compressed LLMs, due to the following unresolved challenges: low computational efficiency, underutilized memory bandwidth, and large compilation overheads.
LLM360: Towards Fully Transparent Open-Source LLMs
The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers.
Rethinking Multiple Instance Learning for Whole Slide Image Classification: A Bag-Level Classifier is a Good Instance-Level Teacher
Based on this idea, we design Iteratively Coupled Multiple Instance Learning (ICMIL) to couple the embedder and the bag classifier at a low cost.
PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices
As neural networks (NN) are deployed across diverse sectors, their energy demand correspondingly grows.
RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs
In this work, we present RedCoast(Redco), a lightweight and user-friendly tool crafted to automate distributed training and inference for LLMs, as well as to simplify ML pipeline development.
Fusing Models with Complementary Expertise
Training AI models that generalize across tasks and domains has long been among the open problems driving AI research.
GridFormer: Towards Accurate Table Structure Recognition via Grid Prediction
In this representation, the vertexes and edges of the grid store the localization and adjacency information of the table.
SlimPajama-DC: Understanding Data Combinations for LLM Training
This paper aims to understand the impacts of various data combinations (e. g., web text, wikipedia, github, books) on the training of large language models using SlimPajama.
Maestro: Uncovering Low-Rank Structures via Trainable Decomposition
We further apply our technique on DNNs and empirically illustrate that Maestro enables the extraction of lower footprint models that preserve model performance while allowing for graceful accuracy-latency tradeoff for the deployment to devices of different capabilities.
Cuttlefish: Low-Rank Model Training without All the Tuning
Cuttlefish leverages the observation that after a few epochs of full-rank training, the stable rank (i. e., an approximation of the true rank) of each layer stabilizes at a constant value.
Iteratively Coupled Multiple Instance Learning from Instance to Bag Classifier for Whole Slide Image Classification
In ICMIL, we use category information in the bag-level classifier to guide the patch-level fine-tuning of the patch feature extractor.
Memory-adaptive Depth-wise Heterogenous Federated Learning
Federated learning is a promising paradigm that allows multiple clients to collaboratively train a model without sharing the local data.
Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach
A recent alternative formulation instead treats federated learning as a distributed inference problem, where the goal is to infer a global posterior from partitioned client data (Al-Shedivat et al., 2021).
Does compressing activations help model parallel training?
Finally, we provide insights for future development of model parallelism compression algorithms.
MPCFormer: fast, performant and private Transformer inference with MPC
Through extensive evaluations, we show that MPCFORMER significantly speeds up Transformer inference in MPC settings while achieving similar ML performance to the input model.
Super-Resolution Based Patch-Free 3D Image Segmentation with High-Frequency Guidance
The framework contains two sub-tasks, of which semantic segmentation is the main task and super resolution is an auxiliary task aiding in rebuilding the high frequency information from the LR input.
AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness
Scaling up model sizes can lead to fundamentally new capabilities in many machine learning (ML) tasks.
CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation
Multimodal sentiment analysis and depression estimation are two important research topics that aim to predict human mental states using multimodal data.
Efficient Federated Learning on Knowledge Graphs via Privacy-preserving Relation Embedding Aggregation
Furthermore, we propose a Federated learning paradigm with privacy-preserving Relation embedding aggregation (FedR) to tackle the privacy issue in FedE.
Rare Gems: Finding Lottery Tickets at Initialization
Frankle & Carbin conjecture that we can avoid this by training "lottery tickets", i. e., special sparse subnetworks found at initialization, that can be trained to high accuracy.
Hformer: Hybrid CNN-Transformer for Fringe Order Prediction in Phase Unwrapping of Fringe Projection
Our work opens an alternative way to deep learning based phase unwrapping methods, which are dominated by CNN in fringe projection 3D measurement.
Mixed Transformer U-Net For Medical Image Segmentation
Therefore, Vision Transformers have emerged as alternative segmentation structures recently, for their innate ability of capturing long-range correlations through Self-Attention (SA).
Solon: Communication-efficient Byzantine-resilient Distributed Training via Redundant Gradients
There has been a growing need to provide Byzantine-resilience in distributed model training.
A Field Guide to Federated Optimization
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection.
Pufferfish: Communication-efficient Models At No Extra Cost
In this work, we present Pufferfish, a communication and computation efficient distributed training framework that incorporates the gradient compression into the model training process via training low-rank, pre-factorized deep networks.
On the Utility of Gradient Compression in Distributed Training Systems
A rich body of prior work has highlighted the existence of communication bottlenecks in synchronous data-parallel training.
BPF for storage: an exokernel-inspired approach
The overhead of the kernel storage path accounts for half of the access latency for new NVMe storage devices.
Operating Systems Databases
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
The techniques usually require choosing a static compression ratio, often requiring users to balance the trade-off between model accuracy and per-iteration speedup.
Few shot domain adaptation for in situ macromolecule structural classification in cryo-electron tomograms
Motivation: Cryo-Electron Tomography (cryo-ET) visualizes structure and spatial organization of macromolecules and their interactions with other subcellular components inside single cells in the close-to-native state at sub-molecular resolution.
FedML: A Research Library and Benchmark for Federated Machine Learning
Federated learning (FL) is a rapidly growing research field in machine learning.
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
Due to its decentralized nature, Federated Learning (FL) lends itself to adversarial attacks in the form of backdoors during training.
Federated Learning with Matched Averaging
Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need to store the data in the cloud.
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation
In this work, we present DETOX, a Byzantine-resilient distributed training framework that combines algorithmic redundancy with robust aggregation.
ErasureHead: Distributed Gradient Descent without Delays Using Approximate Gradient Coding
We present ErasureHead, a new approach for distributed gradient descent (GD) that mitigates system delays by employing approximate gradient coding.
ATOMO: Communication-efficient Learning via Atomic Sparsification
We present ATOMO, a general framework for atomic sparsification of stochastic gradients.
The Effect of Network Width on the Performance of Large-batch Training
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training.
DRACO: Byzantine-resilient Distributed Training via Redundant Gradients
Distributed model training is vulnerable to byzantine system failures and adversarial compute nodes, i. e., nodes that use malicious updates to corrupt the global model stored at a parameter server (PS).
