Search Results for author:
Found 109 papers, 63 papers with code
Don't Waste Your Bits! Squeeze Activations and Gradients for Deep Neural Networks via TinyScript
Recent years have witnessed intensive research interests on training deep neural networks (DNNs) more efficiently by quantization-based compression methods, which facilitate DNNs training in two ways: (1) activations are quantized to shrink the memory consumption, and (2) gradients are quantized to decrease the communication cost.
VersaTune: An Efficient Data Composition Framework for Training Multi-Capability LLMs
Despite their potential, existing work mainly focuses on domain-specific enhancements during fine-tuning, the challenge of which lies in catastrophic forgetting of knowledge across other domains.
Nova: A Practical and Advanced Alignment
Importantly, Qwen2-Nova-72B and Llama3-PBM-Nova-70B are instruct versions of the Qwen2-72B and Llama-3-70B base models, optimized through Nova.
Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning
While current research on function calling by LLMs primarily focuses on single-turn interactions, this paper addresses the overlooked necessity for LLMs to engage in multi-turn function calling--critical for handling compositional, real-world queries that require planning with functions but not only use functions.
FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedback
Human feedback is crucial in the interactions between humans and Large Language Models (LLMs).
IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation
IterComp opens new research avenues in reward feedback learning for diffusion models and compositional generation.
Universal Medical Image Representation Learning with Compositional Decoders
To address these challenges, we have developed a decomposed-composed universal medical imaging paradigm (UniMed) that supports tasks at all levels.
Data Proportion Detection for Optimized Data Management for Large Language Models
Large language models (LLMs) have demonstrated exceptional performance across a wide range of tasks and domains, with data preparation playing a critical role in achieving these results.
Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning
Recently, using large language models (LLMs) for data augmentation has led to considerable improvements in unsupervised sentence embedding models.
Efficient Multi-Task Large Model Training via Data Heterogeneity-aware Model Management
In this paper, we investigate how to achieve high-performance training of large-scale MT MM models through data heterogeneity-aware model management optimization.
DataSculpt: Crafting Data Landscapes for Long-Context LLMs through Multi-Objective Partitioning
Through extensive experimental analysis, we identified three key challenges in designing effective data management strategies that enable the model to achieve long-context capability without sacrificing performance in other tasks: (1) a shortage of long documents across multiple domains, (2) effective construction of context windows, and (3) efficient organization of large-scale datasets.
SysBench: Can Large Language Models Follow System Messages?
Finally, we conduct extensive evaluation across various existing LLMs, measuring their ability to follow specified constraints given in system messages.
MathScape: Evaluating MLLMs in multimodal Math Scenarios through a Hierarchical Benchmark
To address this gap, we proposed MathScape, a new benchmark that emphasizes the understanding and application of combined visual and textual information.
CFBench: A Comprehensive Constraints-Following Benchmark for LLMs
The adeptness of Large Language Models (LLMs) in comprehending and following natural language instructions is critical for their deployment in sophisticated real-world applications.
SynthVLM: High-Efficiency and High-Quality Synthetic Data for Vision Language Models
Crucially, our method's reliance on purely generated data ensures the preservation of privacy, achieving SoTA performance with just 100k data points (only 18% of the official dataset size).
Efficiently Training 7B LLM with 1 Million Sequence Length on 8 GPUs
By leveraging fine-grained activation memory management, MEMO facilitates efficient training of 7B LLM with 1 million sequence length on just 8 A800 GPUs, achieving an MFU of 52. 30%.
PAS: Data-Efficient Plug-and-Play Prompt Augmentation System
This combination of high performance, efficiency, and flexibility makes PAS a valuable system for enhancing the usability and effectiveness of LLMs through improved prompt engineering.
KeyVideoLLM: Towards Large-scale Video Keyframe Selection
Recently, with the rise of web videos, managing and understanding large-scale video datasets has become increasingly important.
Efficient-Empathy: Towards Efficient and Effective Selection of Empathy Data
By integrating sensibility and rationality data with a MoE structure, we achieve even higher performance, demonstrating the effectiveness of our Efficient-Empathy algorithm.
Consistency Flow Matching: Defining Straight Flows with Velocity Consistency
Additionally, we propose a multi-segment training approach for Consistency-FM to enhance expressiveness, achieving a better trade-off between sampling quality and speed.
PQCache: Product Quantization-based KVCache for Long Context LLM Inference
During the prefilling phase, we apply PQ to tokens' keys for each LLM layer and head.
RobGC: Towards Robust Graph Condensation
Graph neural networks (GNNs) have attracted widespread attention for their impressive capability of graph representation learning.
Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models
We introduce Buffer of Thoughts (BoT), a novel and versatile thought-augmented reasoning approach for enhancing accuracy, efficiency and robustness of large language models (LLMs).
VideoTetris: Towards Compositional Text-to-Video Generation
Diffusion models have demonstrated great success in text-to-video (T2V) generation.
A Survey of Multimodal Large Language Model from A Data-centric Perspective
Multimodal large language models (MLLMs) enhance the capabilities of standard large language models by integrating and processing data from multiple modalities, including text, vision, audio, video, and 3D environments.
Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling
First, we raise the scaling law between batch sizes and optimal learning rates in the sign of gradient case, in which we prove that the optimal learning rate first rises and then falls as the batch size increases.
Acceleration Algorithms in GNNs: A Survey
To address the critical challenges, a range of algorithms have been proposed to accelerate training and inference of GNNs, attracting increasing attention from the research community.
Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge
Consequently, the GPU spends most of its time on memory transfer instead of computation.
SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
In this work, we found that by identifying the importance of attention layers, we could optimize the KV-cache jointly from two dimensions, i. e., sequence-wise and layer-wise.
LIST: Learning to Index Spatio-Textual Data for Embedding based Spatial Keyword Queries
With the proliferation of spatio-textual data, Top-k KNN spatial keyword queries (TkQs), which return a list of objects based on a ranking function that considers both spatial and textual relevance, have found many real-life applications.
Retrieval-Augmented Generation for AI-Generated Content: A Survey
We first classify RAG foundations according to how the retriever augments the generator, distilling the fundamental abstractions of the augmentation methodologies for various retrievers and generators.
Structure-Guided Adversarial Training of Diffusion Models
In this pioneering approach, we compel the model to learn manifold structures between samples in each training batch.
Contextualized Diffusion Models for Text-Guided Image and Video Generation
To address this issue, we propose a novel and general contextualized diffusion model (ContextDiff) by incorporating the cross-modal context encompassing interactions and alignments between text condition and visual sample into forward and reverse processes.
RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models
In this paper, we propose RealCompo, a new training-free and transferred-friendly text-to-image generation framework, which aims to leverage the respective advantages of text-to-image models and spatial-aware image diffusion models (e. g., layout, keypoints and segmentation maps) to enhance both realism and compositionality of the generated images.
Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs
In this paper, we propose a brand new training-free text-to-image generation/editing framework, namely Recaption, Plan and Generate (RPG), harnessing the powerful chain-of-thought reasoning ability of multimodal LLMs to enhance the compositionality of text-to-image diffusion models.
Improving Diffusion-Based Image Synthesis with Context Prediction
In this way, each point can better reconstruct itself by preserving its semantic connections with neighborhood context.
Ranked #1 on
Image Inpainting
on CelebA
(LPIPS metric)
On-Device Recommender Systems: A Tutorial on The New-Generation Recommendation Paradigm
Given the sheer volume of contemporary e-commerce applications, recommender systems (RSs) have gained significant attention in both academia and industry.
CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Models
Guided by our design philosophy, we further propose a multi-level hash embedding framework to optimize the embedding tables of non-hot features.
Unveiling Vulnerabilities of Contrastive Recommender Systems to Poisoning Attacks
Our analysis indicates that this vulnerability is attributed to the uniform spread of representations caused by the InfoNCE loss.
Experimental Analysis of Large-scale Learnable Vector Storage Compression
Learnable embedding vector is one of the most important applications in machine learning, and is widely used in various database-related domains.
SpotServe: Serving Generative Large Language Models on Preemptible Instances
This paper aims to reduce the monetary cost for serving LLMs by leveraging preemptible GPU instances on modern clouds, which offer accesses to spare GPUs at a much cheaper price than regular instances but may be preempted by the cloud at any time.
Accelerating Scalable Graph Neural Network Inference with Node-Adaptive Propagation
To further accelerate Scalable GNNs inference in this inductive setting, we propose an online propagation framework and two novel node-adaptive propagation methods that can customize the optimal propagation depth for each node based on its topological information and thereby avoid redundant feature propagation.
Model-enhanced Vector Index
We empirically show that our model achieves better performance on the commonly used academic benchmarks MSMARCO Passage and Natural Questions, with comparable serving latency to dense retrieval solutions.
Towards General and Efficient Online Tuning for Spark
The distributed data analytic system -- Spark is a common choice for processing massive volumes of heterogeneous data, while it is challenging to tune its parameters to achieve high performance.
Mitigating Semantic Confusion from Hostile Neighborhood for Graph Active Learning
Graph Active Learning (GAL), which aims to find the most informative nodes in graphs for annotation to maximize the Graph Neural Networks (GNNs) performance, has attracted many research efforts but remains non-trivial challenges.
VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs
To address this issue, we propose to learn a new powerful graph representation space by directly labeling nodes' diverse local structures for GNN-to-MLP distillation.
BOURNE: Bootstrapped Self-supervised Learning Framework for Unified Graph Anomaly Detection
By swapping the context embeddings between nodes and edges and measuring the agreement in the embedding space, we enable the mutual detection of node and edge anomalies.
Improving Automatic Parallel Training via Balanced Memory Workload Optimization
Transformer models have emerged as the leading approach for achieving state-of-the-art performance across various application domains, serving as the foundation for advanced large-scale deep learning (DL) models.
StreamE: Learning to Update Representations for Temporal Knowledge Graphs in Streaming Scenarios
To reduce the parameter size, entity representations in StreamE are decoupled from the model training to serve as the memory module to store the historical information of entities.
Individual and Structural Graph Information Bottlenecks for Out-of-Distribution Generalization
To remove class spurious feature caused by distribution shifts, we propose Individual Graph Information Bottleneck (I-GIB) which discards irrelevant information by minimizing the mutual information between the input graph and its embeddings.
Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference
The key intuition behind our approach is to utilize the semantic mapping between the minor modifications on the input text and the affected regions on the output image.
OpenBox: A Python Toolkit for Generalized Black-box Optimization
Black-box optimization (BBO) has a broad range of applications, including automatic machine learning, experimental design, and database knob tuning.
FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
We first present an empirical analysis on the problems and opportunities of training MoE models, which motivates us to overcome the routing imbalance and fluctuation problems by a dynamic expert management and device placement mechanism.
A Unified and Efficient Coordinating Framework for Autonomous DBMS Tuning
To tune different components for DBMS, a coordinating mechanism is needed to make the multiple agents cognizant of each other.
Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent
Recent years have witnessed the unprecedented achievements of large-scale pre-trained models, especially the Transformer models.
Transfer Learning for Bayesian Optimization: A Survey
A wide spectrum of design and decision problems, including parameter tuning, A/B testing and drug design, intrinsically are instances of black-box optimization.
Rover: An online Spark SQL tuning service via generalized transfer learning
When applying transfer learning to accelerate the tuning process, we notice two domain-specific challenges: 1) most previous work focus on transferring tuning history, while expert knowledge from Spark engineers is of great potential to improve the tuning performance but is not well studied so far; 2) history tasks should be carefully utilized, where using dissimilar ones lead to a deteriorated performance in production.
DivBO: Diversity-aware CASH for Ensemble Learning
To tackle this issue and further enhance the ensemble performance, we propose DivBO, a diversity-aware framework to inject explicit search of diversity into the CASH problems.
Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism
Transformer models have achieved state-of-the-art performance on various domains of applications and gradually becomes the foundations of the advanced large deep learning (DL) models.
Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training
Generating images from graph-structured inputs, such as scene graphs, is uniquely challenging due to the difficulty of aligning nodes and connections in graphs with objects and their relations in images.
Efficient Graph Neural Network Inference at Large Scale
Graph neural networks (GNNs) have demonstrated excellent performance in a wide range of applications.
Distributed Graph Neural Network Training: A Survey
In recent years, many efforts have been made on distributed GNN training, and an array of training algorithms and systems have been proposed.
CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention
Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual representations with great transferability, which achieves promising accuracy for zero-shot classification.
Ranked #4 on
Training-free 3D Point Cloud Classification
on ScanObjectNN
(using extra training data)
Training-free 3D Point Cloud Classification
Transfer Learning
+1
Diffusion Models: A Comprehensive Survey of Methods and Applications
This survey aims to provide a contextualized, in-depth look at the state of diffusion models, identifying the key areas of focus and pointing to potential areas for further exploration.
Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updates
Vertical federated learning (VFL) is an emerging paradigm that allows different parties (e. g., organizations or enterprises) to collaboratively build machine learning models with privacy protection.
Efficient End-to-End AutoML via Scalable Search Space Decomposition
End-to-end AutoML has attracted intensive interests from both academia and industry which automatically searches for ML pipelines in a space induced by feature engineering, algorithm/model selection, and hyper-parameter tuning.
DFG-NAS: Deep and Flexible Graph Neural Architecture Search
Graph neural networks (GNNs) have been intensively applied to various graph-based applications.
NAFS: A Simple yet Tough-to-beat Baseline for Graph Representation Learning
First, GNNs can learn higher-order structural information by stacking more layers but can not deal with large depth due to the over-smoothing issue.
BlindFL: Vertical Federated Machine Learning without Peeking into Your Data
First, to address the functionality of VFL models, we propose the federated source layers to unite the data from different parties.
Graph Attention Multi-Layer Perceptron
Graph neural networks (GNNs) have achieved great success in many graph-based applications.
Ranked #12 on
Node Property Prediction
on ogbn-mag
Model Degradation Hinders Deep Graph Neural Networks
Graph Neural Networks (GNNs) have achieved great success in various graph mining tasks. However, drastic performance degradation is always observed when a GNN is stacked with many layers.
TransBO: Hyperparameter Optimization via Two-Phase Transfer Learning
With the extensive applications of machine learning models, automatic hyperparameter optimization (HPO) has become increasingly important.
Transfer Learning based Search Space Design for Hyperparameter Tuning
The extensive experiments show that our approach considerably boosts BO by designing a promising and compact search space instead of using the entire space, and outperforms the state-of-the-arts on a wide range of benchmarks, including machine learning and deep learning tuning tasks, and neural architecture search.
Instance-wise Prompt Tuning for Pretrained Language Models
Prompt Learning has recently gained great popularity in bridging the gap between pretraining tasks and various downstream tasks.
AutoML for Deep Recommender Systems: A Survey
To tackle this problem, Automated Machine Learning (AutoML) is introduced to automatically search for the proper candidates for different parts of deep recommender systems.
ZOOMER: Boosting Retrieval on Web-scale Graphs by Regions of Interest
We introduce ZOOMER, a system deployed at Taobao, the largest e-commerce platform in China, for training and serving GNN-based recommendations over web-scale graphs.
Information Gain Propagation: a new way to Graph Active Learning with Soft Labels
Graph Neural Networks (GNNs) have achieved great success in various tasks, but their performance highly relies on a large number of labeled nodes, which typically requires considerable human effort.
PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm
Through deconstructing the message passing mechanism, PasCa presents a novel Scalable Graph Neural Architecture Paradigm (SGAP), together with a general architecture design space consisting of 150k different designs.
Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale
The ever-growing demand and complexity of machine learning are putting pressure on hyper-parameter tuning systems: while the evaluation cost of models continues to increase, the scalability of state-of-the-arts starts to become a crucial bottleneck.
EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate
Then it diversifies the experts and continues to train the MoE with a novel Dense-to-Sparse gate (DTS-Gate).
K-Core Decomposition on Super Large Graphs with Limited Resources
For example, the distributed K-core decomposition algorithm can scale to a large graph with 136 billion edges without losing correctness with our divide-and-conquer technique.
HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework
Embedding models have been an effective learning paradigm for high-dimensional data.
PointCLIP: Point Cloud Understanding by CLIP
On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D.
Ranked #3 on
3D Open-Vocabulary Instance Segmentation
on STPLS3D
3D Open-Vocabulary Instance Segmentation
Few-Shot Learning
+7
RIM: Reliable Influence-based Active Learning on Graphs
Message passing is the core of most graph models such as Graph Convolutional Network (GCN) and Label Propagation (LP), which usually require a large number of clean labeled data to smooth out the neighborhood over the graph.
Node Dependent Local Smoothing for Scalable Graph Learning
Recent works reveal that feature or label smoothing lies at the core of Graph Neural Networks (GNNs).
ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
Designing neural architectures requires immense manual efforts.
Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation
Therefore, we propose a new metric P-Error to evaluate the performance of CardEst methods, which overcomes the limitation of Q-Error and is able to reflect the overall end-to-end performance of CardEst methods.
Graph Attention MLP with Reliable Label Utilization
Graph neural networks (GNNs) have recently achieved state-of-the-art performance in many graph-based applications.
Evaluating Deep Graph Neural Networks
Based on the experimental results, we answer the following two essential questions: (1) what actually leads to the compromised performance of deep GNNs; (2) when we need and how to build deep GNNs.
Grain: Improving Data Efficiency of Graph Neural Networks via Diversified Influence Maximization
Data selection methods, such as active learning and core-set selection, are useful tools for improving the data efficiency of deep learning models on large-scale datasets.
ROD: Reception-aware Online Distillation for Sparse Graphs
Unfortunately, many real-world networks are sparse in terms of both edges and labels, leading to sub-optimal performance of GNNs.
VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition
End-to-end AutoML has attracted intensive interests from both academia and industry, which automatically searches for ML pipelines in a space induced by feature engineering, algorithm/model selection, and hyper-parameter tuning.
OpenBox: A Generalized Black-box Optimization Service
Black-box optimization (BBO) has a broad range of applications, including automatic machine learning, engineering, physics, and experimental design.
CausCF: Causal Collaborative Filtering for RecommendationEffect Estimation
It is based on the idea that similar users not only have a similar taste on items, but also have similar treatment effect under recommendations.
GMLP: Building Scalable and Flexible Graph Neural Networks with Feature-Message Passing
In recent studies, neural message passing has proved to be an effective way to design graph neural networks (GNNs), which have achieved state-of-the-art performance in many graph-based tasks.
Explore User Neighborhood for Real-time E-commerce Recommendation
On the one hand, it utilizes UI relations and user neighborhood to capture both global and local information.
Efficient Automatic CASH via Rising Bandits
In this framework, the BO methods are used to solve the HPO problem for each ML algorithm separately, incorporating a much smaller hyperparameter space for BO methods.
Efficient and Scalable Structure Learning for Bayesian Networks: Algorithms and Applications
To resolve this, we propose a new structure learning algorithm LEAST, which comprehensively fulfills our business requirements as it attains high accuracy, efficiency and scalability at the same time.
MFES-HB: Efficient Hyperband with Multi-Fidelity Quality Measurements
Instead of sampling configurations randomly in HB, BOHB samples configurations based on a BO surrogate model, which is constructed with the high-fidelity measurements only.
FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation
Despite decades of research, existing methods either over simplify the models only using independent factorization which leads to inaccurate estimates, or over complicate them by lossless conditional factorization without any independent assumption which results in slow probability computation.
Graph Neural Networks in Recommender Systems: A Survey
With the explosive growth of online information, recommender systems play a key role to alleviate such information overload.
Contrastive Learning for Sequential Recommendation
Sequential recommendation methods play a crucial role in modern recommender systems because of their ability to capture a user's dynamic interest from her/his historical interactions.
UniNet: Scalable Network Representation Learning with Metropolis-Hastings Sampling
Finally, with the new edge sampler and random walk model abstraction, we carefully implement a scalable NRL framework called UniNet.
DeGNN: Characterizing and Improving Graph Neural Networks with Graph Decomposition
Despite the wide application of Graph Convolutional Network (GCN), one major limitation is that it does not benefit from the increasing depth and suffers from the oversmoothing problem.
X-Forest: Approximate Random Projection Trees for Similarity Measurement
First, we introduced RP Trees into the tasks of similarity measurement such that accuracy is improved.
AHash: A Load-Balanced One Permutation Hash
Unlike the densification to fill the empty bins after they undesirably occur, our design goal is to balance the load so as to reduce the empty bins in advance.
An Experimental Evaluation of Large Scale GBDT Systems
Gradient boosting decision tree (GBDT) is a widely-used machine learning algorithm in both data analytic competitions and real-world industrial applications.
Fast Hyperparameter Optimization of Deep Neural Networks via Ensembling Multiple Surrogates
The performance of deep neural networks crucially depends on good hyperparameter configurations.
