Search Results for author:
Found 35 papers, 19 papers with code
Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback
In this work, we make the first attempt to fine-tune all-modality models (i. e. input and output with any modality, also named any-to-any models) using human preference data across all modalities (including text, image, audio, and video), ensuring its behavior aligns with human intentions.
RAGServe: Fast Quality-Aware RAG Systems with Configuration Adaptation
RAG (Retrieval Augmented Generation) allows LLMs (large language models) to generate better responses with external knowledge, but using more external knowledge often improves generation quality at the expense of response delay.
Residual Channel Boosts Contrastive Learning for Radio Frequency Fingerprint Identification
In order to address the issue of limited data samples for the deployment of pre-trained models in unseen environments, this paper proposes a residual channel-based data augmentation strategy for Radio Frequency Fingerprint Identification (RFFI), coupled with a lightweight SimSiam contrastive learning framework.
Marconi: Prefix Caching for the Era of Hybrid LLMs
Key to Marconi are its novel admission and eviction policies that more judiciously assess potential cache entries based not only on recency, but also on (1) forecasts of their reuse likelihood across a taxonomy of different hit scenarios, and (2) the compute savings that hits deliver relative to memory footprints.
Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code
The underlying cause of this issue is the gap between natural language to programming language gap (NL-PL Gap), which is especially pronounced in LRPLs due to limited aligned data.
Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling
Despite this, MoE models are hindered by high communication overhead from all-to-all operations, low GPU utilization due to the synchronous communication constraint, and complications from heterogeneous GPU environments.
Personalized Visual Instruction Tuning
In this paper, we introduce Personalized Visual Instruction Tuning (PVIT), a novel data curation and training framework designed to enable MLLMs to identify target individuals within an image and engage in personalized and coherent dialogues.
AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy
We find that the previously released AstroLLaMA series, based on LLaMA-2-7B, underperforms compared to the base model.
CollaMamba: Efficient Collaborative Perception with Cross-Agent Spatial-Temporal State Space Model
Initially, we construct a foundational backbone network based on spatial SSM.
TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data
Achieving this goal poses non-trivial challenges: 1) data selection requires accurate data representations that reflect the training samples' quality, 2) considering the diverse nature of instruction datasets, and 3) ensuring the efficiency of the coreset selection algorithm for large models.
AstroMLab 1: Who Wins Astronomy Jeopardy!?
This dataset comprises 4, 425 multiple-choice questions curated from the Annual Review of Astronomy and Astrophysics, covering a broad range of astrophysical topics.
TheoremLlama: Transforming General-Purpose LLMs into Lean4 Experts
However, due to the scarcity of aligned NL and Formal Language (FL) theorem-proving data most modern LLMs exhibit suboptimal performance. This scarcity results in a paucity of methodologies for training LLMs and techniques to fully utilize their capabilities in composing formal proofs.
ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting
Bilevel optimization has shown its utility across various machine learning settings, yet most algorithms in practice require second-order information, making it challenging to scale them up.
AdaGrad under Anisotropic Smoothness
Despite the huge success in practice, their theoretical advantages over classical gradient methods with uniform step sizes across all coordinates (e. g. SGD) have not been fully understood, especially in the large batch-size setting commonly used in practice.
Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions
Image description datasets play a crucial role in the advancement of various applications such as image understanding, text-to-image generation, and text-image retrieval.
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
Attempting to complement this deficiency, we investigate the layerwise properties of LoRA on fine-tuning tasks and observe an unexpected but consistent skewness of weight norms across different layers.
A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniques
A huge amount of statistical methods for massive data computation have been rapidly developed in the past decades.
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
To mitigate this issue, we propose Bootstrapped Preference Optimization (BPO), which conducts preference learning with datasets containing negative responses bootstrapped from the model itself.
Ranked #100 on
Visual Question Answering
on MM-Vet
The Instinctive Bias: Spurious Images lead to Illusion in MLLMs
In this paper, we identify a typical class of inputs that baffles MLLMs, which consist of images that are highly relevant but inconsistent with answers, causing MLLMs to suffer from visual illusion.
MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
The deployment of multimodal large language models (MLLMs) has brought forth a unique vulnerability: susceptibility to malicious attacks through visual inputs.
AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets
We explore the potential of enhancing LLM performance in astronomy-focused question-answering through targeted, continual pre-training.
Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise
This means SGD with heavy-ball momentum is useful in the large-batch settings such as distributed machine learning or federated learning, where a smaller number of iterations can significantly reduce the number of communication rounds, leading to acceleration in practice.
Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving
Machine learning (ML) inference platforms are tasked with balancing two competing goals: ensuring high throughput given many requests, and delivering low-latency responses to support interactive applications.
Plum: Prompt Learning using Metaheuristic
Since the emergence of large language models, prompt learning has become a popular method for optimizing and customizing these models.
Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?
Vision-Language Models (VLMs) are trained on vast amounts of data captured by humans emulating our understanding of the world.
Mitigating the Alignment Tax of RLHF
Building on the analysis and the observation that averaging different layers of the transformer leads to significantly different alignment-forgetting trade-offs, we propose Heterogeneous Model Averaging (HMA) to Heterogeneously find various combination ratios of model layers.
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models
As the number of available foundation models and specialized tasks keeps growing, the job of training scientific language models becomes highly nontrivial.
DetGPT: Detect What You Need via Reasoning
Overall, our proposed paradigm and DetGPT demonstrate the potential for more sophisticated and intuitive interactions between humans and machines.
Effective Bilevel Optimization via Minimax Reformulation
To address this issue, we propose a reformulation of bilevel optimization as a minimax problem, effectively decoupling the outer-inner dependency.
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
Utilizing a reward model and a sufficient number of samples, our approach selects the high-quality samples, discarding those that exhibit undesired behavior, and subsequently enhancing the model by fine-tuning on these filtered samples.
Active Prompting with Chain-of-Thought for Large Language Models
For this purpose, we propose a solution to the key problem of determining which questions are the most important and helpful ones to annotate from a pool of task-specific queries.
ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT
In this paper, we present ExtremeBERT, a toolkit for accelerating and customizing BERT pretraining.
Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums
In this paper, we propose Eigencurve, the first family of learning rate schedules that can achieve minimax optimal convergence rates (up to a constant) for SGD on quadratic objectives when the eigenvalue distribution of the underlying Hessian matrix is skewed.
