Search Results for author:
Found 25 papers, 13 papers with code
Self-Adjust Softmax
We conducted experiments to evaluate the empirical performance of Transformer models using SA-Softmax compared to the vanilla softmax function.
SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator
This observation suggests that information of the segments between these separator tokens can be effectively condensed into the separator tokens themselves without significant information loss.
Efficient Multi-modal Large Language Models via Visual Token Grouping
The development of Multi-modal Large Language Models (MLLMs) enhances Large Language Models (LLMs) with the ability to perceive data formats beyond text, significantly advancing a range of downstream applications, such as visual question answering and image captioning.
DAPE V2: Process Attention Score as Feature Map for Length Extrapolation
The attention mechanism is a fundamental component of the Transformer model, contributing to interactions among distinct tokens, in contrast to earlier feed-forward neural networks.
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
In this paper, we propose a training-free probabilistic parallel decoding algorithm, Speculative Jacobi Decoding (SJD), to accelerate auto-regressive text-to-image generation.
QuickLLaMA: Query-aware Inference Acceleration for Large Language Models
On widely recognized benchmarks, Q-LLM improved by 7. 17% compared to the current state-of-the-art on LLaMA3, and by 3. 26% on Mistral on the $\infty$-bench.
DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis
In addition, to further improve training efficiency for high-resolution image generation with DiM, we investigate "weak-to-strong" training strategy that pretrains DiM on low-resolution images ($256\times 256$) and then finetune it on high-resolution images ($512 \times 512$).
DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
Positional encoding plays a crucial role in transformers, significantly impacting model performance and length generalization.
AlgoFormer: An Efficient Transformer Framework with Algorithmic Structures
Furthermore, some theoretical and empirical results are presented to show that the designed transformer has the potential to perform algorithm representation and learning.
Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models
Recently, diffusion models have garnered significant interest in the field of text processing due to their many potential advantages compared to conventional autoregressive models.
BYOM: Building Your Own Multi-Task Model For Free
Recently, various merging methods have been proposed to build a multi-task model from task-specific finetuned models without retraining.
LEGO-Prover: Neural Theorem Proving with Growing Libraries
Our ablation study indicates that these newly added skills are indeed helpful for proving theorems, resulting in an improvement from a success rate of 47. 1% to 50. 4%.
Ranked #1 on
Automated Theorem Proving
on miniF2F-valid
(Pass@100 metric)
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
Our MetaMath-7B model achieves 66. 4% on GSM8K and 19. 4% on MATH, exceeding the state-of-the-art models of the same size by 11. 5% and 8. 7%.
Ranked #59 on
Arithmetic Reasoning
on GSM8K
(using extra training data)
GrowCLIP: Data-aware Automatic Model Growing for Large-scale Contrastive Language-Image Pre-training
Compared with the existing methods, GrowCLIP improves 2. 3% average top-1 accuracy on zero-shot image classification of 9 downstream tasks.
Forward-Backward Reasoning in Large Language Models for Mathematical Verification
Instead of using forward or backward reasoning alone, we propose FOBAR to combine FOrward and BAckward Reasoning for verification.
DiffFlow: A Unified SDE Framework for Score-Based Diffusion Models and Generative Adversarial Networks
Generative models can be categorized into two types: explicit generative models that define explicit density forms and allow exact likelihood inference, such as score-based diffusion models (SDMs) and normalizing flows; implicit generative models that directly learn a transformation from the prior to the data distribution, such as generative adversarial nets (GANs).
Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines
Data pipelines are widely employed in modern enterprises to power a variety of Machine-Learning (ML) and Business-Intelligence (BI) applications.
DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning
This paper proposes DiffFit, a parameter-efficient strategy to fine-tune large pre-trained diffusion models that enable fast adaptation to new domains.
Efficient Diffusion Personalization
parameter-efficient fine-tuning
Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism
To make ROSETTA automatically determine which experience is available and useful, a prototypical task correlation guided Gating Diversity Controller(GDC) is introduced to adaptively adjust the diversity of gates for the new task based on class-specific prototypes.
Revisiting Over-smoothing in BERT from the Perspective of Graph
Recently over-smoothing phenomenon of Transformer-based models is observed in both vision and language fields.
AutoBERT-Zero: Evolving BERT Backbone from Scratch
Transformer-based pre-trained language models like BERT and its variants have recently achieved promising performance in various natural language processing (NLP) tasks.
Ranked #10 on
Semantic Textual Similarity
on MRPC
SparseBERT: Rethinking the Importance Analysis in Self-attention
A surprising result is that diagonal elements in the attention map are the least important compared with other attention positions.
Effective Decoding in Graph Auto-Encoder using Triadic Closure
We propose the triad decoder, which considers and predicts the three edges involved in a local triad together.
Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
In this work, we propose BONAS (Bayesian Optimized Neural Architecture Search), a sample-based NAS framework which is accelerated using weight-sharing to evaluate multiple related architectures simultaneously.
Multi-objective Neural Architecture Search via Predictive Network Performance Optimization
Inspired by the nature of the graph structure of a neural network, we propose BOGCN-NAS, a NAS algorithm using Bayesian Optimization with Graph Convolutional Network (GCN) predictor.
