Search Results for author:
Found 37 papers, 17 papers with code
Event Graph based Sentence Fusion
Sentence fusion is a conditional generation task that merges several related sentences into a coherent one, which can be deemed as a summary sentence.
A Comprehensive Survey on Long Context Language Modeling
With the growing number of long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (LCLMs) that can process and analyze extensive inputs in an effective and efficient way.
Predictable Scale: Part I -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
Through extensive empirical studies involving grid searches across diverse configurations, we discover universal scaling laws governing these hyperparameters: optimal learning rate follows a power-law relationship with both model parameters and data sizes, while optimal batch size scales primarily with data sizes.
SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
To address this gap, we present SuperGPQA, a comprehensive benchmark that evaluates graduate-level knowledge and reasoning capabilities across 285 disciplines.
Multi-matrix Factorization Attention
Existing variants for standard Multi-Head Attention (MHA), including SOTA methods like MLA, fail to maintain as strong performance under stringent Key-Value cache (KV cache) constraints.
Continuous Speculative Decoding for Autoregressive Image Generation
Experimental results show that our continuous speculative decoding achieves a remarkable $2. 33\times$ speed-up on off-the-shelf models while maintaining the output distribution.
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
To address the gap, we introduce OpenCoder, a top-tier code LLM that not only achieves performance comparable to leading models but also serves as an "open cookbook" for the research community.
Post-hoc Reward Calibration: A Case Study on Length Bias
Central to this process is the reward model (RM), which translates human feedback into training signals for optimising LLM behaviour.
Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis
These challenges encompass maintaining the content consistency between the input video and the generated audio, as well as the alignment of temporal and loudness properties within the video.
Layerwise Recurrent Router for Mixture-of-Experts
The scaling of large language models (LLMs) has revolutionized their capabilities in various tasks, yet this growth must be matched with efficient computational strategies.
AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation
By characterizing attention maps of the network, we identify two key obstacles in AVS models: 1) attention dissipation, corresponding to the over-concentrated attention weights by Softmax within restricted frames, and 2) inefficient, burdensome transformer decoder, caused by narrow focus patterns in early stages.
R3D-AD: Reconstruction via Diffusion for 3D Anomaly Detection
However, there are two major challenges to the practical application of the current approaches: 1) the embedded models suffer the prohibitive computational and storage due to the memory bank structure; 2) the reconstructive models based on the MAE mechanism fail to detect anomalies in the unmasked regions.
Ranked #2 on
3D Anomaly Detection
on Anomaly-ShapeNet
A Closer Look into Mixture-of-Experts in Large Language Models
Mixture-of-experts (MoE) is gaining increasing attention due to its unique properties and remarkable performance, especially for language tasks.
GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory
Therefore, these tokens can acquire the necessary knowledge from any expert during inference and become less sensitive to the choice.
AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning
In-context learning (ICL) facilitates Large Language Models (LLMs) exhibiting emergent ability on downstream tasks without updating billions of parameters.
Lyapunov Neural Network with Region of Attraction Search
Deep learning methods have been widely used in robotic applications, making learning-enabled control design for complex nonlinear systems a promising direction.
Beyond Language Models: Byte Models are Digital World Simulators
Traditional deep learning often overlooks bytes, the basic units of the digital world, where all forms of information and operations are encoded and manipulated in binary format.
m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers
Additionally, the V-MoE-Base model trained with m2mKD achieves 3. 5% higher accuracy than end-to-end training on ImageNet-1k.
ChatMusician: Understanding and Generating Music Intrinsically with LLM
It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language.
HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts
The Mixture of Experts (MoE) for language models has been proven effective in augmenting the capacity of models by dynamically routing each input token to a specific subset of experts for processing.
A Novel Garment Transfer Method Supervised by Distilled Knowledge of Virtual Try-on Model
This paper proposes a novel garment transfer method supervised with knowledge distillation from virtual try-on.
Personalized Large Language Model Assistant with Evolving Conditional Memory
With the rapid development of large language models, AI assistants like ChatGPT have become increasingly integrated into people's works and lives but are limited in personalized services.
GSDC Transformer: An Efficient and Effective Cue Fusion for Monocular Multi-Frame Depth Estimation
To address these issues, we propose the GSDC Transformer, an efficient and effective component for cue fusion in monocular multi-frame depth estimation.
MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Remarkable progress has been made on automated problem solving through societies of agents based on large language models (LLMs).
Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs).
MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored.
RefGPT: Dialogue Generation of GPT, by GPT, and for GPT
Therefore, we propose a method called RefGPT to generate enormous truthful and customized dialogues without worrying about factual errors caused by the model hallucination.
Basic syntax from speech: Spontaneous concatenation in unsupervised deep neural networks
We introduce spontaneous concatenation: a phenomenon where convolutional neural networks (CNNs) trained on acoustic recordings of individual words start generating outputs with two or even three words concatenated without ever accessing data with multiple words in the input.
PG-VTON: A Novel Image-Based Virtual Try-On Method via Progressive Inference Paradigm
To address these limitations, we propose a novel virtual try-on method via progressive inference paradigm (PGVTON) that leverages a top-down inference pipeline and a general garment try-on strategy.
A Cross-Scale Hierarchical Transformer with Correspondence-Augmented Attention for inferring Bird's-Eye-View Semantic Segmentation
To save the computation increase caused by this hierarchical framework, we exploit the cross-scale Transformer to learn feature relationships in a reversed-aligning way, and leverage the residual connection of BEV features to facilitate information transmission between scales.
Few-shot Query-Focused Summarization with Prefix-Merging
Drawn inspiration from prefix-tuning, we are allowed to integrate the task knowledge from text summarization and question answering into a properly designed prefix and apply the merged prefix to query-focused summarization.
Physical Logic Enhanced Network for Small-Sample Bi-Layer Metallic Tubes Bending Springback Prediction
In the second stage, under the physical logic, the PE-NET is assembled by ES-NET and SP-NET and then fine-tuned with the small sample BMT dataset and composite loss function.
Digital-twin-enhanced metal tube bending forming real-time prediction method based on Multi-source-input MTL
To address this issue, a digital-twin-enhanced (DT-enhanced) metal tube bending forming real-time prediction method based on multi-source-input multi-task learning (MTL) is proposed.
Bearing-Based Formation Control with Optimal Motion Trajectory
We form and solve a nonlinear optimization problem with the sum of path lengths of the agent trajectories as the objective and subject to the original equilibria and global convergence conditions for formation control.
Fact-level Extractive Summarization with Hierarchical Graph Mask on BERT
We also introduce a hierarchical structure, which incorporates the multi-level of granularities of the textual information into the model.
Query-aware Tip Generation for Vertical Search
Existing work on tip generation does not take query into consideration, which limits the impact of tips in search scenarios.
MSnet: A BERT-based Network for Gendered Pronoun Resolution
In stage 1 of the gendered pronoun resolution task, a variant of this model, trained in the fine-tuning approach, reduced the multi-class logarithmic loss to 0. 3033 in the 5-fold cross-validation of training set and 0. 2795 in testing set.
