Search Results for author:
Found 89 papers, 29 papers with code
Optimizing Compound Retrieval Systems
We focus on the optimization of compound retrieval system design which uniquely involves learning where to apply the component models and how to aggregate their predictions into a final ranking.
Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs?
While metrics available during pre-training, such as perplexity, correlate well with model performance at scaling-laws studies, their predictive capacities at a fixed model size remain unclear, hindering effective model selection and development.
Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation
Large language models (LLMs) are increasingly integral to information retrieval (IR), powering ranking, evaluation, and AI-assisted content creation.
Vertical Federated Learning in Practice: The Good, the Bad, and the Ugly
Based on these observations, we outline key research directions aimed at bridging the gap between current VFL research and real-world applications.
LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
In this work, we introduce a novel direct optimization approach for LLM alignment by drawing on established Information Retrieval (IR) principles.
MiniMax-01: Scaling Foundation Models with Lightning Attention
This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens.
Tensor Product Attention Is All You Need
Scaling language models to handle longer input sequences typically necessitates large key-value (KV) caches, resulting in substantial memory overhead during inference.
SeMi: When Imbalanced Semi-Supervised Learning Meets Mining Hard Examples
This method distinguishes the entropy differences among logits of hard and easy examples, thereby identifying hard examples and increasing the utility of unlabeled data, better addressing the imbalance problem in CISSL.
Deep Non-rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling
In this paper, we revisit deep NRSfM from two perspectives to address the limitations of current deep NRSfM methods : (1) canonicalization and (2) sequence modeling.
Scaling Image Tokenizers with Grouped Spherical Quantization
Vision tokenizers have gained a lot of attraction due to their scalability and compactness; previous works depend on old-school GAN-based hyperparameters, biased comparisons, and a lack of comprehensive analysis of the scaling behaviours.
Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures
Furthermore, we develop a reverse selection-based expert assignment (RSEA) strategy, which enables data-driven model architecture adjustment during fine-tuning by allowing domain experts to select clients that best align with their knowledge domains.
Robust Low-rank Tensor Train Recovery
We first establish the $\ell_1/\ell_2$-restricted isometry property (RIP) for Gaussian measurement operators, demonstrating that the information in the TT format tensor can be preserved using a number of measurements that grows linearly with $N$.
Federated Data-Efficient Instruction Tuning for Large Language Models
Instruction tuning helps improve pretrained large language models (LLMs) in terms of the responsiveness to human instructions, which is benefited from diversified instruction data.
Integrating Planning into Single-Turn Long-Form Text Generation
To overcome the scarcity of training data for these intermediate steps, we leverage LLMs to generate synthetic intermediate writing data such as outlines, key information and summaries from existing full articles.
Inference Scaling for Long-Context Retrieval Augmented Generation
Our observations reveal that increasing inference computation leads to nearly linear gains in RAG performance when optimally allocated, a relationship we describe as the inference scaling laws for RAG.
RRM: Robust Reward Model Training Mitigates Reward Hacking
Our RRM improves the performance of a pairwise reward model trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80. 61% to 84. 15%.
Building Math Agents with Multi-Turn Iterative Preference Learning
Recent studies have shown that large language models' (LLMs) mathematical problem-solving capabilities can be enhanced by integrating external tools, such as code interpreters, and employing multi-turn Chain-of-Thought (CoT) reasoning.
LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification
We introduce LAMPO, a novel paradigm that leverages Large Language Models (LLMs) for solving few-shot multi-class ordinal classification tasks.
Multilingual Fine-Grained News Headline Hallucination Detection
In this study, we introduce the first multilingual, fine-grained news headline hallucination detection dataset that contains over 11 thousand pairs in 5 languages, each annotated with detailed hallucination types by experts.
Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation
Unlike traditional methods, which generate two responses before obtaining the preference label, RMBoost first generates one response and selects a preference label, followed by generating the second more (or less) preferred response conditioned on the pre-selected preference label and the first response.
The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective
As LLMs and MLLMs rely on vast amounts of model parameters and data to achieve emergent capabilities, the importance of data is receiving increasingly widespread attention and recognition.
Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I
In this work, we propose two methods based on prediction-powered inference and conformal risk control that utilize computer-generated relevance annotations to place reliable confidence intervals (CIs) around IR evaluation metrics.
Scaling Laws for Linear Complexity Language Models
In this study, we present the scaling laws for linear complexity language models to establish a foundation for their scalability.
Computational and Statistical Guarantees for Tensor-on-Tensor Regression with Tensor Train Decomposition
However, the exponential growth in tensor complexity poses challenges for storage and computation in ToT regression.
PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs
Through extensive experiments on two sequence generation tasks and with various LLMs, we demonstrate the effectiveness of our proposed PLaD framework.
You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet
Linear attention mechanisms have gained prominence in causal language models due to their linear computational complexity and enhanced speed.
Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention
This eliminates the need for cumsum in the linear attention calculation.
Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective
Finally, the memory state is projected back to a low-dimensional space in the Shrink stage.
TAVGBench: Benchmarking Text to Audible-Video Generation
To support research in this field, we have developed a comprehensive Text to Audible-Video Generation Benchmark (TAVGBench), which contains over 1. 7 million clips with a total duration of 11. 8 thousand hours.
Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing
Our method takes both LLM generated relevance labels and pairwise preferences.
HGRN2: Gated Linear RNNs with State Expansion
Hierarchically gated linear RNN (HGRN, \citealt{HGRN}) has demonstrated competitive training speed and performance in language modeling while offering efficient inference.
Linear Attention Sequence Parallelism
However, for linear sequence modeling methods like linear attention, existing SP approaches do not take advantage of their right-product-first feature, resulting in sub-optimal communication efficiency and usability.
LiPO: Listwise Preference Optimization through Learning-to-Rank
In this work, we formulate the LM alignment as a \textit{listwise} ranking problem and describe the LiPO framework, where the policy can potentially learn more effectively from a ranked list of plausible responses given the prompt.
CO2: Efficient Distributed Training with Full Communication-Computation Overlap
CO2 is able to attain a high scalability even on extensive multi-node clusters constrained by very limited communication bandwidth.
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
With its ability to process tokens in linear computational complexities, linear attention, in theory, can handle sequences of unlimited length without sacrificing speed, i. e., maintaining a constant training speed for various sequence lengths with a fixed memory consumption.
Guaranteed Nonconvex Factorization Approach for Tensor Train Recovery
We first delve into the TT factorization problem and establish the local linear convergence of RGD.
Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes
Pre-trained large language models (LLMs) need fine-tuning to improve their responsiveness to natural language instructions.
Learning Multi-Pattern Normalities in the Frequency Domain for Efficient Time Series Anomaly Detection
Anomaly detection significantly enhances the robustness of cloud systems.
Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks
Enforcing orthonormal or isometric property for the weight matrices has been shown to enhance the training of deep neural networks by mitigating gradient exploding/vanishing and increasing the robustness of the learned networks.
Can Query Expansion Improve Generalization of Strong Cross-Encoder Rankers?
In this paper, we re-examine this conclusion and raise the following question: Can query expansion improve generalization of strong cross-encoder rankers?
Accelerating Toeplitz Neural Network with Constant-time Inference Complexity
On the other hand, State Space Models (SSMs) achieve lower performance than TNNs in language modeling but offer the advantage of constant inference complexity.
Predicting Text Preference Via Structured Comparative Reasoning
Comparative reasoning plays a crucial role in text preference prediction; however, large language models (LLMs) often demonstrate inconsistencies in their reasoning.
Explanation-aware Soft Ensemble Empowers Large Language Model In-context Learning
To fully unleash the power of explanations, we propose EASE, an Explanation-Aware Soft Ensemble framework to empower in-context learning with LLMs.
Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimization
Yet, existing analyses of signSGD rely on assuming that data are sampled with replacement in each iteration, contradicting the practical implementation where data are randomly reshuffled and sequentially fed into the algorithm.
PaRaDe: Passage Ranking using Demonstrations with Large Language Models
Recent studies show that large language models (LLMs) can be instructed to effectively perform zero-shot passage re-ranking, in which the results of a first stage retrieval method, such as BM25, are rated and reordered to improve relevance.
Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels
We propose to incorporate fine-grained relevance labels into the prompt for LLM rankers, enabling them to better differentiate among documents with different levels of relevance to the query and thus derive a more accurate ranking.
LARA: A Light and Anti-overfitting Retraining Approach for Unsupervised Time Series Anomaly Detection
Retraining a large neural network model with limited data is vulnerable to overfitting.
Resisting Backdoor Attacks in Federated Learning via Bidirectional Elections and Individual Perspective
Existing approaches defend against backdoor attacks in federated learning (FL) mainly through a) mitigating the impact of infected models, or b) excluding infected models.
All-pairs Consistency Learning for Weakly Supervised Semantic Segmentation
Given a pair of augmented views, our approach regularizes the activation intensities between a pair of augmented views, while also ensuring that the affinity across regions within each view remains consistent.
Ranked #16 on
Weakly-Supervised Semantic Segmentation
on COCO 2014 val
TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer
TransNormerLLM evolves from the previous linear attention architecture TransNormer by making advanced modifications that include positional embedding, linear attention acceleration, gating mechanisms, tensor normalization, and inference acceleration and stabilization.
Exploring Transformer Extrapolation
While these methods perform well on a variety of corpora, the conditions for length extrapolation have yet to be investigated.
Linearized Relative Positional Encoding
Meanwhile, it emphasizes a general paradigm for designing broadly more relative positional encoding methods that are applicable to linear transformers.
PePNet: A Periodicity-Perceived Workload Prediction Network Supporting Rare Occurrence of Heavy Workload
This impairs the prediction accuracy of neural network-based models on heavy workload.
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting
Ranking documents using Large Language Models (LLMs) by directly feeding the query and candidate documents into the prompt is an interesting and practical problem.
Learning to Rank when Grades Matter
In this paper, we conduct a rigorous study of learning to rank with grades, where both ranking performance and grade prediction performance are important.
Toeplitz Neural Network for Sequence Modeling
Sequence modeling has important applications in natural language processing and computer vision.
Query Expansion by Prompting Large Language Models
Query expansion is a widely used technique to improve the recall of search systems.
Dynamic Compressive Sensing based on RLS for Underwater Acoustic Communications
Sparse structures are widely recognized and utilized in channel estimation.
Fine-grained Audible Video Description
We explore a new task for audio-visual-language modeling called fine-grained audible video description (FAVD).
Towards Disentangling Relevance and Bias in Unbiased Learning to Rank
We give both theoretical analysis and empirical results to show the negative effects on relevance tower due to such a correlation.
Regression Compatible Listwise Objectives for Calibrated Ranking with Binary Relevance
As Learning-to-Rank (LTR) approaches primarily seek to improve ranking quality, their output scores are not scale-calibrated by design.
Proportionate Recursive Maximum Correntropy Criterion Adaptive Filtering Algorithms and their Performance Analysis
Motivated by the success of our recently proposed proportionate recursive least squares (PRLS) algorithm for sparse system identification, we propose to introduce the proportionate updating (PU) mechanism into the RMCC, leading to two sparsity-aware RMCC algorithms: the proportionate recursive MCC (PRMCC) algorithm and the combinational PRMCC (CPRMCC) algorithm.
The Devil in Linear Transformer
In this paper, we examine existing kernel-based linear transformers and identify two key issues that lead to such performance gaps: 1) unbounded gradients in the attention computation adversely impact the convergence of linear transformer models; 2) attention dilution which trivially distributes attention scores over long sequences while neglecting neighbouring structures.
Linear Video Transformer with Feature Fixation
Therefore, we propose a feature fixation module to reweight the feature importance of the query and key before computing linear attention.
RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses
Recently, substantial progress has been made in text ranking based on pretrained language models such as BERT.
Retrieval Augmentation for T5 Re-ranker using External Sources
Retrieval augmentation has shown promising improvements in different tasks.
A Validation Approach to Over-parameterized Matrix and Image Recovery
This paper studies the problem of recovering a low-rank matrix from several noisy random linear measurements.
Neural Architecture Search on Efficient Transformers and Beyond
To address this issue, we propose a new framework to find optimal architectures for efficient Transformers with the neural architecture search (NAS) technique.
Error Analysis of Tensor-Train Cross Approximation
Tensor train decomposition is widely used in machine learning and quantum physics due to its concise representation of high-dimensional tensors, overcoming the curse of dimensionality.
Vicinity Vision Transformer
Based on this observation, we present a Vicinity Attention that introduces a locality bias to vision transformers with linear complexity.
Ranked #294 on
Image Classification
on ImageNet
(Top 1 Accuracy metric)
ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference
This results in significant inference time speedups since the decoder-only architecture only needs to learn to interpret static encoder embeddings during inference.
cosFormer: Rethinking Softmax in Attention
As one of its core components, the softmax attention helps to capture long-range dependencies yet prohibits its scale-up due to the quadratic space and time complexity to the sequence length.
Ranked #6 on
D4RL
on D4RL
Transformer Memory as a Differentiable Search Index
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model.
Rank4Class: A Ranking Formulation for Multiclass Classification
Multiclass classification (MCC) is a fundamental machine learning problem of classifying each instance into one of a predefined set of classes.
Improving Neural Ranking via Lossless Knowledge Distillation
We explore a novel perspective of knowledge distillation (KD) for learning to rank (LTR), and introduce Self-Distilled neural Rankers (SDR), where student rankers are parameterized identically to their teachers.
Rank4Class: Examining Multiclass Classification through the Lens of Learning to Rank
We further demonstrate that the most popular MCC architecture in deep learning can be mathematically formulated as a LTR pipeline equivalently, with a specific set of choices in terms of ranking model architecture and loss function.
Are Pretrained Convolutions Better than Pretrained Transformers?
In the context of language models, are convolutional models competitive to Transformers when pre-trained?
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model.
Ranked #3 on
Paraphrase Identification
on Quora Question Pairs
Are Pre-trained Convolutions Better than Pre-trained Transformers?
In the context of language models, are convolutional models competitive to Transformers when pre-trained?
OmniNet: Omnidirectional Representations from Transformers
In OmniNet, instead of maintaining a strictly horizontal receptive field, each token is allowed to attend to all tokens in the entire network.
Ranked #1 on
Machine Translation
on WMT2017 Russian-English
Neural Rankers are hitherto Outperformed by Gradient Boosted Decision Trees
We first validate this concern by showing that most recent neural LTR models are, by a large margin, inferior to the best publicly available Gradient Boosted Decision Trees (GBDT) in terms of their reported ranking accuracy on benchmark datasets.
DeepKeyGen: A Deep Learning-based Stream Cipher Generator for Medical Image Encryption and Decryption
In this paper, a novel deep learning-based key generation network (DeepKeyGen) is proposed as a stream cipher generator to generate the private key, which can then be used for encrypting and decrypting of medical images.
Diversify Question Generation with Continuous Content Selectors and Question Type Modeling
However, question generation is actually a one-to-many problem, as it is possible to raise questions with different focuses on contexts and various means of expression.
Do RNN and LSTM have Long Memory?
The LSTM network was proposed to overcome the difficulty in learning long-term dependence, and has made significant advancements in applications.
Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction
Recent advances in unbiased learning to rank (LTR) count on Inverse Propensity Scoring (IPS) to eliminate bias in implicit feedback.
An Improvement of PAA on Trend-Based Approximation for Time Series
In this paper, we propose two new approaches for time series that utilize approximate trend feature information.
Multi-Task Learning for Email Search Ranking with Auxiliary Query Clustering
In this paper, we study how to obtain query type in an unsupervised fashion and how to incorporate this information into query-dependent ranking models.
An Online Learned Elementary Grouping Model for Multi-target Tracking
We introduce an online approach to learn possible elementary groups (groups that contain only two targets) for inferring high level context that can be used to improve multi-target tracking in a data-association based framework.
Efficient Online Bootstrapping for Large Scale Learning
Bootstrapping is a useful technique for estimating the uncertainty of a predictor, for example, confidence intervals for prediction.
