Search Results for author:
Found 32 papers, 26 papers with code
Matching-oriented Embedding Quantization For Ad-hoc Retrieval
In this work, we propose the Matching-oriented Product Quantization (MoPQ), where a novel objective Multinoulli Contrastive Loss (MCL) is formulated.
Lighter And Better: Towards Flexible Context Adaptation For Retrieval Augmented Generation
In this approach, the retrieved contexts are compressed into compact embeddings before being encoded by the LLMs.
Making Text Embedders Few-Shot Learners
To this end, we introduce a novel model bge-en-icl, which employs few-shot examples to produce high-quality text embeddings.
OmniGen: Unified Image Generation
In this work, we introduce OmniGen, a new diffusion model for unified image generation.
Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment
In this work, we conduct a comprehensive empirical study on a wide range of retrieval tasks, including in domain accuracy, data efficiency, zero shot generalization, lengthy retrieval, instruction based retrieval, and multi task learning.
SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking
In contrast, human brains, which contain approximately 86 billion biological neurons, exhibit significantly greater energy efficiency compared to LLMs with a similar number of parameters.
MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding
To address the above problems, we propose a new benchmark, called MLVU (Multi-task Long Video Understanding Benchmark), for the comprehensive and in-depth evaluation of LVU.
VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval
Thirdly, we introduce a multi-stage training algorithm, which first aligns the visual token embedding with the text encoder using massive weakly labeled data, and then develops multi-modal representation capability using the generated composed image-text data.
Ranked #8 on
Image Retrieval
on CIRR
SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms
We plug this elastic bi-spiking mechanism in language modeling, named SpikeLM.
Compressing Lengthy Context With UltraGist
Compressing lengthy context is a critical but technically challenging problem.
Extending Llama-3's Context Ten-Fold Overnight
We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning.
Extensible Embedding: A Flexible Multipler For LLM's Context Length
2) Strong sample efficiency of training, which enables the embedding model to be learned in a cost-effective way.
BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models
In this work, we proposeExtensible Embedding, which realizes high-quality extension of LLM's context with strong flexibility and cost-effectiveness.
BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
It can simultaneously perform the three common retrieval functionalities of embedding model: dense retrieval, multi-vector retrieval, and sparse retrieval, which provides a unified model foundation for real-world IR applications.
Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization
Extensible Tokenization stands as a midware in between of the tokenized context and the LLM, which transforms the raw token embeddings into the extensible embeddings.
Long Context Compression with Activation Beacon
In this paper, we propose Activation Beacon, a plug-in module for transformer-based LLMs that targets effective, efficient, and flexible compression of long contexts.
Making Large Language Models A Better Foundation For Dense Retrieval
LLaRA consists of two pretext tasks: EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), where the text embeddings from LLM are used to reconstruct the tokens for the input sentence and predict the tokens for the next sentence, respectively.
LM-Cocktail: Resilient Tuning of Language Models via Model Merging
Despite simplicity, LM-Cocktail is surprisingly effective: the resulted model is able to achieve a strong empirical performance in the whole scope of general tasks while preserving a superior capacity in its targeted domain.
Retrieve Anything To Augment Large Language Models
On the other hand, the task-specific retrievers lack the required versatility, hindering their performance across the diverse retrieval augmentation scenarios.
C-Pack: Packed Resources For General Chinese Embeddings
Along with our resources on general Chinese embedding, we release our data and models for English text embeddings.
RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models
It is designed to improve the quality of semantic representation where all contextualized embeddings of the pre-trained model can be leveraged.
RetroMAE v2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models
DupMAE, which targets on improving the semantic representation capacity for the contextualized embeddings of both [CLS] and ordinary tokens.
Ranked #1 on
Information Retrieval
on MS MARCO
(MRR@10 metric)
Hybrid Inverted Index Is a Robust Accelerator for Dense Retrieval
Based on comprehensive experiments on popular retrieval benchmarks, we verify that clusters and terms indeed complement each other, enabling HI$^2$ to achieve lossless retrieval quality with competitive efficiency across various index settings.
RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder
The sentence embedding is generated from the encoder's masked input; then, the original sentence is recovered based on the sentence embedding and the decoder's masked input via masked language modeling.
Ranked #1 on
Information Retrieval
on MSMARCO
Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings
We perform comprehensive explorations for the optimal conduct of knowledge distillation, which may provide useful insights for the learning of VQ based ANN index.
A Mutually Reinforced Framework for Pretrained Sentence Embeddings
Instead of relying on annotation heuristics defined by humans, it leverages the sentence representation model itself and realizes the following iterative self-supervision process: on one hand, the improvement of sentence representation may contribute to the quality of data annotation; on the other hand, more effective data annotation helps to generate high-quality positive samples, which will further improve the current sentence representation model.
Uni-Retriever: Towards Learning The Unified Embedding Based Retriever in Bing Sponsored Search
On the other hand, the capability of making high-CTR retrieval is optimized by learning to discriminate user's clicked ads from the entire corpus.
Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval
In this work, we tackle this problem with Bi-Granular Document Representation, where the lightweight sparse embeddings are indexed and standby in memory for coarse-grained candidate search, and the heavyweight dense embeddings are hosted in disk for fine-grained post verification.
LECF: Recommendation via Learnable Edge Collaborative Filtering
In this paper, we model an interaction between user and item as an edge and propose a novel CF framework, called learnable edge collaborative filtering (LECF).
GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph
The representation learning on textual graph is to generate low-dimensional embeddings for the nodes based on the individual textual features and the neighbourhood information.
Matching-oriented Product Quantization For Ad-hoc Retrieval
In this work, we propose the Matching-oriented Product Quantization (MoPQ), where a novel objective Multinoulli Contrastive Loss (MCL) is formulated.
Training Large-Scale News Recommenders with Pretrained Language Models in the Loop
Secondly, it improves the data efficiency of the training workflow, where non-informative data can be eliminated from encoding.
