Search Results for author:
Found 313 papers, 190 papers with code
S^4-Tuning: A Simple Cross-lingual Sub-network Tuning Method
The emergence of multilingual pre-trained language models makes it possible to adapt to target languages with only few labeled examples. However, vanilla fine-tuning tends to achieve degenerated and unstable results, owing to the Language Interference among different languages, and Parameter Overload under the few-sample transfer learning scenarios. To address two problems elegantly, we propose S^4-Tuning, a Simple Cross-lingual Sub-network Tuning method.
Good Visual Guidance Make A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction
Multimodal named entity recognition and relation extraction (MNER and MRE) is a fundamental and crucial branch in information extraction.
PALM: Pre-training an Autoencoding\&Autoregressive Language Model for Context-conditioned Generation
An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.
Abstractive Text Summarization
Conversational Response Generation
+9
Bilingual Methods for Adaptive Training Data Selection for Machine Translation
In this paper, we propose a new data selection method which uses semi-supervised convolutional neural networks based on bitokens (Bi-SSCNNs) for training machine translation systems from a large bilingual corpus.
A Unified Encoding of Structures in Transition Systems
Transition systems usually contain various dynamic structures (e. g., stacks, buffers).
Rethinking Denoised Auto-Encoding in Language Pre-Training
Pre-trained self-supervised models such as BERT have achieved striking success in learning sequence representations, especially for natural language processing.
Word Reordering for Zero-shot Cross-lingual Structured Prediction
Adapting word order from one language to another is a key problem in cross-lingual structured prediction.
TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence
Recognizing that the social world follows a distinct timeline and requires a richer blend of cognitive modes (from intuitive reactions (System 1) and surface-level thinking to deliberate thinking (System 2)) than mathematics, which primarily relies on System 2 cognition (careful, step-by-step reasoning), we introduce Temporal-aware Hierarchical Cognitive Reinforcement Learning (TimeHC-RL) for enhancing LLMs' social intelligence.
Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns
Therefore, PRMs are required to identify errors under various reasoning patterns during the reasoning process.
ChARM: Character-based Act-adaptive Reward Modeling for Advanced Role-Playing Language Agents
Role-Playing Language Agents (RPLAs) aim to simulate characters for realistic and engaging human-computer interactions.
VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning
As RL has been proven to be beneficial for model reasoning, we introduce VRAG-RL, a novel RL framework tailored for complex reasoning across visually rich information.
WebDancer: Towards Autonomous Information Seeking Agency
We instantiate this framework in a web agent based on the ReAct, WebDancer.
Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration
However, the limited context window of LLMs obstructs scaling the amount of external knowledge input, prohibiting further improvement, especially for tasks requiring significant amount of external knowledge.
OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction
Role-Playing Agents (RPAs), benefiting from large language models, is an emerging interactive AI system that simulates roles or characters with diverse personalities.
MASKSEARCH: A Universal Pre-Training Framework to Enhance Agentic Search Capability
In the pre-training stage, we introduce the Retrieval Augmented Mask Prediction (RAMP) task, where the model learns to leverage search tools to fill masked spans on a large number of pre-training data, thus acquiring universal retrieval and reasoning capabilities for LLMs.
Marginal Fairness: Fair Decision-Making under Risk Measures
This paper introduces marginal fairness, a new individual fairness notion for equitable decision-making in the presence of protected attributes such as gender, race, and religion.
QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning
To bridge this gap, we first formalize the paradigm of long-context reasoning RL, and identify key challenges in suboptimal training efficiency and unstable optimization process.
VLM-R$^3$: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought
Recently, reasoning-based MLLMs have achieved a degree of success in generating long-form textual reasoning chains.
Mobile-Agent-V: A Video-Guided Approach for Effortless and Efficient Operational Knowledge Injection in Mobile Automation
The exponential rise in mobile device usage necessitates streamlined automation for effective task management, yet many AI frameworks fall short due to inadequate operational expertise.
SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization
SoLoPO is compatible with mainstream preference optimization algorithms, while substantially improving the efficiency of data construction and training processes.
WorldPM: Scaling Human Preference Modeling
Motivated by scaling laws in language modeling that demonstrate how test loss scales as a power law with model and dataset sizes, we find that similar laws exist in preference modeling.
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
Gating mechanisms have been widely utilized, from early models like LSTMs and Highway Networks to recent state space models, linear attention, and also softmax attention.
ZeroSearch: Incentivize the Search Capability of LLMs without Searching
To address these challenges, we introduce ZeroSearch, a novel RL framework that incentivizes the capabilities of LLMs to use a real search engine with simulated searches during training.
Adaptive Thinking via Mode Policy Optimization for Social Language Agents
To address this, we propose an $\textbf{A}$daptive $\textbf{M}$ode $\textbf{L}$earning ($\textbf{AML}$) framework in this paper, aiming to improve the adaptive thinking ability of language agents in dynamic social interactions.
PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts
In this paper, we introduce PolyMath, a multilingual mathematical reasoning benchmark covering 18 languages and 4 easy-to-hard difficulty levels.
Supervised Optimism Correction: Be Confident When LLMs Are Sure
In this work, we establish a novel theoretical connection between supervised fine-tuning and offline reinforcement learning under the token-level Markov decision process, revealing that large language models indeed learn an implicit $Q$-function for inference.
SynWorld: Virtual Scenario Synthesis for Agentic Action Knowledge Refinement
In the interaction between agents and their environments, agents expand their capabilities by planning and executing actions.
Agentic Knowledgeable Self-awareness
Our experiments demonstrate that KnowSelf can outperform various strong baselines on different tasks and models with minimal use of external knowledge.
Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute
Recent advancements in software engineering agents have demonstrated promising capabilities in automating program improvements.
AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization
Specifically, we first design mapping function between models to apply model merging on MLLMs with different architecture.
Long Is More Important Than Difficult for Training Reasoning Models
Second, we identify a scaling law on reasoning length, showing that model performance increases in a log-linear fashion as the reasoning data length grows.
WritingBench: A Comprehensive Benchmark for Generative Writing
Recent advancements in large language models (LLMs) have significantly enhanced text generation capabilities, yet evaluating their performance in generative writing remains a challenge.
AutoHete: An Automatic and Efficient Heterogeneous Training System for LLMs
Transformer-based large language models (LLMs) have demonstrated exceptional capabilities in sequence modeling and text generation, with improvements scaling proportionally with model size.
Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference
To mitigate the dependence on retrieval and simultaneously maintain, or even improve, the performance benefits provided by retrieval, we propose a method to detect the knowledge boundary of VLLMs, allowing for more efficient use of techniques like RAG.
AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models
While logical reasoning evaluation of Large Language Models (LLMs) has attracted significant attention, existing benchmarks predominantly rely on multiple-choice formats that are vulnerable to random guessing, leading to overestimated performance and substantial performance fluctuations.
Mobile-Agent-V: Learning Mobile Device Operation Through Video-Guided Multi-Agent Collaboration
The rapid increase in mobile device usage necessitates improved automation for seamless task management.
Enhancing Language Multi-Agent Learning with Multi-Agent Credit Re-Assignment for Interactive Environment Generalization
To address these issues, we propose CollabUIAgents, a multi-agent reinforcement learning framework with a novel multi-agent credit re-assignment (CR) strategy, assigning process rewards with LLMs rather than environment-specific rewards and learning with synthesized preference data, in order to foster generalizable, collaborative behaviors among the role-free agents' policies.
PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC
From the decision-making perspective, to handle complex user instructions and interdependent subtasks more effectively, we propose a hierarchical multi-agent collaboration architecture that decomposes decision-making processes into Instruction-Subtask-Action levels.
EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning
Large Language Models (LLMs) have shown impressive reasoning capabilities in well-defined problems with clear solutions, such as mathematics and coding.
LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs - No Silver Bullet for LC or RAG Routing
However, the advancements in context window size for LLMs offer an alternative approach, raising the question of whether RAG remains necessary for effectively handling external knowledge.
IU4Rec: Interest Unit-Based Product Organization and Recommendation for E-Commerce Platform
These IUs are then integrated into the Recommendation system, delivering both product and technological innovations.
Multilingual Non-Autoregressive Machine Translation without Knowledge Distillation
To this end, we propose an M-DAT approach to non-autoregressive multilingual machine translation.
Qwen2.5-1M Technical Report
By leveraging our inference framework, the Qwen2. 5-1M models achieve a remarkable 3x to 7x prefill speedup in scenarios with 1 million tokens of context.
RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques
In this work, we introduce a new benchmark designed to assess the critique capabilities of LLMs.
FishBargain: An LLM-Empowered Bargaining Agent for Online Fleamarket Platform Sellers
Different from traditional Business-to-Consumer e-commerce platforms~(e. g., Amazon), online fleamarket platforms~(e. g., Craigslist) mainly focus on individual sellers who are lack of time investment and business proficiency.
Debate Helps Weak-to-Strong Generalization
Specifically, we investigate ways of improving human supervision with a strong pretrained model and then supervise the strong model with enhanced weak human supervision.
Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks
By hierarchical, we mean an explicit separation of high-level planning and low-level action execution.
OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking
Machine writing with large language models often relies on retrieval-augmented generation.
Unsupervised Query Routing for Retrieval Augmented Generation
This evaluation enables the decision of the most suitable search engine for a given query.
WebWalker: Benchmarking LLMs in Web Traversal
Extensive experimental results show that WebWalkerQA is challenging and demonstrates the effectiveness of RAG combined with WebWalker, through the horizontal and vertical integration in real-world scenarios.
Enabling Scalable Oversight via Self-Evolving Critic
Despite their remarkable performance, the development of Large Language Models (LLMs) faces a critical challenge in scalable oversight: providing effective feedback for tasks where human evaluation is difficult or where LLMs outperform humans.
OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment across Language with Real-time Self-Aware Emotional Speech Synthesis
In the alignment phase, a pre-trained speech model is further trained on text-image tasks to generalize from vision to speech in a (near) zero-shot manner, outperforming models trained on tri-modal datasets.
SDPO: Segment-Level Direct Preference Optimization for Social Agents
To address these limitations, we propose Segment-Level Direct Preference Optimization (SDPO), which focuses on specific key segments within interactions to optimize multi-turn agent behavior while minimizing training noise.
Unfolding the Headline: Iterative Self-Questioning for News Retrieval and Timeline Summarization
In the fast-changing realm of information, the capacity to construct coherent timelines from extensive event-related content has become increasingly significant and challenging.
Qwen2.5 Technical Report
In addition, for hosted solutions, the proprietary models currently include two mixture-of-experts (MoE) variants: Qwen2. 5-Turbo and Qwen2. 5-Plus, both available from Alibaba Cloud Model Studio.
Ranked #7 on
on GPQA
DISC: Plug-and-Play Decoding Intervention with Similarity of Characters for Chinese Spelling Check
One key characteristic of the Chinese spelling check (CSC) task is that incorrect characters are usually similar to the correct ones in either phonetics or glyph.
STORM: A Spatio-Temporal Factor Model Based on Dual Vector Quantized Variational Autoencoders for Financial Trading
In financial trading, factor models are widely used to price assets and capture excess returns from mispricing.
DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling
Despite the large volumes of dialogue-related studies, there is a lack of benchmark that encompasses comprehensive dialogue elements, which hinders precise modeling, generation and systematic evaluation.
LLMs as Continuous Learners: Improving the Reproduction of Defective Code in Software Issues
Reproducing buggy code is the first and crucially important step in issue resolving, as it aids in identifying the underlying problems and validating that generated patches resolve the problem.
SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization
This forces the model to carefully understand the demonstration images and establish a relationship between the images and the symbols to answer questions correctly.
P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs
Recent advancements in large language models (LLMs) showcase varied multilingual capabilities across tasks like translation, code generation, and reasoning.
Exploring Knowledge Boundaries in Large Language Models for Retrieval Judgment
Actually, we find that the impact of RAG on the question answering capabilities of LLMs can be categorized into three groups: beneficial, neutral, and harmful.
IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization
In the realm of large language models (LLMs), the ability of models to accurately follow instructions is paramount as more agents and applications leverage LLMs for construction, where the complexity of instructions are rapidly increasing.
Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of "dynamic" questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers.
Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement
The results demonstrate that Lingma SWE-GPT 72B successfully resolves 30. 20% of the GitHub issues, marking a significant improvement in automatic issue resolution (22. 76% relative improvement compared to Llama 3. 1 405B), approaching the performance of closed-source models (31. 80\% issues of GPT-4o resolved).
EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations
These evaluations help practitioners select superior LLMs in specific domains and discover the shortcomings of existing LLMs.
An Adaptive Framework for Generating Systematic Explanatory Answer in Online Q&A Platforms
The naive RAG models, although effective in information retrieval, struggle with complex questions that require comprehensive and in-depth answers.
IPL: Leveraging Multimodal Large Language Models for Intelligent Product Listing
To this end, we develop IPL, an Intelligent Product Listing tool tailored to generate descriptions using various product attributes such as category, brand, color, condition, etc.
On the Role of Attention Heads in Large Language Model Safety
In light of this, recent research on safety mechanisms has emerged, revealing that when safety representations or component are suppressed, the safety capability of LLMs are compromised.
StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization
Retrieval-augmented generation (RAG) is a key means to effectively enhance large language models (LLMs) in many knowledge-based tasks.
Benchmarking Agentic Workflow Generation
Large Language Models (LLMs), with their exceptional ability to handle a wide range of tasks, have driven significant advancements in tackling reasoning and planning tasks, wherein decomposing complex problems into executable workflows is a crucial step in this process.
A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction Based on Large Language Models
This work proposes a simple training-free prompt-free approach to leverage large language models (LLMs) for the Chinese spelling correction (CSC) task, which is totally different from all previous CSC approaches.
Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?
Code completion, a key downstream task in code generation, is one of the most frequent and impactful methods for enhancing developer productivity in software development.
In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks
To address this, inspired by transfer learning, we propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks.
The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trends
In the era of large language models (LLMs), a vast amount of conversation logs will be accumulated thanks to the rapid development trend of language UI.
Qwen2.5-Coder Technical Report
In this report, we introduce the Qwen2. 5-Coder series, a significant upgrade from its predecessor, CodeQwen1. 5.
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
This framework iteratively improve data quality through a refined combination of fine-grained perception, cognitive reasoning, and interaction evolution, generating a more complex and diverse image-text instruction dataset that empowers MLLMs with enhanced capabilities.
mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding
Multimodel Large Language Models(MLLMs) have achieved promising OCR-free Document Understanding performance by increasing the supported resolution of document images.
document understanding
Optical Character Recognition (OCR)
+1
Platypus: A Generalized Specialist Model for Reading Text in Various Forms
Recently, generalist models (such as GPT-4V), trained on tremendous data in a unified way, have shown enormous potential in reading text in various scenarios, but with the drawbacks of limited accuracy and low efficiency.
MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model
This paper presents MaVEn, an innovative Multi-granularity Visual Encoding framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning.
Ranked #205 on
Visual Question Answering
on MM-Vet
Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion for Efficient Inference Intervention in Large Language Model
Transformer-based large language models (LLMs) exhibit limitations such as generating unsafe responses, unreliable reasoning, etc.
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models
Multi-modal Large Language Models (MLLMs) have demonstrated remarkable capabilities in executing instructions for a variety of single-image tasks.
Ranked #7 on
Visual Question Answering (VQA)
on VLM2-Bench
mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained in a native 8192-token context (longer than 512 of previous multilingual encoders).
Knowledge Mechanisms in Large Language Models: A Survey and Perspective
Understanding knowledge mechanisms in Large Language Models (LLMs) is crucial for advancing towards trustworthy AGI.
MIBench: Evaluating Multimodal Large Language Models over Multiple Images
However, most existing MLLMs and benchmarks primarily focus on single-image input scenarios, leaving the performance of MLLMs when handling realistic multiple images underexplored.
Visual Text Generation in the Wild
However, it is still challenging to render high-quality text images in real-world scenarios, as three critical criteria should be satisfied: (1) Fidelity: the generated text images should be photo-realistic and the contents are expected to be the same as specified in the given conditions; (2) Reasonability: the regions and contents of the generated text should cohere with the scene; (3) Utility: the generated text images can facilitate related tasks (e. g., text detection and recognition).
Qwen2 Technical Report
This report introduces the Qwen2 series, the latest addition to our large language models and large multimodal models.
Ranked #3 on
Arithmetic Reasoning
on GSM8K
(using extra training data)
Retrieved In-Context Principles from Previous Mistakes
In RICP, the teacher model analyzes mistakes from the student model to generate reasons and insights for preventing similar mistakes.
ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification Questions
To address this task, we propose ProductAgent, a conversational information seeking agent equipped with abilities of strategic clarification question generation and dynamic product retrieval.
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
Long-context modeling capabilities have garnered widespread attention, leading to the emergence of Large Language Models (LLMs) with ultra-context windows.
FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents
Motivated by this, we formalize different formats of workflow knowledge and present FlowBench, the first benchmark for workflow-guided planning.
Query Routing for Homogeneous Tools: An Instantiation in the RAG Scenario
Current research on tool learning primarily focuses on selecting the most effective tool from a wide array of options, often overlooking cost-effectiveness, a crucial factor in human problem-solving.
Supportiveness-based Knowledge Rewriting for Retrieval-augmented Language Modeling
Retrieval-augmented language models (RALMs) have recently shown great potential in mitigating the limitations of implicit knowledge in LLMs, such as untimely updating of the latest expertise and unreliable retention of long-tail knowledge.
How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States
Large language models (LLMs) rely on safety alignment to avoid responding to malicious user inputs.
Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
However, the two major navigation challenges in mobile device operation tasks, task progress navigation and focus content navigation, are significantly complicated under the single-agent architecture of existing work.
Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration
This paper presents Alibaba LingmaAgent, a novel Automated Software Engineering method designed to comprehensively understand and utilize whole software repositories for issue resolution.
DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
Our experiments reveal these LLMs' coding abilities in real-world code repositories.
Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment
Effectively aligning Large Language Models (LLMs) with human-centric values while preventing the degradation of abilities acquired through Pre-training and Supervised Fine-tuning (SFT) poses a central challenge in Reinforcement Learning from Human Feedback (RLHF).
Agent Planning with World Knowledge Model
Imitating humans' mental world knowledge model which provides global prior knowledge before the task and maintains local dynamic knowledge during the task, in this paper, we introduce parametric World Knowledge Model (WKM) to facilitate agent planning.
RaFe: Ranking Feedback Improves Query Rewriting for RAG
As Large Language Models (LLMs) and Retrieval Augmentation Generation (RAG) techniques have evolved, query rewriting has been widely incorporated into the RAG system for downstream tasks like open-domain QA.
WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
In WISE, we design a dual parametric memory scheme, which consists of the main memory for the pretrained knowledge and a side memory for the edited knowledge.
Why You Should Not Trust Interpretations in Machine Learning: Adversarial Attacks on Partial Dependence Plots
The adoption of artificial intelligence (AI) across industries has led to the widespread use of complex black-box models and interpretation tools for decision making.
TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning
Charts are important for presenting and explaining complex data relationships.
A Survey on Self-Evolution of Large Language Models
To address this issue, self-evolution approaches that enable LLM to autonomously acquire, refine, and learn from experiences generated by the model itself are rapidly growing.
Exploring Key Point Analysis with Pairwise Generation and Graph Partitioning
Our objective is to train a generative model that can simultaneously provide a score indicating the presence of shared key point between a pair of arguments and generate the shared key point.
Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts
With our method, the origin language models can cover several times longer contexts while keeping the computing requirements close to the baseline.
Fine-Tuning Language Models with Reward Learning on Policy
RLHF contains three steps, i. e., human preference collecting, reward learning, and policy optimization, which are usually performed serially.
OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition
Recently, visually-situated text parsing (VsTP) has experienced notable advancements, driven by the increasing demand for automated document understanding and the emergence of Generative Large Language Models (LLMs) capable of processing document-based questions.
ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomy
In WebShop, the 1-shot performance of the A$^3$T agent matches human average, and 4 rounds of iterative refinement lead to the performance approaching human experts.
SocialBench: Sociality Evaluation of Role-Playing Conversational Agents
In this paper, we introduce SocialBench, the first benchmark designed to systematically evaluate the sociality of role-playing conversational agents at both individual and group levels of social interactions.
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding
In this work, we emphasize the importance of structure information in Visual Document Understanding and propose the Unified Structure Learning to boost the performance of MLLMs.
Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment
Additionally, the concept of diversity for prompts can be more complex than responses that are typically quantified by single digits.
Improving Cross-lingual Representation for Semantic Retrieval with Code-switching
Semantic Retrieval (SR) has become an indispensable part of the FAQ system in the task-oriented question-answering (QA) dialogue scenario.
Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training
In vision-language pre-training (VLP), masked image modeling (MIM) has recently been introduced for fine-grained cross-modal alignment.
Effective Two-Stage Knowledge Transfer for Multi-Entity Cross-Domain Recommendation
In the end, the extracted common knowledge is adopted for target entity model training.
Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark
To promote the improvement of Chinese LLMs' ability to answer dynamic questions, in this paper, we introduce CDQA, a Chinese Dynamic QA benchmark containing question-answer pairs related to the latest news on the Chinese Internet.
Training-Free Long-Context Scaling of Large Language Models
The ability of Large Language Models (LLMs) to process and generate coherent text is markedly weakened when the number of input tokens exceeds their pretraining length.
Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval
In this work, we propose the UNIFY framework, which learns lexicon representations to capture fine-grained semantics and combines the strengths of latent and lexicon representations for video-text retrieval.
Budget-Constrained Tool Learning with Planning
Despite intensive efforts devoted to tool learning, the problem of budget-constrained tool learning, which focuses on resolving user queries within a specific budget constraint, has been widely overlooked.
Self-Retrieval: End-to-End Information Retrieval with One Large Language Model
However, current interactions between IR systems and LLMs remain limited, with LLMs merely serving as part of components within IR systems, and IR systems being constructed independently of LLMs.
PANDA: Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs
While Large language models (LLMs) have demonstrated considerable capabilities across various natural language tasks, they often fall short of the performance achieved by domain-specific state-of-the-art models.
Model Composition for Multimodal Large Language Models
In this paper, we propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model.
Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion
With the bloom of Large Language Models (LLMs), Multimodal Large Language Models (MLLMs) that incorporate LLMs with pre-trained vision models have recently demonstrated impressive performance across diverse vision-language tasks.
Enabling Weak LLMs to Judge Response Reliability via Meta Ranking
In model cascading, we combine open- and closed-source LLMs to achieve performance comparable to GPT-4-turbo with lower costs.
AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator
Artificial intelligence has significantly advanced healthcare, particularly through large language models (LLMs) that excel in medical question answering benchmarks.
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
To assess the performance of Mobile-Agent, we introduced Mobile-Eval, a benchmark for evaluating mobile device operations.
Small LLMs Are Weak Tool Learners: A Multi-LLM Agent
Each component is implemented by a single LLM that focuses on a specific capability and collaborates with others to accomplish the task.
Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection
Vision Transformers (ViTs) have become increasingly popular in large-scale Vision and Language Pre-training (VLP) models.
LORE++: Logical Location Regression Network for Table Structure Recognition with Pre-training
We model TSR as a logical location regression problem and propose a new TSR framework called LORE, standing for LOgical location REgression network, which for the first time regresses logical location as well as spatial location of table cells in a unified network.
DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever
Recently, substantial advancements in pre-trained vision-language models have greatly enhanced the capabilities of multi-modal dialog systems.
Unifying Structured Data as Graph for Data-to-Text Pre-Training
In this paper, we unify different types of structured data (i. e., table, key-value data, knowledge graph) into the graph format and cast different data-to-text generation tasks as graph-to-text generation.
A Comprehensive Study of Knowledge Editing for Large Language Models
In this paper, we first define the knowledge editing problem and then provide a comprehensive review of cutting-edge approaches.
Ranked #1 on
knowledge editing
on zsRE
(using extra training data)
OmniParser: A Unified Framework for Text Spotting Key Information Extraction and Table Recognition
Recently visually-situated text parsing (VsTP) has experienced notable advancements driven by the increasing demand for automated document understanding and the emergence of Generative Large Language Models (LLMs) capable of processing document-based questions.
EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data
Experimental results demonstrate the effectiveness of continual pre-training of E-commerce LLMs and the efficacy of our devised data mixing strategy.
One-Shot Learning as Instruction Data Prospector for Large Language Models
Contemporary practices in instruction tuning often hinge on enlarging data scaling without a clear strategy for ensuring data quality, inadvertently introducing noise that may compromise model performance.
Hallucination Augmented Contrastive Learning for Multimodal Large Language Model
We first analyzed the representation distribution of textual and visual tokens in MLLM, revealing two important findings: 1) there is a significant gap between textual and visual representations, indicating unsatisfactory cross-modal representation alignment; 2) representations of texts that contain and do not contain hallucinations are entangled, making it challenging to distinguish them.
Ranked #205 on
Visual Question Answering
on MM-Vet
Jointly spatial-temporal representation learning for individual trajectories
The proposed ST-GraphRL consists of three compositions: (i) a weighted directed spatial-temporal graph to explicitly construct mobility interactions in both space and time dimensions; (ii) a two-stage jointly encoder (i. e., decoupling and fusion), to learn entangled spatial-temporal dependencies by independently decomposing and jointly aggregating space and time information; (iii) a decoder guides ST-GraphRL to learn explicit mobility regularities by simulating the spatial-temporal distributions of trajectories.
Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use
Specifically, the crucial information in the context will be potentially overlooked by model when it is positioned in the trough zone of the attention waveform, leading to decreased performance.
Ranked #2 on
Trajectory Planning
on ToolBench
mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model
In this work, towards a more versatile copilot for academic paper writing, we mainly focus on strengthening the multi-modal diagram analysis ability of Multimodal LLMs.
Speculative Contrastive Decoding
Large language models~(LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias.
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Multi-modal Large Language Models (MLLMs) have demonstrated impressive instruction abilities across various open-ended tasks.
Ranked #10 on
Long-Context Understanding
on MMNeedle
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
We experiment with encoder- and decoder-based LMs, showing that: (1) SFT delta parameter value ranges are typically small (within 0. 002) with extreme redundancy, and DARE can effortlessly eliminate 90% or even 99% of them; (2) DARE can merge multiple task-specific LMs into one LM with diverse capabilities.
Improving Factual Consistency of Text Summarization by Adversarially Decoupling Comprehension and Embellishment Abilities of LLMs
Despite the recent progress in text summarization made by large language models (LLMs), they often generate summaries that are factually inconsistent with original articles, known as "hallucinations" in text generation.
Diversify Question Generation with Retrieval-Augmented Style Transfer
Given a textual passage and an answer, humans are able to ask questions with various expressions, but this ability is still challenging for most question generation (QG) systems.
Improving Seq2Seq Grammatical Error Correction via Decoding Interventions
In this paper, we propose a unified decoding intervention framework that employs an external critic to assess the appropriateness of the token to be generated incrementally, and then dynamically influence the choice of the next token.
Ranked #1 on
Grammatical Error Correction
on MuCGEC
Improving Question Generation with Multi-level Content Planning
Previous studies have suggested that key phrase selection is essential for question generation (QG), yet it is still challenging to connect such disjointed phrases into meaningful questions, particularly for long context.
FactCHD: Benchmarking Fact-Conflicting Hallucination Detection
Despite their impressive generative capabilities, LLMs are hindered by fact-conflicting hallucinations in real-world applications.
Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection
Out-of-distribution (OOD) detection is essential for reliable and trustworthy machine learning.
Constructive Large Language Models Alignment with Diverse Feedback
This limitation leads to suboptimal performance, even when ample training data is available.
UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
Text is ubiquitous in our visual world, conveying crucial information, such as in documents, websites, and everyday photographs.
Editing Personality for Large Language Models
This task seeks to adjust the models' responses to opinion-related questions on specified topics since an individual's personality often manifests in the form of their expressed opinions, thereby showcasing different personality traits.
Qwen Technical Report
Large language models (LLMs) have revolutionized the field of artificial intelligence, enabling natural language processing tasks that were previously thought to be exclusive to humans.
Ranked #3 on
Multi-Label Text Classification
on CC3M-TagMask
VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialogue
Visually-grounded dialog systems, which integrate multiple modes of communication such as text and visual inputs, have become an increasingly popular area of investigation.
Geo-Encoder: A Chunk-Argument Bi-Encoder Framework for Chinese Geographic Re-Ranking
Chinese geographic re-ranking task aims to find the most relevant addresses among retrieved candidates, which is crucial for location-related services such as navigation maps.
ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Models
Large language models (LLMs) have recently demonstrated remarkable capabilities to comprehend human intentions, engage in reasoning, and design planning-like behavior.
SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understanding
However, LLMs are sometimes too footloose for natural language understanding (NLU) tasks which always have restricted output and input format.
EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commerce
EcomInstruct scales up the data size and task diversity by constructing atomic tasks with E-commerce basic data types, such as product information, user reviews.
A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment
Training large language models (LLMs) with open-domain instruction data has yielded remarkable success in aligning to end tasks and human preferences.
Wider and Deeper LLM Networks are Fairer LLM Evaluators
Each perspective corresponds to the role of a specific LLM neuron in the first layer.
CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility
In this paper, we present CValues, the first Chinese human values evaluation benchmark to measure the alignment ability of LLMs in terms of both safety and responsibility criteria.
BUS:Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarization
Specifically, We incorporate a Text-Semantics-Aware Patch Selector (TSPS) into the ViT backbone to perform a coarse-grained visual token extraction and then attach a flexible Transformer-based Patch Abstraction Decoder (PAD) upon the backbone for top-level visual abstraction.
DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering
Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability.
PolyLM: An Open Source Polyglot Large Language Model
Large language models (LLMs) demonstrate remarkable ability to comprehend, reason, and generate following nature language instructions.
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
Nevertheless, without in-domain training, these models tend to ignore fine-grained OCR features, such as sophisticated tables or large blocks of text, which are essential for OCR-free document understanding.
Improving Text Matching in E-Commerce Search with A Rationalizable, Intervenable and Fast Entity-Based Relevance Model
We also show that pretraining the QE module with auto-generated QE data from user logs can further improve the overall performance.
Preference Ranking Optimization for Human Alignment
In this manner, PRO effectively transforms human alignment into aligning the probability ranking of n responses generated by LLM with the preference ranking of humans towards these responses.
Unified Language Representation for Question Answering over Text, Tables, and Images
When trying to answer complex questions, people often rely on multiple sources of information, such as visual, textual, and tabular data.
DiffDTM: A conditional structure-free framework for bioactive molecules generation targeted for dual proteins
However, molecule generation targeted for dual protein targets still faces formidable challenges including protein 3D structure data requisition for model training, auto-regressive sampling, and model generalization for unseen targets.
CATS: A Pragmatic Chinese Answer-to-Sequence Dataset with Large Scale and High Quality
To alleviate these limitations, in this paper, we present CATS, a pragmatic Chinese answer-to-sequence dataset with large scale and high quality.
Universal Information Extraction with Meta-Pretrained Self-Retrieval
Inspired by the fact that large amount of knowledge are stored in the pretrained language models~(PLM) and can be retrieved explicitly, in this paper, we propose MetaRetriever to retrieve task-specific knowledge from PLMs to enhance universal IE.
Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks
In addition, to facilitate a comprehensive evaluation of video-language models, we carefully build the largest human-annotated Chinese benchmarks covering three popular video-language tasks of cross-modal retrieval, video captioning, and video category classification.
Multimodal Recommendation Dialog with Subjective Preference: A New Challenge and Benchmark
Existing multimodal task-oriented dialog data fails to demonstrate the diverse expressions of user subjective preferences and recommendation acts in the real-life shopping scenario.
NaSGEC: a Multi-Domain Chinese Grammatical Error Correction Dataset from Native Speaker Texts
We introduce NaSGEC, a new dataset to facilitate research on Chinese grammatical error correction (CGEC) for native speaker texts from multiple domains.
PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts
It utilizes a combination of several fundamental experts to accommodate multiple dialogue-related tasks and can be pre-trained using limited dialogue and extensive non-dialogue multi-modal data.
Ranked #1 on
Response Generation
on SIMMC2.0
Optimizing Non-Autoregressive Transformers with Contrastive Learning
In this paper, we propose to ease the difficulty of modality learning via sampling from the model distribution instead of the data distribution.
SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents
SpokenWOZ further incorporates common spoken characteristics such as word-by-word processing and reasoning in spoken language.
Iterative Forward Tuning Boosts In-Context Learning in Language Models
Despite the advancements in in-context learning (ICL) for large language models (LLMs), current research centers on specific prompt engineering, such as demonstration selection, with the expectation that a single iteration of demonstrations processing can generalize effectively to a given test sample.
Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment
In this paper, we propose Speech-text dialog Pre-training for spoken dialog understanding with ExpliCiT cRoss-Modal Alignment (SPECTRA), which is the first-ever speech-text dialog pre-training model.
Ranked #2 on
Multimodal Sentiment Analysis
on CMU-MOSI
(Acc-2 metric, using extra
training data)
cross-modal alignment
Emotion Recognition in Conversation
+2
Causal Document-Grounded Dialogue Pre-training
To tackle this issue, we are the first to present a causally-complete dataset construction strategy for building million-level DocGD pre-training corpora.
Knowledge Rumination for Pre-trained Language Models
Previous studies have revealed that vanilla pre-trained language models (PLMs) lack the capacity to handle knowledge-intensive NLP tasks alone; thus, several works have attempted to integrate external knowledge into PLMs.
Distinguish Before Answer: Generating Contrastive Explanation as Knowledge for Commonsense Question Answering
Existing knowledge-enhanced methods have achieved remarkable results in certain QA tasks via obtaining diverse knowledge from different knowledge bases.
Domain Incremental Lifelong Learning in an Open World
Specifically, we dedicate task-level prompts to capture task-specific knowledge to retain high LL performances and maintain instance-level prompts to learn knowledge shared across input samples to improve the model's generalization performance.
Long-Tailed Question Answering in an Open World
A retrieve-then-rerank frame is further introduced to select in-context examples, which guild the LM to generate text that express knowledge for QA tasks.
GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmark
With a fast developing pace of geographic applications, automatable and intelligent models are essential to be designed to handle the large volume of information.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Cross-modal contrastive learning in vision language pretraining (VLP) faces the challenge of (partial) false negatives.
DAMO-NLP at SemEval-2023 Task 2: A Unified Retrieval-augmented System for Multilingual Named Entity Recognition
Also, we discover that the limited context length causes the retrieval knowledge to be invisible to the model.
Multilingual Named Entity Recognition
named-entity-recognition
+4
Out-of-Domain Intent Detection Considering Multi-Turn Dialogue Contexts
Out-of-Domain (OOD) intent detection is vital for practical dialogue systems, and it usually requires considering multi-turn dialogue contexts.
A Survey on Out-of-Distribution Detection in NLP
Out-of-distribution (OOD) detection is essential for the reliable and safe deployment of machine learning systems in the real world.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
+1
Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
Our emphasis on database values highlights the new challenges of dirty database contents, external knowledge between NL questions and database contents, and SQL efficiency, particularly in the context of massive databases.
Ranked #1 on
Text-To-SQL
on BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQL Evaluation)
(Execution Accurarcy (Human) metric)
Unsupervised Dialogue Topic Segmentation with Topic-aware Utterance Representation
Dialogue Topic Segmentation (DTS) plays an essential role in a variety of dialogue modeling tasks.
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality
Our code, pre-trained model, instruction-tuned models, and evaluation set are available at https://github. com/X-PLUG/mPLUG-Owl.
Ranked #1 on
Visual Question Answering (VQA)
on HallusionBench
(Question Pair Acc metric)
Visual Question Answering (VQA)
Zero-Shot Video Question Answer
Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation
Non-AutoRegressive (NAR) text generation models have drawn much attention because of their significantly faster decoding speed and good generation quality in machine translation.
VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning
Recent studies have demonstrated the potential of cross-lingual transferability by training a unified Transformer encoder for multiple languages.
ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human
In this paper, we present ChatPLUG, a Chinese open-domain dialogue system for digital human applications that instruction finetunes on a wide range of dialogue tasks in a unified internet-augmented format.
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
(2) How can we enhance LLMs' ability to utilize tools?
RRHF: Rank Responses to Align Language Models with Human Feedback without tears
Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models.
