Search Results for author:
Found 28 papers, 14 papers with code
Baichuan-M1: Pushing the Medical Capability of Large Language Models
The current generation of large language models (LLMs) is typically designed for broad, general-purpose applications, while domain-specific LLMs, especially in vertical fields like medicine, remain relatively scarce.
LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation
To address these challenges, we propose Long Context Pre-training with Restoration Distillation (LongReD), a novel approach designed to mitigate short-text performance degradation through minimizing the distribution discrepancy between the extended and original models.
Virgo: A Preliminary Exploration on Reproducing o1-like MLLM
Moreover, it seems that such textual reasoning data can be even more effective than visual reasoning data in eliciting the slow-thinking capacities of MLLMs.
KV Shifting Attention Enhances Language Modeling
In order to more efficiently implement the ability of the model's induction, we revisit the induction heads mechanism and proposed a KV shifting attention.
LLaSA: Large Language and Structured Data Assistant
Specifically, we represent various types of structured data in a unified hypergraph format, and use self-supervised learning to pretrain a hypergraph encoder, and a G-Former compressing encoded hypergraph representations with cross-attention.
Beyond Filtering: Adaptive Image-Text Quality Enhancement for MLLM Pretraining
A critical factor in training MLLMs is the quality of image-text pairs within multimodal pretraining datasets.
Exploring the Design Space of Visual Context Representation in Video MLLMs
Then, we explore the scaling effects in frame selection and token selection respectively, and fit the corresponding function curve by conducting extensive empirical experiments.
Extracting and Transferring Abilities For Building Multi-lingual Ability-enhanced Large Language Models
In the extraction stage, we firstly locate key neurons that are highly related to specific abilities, and then employ them to extract the transferable ability-specific weights.
BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline
The general capabilities of Large Language Models (LLM) highly rely on the composition and selection on extensive pretraining datasets, treated as commercial secrets by several institutions.
Towards Event-oriented Long Video Understanding
Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench.
Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
Synthetic data has been proposed as a solution to address the issue of high-quality data scarcity in the training of large language models (LLMs).
Full-ECE: A Metric For Token-level Calibration on Large Language Models
Deep Neural Networks (DNNs) excel in various domains but face challenges in providing accurate uncertainty estimates, which are crucial for high-stakes applications.
MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic
The advent of large language models (LLMs) like GPT-4 has catalyzed the exploration of multi-task learning (MTL), in which a single model demonstrates proficiency across diverse tasks.
Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs
In this paper, we propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation.
Accurate and Reliable Predictions with Mutual-Transport Ensemble
Deep Neural Networks (DNNs) have achieved remarkable success in a variety of tasks, especially when it comes to prediction accuracy.
Base of RoPE Bounds Context Length
We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, we derive that the \textit{base of RoPE bounds context length}: there is an absolute lower bound for the base value to obtain certain context length capability.
Checkpoint Merging via Bayesian Optimization in LLM Pretraining
The rapid proliferation of large language models (LLMs) such as GPT-4 and Gemini underscores the intense demand for resources during their training processes, posing significant challenges due to substantial computational and environmental costs.
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
As Large Language Models (LLMs) continue to advance in performance, their size has escalated significantly, with current LLMs containing billions or even trillions of parameters.
Baichuan 2: Open Large-scale Language Models
Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering.
T2Ranking: A large-scale Chinese Benchmark for Passage Ranking
T2Ranking comprises more than 300K queries and over 2M unique passages from real-world search engines.
ChiQA: A Large Scale Image-based Real-World Question Answering Dataset for Multi-Modal Understanding
However, in most of the public visual question answering datasets such as VQA, CLEVR, the questions are human generated that specific to the given image, such as `What color are her eyes?'.
Bridging the Gap Between Training and Inference of Bayesian Controllable Language Models
Large-scale pre-trained language models have achieved great success on natural language generation tasks.
ComQA:Compositional Question Answering via Hierarchical Graph Neural Networks
In compositional question answering, the systems should assemble several supporting evidence from the document to generate the final answer, which is more difficult than sentence-level or phrase-level QA.
SenSeNet: Neural Keyphrase Generation with Document Structure
Keyphrase Generation (KG) is the task of generating central topics from a given document or literary work, which captures the crucial information necessary to understand the content.
Tasty Burgers, Soggy Fries: Probing Aspect Robustness in Aspect-Based Sentiment Analysis
Based on the SemEval 2014 dataset, we construct the Aspect Robustness Test Set (ARTS) as a comprehensive probe of the aspect robustness of ABSA models.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
Sogou Machine Reading Comprehension Toolkit
In this paper, we present a Sogou Machine Reading Comprehension (SMRC) toolkit that can be used to provide the fast and efficient development of modern machine comprehension models, including both published models and original prototypes.
