Search Results for author:
Found 88 papers, 52 papers with code
MuPT: A Generative Symbolic Music Pretrained Transformer
In this paper, we explore the application of Large Language Models (LLMs) to the pre-training of music.
Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs.
CodeEditorBench: Evaluating Code Editing Capability of Large Language Models
Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability.
Long-context LLMs Struggle with Long In-context Learning
Our study reveals that long context understanding and reasoning is still a challenging task for the existing LLMs.
The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis
To address the aforementioned limitations, this paper undertakes a comprehensive comparison of model capabilities at various pretraining intermediate checkpoints.
MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions
Image retrieval, i. e., finding desired images given a reference image, inherently encompasses rich, multi-faceted search intents that are difficult to capture solely using image-based measures.
COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning
To bridge this gap, we introduce COIG-CQIA, a high-quality Chinese instruction tuning dataset.
AnyV2V: A Plug-and-Play Framework For Any Video-to-Video Editing Tasks
In the second stage, AnyV2V can plug in any existing image-to-video models to perform DDIM inversion and intermediate feature injection to maintain the appearance and motion consistency with the source video.
Reward Guided Latent Consistency Distillation
By distilling a latent consistency model (LCM) from a pre-trained teacher latent diffusion model (LDM), LCD facilitates the generation of high-fidelity images within merely 2 to 4 inference steps.
DEEP-ICL: Definition-Enriched Experts for Language Model In-Context Learning
It has long been assumed that the sheer number of parameters in large language models (LLMs) drives in-context learning (ICL) capabilities, enabling remarkable performance improvements by leveraging task-specific demonstrations.
StructLM: Towards Building Generalist Models for Structured Knowledge Grounding
Utilizing this dataset, we train a series of models, referred to as StructLM, based on the Code-LLaMA architecture, ranging from 7B to 34B parameters.
ChatMusician: Understanding and Generating Music Intrinsically with LLM
It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language.
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
However, open-source models often lack the execution capabilities and iterative refinement of advanced systems like the GPT-4 Code Interpreter.
CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models
The advancement of large language models (LLMs) has enhanced the ability to generalize across a wide range of unseen natural language processing (NLP) tasks through instruction-following.
ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation
To verify the effectiveness of our method, we propose I2V-Bench, a comprehensive evaluation benchmark for I2V generation.
Understanding the Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation
To understand how pre-training with a next-token prediction objective contributes to the emergence of such reasoning capability, we propose that we can view an LM as deriving new conclusions by aggregating indirect reasoning paths seen at pre-training time.
SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval
We further annotate the image-text pairs with two-level subset-subcategory hierarchy annotations to facilitate a more comprehensive evaluation of the baselines.
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
We introduce CMMMU, a new Chinese Massive Multi-discipline Multimodal Understanding benchmark designed to evaluate LMMs on tasks demanding college-level subject knowledge and deliberate reasoning in a Chinese context.
E^2-LLM: Efficient and Extreme Length Extension of Large Language Models
Typically, training LLMs with long context sizes is computationally expensive, requiring extensive training hours and GPU resources.
Kun: Answer Polishment for Chinese Self-Alignment with Instruction Back-Translation
In this paper, we introduce Kun, a novel approach for creating high-quality instruction-tuning datasets for large language models (LLMs) without relying on manual annotations.
Instruct-Imagen: Image Generation with Multi-modal Instruction
We introduce *multi-modal instruction* for image generation, a task representation articulating a range of generation intents with precision.
VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation
We evaluate VIESCORE on seven prominent tasks in conditional image tasks and found: (1) VIESCORE (GPT4-v) achieves a high Spearman correlation of 0. 3 with human evaluations, while the human-to-human correlation is 0. 45.
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
Existing information retrieval (IR) models often assume a homogeneous format, limiting their applicability to diverse user needs, such as searching for images with text descriptions, searching for a news article with a headline image, or finding a similar photo with a query image.
MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning.
Kosmos-G: Generating Images in Context with Multimodal Large Language Models
These limitations keep them far from the ultimate goal of "image as a foreign language in image generation."
ImagenHub: Standardizing the evaluation of conditional image generation models
Recently, a myriad of conditional image generation and editing models have been developed to serve different downstream tasks, including text-to-image generation, text-guided image editing, subject-driven image generation, control-guided image generation, etc.
TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks
To quantitatively assess our metric, we evaluate its correlation with human ratings on 5 held-in datasets, 2 held-out datasets and show that TIGERScore can achieve the open-source SoTA correlation with human ratings across these datasets and almost approaches GPT-4 evaluator.
RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models
The advent of Large Language Models (LLMs) has paved the way for complex tasks such as role-playing, which enhances user interactions by enabling models to imitate various characters.
MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response
Large Language Models (LLMs) have shown immense potential in multimodal applications, yet the convergence of textual and musical domains remains not well-explored.
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
The MAmmoTH models are trained on MathInstruct, our meticulously curated instruction tuning dataset.
Augmenting Black-box LLMs with Medical Textbooks for Clinical Question Answering
In this study, we present a system called LLMs Augmented with Medical Textbooks (LLM-AMT) designed to enhance the proficiency of LLMs in specialized domains.
LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT
We introduce LyricWhiz, a robust, multilingual, and zero-shot automatic lyrics transcription method achieving state-of-the-art performance on various lyrics transcription datasets, even in challenging genres such as rock and metal.
DreamEdit: Subject-driven Image Editing
In this work, we aspire to fill the void and propose two novel subject-driven sub-tasks, i. e., Subject Replacement and Subject Addition.
MARBLE: Music Audio Representation Benchmark for Universal Evaluation
This is evident in the limited work on deep music representations, the scarcity of large-scale datasets, and the absence of a universal and community-driven benchmark.
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing
To address this issue, we introduce MagicBrush (https://osu-nlp-group. github. io/MagicBrush/), the first large-scale, manually annotated dataset for instruction-guided real image editing that covers diverse scenarios: single-turn, multi-turn, mask-provided, and mask-free editing.
MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored.
Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models
This paper investigates the capabilities of Large Language Models (LLMs) in the context of understanding their own knowledge and measuring their uncertainty.
EDIS: Entity-Driven Image Search over Multimodal Web Content
Making image retrieval methods practical for real-world search applications requires significant progress in dataset scales, entity comprehension, and multimodal information fusion.
On the Risk of Misinformation Pollution with Large Language Models
In this paper, we comprehensively investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on information-intensive applications, particularly Open-Domain Question Answering (ODQA) systems.
Interactive Natural Language Processing
Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP, aimed at addressing limitations in existing frameworks while aligning with the ultimate goals of artificial intelligence.
TheoremQA: A Theorem-driven Question Answering dataset
We evaluate a wide spectrum of 16 large language and code models with different prompting strategies like Chain-of-Thoughts and Program-of-Thoughts.
Ranked #1 on
Natural Questions
on TheoremQA
Few-shot In-context Learning for Knowledge Base Question Answering
On GrailQA and WebQSP, our model is also on par with other fully-trained models.
DePlot: One-shot visual language reasoning by plot-to-table translation
Compared with a SOTA model finetuned on more than >28k data points, DePlot+LLM with just one-shot prompting achieves a 24. 0% improvement over finetuned SOTA on human-written queries from the task of chart QA.
Chart Question Answering
Factual Inconsistency Detection in Chart Captioning
+3
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
By combining PoT with self-consistency decoding, we can achieve SoTA performance on all math problem datasets and near-SoTA performance on financial datasets.
Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models
Conditioned diffusion models have demonstrated state-of-the-art text-to-image synthesis capacity.
Ranked #1 on
Story Visualization
on Pororo
Large Language Models are few(1)-shot Table Reasoners
Specifically, we evaluated LLMs on popular table QA and fact verification datasets like WikiTableQuestion, FetaQA, TabFact, and FEVEROUS and found that LLMs are competent at complex reasoning over table structures, though these models are not pre-trained on any table corpus.
Explanations from Large Language Models Make Small Reasoners Better
Integrating free-text explanations to in-context learning of large language models (LLM) is shown to elicit strong reasoning capabilities along with reasonable explanations.
Controllable Dialogue Simulation with In-Context Learning
Experimental results on the MultiWOZ dataset demonstrate that training a model on the simulated dialogues leads to even better performance than using the same amount of human-generated dialogues under the challenging low-resource settings, with as few as 85 dialogues as a seed.
MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
While language Models store a massive amount of world knowledge implicitly in their parameters, even very large models often fail to encode information about rare entities and events, while incurring huge computational costs.
Re-Imagen: Retrieval-Augmented Text-to-Image Generator
To further evaluate the capabilities of the model, we introduce EntityDrawBench, a new benchmark that evaluates image generation for diverse entities, from frequent to rare, across multiple object categories including dogs, foods, landmarks, birds, and characters.
Ranked #3 on
Text-to-Image Generation
on MS COCO
QA Is the New KR: Question-Answer Pairs as Knowledge Bases
In this position paper, we propose a new approach to generating a type of knowledge base (KB) from text, based on question generation and entity linking.
HybriDialogue: An Information-Seeking Dialogue Dataset Grounded on Tabular and Textual Data
A pressing challenge in current dialogue systems is to successfully converse with users on topics with information distributed across different modalities.
Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering
Retrieval augmented language models have recently become the standard for knowledge intensive tasks.
Attacking Open-domain Question Answering by Injecting Misinformation
We curate both human-written and model-generated false documents that we inject into the evidence corpus of QA models and assess the impact on the performance of these systems.
Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding
Task-adaptive pre-training (TAPT) and Self-training (ST) have emerged as the major semi-supervised approaches to improve natural language understanding (NLU) tasks with massive amount of unlabeled data.
FinQA: A Dataset of Numerical Reasoning over Financial Data
In contrast to existing tasks on general domain, the finance domain includes complex numerical reasoning and understanding of heterogeneous representations.
Ranked #4 on
Question Answering
on FinQA
Local Explanation of Dialogue Response Generation
To gain insights into the reasoning process of a generation model, we propose a new method, local explanation of response generation (LERG) that regards the explanations as the mutual interaction of segments in input and output sentences.
Counterfactual Maximum Likelihood Estimation for Training Deep Networks
Although deep learning models have driven state-of-the-art performance on a wide array of tasks, they are prone to spurious correlations that should not be learned as predictive clues.
A Systematic Investigation of KB-Text Embedding Alignment at Scale
Knowledge bases (KBs) and text often contain complementary knowledge: KBs store structured knowledge that can support long range reasoning, while text stores more comprehensive and timely knowledge in an unstructured way.
Zero-shot Fact Verification by Claim Generation
However, for each new domain that requires fact verification, creating a dataset by manually writing claims and linking them to their supporting evidence is expensive.
Unsupervised Multi-hop Question Answering by Question Generation
Obtaining training data for multi-hop question answering (QA) is time-consuming and resource-intensive.
Open Question Answering over Tables and Text
In open question answering (QA), the answer to a question is produced by retrieving and then analyzing documents that might contain answers to the question.
Ranked #1 on
Question Answering
on OTT-QA
Modeling Token-level Uncertainty to Learn Unknown Concepts in SLU via Calibrated Dirichlet Prior RNN
We design a Dirichlet Prior RNN to model high-order uncertainty by degenerating as softmax layer for RNN model training.
KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation
We propose a knowledge-grounded pre-training (KGPT), which consists of two parts, 1) a general knowledge-grounded generation model to generate knowledge-enriched text.
Ranked #9 on
KG-to-Text Generation
on WebNLG 2.0 (Unconstrained)
Logic2Text: High-Fidelity Natural Language Generation from Logical Forms
If only provided with the table, it is hard for existing models to produce controllable and high-fidelity logical generations.
Logical Natural Language Generation from Open-Domain Tables
To facilitate the study of the proposed logical NLG problem, we use the existing TabFact dataset \cite{chen2019tabfact} featured with a wide range of logical/symbolic inferences as our testbed, and propose new automatic metrics to evaluate the fidelity of generation models w. r. t.\ logical inference.
HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
3) a hybrid model that combines heterogeneous information to find the answer.
Ranked #4 on
Question Answering
on HybridQA
VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
We introduce a new task, Video-and-Language Inference, for joint multimodal understanding of video and text.
Generative Adversarial Zero-Shot Relational Learning for Knowledge Graphs
Large-scale knowledge graphs (KGs) are shown to become more important in current information systems.
Meta Module Network for Compositional Visual Reasoning
To design a more powerful NMN architecture for practical use, we propose Meta Module Network (MMN) centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically.
TabFact: A Large-scale Dataset for Table-based Fact Verification
To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED.
Ranked #10 on
Table-based Fact Verification
on TabFact
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation.
Ranked #27 on
Image Generation
on ImageNet 64x64
(Bits per dim metric)
Global Textual Relation Embedding for Relational Understanding
Pre-trained embeddings such as word embeddings and sentence embeddings are fundamental tools facilitating a wide range of downstream NLP tasks.
Ranked #142 on
Action Classification
on Kinetics-400
Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention
Semantically controlled neural response generation on limited-domain has achieved great performance.
Ranked #5 on
Data-to-Text Generation
on MULTIWOZ 2.1
Few-Shot NLG with Pre-Trained Language Model
Neural-based end-to-end approaches to natural language generation (NLG) from structured data or knowledge are data-hungry, making their adoption for real-world applications difficult with limited data.
How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection
Under deep neural networks, a pre-defined vocabulary is required to vectorize text inputs.
A Variational Dirichlet Framework for Out-of-Distribution Detection
With the recently rapid development in deep learning, deep neural networks have been widely adopted in many real-life applications.
Approximate Distribution Matching for Sequence-to-Sequence Learning
Then, we interpret sequence-to-sequence learning as learning a transductive model to transform the source local latent distributions to match their corresponding target distributions.
XL-NBT: A Cross-lingual Neural Belief Tracking Framework
Then, we pre-train a state tracker for the source language as a teacher, which is able to exploit easy-to-access parallel data.
Generative Bridging Network for Neural Sequence Prediction
In order to alleviate data sparsity and overfitting problems in maximum likelihood estimation (MLE) for sequence prediction tasks, we propose the Generative Bridging Network (GBN), in which a novel bridge module is introduced to assist the training of the sequence prediction model (the generator network).
Triangular Architecture for Rare Language Translation
Neural Machine Translation (NMT) performs poor on the low-resource language pair $(X, Z)$, especially when $Z$ is a rare language.
No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
Though impressive results have been achieved in visual captioning, the task of generating abstract stories from photo streams is still a little-tapped problem.
Ranked #13 on
Visual Storytelling
on VIST
Variational Knowledge Graph Reasoning
Inferring missing links in knowledge graphs (KG) has attracted a lot of attention from the research community.
Video Captioning via Hierarchical Reinforcement Learning
Video captioning is the task of automatically generating a textual description of the actions in a video.
Hierarchical Reinforcement Learning
reinforcement-learning
+2
Generative Bridging Network in Neural Sequence Prediction
In order to alleviate data sparsity and overfitting problems in maximum likelihood estimation (MLE) for sequence prediction tasks, we propose the Generative Bridging Network (GBN), in which a novel bridge module is introduced to assist the training of the sequence prediction model (the generator network).
A Semi-supervised Framework for Image Captioning
We here propose a novel way of using such textual data by artificially generating missing visual information.
Guided Alignment Training for Topic-Aware Neural Machine Translation
In this paper, we propose an effective way for biasing the attention mechanism of a sequence-to-sequence neural machine translation (NMT) model towards the well-studied statistical word alignment models.
