Search Results for author:
Found 76 papers, 34 papers with code
Finding the Dominant Winning Ticket in Pre-Trained Language Models
Empirically, we show that (a) the dominant winning ticket can achieve performance that is comparable with that of the full-parameter model, (b) the dominant winning ticket is transferable across different tasks, (c) and the dominant winning ticket has a natural structure within each parameter matrix.
Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models
We introduce Sigma, an efficient large language model specialized for the system domain, empowered by a novel architecture including DiffQKV attention, and pre-trained on our meticulously collected system domain data.
Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation
However, they still struggle to coherently present multiple sequential events in the stories specified by the prompts, which is foreseeable an essential capability for future long video generation scenarios.
Mojito: Motion Trajectory and Intensity Control for Video Generation
Recent advancements in diffusion models have shown great promise in producing high-quality video content.
StreamAdapter: Efficient Test Time Adaptation from Contextual Streams
The superior task adaptation and context encoding capabilities of StreamAdapter on both language understanding and generation tasks provides a new perspective for adapting LLMs at test time using context, allowing for more efficient adaptation across scenarios and more cost-effective inference
Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation
This paper delves into the effects of decoding temperatures on speculative decoding's efficacy.
MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
Parameter-efficient fine-tuning (PEFT) has been widely employed for domain adaptation, with LoRA being one of the most prominent methods due to its simplicity and effectiveness.
LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy
Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages, which requires extensive parameter tuning thus unsuitable for pre-trained LLMs; (2) KV cache compression at test time, primarily through token eviction policies, which often overlook inter-layer dependencies and can be task-specific.
Demo: SGCode: A Flexible Prompt-Optimizing System for Secure Generation of Code
This paper introduces SGCode, a flexible prompt-optimizing system to generate secure code with large language models (LLMs).
Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning
Reinforcement learning with human feedback (RLHF), as a widely adopted approach in current large language model pipelines, is \textit{bottlenecked by the size of human preference data}.
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
When finetuned on 4K-length sequences, Samba efficiently extrapolates to a 256K context length with perfect memory recall on the Passkey Retrieval task, and exhibits superior retrieval extrapolation on the challenging Phonebook task compared to full-attention models.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
We introduce phi-3-mini, a 3. 8 billion parameter language model trained on 3. 3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3. 5 (e. g., phi-3-mini achieves 69% on MMLU and 8. 38 on MT-bench), despite being small enough to be deployed on a phone.
Ranked #5 on
MMR total
on MRR-Benchmark
(using extra training data)
Rho-1: Not All Tokens Are What You Need
Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution.
Exploring the Mystery of Influential Data for Mathematical Reasoning
Additionally, we showcase the use of QaDS in creating efficient fine-tuning mixtures with various selection ratios, and analyze the quality of a wide range of open-source datasets, which can perform as a reference for future works on mathematical reasoning tasks.
Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models
Subsequently, the retrieval model correlates new inputs with relevant guidelines, which guide LLMs in response generation to ensure safe and high-quality outputs, thereby aligning with human values.
Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning
Large language models (LLMs) have shown great potential in complex reasoning tasks, yet their performance is often hampered by the scarcity of high-quality and reasoning-focused training datasets.
Ranked #51 on
Math Word Problem Solving
on MATH
Multi-LoRA Composition for Image Generation
Low-Rank Adaptation (LoRA) is extensively utilized in text-to-image models for the accurate rendition of specific elements like distinct characters or unique styles in generated images.
Competition-Level Problems are Effective LLM Evaluators
Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about these abilities and the potential data contamination problem recently.
Language Models can be Logical Solvers
Despite their impressive performance, any parsing errors will inevitably result in the failure of the execution of the external logical solver and no answer to the logical questions.
Adapting LLM Agents with Universal Feedback in Communication
To optimize agent interactions for task-specific learning with our universal buffer and pipeline, we introduce diverse communication patterns tailored for both single-agent and multi-agent environments.
ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
Large language models have made significant progress in various language tasks, yet they still struggle with complex mathematics.
Ranked #27 on
Arithmetic Reasoning
on GSM8K
(using extra training data)
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
We find that scaling LMM consistently enhances model performance and improves language capabilities, and performance of LoRA/QLoRA tuning of LMM are comparable to the performance of full-model fine-tuning.
Ranked #141 on
Visual Question Answering
on MM-Vet
Efficient RLHF: Reducing the Memory Usage of PPO
To address this issue, we present a comprehensive analysis the memory usage, performance, and training time of memory-savings techniques for PPO.
GRILL: Grounded Vision-language Pre-training via Aligning Text and Image Regions
Generalization to unseen tasks is an important ability for few-shot learners to achieve better zero-/few-shot performance on diverse tasks.
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
In this paper, we show that strong performance can be achieved by a method we call Iter-RetGen, which synergizes retrieval and generation in an iterative manner.
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Unlike these models, humans typically utilize external tools to cross-check and refine their initial content, like using a search engine for fact-checking, or a code interpreter for debugging.
AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance.
In-Context Learning Unlocked for Diffusion Models
We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models.
Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models
By utilizing iterative bootstrapping, our approach enables LLMs to autonomously rectify errors, resulting in more precise and comprehensive reasoning chains.
What Matters In The Structured Pruning of Generative Language Models?
Auto-regressive large language models such as GPT-3 require enormous computational resources to use.
Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models
However, the quality of the prompts depends on the demonstrations given to the models, and creating many of them by hand is costly.
Generation-Augmented Query Expansion For Code Retrieval
Pre-trained language models have achieved promising success in code retrieval tasks, where a natural language documentation query is given to find the most relevant existing code snippet.
GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation
We introduce GENIUS: a conditional text generation model using sketches as input, which can fill in the missing contexts for a given sketch (key information consisting of textual spans, phrases, or words, concatenated by mask tokens).
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Thus, we propose a simple ambiguous negatives sampling method, SimANS, which incorporates a new sampling probability distribution to sample more ambiguous negatives.
Soft-Labeled Contrastive Pre-training for Function-level Code Representation
In this paper, we present \textbf{SCodeR}, a \textbf{S}oft-labeled contrastive pre-training framework with two positive sample construction methods to learn functional-level \textbf{Code} \textbf{R}epresentation.
Explanations from Large Language Models Make Small Reasoners Better
Integrating free-text explanations to in-context learning of large language models (LLM) is shown to elicit strong reasoning capabilities along with reasonable explanations.
Joint Generator-Ranker Learning for Natural Language Generation
Generate-then-rank is a widely used mechanism for text generation, where a generator produces multiple text candidates and a ranker chooses the best one among the text candidates.
A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation
To further illustrate the commercial value of our approach, we conduct experiments on three generation tasks in real-world advertisements applications.
CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing
To retain ensemble benefits while maintaining a low memory cost, we propose a consistency-regularized ensemble learning approach based on perturbed models, named CAMERO.
Controllable Natural Language Generation with Contrastive Prefixes
We propose a novel supervised method and also an unsupervised method to train the prefixes for single-aspect control while the combination of these two methods can achieve multi-aspect control.
CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search
For bimodal contrastive learning, we leverage the documentation and in-line comments of code to build code-text pairs.
Knowledge-Grounded Dialogue Generation with a Unified Knowledge Representation
To address these challenges, we present PLUG, a language model that homogenizes different knowledge sources to a unified knowledge representation for knowledge-grounded dialogue generation tasks.
A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models
Large pre-trained vision-language (VL) models can learn a new task with a handful of examples and generalize to a new task without fine-tuning.
Ranked #4 on
Image Captioning
on Flickr30k Captions test
(SPICE metric)
Adversarial Retriever-Ranker for dense text retrieval
To address these challenges, we present Adversarial Retriever-Ranker (AR2), which consists of a dual-encoder retriever plus a cross-encoder ranker.
LoRA: Low-Rank Adaptation of Large Language Models
We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.
Ranked #2 on
Mathematical Reasoning
on AMC23
Memory-Efficient Differentiable Transformer Architecture Search
To this end, we propose a multi-split reversible network and combine it with DARTS.
Poolingformer: Long Document Modeling with Pooling Attention
We first evaluate Poolingformer on two long sequence QA tasks: the monolingual NQ and the multilingual TyDi QA.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively.
Ranked #1 on
Open-Domain Question Answering
on TriviaQA
Rider: Reader-Guided Passage Reranking for Open-Domain Question Answering
Current open-domain question answering systems often follow a Retriever-Reader architecture, where the retriever first retrieves relevant passages and the reader then reads the retrieved passages to form an answer.
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
Adversarial Attacks on Deep Graph Matching
Despite achieving remarkable performance, deep graph learning models, such as node classification and network embedding, suffer from harassment caused by small adversarial perturbations.
CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
To verify the effectiveness of the proposed framework, we apply CoDA to Transformer-based models on a wide range of natural language understanding tasks.
Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
We prove, from a theoretical perspective, that the gradients derived from this new masking schema have a smaller variance and can lead to more efficient self-supervised training.
Ranked #1 on
Sentence Classification
on ACL-ARC
A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
Adversarial training has been shown effective at endowing the learned representations with stronger generalization ability.
Ranked #9 on
Machine Translation
on IWSLT2014 German-English
Generation-Augmented Retrieval for Open-domain Question Answering
We demonstrate that the generated contexts substantially enrich the semantics of the queries and GAR with sparse representations (BM25) achieves comparable or better performance than state-of-the-art dense retrieval methods such as DPR.
Ranked #9 on
Passage Retrieval
on Natural Questions
Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension
In this paper, we study machine reading comprehension (MRC) on long texts, where a model takes as inputs a lengthy document and a question and then extracts a text span from the document as an answer.
MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
Generating multi-sentence descriptions for videos is one of the most challenging captioning tasks due to its high requirements for not only visual relevance but also discourse-based coherence across the sentences in the paragraph.
Ranked #5 on
Video Captioning
on ActivityNet Captions
Unsupervised Deep Structured Semantic Models for Commonsense Reasoning
Commonsense reasoning is fundamental to natural language understanding.
Ranked #2 on
Natural Language Understanding
on PDP60
StoryGAN: A Sequential Conditional GAN for Story Visualization
We therefore propose a new story-to-image-sequence generation model, StoryGAN, based on the sequential conditional GAN framework.
Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
We propose a multi-task learning framework to learn a joint Machine Reading Comprehension (MRC) model that can be applied to a wide range of MRC tasks in different domains.
M-Walk: Learning to Walk over Graphs using Monte Carlo Tree Search
In order to effectively train the agent from sparse rewards, we combine MCTS with the neural policy to generate trajectories yielding more positive rewards.
Ranked #45 on
Link Prediction
on WN18RR
(Hits@3 metric)
Stochastic Answer Networks for Machine Reading Comprehension
We propose a simple yet robust stochastic answer network (SAN) that simulates multi-step reasoning in machine reading comprehension.
Ranked #24 on
Question Answering
on SQuAD1.1 dev
FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension
This paper introduces a new neural structure called FusionNet, which extends existing attention approaches from three perspectives.
Ranked #26 on
Question Answering
on SQuAD1.1 dev
Language-Based Image Editing with Recurrent Attentive Models
First, we introduce a synthetic dataset, called CoSaL, to evaluate the end-to-end performance of our LBIE system.
Dynamic Fusion Networks for Machine Reading Comprehension
This paper presents a novel neural model - Dynamic Fusion Network (DFN), for machine reading comprehension (MRC).
An Empirical Analysis of Multiple-Turn Reasoning Strategies in Reading Comprehension Tasks
Using a state-of-the-art RC model, we empirically investigate the performance of single-turn and multiple-turn reasoning on the SQuAD and MS MARCO datasets.
Modeling Large-Scale Structured Relationships with Shared Memory for Knowledge Base Completion
However, due to the size of knowledge bases, learning multi-step relations directly on top of observed triplets could be costly.
Link Prediction using Embedded Knowledge Graphs
Since large knowledge bases are typically incomplete, missing facts need to be inferred from observed facts in a task called knowledge base completion.
ReasoNet: Learning to Stop Reading in Machine Comprehension
Teaching a computer to read and answer general questions pertaining to a document is a challenging yet unsolved problem.
Ranked #7 on
Question Answering
on CNN / Daily Mail
End-to-end Learning of LDA by Mirror-Descent Back Propagation over a Deep Architecture
We develop a fully discriminative learning approach for supervised Latent Dirichlet Allocation (LDA) model using Back Propagation (i. e., BP-sLDA), which maximizes the posterior probability of the prediction variable given the input document.
A Deep Embedding Model for Co-occurrence Learning
Co-occurrence Data is a common and important information source in many areas, such as the word co-occurrence in the sentences, friends co-occurrence in social networks and products co-occurrence in commercial transaction data, etc, which contains rich correlation and clustering information about the items.
Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval
The results show that the proposed method in this paper significantly outperforms it for web document retrieval task.
