Search Results for author:
Found 15 papers, 8 papers with code
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
The quadratic complexity and weak length extrapolation of Transformers limits their ability to scale to long sequences, and while sub-quadratic solutions like linear attention and state space models exist, they empirically underperform Transformers in pretraining efficiency and downstream task accuracy.
Jointly Training Large Autoregressive Multimodal Models
In recent years, advances in the large-scale pretraining of language and text-to-image models have revolutionized the field of machine learning.
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning
It is also a general-purpose model that can do both text-to-image and image-to-text generation, allowing us to introduce self-contained contrastive decoding methods that produce high-quality outputs.
Ranked #2 on
Text-to-Image Generation
on MS COCO
LIMA: Less Is More for Alignment
Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences.
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Autoregressive transformers are spectacular models for short sequences but scale poorly to long sequences such as high-resolution images, podcasts, code, or books.
VideoOFA: Two-Stage Pre-Training for Video-to-Text Generation
We propose a new two-stage pre-training framework for video-to-text generation tasks such as video captioning and video question answering: A generative encoder-decoder model is first jointly pre-trained on massive image-text data to learn fundamental vision-language concepts, and then adapted to video data in an intermediate video-text pre-training stage to learn video-specific skills such as spatio-temporal reasoning.
Scaling Laws for Generative Mixed-Modal Language Models
To better understand the scaling properties of such mixed-modal models, we conducted over 250 experiments using seven different modalities and model sizes ranging from 8 million to 30 billion, trained on 5-100 billion tokens.
Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences
Lack of factual correctness is an issue that still plagues state-of-the-art summarization systems despite their impressive progress on generating seemingly fluent summaries.
Nutri-bullets Hybrid: Consensual Multi-document Summarization
We present a method for generating comparative summaries that highlight similarities and contradictions in input documents.
Nutribullets Hybrid: Multi-document Health Summarization
We present a method for generating comparative summaries that highlights similarities and contradictions in input documents.
Nutri-bullets: Summarizing Health Studies by Composing Segments
We introduce \emph{Nutri-bullets}, a multi-document summarization task for health and nutrition.
Rationalizing Text Matching: Learning Sparse Alignments via Optimal Transport
Selecting input features of top relevance has become a popular method for building self-explaining models.
Interactive Classification by Asking Informative Questions
We study the potential for interaction in natural language classification.
Building a Production Model for Retrieval-Based Chatbots
Response suggestion is an important task for building human-computer conversation systems.
On the evolution of word usage of classical Chinese poetry
The primary goal of this study is to provide quantitative evidence of the evolutionary linkages, with emphasis on character usage, among different period genres of classical Chinese poetry.
