Search Results for author:
Found 56 papers, 19 papers with code
The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval
For long context modeling, the range of positions may vary a lot, and thus RoPE rotates some dimensions by a great range of angles.
The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities
We hypothesize that models acquire this capability through learning a shared representation space across heterogeneous data types (e. g., different languages and modalities), which places semantically similar inputs near one another, even if they are from different modalities/languages.
Causal Interventions on Causal Paths: Mapping GPT-2's Reasoning From Syntax to Semantics
While interpretability research has shed light on some internal algorithms utilized by transformer-based LLMs, reasoning in natural language, with its deep contextuality and ambiguity, defies easy categorization.
LocateBench: Evaluating the Locating Ability of Vision Language Models
The ability to locate an object in an image according to natural language instructions is crucial for many real-world applications.
Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models
We introduce Vibe-Eval: a new open benchmark and framework for evaluating multimodal chat models.
IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations
Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs.
Understanding In-Context Learning with a Pelican Soup Framework
In this framework, we introduce (1) the notion of a common sense knowledge base, (2) a general formalism for natural language classification tasks, and the notion of (3) meaning association.
DeLLMa: Decision Making Under Uncertainty with Large Language Models
The potential of large language models (LLMs) as decision support tools is increasingly being explored in fields such as business, engineering, and medicine, which often face challenging tasks of decision-making under uncertainty.
On Retrieval Augmentation and the Limitations of Language Model Training
Augmenting a language model (LM) with $k$-nearest neighbors ($k$NN) retrieval on its training data alone can decrease its perplexity, though the underlying reasons for this remain elusive.
The Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretraining
Via a synthetic dataset, our analysis suggests that distributional property indeed leads to the better sample efficiency of pretrained masked language models, but does not fully explain the generalization capability.
Interpretable Diffusion via Information Decomposition
For diffusion models, we show that a natural non-negative decomposition of mutual information emerges, allowing us to quantify informative relationships between words and pixels in an image.
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
There have been a lot of interest in the scaling properties of Transformer models.
Emergent Abilities of Large Language Models
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks.
Language Models Can See: Plugging Visual Controls in Text Generation
MAGIC is a flexible framework and is theoretically compatible with any text generation tasks that incorporate image grounding.
High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on robots.
A Contrastive Framework for Neural Text Generation
Text generation is of great importance to many natural language processing applications.
Relational Memory Augmented Language Models
We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph.
Balancing Average and Worst-case Accuracy in Multitask Learning
When training and evaluating machine learning models on a large number of tasks, it is important to not only look at average task accuracy -- which may be biased by easy or redundant tasks -- but also worst-case accuracy (i. e. the performance on the task with the lowest accuracy).
ABC: Attention with Bounded-memory Control
One way to improve the efficiency is to bound the memory size.
Scale Efficiently: Insights from Pretraining and Finetuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering
We model retrieval decisions as latent variables over sets of relevant documents.
Finetuning Pretrained Transformers into RNNs
Specifically, we propose a swap-then-finetune procedure: in an off-the-shelf pretrained transformer, we replace the softmax attention with its linear-complexity recurrent alternative and then finetune.
Ranked #2 on
Machine Translation
on WMT2017 Chinese-English
Random Feature Attention
RFA can be used as a drop-in replacement for conventional softmax attention and offers a straightforward way of learning with recency bias through an optional gating mechanism.
Ranked #28 on
Machine Translation
on IWSLT2014 German-English
Adaptive Semiparametric Language Models
We present a language model that combines a large parametric neural network (i. e., a transformer) with a non-parametric episodic memory component in an integrated architecture.
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Hence, given the compilation of ever-larger language modelling datasets, combined with the growing list of language-model-based NLP applications that require up-to-date factual knowledge about the world, we argue that now is the right time to rethink the static way in which we currently train and evaluate our language models, and develop adaptive language models that can remain up-to-date with respect to our ever-changing and non-stationary world.
Syntactic Structure Distillation Pretraining For Bidirectional Encoders
Textual representation learners trained on large amounts of data have achieved notable success on downstream tasks; intriguingly, they have also performed well on challenging tests of syntactic competence.
A Call for More Rigor in Unsupervised Cross-lingual Learning
We review motivations, definition, approaches, and methodology for unsupervised cross-lingual learning and call for a more rigorous position in each of them.
Modelling Latent Skills for Multitask Language Generation
We show that our latent task variable model outperforms other sequence-to-sequence baselines on average across tasks in the multitask setting.
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation
This paper aims to quantify and reduce a particular type of bias exhibited by language models: bias in the sentiment of generated text.
On the Cross-lingual Transferability of Monolingual Representations
This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions.
A Mutual Information Maximization Perspective of Language Representation Learning
We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i. e., a sentence).
Relative Pixel Prediction For Autoregressive Image Generation
In natural images, transitions between adjacent pixels tend to be smooth and gradual, a fact that has long been exploited in image compression models based on predictive coding.
Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation
Neural networks are part of many contemporary NLP systems, yet their empirical successes come at the price of vulnerability to adversarial attacks.
Episodic Memory in Lifelong Language Learning
We introduce a lifelong language learning setup where a model needs to learn from a stream of text examples without any dataset identifier.
Learning and Evaluating General Linguistic Intelligence
We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly.
Variational Smoothing in Recurrent Neural Network Language Models
We present a new theoretical perspective of data noising in recurrent neural network language models (Xie et al., 2017).
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better
Language exhibits hierarchical structure, but recent work using a subject-verb agreement diagnostic argued that state-of-the-art language models, LSTMs, fail to learn long-range syntax sensitive dependencies.
Memory Architectures in Recurrent Neural Network Language Models
We compare and analyze sequential, random access, and stack memory architectures for recurrent neural network language models.
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Solving algebraic word problems requires executing a series of arithmetic operations{---}a program{---}to obtain a final answer.
Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs
It can therefore be seen as a tree-based RNN that is unsupervised with respect to the parse trees.
Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems
Solving algebraic word problems requires executing a series of arithmetic operations---a program---to obtain a final answer.
Generative and Discriminative Text Classification with Recurrent Neural Networks
We empirically characterize the performance of discriminative and generative LSTM models for text classification.
Learning to Compose Words into Sentences with Reinforcement Learning
We use reinforcement learning to learn tree-structured neural networks for computing representations of natural language sentences.
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech--two vastly different languages.
Sparse Overcomplete Word Vector Representations
Current distributed representations of words show little resemblance to theories of lexical semantics.
Bayesian Optimization of Text Representations
When applying machine learning to problems in NLP, there are many choices to make about how to represent input texts.
Learning Word Representations with Hierarchical Sparse Coding
We propose a new method for learning word representations using hierarchical regularization in sparse coding inspired by the linguistic study of word meanings.
Dynamic Language Models for Streaming Text
We present a probabilistic language model that captures temporal dynamics and conditions on arbitrary non-linguistic context features.
A Sparse and Adaptive Prior for Time-Dependent Model Parameters
We consider the scenario where the parameters of a probabilistic model are expected to vary over time.
