Search Results for author:
Found 55 papers, 29 papers with code
Self-training with Few-shot Rationalization
While pre-trained language models have obtained state-of-the-art performance for several natural language understanding tasks, they are quite opaque in terms of their decision-making process.
Measuring Morphological Fusion Using Partial Information Decomposition
Morphological systems across languages vary when it comes to the relation between form and meaning.
Unsupervised Dependency Graph Network
We introduce a new model, the Unsupervised Dependency Graph Network (UDGN), that can induce dependency structures from raw corpora and the masked language modeling task.
Not All LLM Reasoners Are Created Equal
We study the depth of grade-school math (GSM) problem-solving capabilities of LLMs.
VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment
In this work, we systematically evaluate the efficacy of value networks and reveal their significant shortcomings in reasoning-heavy LLM tasks, showing that they barely outperform a random baseline when comparing alternative steps.
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to a particular domain or task.
Improving Context-Aware Preference Modeling for Language Models
To address these challenges, we consider the two-step preference modeling procedure that first resolves the under-specification by selecting a context, and then evaluates preference with respect to the chosen context.
Efficient Adversarial Training in LLMs with Continuous Attacks
We propose a fast adversarial training algorithm (C-AdvUL) composed of two losses: the first makes the model robust on continuous embedding attacks computed on an adversarial behaviour dataset; the second ensures the usefulness of the final model by fine-tuning on utility data.
Towards Modular LLMs by Building and Reusing a Library of LoRAs
The growing number of parameter-efficient adaptations of a base large language model (LLM) calls for studying whether we can reuse such trained adapters to improve performance for new tasks.
V-STaR: Training Verifiers for Self-Taught Reasoners
Common self-improvement approaches for large language models (LLMs), such as STaR, iteratively fine-tune LLMs on self-generated solutions to improve their problem-solving ability.
Guiding Language Model Reasoning with Planning Tokens
To encourage a more structural generation of CoT steps, we propose a hierarchical generation scheme: we let the LM generate a planning token at the start of each reasoning step, intuitively serving as a high-level plan of the current step, and add their embeddings to the model parameters.
Joint Prompt Optimization of Stacked LLMs using Variational Inference
Thus, they can be seen as stochastic language layers in a language network, where the learnable parameters are the natural language prompts at each layer.
On the Compositional Generalization Gap of In-Context Learning
In this work, we look at the gap between the in-distribution (ID) and out-of-distribution (OOD) performance of such models in semantic parsing tasks with in-context learning.
Multi-Head Adapter Routing for Cross-Task Generalization
We find that routing is most beneficial during multi-task pre-training rather than during few-shot adaptation and propose $\texttt{MHR}$-$\mu$, which discards routing and fine-tunes the average of the pre-trained adapters on each downstream tasks.
Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods
We address the problem of evaluating the quality of self-supervised learning (SSL) models without access to supervised labels, while being agnostic to the architecture, learning algorithm or data manipulation used during training.
Evaluating Distributional Distortion in Neural Language Modeling
To address this gap, we develop a controlled evaluation scheme which uses generative models trained on natural data as artificial languages from which we can exactly compute sequence probabilities.
Better Language Model with Hypernym Class Prediction
Class-based language models (LMs) have been long devised to address context sparsity in $n$-gram LMs.
Combining Modular Skills in Multitask Learning
By jointly learning these and a task-skill allocation matrix, the network for each task is instantiated as the average of the parameters of active skills.
Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge
Transformer models pre-trained with a masked-language-modeling objective (e. g., BERT) encode commonsense knowledge as evidenced by behavioral probes; however, the extent to which this knowledge is acquired by systematic inference over the semantics of the pre-training corpora is an open question.
Learning to Dequantise with Truncated Flows
Addtionally, since the support of the marginal $q(z)$ is bounded and the support of prior $p(z)$ is not, we propose renormalising the prior distribution over the support of $q(z)$.
Learnability and Expressiveness in Self-Supervised Learning
In this work, we argue that representations induced by self-supervised learning (SSL) methods should both be expressive and learnable.
Self-training with Few-shot Rationalization: Teacher Explanations Aid Student in Few-shot NLU
While pre-trained language models have obtained state-of-the-art performance for several natural language understanding tasks, they are quite opaque in terms of their decision-making process.
The Emergence of the Shape Bias Results from Communicative Efficiency
By the age of two, children tend to assume that new word categories are based on objects' shape, rather than their color or texture; this assumption is called the shape bias.
Decomposed Mutual Information Estimation for Contrastive Representation Learning
We propose decomposing the full MI estimation problem into a sum of smaller estimation problems by splitting one of the views into progressively more informed subviews and by applying the chain rule on MI between the decomposed views.
Understanding by Understanding Not: Modeling Negation in Language Models
To improve language models in this regard, we propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences from a raw text corpus.
Linguistic Dependencies and Statistical Dependence
Are pairs of words that tend to occur together also likely to stand in a linguistic dependency?
Decomposing Mutual Information for Representation Learning
In this paper, we transform each view into a set of subviews and then decompose the original MI bound into a sum of bounds involving conditional MI between the subviews.
Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle
In the present work, we propose a new syntax-aware language model: Syntactic Ordered Memory (SOM).
Recursive Top-Down Production for Sentence Generation with Latent Trees
We model the recursive production property of context-free grammars for natural and synthetic languages.
Exploring and Predicting Transferability across NLP Tasks
We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task, and we validate their effectiveness in experiments controlled for source and target data size.
Hybrid Generative-Retrieval Transformers for Dialogue Domain Adaptation
Domain adaptation has recently become a key problem in dialogue systems research.
Report on the First HIPstIR Workshop on the Future of Information Retrieval
The vision of HIPstIR is that early stage information retrieval (IR) researchers get together to develop a future for non-mainstream ideas and research agendas in IR.
Increasing Robustness to Spurious Correlations using Forgettable Examples
Neural NLP models tend to rely on spurious correlations between labels and input features to perform their tasks.
Natural Language Inference
Natural Language Understanding
+2
Ordered Memory
Inspired by Ordered Neurons (Shen et al., 2018), we introduce a new attention-based mechanism and use its cumulative probability to control the writing and erasing operation of the memory.
Metalearned Neural Memory
We augment recurrent neural networks with an external memory mechanism that builds upon recent progress in metalearning.
An Empirical Study of Example Forgetting during Deep Neural Network Learning
Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks.
Towards Text Generation with Adversarially Learned Neural Outlines
We generate outlines with an adversarial model trained to approximate the distribution of sentences in a latent space induced by general-purpose sentence encoders.
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed.
Counting to Explore and Generalize in Text-based Games
We propose a recurrent RL agent with an episodic exploration mechanism that helps discovering good policies in text-based game environments.
Focused Hierarchical RNNs for Conditional Sequence Processing
We evaluate this method on several types of tasks with different attributes.
Ranked #3 on
Open-Domain Question Answering
on SearchQA
(Unigram Acc metric)
Straight to the Tree: Constituency Parsing with Neural Syntactic Distance
In this work, we propose a novel constituency parsing scheme.
Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
Learning inter-domain mappings from unpaired data can improve performance in structured prediction tasks, such as image segmentation, by reducing the need for paired data.
Z-Forcing: Training Stochastic Recurrent Networks
Stochastic recurrent models have been successful in capturing the variability observed in natural sequential data such as speech.
Twin Networks: Matching the Future for Sequence Generation
We propose a simple technique for encouraging generative RNNs to plan ahead.
Learning Algorithms for Active Learning
We introduce a model that learns active learning algorithms via metalearning.
Machine Comprehension by Text-to-Text Neural Question Generation
We propose a recurrent neural model that generates natural-language questions from documents, conditioned on answers.
Towards Information-Seeking Agents
We develop a general problem setting for training and testing the ability of agents to gather information efficiently.
NewsQA: A Machine Comprehension Dataset
We present NewsQA, a challenging machine comprehension dataset of over 100, 000 human-generated question-answer pairs.
Iterative Alternating Neural Attention for Machine Reading
We propose a novel neural attention architecture to tackle machine comprehension tasks, such as answering Cloze-style queries with respect to a document.
Ranked #3 on
Question Answering
on Children's Book Test
(Accuracy-NE metric)
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
Sequential data often possesses a hierarchical structure with complex dependencies between subsequences, such as found between the utterances in a dialogue.
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
We investigate the task of building open domain, conversational dialogue systems based on large dialogue corpora using generative models.
A Hierarchical Recurrent Encoder-Decoder For Generative Context-Aware Query Suggestion
Our novel hierarchical recurrent encoder-decoder architecture allows the model to be sensitive to the order of queries in the context while avoiding data sparsity.
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
We introduce Discriminative BLEU (deltaBLEU), a novel metric for intrinsic evaluation of generated text in tasks that admit a diverse range of possible outputs.
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
We present a novel response generation system that can be trained end to end on large quantities of unstructured Twitter conversations.
