Search Results for author:
Found 39 papers, 8 papers with code
Depth-Bounded Statistical PCFG Induction as a Model of Human Grammar Acquisition
Abstract This article describes a simple PCFG induction model with a fixed category domain that predicts a large majority of attested constituent boundaries, and predicts labels consistent with nearly half of attested constituent labels on a standard evaluation data set of child-directed speech.
Contributions of Propositional Content and Syntactic Category Information in Sentence Processing
Expectation-based theories of sentence processing posit that processing difficulty is determined by predictability in context.
Character-based PCFG Induction for Modeling the Syntactic Acquisition of Morphologically Rich Languages
A subsequent evaluation on multilingual treebanks shows that the model with subword information achieves state-of-the-art results on many languages, further supporting a distributional model of syntactic acquisition.
Coreference-aware Surprisal Predicts Brain Response
Recent evidence supports a role for coreference processing in guiding human expectations about upcoming words during reading, based on covariation between reading times and word surprisal estimated by a coreference-aware semantic processing model (Jaffe et al. 2020). The present study reproduces and elaborates on this finding by (1) enabling the parser to process subword information that might better approximate human morphological knowledge, and (2) extending evaluation of coreference effects from self-paced reading to human brain imaging data.
Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times
Additionally, training dynamics reveal that during later training steps, all model variants learn to predict rare words and that larger model variants do so more accurately, which explains the detrimental effect of both training data amount and model size on fit to reading times.
Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions
While there is much recent interest in studying why Transformer-based large language models make predictions the way they do, the complex computations performed within each layer have made their behavior somewhat opaque.
Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens
Recent psycholinguistic studies have drawn conflicting conclusions about the relationship between the quality of a language model and the ability of its surprisal estimates to predict human reading times, which has been speculated to be due to the large gap in both the amount of training data and model capacity across studies.
Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?
This work presents a detailed linguistic analysis into why larger Transformer-based pre-trained language models with more parameters and lower perplexity nonetheless yield surprisal estimates that are less predictive of human reading times.
Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisal
Transformer-based large language models are trained to make predictions about the next word by aggregating representations of previous tokens through their self-attention mechanism.
A Deep Learning Approach to Analyzing Continuous-Time Systems
Scientists often use observational time series data to study complex natural processes, but regression analyses often assume simplistic dynamics.
Surprisal Estimators for Human Reading Times Need Character Models
While the use of character models has been popular in NLP applications, it has not been explored much in the context of psycholinguistic modeling.
Grounded PCFG Induction with Images
Recent work in unsupervised parsing has tried to incorporate visual information into learning, but results suggest that these models need linguistic bias to compete against models that only rely on text.
Coreference information guides human expectations during natural reading
Models of human sentence processing effort tend to focus on costs associated with retrieving structures and discourse referents from memory (memory-based) and/or on costs associated with anticipating upcoming words and structures based on contextual cues (expectation-based) (Levy, 2008).
Memory-bounded Neural Incremental Parsing for Psycholinguistic Prediction
Syntactic surprisal has been shown to have an effect on human sentence processing, and can be predicted from prefix probabilities of generative incremental parsers.
The Importance of Category Labels in Grammar Induction with Child-directed Utterances
Recent progress in grammar induction has shown that grammar induction is possible without explicit assumptions of language-specific knowledge.
A Corpus of Encyclopedia Articles with Logical Forms
People can extract precise, complex logical meanings from text in documents such as tax forms and game rules, but language processing systems lack adequate training and evaluation resources to do these kinds of tasks reliably.
Unsupervised Learning of PCFGs with Normalizing Flow
This paper describes a neural PCFG inducer which employs context embeddings (Peters et al., 2018) in a normalizing flow model (Dinh et al., 2015) to extend PCFG induction to use semantic and morphological information.
Variance of Average Surprisal: A Better Predictor for Quality of Grammar from Unsupervised PCFG Induction
In unsupervised grammar induction, data likelihood is known to be only weakly correlated with parsing accuracy, especially at convergence after multiple runs.
Deconvolutional Time Series Regression: A Technique for Modeling Temporally Diffuse Effects
Researchers in computational psycholinguistics frequently use linear models to study time series data generated by human subjects.
Depth-bounding is effective: Improvements and evaluation of unsupervised PCFG induction
There have been several recent attempts to improve the accuracy of grammar induction systems by bounding the recursive complexity of the induction model (Ponvert et al., 2011; Noji and Johnson, 2016; Shain et al., 2016; Jin et al., 2018).
Unsupervised Grammar Induction with Depth-bounded PCFG
There has been recent interest in applying cognitively or empirically motivated bounds on recursion depth to limit the search space of grammar induction models (Ponvert et al., 2011; Noji and Johnson, 2016; Shain et al., 2016).
Memory-Bounded Left-Corner Unsupervised Grammar Induction on Child-Directed Input
This paper presents a new memory-bounded left-corner parsing model for unsupervised raw-text syntax induction, using unsupervised hierarchical hidden Markov models (UHHMM).
Addressing surprisal deficiencies in reading time models
This study demonstrates a weakness in how n-gram and PCFG surprisal are used to predict reading times in eye-tracking data.
Memory access during incremental sentence processing causes reading time latency
Studies on the role of memory as a predictor of reading time latencies (1) differ in their predictions about when memory effects should occur in processing and (2) have had mixed results, with strong positive effects emerging from isolated constructed stimuli and weak or even negative effects emerging from naturally-occurring stimuli.
