Guiding Symbolic Natural Language Grammar Induction via Transformer-Based Sequence Probabilities

26 May 2020 Ben Goertzel Andres Suarez Madrigal Gino Yu

A novel approach to automated learning of syntactic rules governing natural languages is proposed, based on using probabilities assigned to sentences (and potentially longer word sequences) by transformer neural network language models to guide symbolic learning processes like clustering and rule induction. This method exploits the learned linguistic knowledge in transformers, without any reference to their inner representations; hence, the technique is readily adaptable to the continuous appearance of more powerful language models... (read more)

PDF Abstract
No code implementations yet. Submit your code now

Tasks


Datasets


  Add Datasets introduced or used in this paper

Results from the Paper


  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods used in the Paper


METHOD TYPE
Residual Connection
Skip Connections
Label Smoothing
Regularization
Multi-Head Attention
Attention Modules
Adam
Stochastic Optimization
ReLU
Activation Functions
Dropout
Regularization
BPE
Subword Segmentation
Dense Connections
Feedforward Networks
Layer Normalization
Normalization
Softmax
Output Functions
Scaled Dot-Product Attention
Attention Mechanisms
Transformer
Transformers