Continuous Learning in a Hierarchical Multiscale Neural Network

ACL 2018  ·  Thomas Wolf, Julien Chaumond, Clement Delangue ·

We reformulate the problem of encoding a multi-scale representation of a sequence in a language model by casting it in a continuous learning framework. We propose a hierarchical multi-scale language model in which short time-scale dependencies are encoded in the hidden state of a lower-level recurrent neural network while longer time-scale dependencies are encoded in the dynamic of the lower-level network by having a meta-learner update the weights of the lower-level neural network in an online meta-learning fashion... We use elastic weights consolidation as a higher-level to prevent catastrophic forgetting in our continuous learning framework. read more

PDF Abstract ACL 2018 PDF ACL 2018 Abstract

Datasets


Results from the Paper


  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods


No methods listed for this paper. Add relevant methods here