Search Results for author:
Found 18 papers, 10 papers with code
Gradient Descent Fails to Learn High-frequency Functions and Modular Arithmetic
In the novel framework, the hardness of a class is usually quantified by the variance of the gradient with respect to a random choice of a target function.
Intractability of Learning the Discrete Logarithm with Gradient-Based Methods
The discrete logarithm problem is a fundamental challenge in number theory with significant implications for cryptographic protocols.
Long-Tail Theory under Gaussian Mixtures
We suggest a simple Gaussian mixture model for data generation that complies with Feldman's long tail theory (2020).
From Hyperbolic Geometry Back to Word Embeddings
We choose random points in the hyperbolic disc and claim that these points are already word representations.
Speeding Up Entmax
Softmax is the de facto standard in modern neural networks for language processing when it comes to normalizing logits.
The Rediscovery Hypothesis: Language Models Need to Meet Linguistics
In this paper, we study whether linguistic knowledge is a necessary condition for the good performance of modern language models, which we call the \textit{rediscovery hypothesis}.
Squashed Shifted PMI Matrix: Bridging Word Embeddings and Hyperbolic Spaces
We show that removing sigmoid transformation in the skip-gram with negative sampling (SGNS) objective does not harm the quality of word vectors significantly and at the same time is related to factorizing a squashed shifted PMI matrix which, in turn, can be treated as a connection probabilities matrix of a random graph.
Semantics- and Syntax-related Subvectors in the Skip-gram Embeddings
We show that the skip-gram embedding of any word can be decomposed into two subvectors which roughly correspond to semantic and syntactic roles of the word.
A Critique of the Smooth Inverse Frequency Sentence Embeddings
We critically review the smooth inverse frequency sentence embedding method of Arora, Liang, and Ma (2017), and show inconsistencies in its setup, derivation, and evaluation.
Low-Rank Approximation of Matrices for PMI-based Word Embeddings
We perform an empirical evaluation of several methods of low-rank approximation in the problem of obtaining PMI-based word embeddings.
Context Vectors are Reflections of Word Vectors in Half the Dimensions
This paper takes a step towards theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec.
Fourier Neural Networks: A Comparative Study
We review neural network architectures which were motivated by Fourier series and integrals and which are referred to as Fourier neural networks.
Reproducing and Regularizing the SCRN Model
We reproduce the Structurally Constrained Recurrent Network (SCRN) model, and then regularize it using the existing widespread techniques, such as naive dropout, variational dropout, and weight tying.
Reusing Weights in Subword-aware Neural Language Models
We propose several ways of reusing subword embeddings and other weights in subword-aware neural language models.
Patterns versus Characters in Subword-aware Neural Language Modeling
Further, for every word we construct a new sequence over an alphabet of patterns.
Syllable-aware Neural Language Models: A Failure to Beat Character-aware Ones
Syllabification does not seem to improve word-level RNN language modeling quality when compared to character-based segmentation.
