Search Results for author:
Found 9 papers, 1 papers with code
Leveraging redundancy in attention with Reuse Transformers
Pairwise dot product-based attention allows Transformers to exchange information between tokens in an input-dependent way, and is key to their success across diverse applications in language and vision.
A Simple and Effective Positional Encoding for Transformers
Our analysis shows that the gain actually comes from moving positional information to attention layer from the input.
O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers
We propose sufficient conditions under which we prove that a sparse attention model can universally approximate any sequence-to-sequence function.
$O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers
We propose sufficient conditions under which we prove that a sparse attention model can universally approximate any sequence-to-sequence function.
Pre-training Tasks for Embedding-based Large-scale Retrieval
We consider the large-scale query-document retrieval problem: given a query (e. g., a question), return the set of relevant documents (e. g., paragraphs containing the answer) from a large document corpus.
Source-Side Left-to-Right or Target-Side Left-to-Right? An Empirical Comparison of Two Phrase-Based Decoding Algorithms
The algorithm produces a translation by processing the source-language sentence in strictly left-to-right order, differing from commonly used approaches that build the target-language sentence in left-to-right order.
A Polynomial-Time Dynamic Programming Algorithm for Phrase-Based Decoding with a Fixed Distortion Limit
Decoding of phrase-based translation models in the general case is known to be NP-complete, by a reduction from the traveling salesman problem (Knight, 1999).
