Randomized Positional Encodings Boost Length Generalization of Transformers
Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, Joel Veness
Abstract
Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, Joel Veness. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2023.
Topics & Concepts
TransformerGeneralizationComputer scienceLibrary scienceArtificial intelligenceMathematicsEngineeringElectrical engineeringMathematical analysisVoltageNatural Language Processing TechniquesTopic ModelingSpeech Recognition and Synthesis