Litcius/Paper detail

Mixhead: Breaking the low-rank bottleneck in multi-head attention language models

Zhong Zhang, Nian Shao, Chongming Gao, Rui Miao, Qinli Yang, Junming Shao

2022Knowledge-Based Systems19 citationsDOI

Topics & Concepts

BottleneckOverfittingComputer scienceMachine translationTransformerRank (graph theory)Head (geology)Artificial intelligenceLanguage modelMachine learningAlgorithmMathematicsEngineeringArtificial neural networkGeologyElectrical engineeringVoltageCombinatoricsEmbedded systemGeomorphologyTopic ModelingDomain Adaptation and Few-Shot LearningNatural Language Processing Techniques
Mixhead: Breaking the low-rank bottleneck in multi-head attention language models | Litcius