Can attention enable MLPs to catch up with CNNs?
Meng-Hao Guo, Zheng-Ning Liu, Tai‐Jiang Mu, Dun Liang, Ralph R. Martin, Shi‐Min Hu
Abstract
In the first week of May 2021, researchers from four different institutions: Google, Tsinghua University, Oxford University, and Facebook shared their latest work [1-4] on arXiv.org at almost the same time, each proposing new learning architectures, consisting mainly of linear layers, claiming them to be comparable or superior to convolutional-based models.This sparked immediate discussion and debate in both academic and industrial communities as to whether MLPs are sufficient, many thinking that learning architectures are returning to MLPs.Is this true?In the following, we give a brief history of learning architectures, including multilayer perceptrons (MLPs), convolutional neural networks (CNNs), and transformers.We then examine what the four newly proposed architectures have in common.Finally, we give our views on challenges and directions for new learning architectures, hoping to inspire future research. Learning architectures for visual tasksMultilayer perceptrons (MLPs) [5] consist of an input layer and an output layer, possibly with multiple hidden layers in between.Layers are typically fully connected using linear transformations and activation functions.MLPs were the basis for neural networks before deep convolutional neural networks (DCNNs) took over, and greatly improved the power