DeepViT: Towards Deeper Vision Transformer
Daquan Zhou, Bingyi Kang, Xiaojie Jin et al.
2021 · arXiv (Cornell University) · 349 citations
Vision transformers (ViTs) have been successfully applied in image classification tasks recently. In this paper, we show that, unlike convolution neural networks (CNNs)that can be improved by stacking more convolutional layers, the performance of ViTs saturate fast when scaled to be deeper. More specifically, we empirically observe that such scaling difficulty is caused by the attention collapse issue: as the transformer goes deeper, the attention maps gradually become similar and even much the same after certain layers. In other words, the feature maps tend to be identical in the top layers…
Explore this paper's citation graph on Constellation.