ConViT: improving vision transformers with soft convolutional inductive biases*

Stéphane d’Ascoli, Hugo Touvron, Matthew L. Leavitt et al.

2022 · Journal of Statistical Mechanics Theory and Experiment · 716 citations

Abstract Convolutional architectures have proven to be extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision transformers rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification. However, they require costly pre-training on large external datasets or distillation from pre-trained convolutional networks. In this paper, we ask the following question: is it possible to combine the strengths of these two architectures while avoidi…

Read the paper →

Explore this paper's citation graph on Constellation.