Conditional Positional Encodings for Vision Transformers

Xiangxiang Chu, Zhi Tian, Bo Zhang et al.

2021 · arXiv (Cornell University) · 407 citations

We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike previous fixed or learnable positional encodings, which are pre-defined and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during training. Besides, CPE can keep the desired translation-invariance in the image classification task, resulting in improved performance. We implement CPE with a simple Position Encoding Generato…

Read the paper →

Explore this paper's citation graph on Constellation.