Positional Encoding in the Transformer

The original Transformer gives every position a fixed 512-dimensional vector of sines and cosines and adds it to the word embedding. Early dimensions change fast and separate neighbouring positions; late dimensions change slowly and separate distant ones.

Sense Wang · September 2026

1. Attention ignores order

A token is the unit of text the model reads: a word or a piece of a word. In self-attention, token i scores every token j with a dot product of a query vector and a key vector:

\[ a_{ij} = \mathrm{softmax}_j\!\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right) \]

Nothing in this formula depends on where token j sits. Swap two tokens and the outputs are swapped too, so “dog bites man” and “man bites dog” look the same to the layer. The paper therefore has to inject position explicitly [1].

2. Add a position vector to each input

\[ x_i = E[t_i]\cdot\sqrt{d_{\text{model}}} + PE(i) \]

\(t_i\) is the id of the i-th token. \(E\) is the learned embedding table, one row per token. \(d_{\text{model}} = 512\) is the model width; the paper multiplies embeddings by \(\sqrt{d_{\text{model}}}\) [1]. \(PE(i)\) is the position vector for position i, also 512-dimensional.

3. The formula

\[ \begin{aligned} PE_{(pos,\,2i)} &= \sin(\omega_i\, pos) \\ PE_{(pos,\,2i+1)} &= \cos(\omega_i\, pos) \\ \omega_i &= 1/10000^{2i/d_{\text{model}}} \end{aligned} \]

Dimensions come in 256 sin/cos pairs; pair i has frequency \(\omega_i\) and period \(2\pi/\omega_i\). The periods run from \(2\pi \approx 6\) positions to about \(10000 \cdot 2\pi\) [1]. The table has no trainable parameters.

Heatmap of the sinusoidal positional-encoding matrix, positions 0 to 63 by dimensions 0 to 511
Figure 1. The first 64 rows of the encoding. Each row is one position; each column is one dimension. Red is +1, blue is −1, white is 0. Computed from the paper’s formula with \(d_{\text{model}} = 512\).

4. Each column is a sine wave

Pull single columns out of Figure 1 and each one is a sinusoid. Only the period differs.

Values of dimensions 0, 32, 128, 256 and 500 plotted against position
Figure 2. Five sin dimensions over positions 0–63. Dots are the values at integer positions. Dimension 0 repeats every ~6 positions; dimension 128 completes one cycle in 64; dimension 500 barely moves.

A binary counter has the same structure: the lowest bit flips every step, the next every 2 steps, the next every 4. Fast dimensions tell neighbours apart, slow dimensions tell distant positions apart, and together each position gets a distinct vector.

5. Periods grow geometrically

\[ \frac{\text{period}_{i+1}}{\text{period}_i} = 10000^{2/512} \approx 1.036 \]

Each pair’s period is 3.6% longer than the previous one, so on a log axis the periods form a straight line. From dimension 130 on, the period is longer than the 64 positions shown in Figure 1.

Period of each sin/cos pair on a log axis, with a line at 64 positions
Figure 3. Period of each dimension pair, log scale. The last pair’s period is about 60,611 positions. The dashed line marks 64 positions.

6. The right half is constant only in short sequences

A dimension whose period is much longer than the sequence stays near its starting value: \(\sin 0 = 0\) (white) or \(\cos 0 = 1\) (red). That is the red-and-white striping on the right of Figure 1.

Range of each dimension over positions 0 to 63, for sin and cos dimensions
Figure 4. How much each dimension changes over positions 0–63 (max minus min). 194 of 512 dimensions change by less than 0.1; 264 change by less than 0.5.

MeasuredIn a 64-token window, position information sits mostly in the early dimensions, and the late dimensions act as near-constant offsets that the learned embedding can use freely. This split comes from the choice of periods; nothing in the architecture enforces it. We did not check whether trained models actually use the dimensions this way.

7. A shift by k is a rotation

\[ \begin{pmatrix} s_{pos+k} \\ c_{pos+k} \end{pmatrix} = \begin{pmatrix} \cos\theta & \sin\theta \\ -\sin\theta & \cos\theta \end{pmatrix} \begin{pmatrix} s_{pos} \\ c_{pos} \end{pmatrix}, \qquad \theta = \omega_i k \]

Here \(s_{pos} = \sin(\omega_i\, pos)\) and \(c_{pos} = \cos(\omega_i\, pos)\) are pair i of \(PE(pos)\). The matrix depends on the offset k and not on pos, so \(PE(pos+k)\) is a fixed linear function of \(PE(pos)\).

PaperThe authors hypothesized that this lets the model learn to attend by relative position. The paper does not test the hypothesis [1].

8. Similarity depends only on distance

\[ PE(pos)\cdot PE(pos+k) = \sum_{i=0}^{255} \cos(\omega_i k) \]

Using \(\sin a \sin b + \cos a \cos b = \cos(a-b)\), the dot product of two position vectors depends only on their distance k. Nearby positions score higher.

Dot product between position vectors as a function of offset k
Figure 5. Dot product between two position vectors versus their offset k. The maximum, 256, is at k = 0. The same curve holds for every starting position; for k = 37 the value is 139.05 at pos = 0, 100 and 300.

9. Learned embeddings did as well

PaperReplacing the sinusoids with a learned positional embedding (one trainable vector per position) gave perplexity 4.92 and BLEU 25.7 on English–German newstest2013, against 4.92 and 25.8 for the sinusoidal base model (Table 3, row E) [1]. The authors kept sinusoids because they may extrapolate to sequences longer than those seen in training; the paper has no experiment on this.

10. Adding mixes word and position

\[ \begin{aligned} (w_1+p_1)\cdot(w_2+p_2) = {} & \underbrace{w_1\cdot w_2}_{\text{word}} + \underbrace{p_1\cdot p_2}_{\text{position}} \\ & + \underbrace{w_1\cdot p_2 + p_1\cdot w_2}_{\text{cross terms}} \end{aligned} \]

Yes, they mix: the cross terms are noise for attention. The paper never compares adding with concatenating. Three reasons it still works, all Inference:

  • Comparable scale. PE entries lie in [−1, 1]. In tensor2tensor, embeddings are initialized with standard deviation \(d_{\text{model}}^{-1/2}\) and multiplied by \(\sqrt{d_{\text{model}}}\), so their entries are also about 1 [2]. The paper does not explain the \(\sqrt{d_{\text{model}}}\).
  • Soft dimension split. Figure 4: in short sequences, late dimensions carry little position signal, leaving room for word information.
  • Learned projections. \(q = xW^Q\) and \(k = xW^K\) can learn to pick out word-related and position-related directions before the dot product.

11. What replaced it: RoPE

\[ \begin{aligned} \tilde q_m = R(m\omega)\, q_m, &\quad \tilde k_n = R(n\omega)\, k_n \\ \tilde q_m \cdot \tilde k_n &= q_m^{\top} R\big((n-m)\omega\big)\, k_n \end{aligned} \]

Rotary Position Embedding (RoPE) [3] applies the rotation from section 7 to each pair of query and key dimensions inside every attention layer, instead of adding a vector at the input. The attention score then sees position only through the offset \(n-m\), and there are no word–position cross terms. LLaMA [4] and Qwen [5] use RoPE.

  1. Vaswani et al., Attention Is All You Need, NeurIPS 2017. Section 3.4 (embedding scale), Section 3.5 (positional encoding and the relative-position hypothesis), Table 3 row (E).
  2. tensor2tensor, layers/modalities.py: embedding initializer hidden_dim**-0.5; sqrt_depth mode multiplies by hidden_size**0.5.
  3. Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, 2021.
  4. Touvron et al., LLaMA: Open and Efficient Foundation Language Models, 2023.
  5. Bai et al., Qwen Technical Report, 2023.

Figures 1–5 are computed by the author from the formula in [1].