← Roadmap

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

Ainslie et al. · 2023

① Intuition

During autoregressive decoding, the transformer generates one token at a time. Each new token must attend to every prior token, which requires storing the key (K) and value (V) projections for all past positions. This KV-cache grows with sequence length, number of layers, and — crucially — the number of attention heads. In standard Multi-Head Attention (MHA) with h heads, you store h separate (K, V) pairs per token per layer, each of dimension d_head. For a 65B-parameter model serving a 4096-token context, this can reach several gigabytes — more than the weights themselves at fp16.

Multi-Query Attention (MQA) proposed by Shazeer (2019) solves this by making all h query heads share a single (K, V) pair. The KV-cache shrinks by a factor of h, yielding dramatic speedups. The catch: training MQA from scratch sometimes degrades quality, and you cannot easily convert an existing MHA checkpoint without retraining from scratch.

Grouped-Query Attention (GQA) splits the difference. Instead of 1 or h KV heads, it uses g groups (1 ≤ g ≤ h), each group sharing one (K, V) pair among h/g query heads. When g = h you recover MHA; when g = 1 you recover MQA. The key insight: you can initialize a GQA model by mean-pooling the pretrained MHA K and V projection matrices within each group — giving a warm start that requires only ~5% of the original pretraining tokens to recover most of the original quality.

Before using the widget: if a model has h=16 query heads and you reduce KV groups from g=16 to g=4, by what factor does the KV-cache shrink, and what happens to the query projections?

Interact with the widget: slide the KV groups slider from h (MHA) down to 1 (MQA) and watch the cache shrink. The current-g column shows you exactly how much memory a given g choice uses versus the two extremes.

MHA cache (g=h)

1024.0 MB

GQA cache (current g)

256.0 MB

MQA cache (g=1)

64.0 MB

Mode: Grouped-Query (GQA)

Fixed: seq_len=4096, n_layers=32, d_head=128, fp16

After exploring the widget: with h=16 and g=4, the GQA cache is 1/4 of MHA. What would happen to decoding throughput if the KV-cache is the memory-bandwidth bottleneck, and why does g=4 often outperform g=1 in practice?

Retrieval check

During autoregressive decoding, why does the KV-cache dominate memory rather than model weights?
How does GQA's mean-pool initialization avoid training from scratch?

Answer all 2 correctly to mark this stage mastered.

② Math

The KV-cache size for a single sequence scales as:

KV-cache bytes=g×dhead×2×L×T×be\text{KV-cache bytes} = g \times d_\text{head} \times 2 \times L \times T \times b_e

where g is the number of KV groups, d_head is the head dimension, the factor 2 accounts for both K and V, L is the number of layers, T is the current sequence length, and b_e is bytes per element (2 for fp16). The reduction ratio from MHA to GQA is simply h/g: replacing h independent KV heads with g shared ones cuts the cache by that factor.

For GQA attention itself, each query head q_i in group k attends using the single shared key and value for that group:

Attn(Qi,Kk,Vk)=softmax ⁣(QiKk⊤dhead)Vk,k=⌊i⋅gh⌋\text{Attn}(Q_i, K_k, V_k) = \text{softmax}\!\left(\frac{Q_i K_k^\top}{\sqrt{d_\text{head}}}\right) V_k, \quad k = \left\lfloor \frac{i \cdot g}{h} \right\rfloor

The mean-pool initialization for a group containing head indices S_k is:

W^kK=1∣Sk∣∑j∈SkWjK,W^kV=1∣Sk∣∑j∈SkWjV\hat{W}^K_k = \frac{1}{|S_k|} \sum_{j \in S_k} W^K_j, \quad \hat{W}^V_k = \frac{1}{|S_k|} \sum_{j \in S_k} W^V_j
What does g=h reduce to in the GQA cache formula, and what does g=1 reduce to?
Show next line
g=h⇒MHA,g=1⇒MQA,ratio=h/gg{=}h \Rightarrow \text{MHA},\quad g{=}1 \Rightarrow \text{MQA},\quad \text{ratio} = h/g
Why does mean-pooling pretrained MHA heads give a better GQA initialization than random weights?
Show next line
W^kK=1∣Sk∣∑j∈SkWjK  ⇒  warm start, not random\hat{W}^K_k = \frac{1}{|S_k|}\sum_{j \in S_k} W^K_j \;\Rightarrow\; \text{warm start, not random}

Retrieval check

In the GQA attention formula, which projections are shared within a group and which are not?
If a model has h=32 query heads, g=8 KV groups, and the total MHA KV-cache at some sequence length is 2 GB, what is the GQA KV-cache?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the four steps of converting a pretrained MHA checkpoint to GQA in the correct order:

  1. Keep all h Q projection matrices unchanged
  2. Partition the h query heads into g equal groups of size h/g each
  3. Fine-tune the GQA model for ~5% of original pretraining tokens to recover quality
  4. For each group, mean-pool the K and V projection matrices: W_K_group = mean(W_K[heads_in_group], axis=0)

Fill in the blank — the mean-pool step for K projections inside the group loop:

def mha_to_gqa(W_K, h, g):
    # W_K: shape (h, d_model, d_head)
    group_size = h // g
    W_K_gqa = []
    for k in range(g):
        heads = W_K[k * group_size : (k + 1) * group_size]
        W_K_gqa.append(_________)
    return stack(W_K_gqa)  # shape (g, d_model, d_head)

Retrieval check

Why is fine-tuning needed after mean-pool initialization, given that the weights are already pretrained?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement the group-mean-pooling of K (or V) projection weights in NumPy. Given W_K of shape (h, d_model, d_head) and the target number of groups g, return W_K_gqa of shape (g, d_model, d_head) where each group is the mean of its h // g consecutive heads.

Retrieval check

In the pool_kv_heads implementation, why is a reshape to (g, group_size, d_model, d_head) followed by mean(axis=1) correct for group mean-pooling?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al. 2023).

Open in new tab · Download

Suggested reading order: §2.2 (Grouped-Query Attention — the definition of g groups, the mean-pool initialization procedure, and how it generalizes MHA and MQA) → §3 (Experiments — model sizes, datasets, and baseline comparisons) → §3.2 (Speed — memory bandwidth analysis and wall-clock decoding throughput results).