① Intuition
During autoregressive decoding, the transformer generates one token at a time. Each new token must attend to every prior token, which requires storing the key (K) and value (V) projections for all past positions. This KV-cache grows with sequence length, number of layers, and — crucially — the number of attention heads. In standard Multi-Head Attention (MHA) with h heads, you store h separate (K, V) pairs per token per layer, each of dimension d_head. For a 65B-parameter model serving a 4096-token context, this can reach several gigabytes — more than the weights themselves at fp16.
Multi-Query Attention (MQA) proposed by Shazeer (2019) solves this by making all h query heads share a single (K, V) pair. The KV-cache shrinks by a factor of h, yielding dramatic speedups. The catch: training MQA from scratch sometimes degrades quality, and you cannot easily convert an existing MHA checkpoint without retraining from scratch.
Grouped-Query Attention (GQA) splits the difference. Instead of 1 or h KV heads, it uses g groups (1 ≤ g ≤ h), each group sharing one (K, V) pair among h/g query heads. When g = h you recover MHA; when g = 1 you recover MQA. The key insight: you can initialize a GQA model by mean-pooling the pretrained MHA K and V projection matrices within each group — giving a warm start that requires only ~5% of the original pretraining tokens to recover most of the original quality.
Before using the widget: if a model has h=16 query heads and you reduce KV groups from g=16 to g=4, by what factor does the KV-cache shrink, and what happens to the query projections?
The KV-cache shrinks by a factor of h/g = 16/4 = 4. The query projections are completely unchanged — all 16 Q heads remain. Only the K and V heads are reduced from 16 independent copies to 4 shared copies. This is why GQA preserves most of MHA’s expressiveness: queries still attend with full diversity, they just share keys and values within their group.
Interact with the widget: slide the KV groups slider from h (MHA) down to 1 (MQA) and watch the cache shrink. The current-g column shows you exactly how much memory a given g choice uses versus the two extremes.
MHA cache (g=h)
1024.0 MB
GQA cache (current g)
256.0 MB
MQA cache (g=1)
64.0 MB
Mode: Grouped-Query (GQA)
Fixed: seq_len=4096, n_layers=32, d_head=128, fp16
After exploring the widget: with h=16 and g=4, the GQA cache is 1/4 of MHA. What would happen to decoding throughput if the KV-cache is the memory-bandwidth bottleneck, and why does g=4 often outperform g=1 in practice?
When KV-cache bandwidth is the bottleneck, reducing the cache size by 4x means 4x less data to read per decode step — approaching a 4x throughput improvement. GQA (g=4) often beats MQA (g=1) in practice because g=1 forces all heads to share identical keys/values, losing the ability to attend to different aspects of the context simultaneously. GQA at g=4 retains multiple independent context summaries per layer while still achieving most of MQA’s memory savings. Llama 2 70B and Mistral 7B both use g=8 for this reason.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
The KV-cache size for a single sequence scales as:
where g is the number of KV groups, d_head is the head dimension, the factor 2 accounts for both K and V, L is the number of layers, T is the current sequence length, and b_e is bytes per element (2 for fp16). The reduction ratio from MHA to GQA is simply h/g: replacing h independent KV heads with g shared ones cuts the cache by that factor.
For GQA attention itself, each query head q_i in group k attends using the single shared key and value for that group:
The mean-pool initialization for a group containing head indices S_k is:
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the four steps of converting a pretrained MHA checkpoint to GQA in the correct order:
- Keep all h Q projection matrices unchanged
- Partition the h query heads into g equal groups of size h/g each
- Fine-tune the GQA model for ~5% of original pretraining tokens to recover quality
- For each group, mean-pool the K and V projection matrices: W_K_group = mean(W_K[heads_in_group], axis=0)
Fill in the blank — the mean-pool step for K projections inside the group loop:
def mha_to_gqa(W_K, h, g):
# W_K: shape (h, d_model, d_head)
group_size = h // g
W_K_gqa = []
for k in range(g):
heads = W_K[k * group_size : (k + 1) * group_size]
W_K_gqa.append(_________)
return stack(W_K_gqa) # shape (g, d_model, d_head)Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement the group-mean-pooling of K (or V) projection weights in NumPy. Given W_K of shape (h, d_model, d_head) and the target number of groups g, return W_K_gqa of shape (g, d_model, d_head) where each group is the mean of its h // g consecutive heads.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §2.2 (Grouped-Query Attention — the definition of g groups, the mean-pool initialization procedure, and how it generalizes MHA and MQA) → §3 (Experiments — model sizes, datasets, and baseline comparisons) → §3.2 (Speed — memory bandwidth analysis and wall-clock decoding throughput results).