An attention sink can mean two opposite things
A small reproduction of a June 2026 paper showing that the same attention stripe can be either a no-op or a broadcast channel.
Attention maps are tempting to read as explanations. A vertical stripe says "many tokens looked at this token," so it is natural to ask what that token was doing.
The June 2026 paper A Unifying View of Attention Sinks: Two Algorithms, Two Solutions makes a useful warning concrete: the same sink stripe can mean two different algorithms.
- Adaptive no-op: route to a sink whose value vector is almost zero, so the attention head effectively writes nothing.
- Broadcast: route to a sink whose value vector carries a payload, so the head writes a shared low-rank update to many tokens.
I reproduced the core diagnostic in a small synthetic attention head. This is not a ViT reproduction; it isolates the mechanism the paper uses to explain why attention alone is ambiguous. Repro code path: /nvme0n1-disk/code/papersreplicate/attention-sinks-two-algorithms.
The controlled setup
I used one row-stochastic attention matrix with a strong sink column. Then I reused that exact attention matrix under two value regimes:
same attention stripe
+ zero sink value -> no-op
+ content-scale value -> broadcastThe paper's diagnostics are simple:
| Diagnostic | No-op prediction | Broadcast prediction |
|---|---|---|
| Sink attention strength | high | high |
| Sink value norm | near zero | content scale |
| Attention output rank | not necessarily low | near rank-1 |
| Token geometry | little movement | shared component appears |
What happened
Both runs had the same mean sink attention strength: 0.9516. The attention map alone cannot distinguish them.
| Metric from the run | No-op sink | Broadcast sink |
|---|---|---|
| Sink value norm ratio | 0.0000 | 1.0000 |
| Relative update norm | 0.0484 | 0.9425 |
| Stable rank of attention output | 7.1971 | 1.0027 |
| Top singular energy fraction | 0.1389 | 0.9973 |
| Mean pairwise cosine after update | -0.0317 | 0.4251 |
That matches the paper's central claim in miniature. With the no-op sink, almost all attention goes to a value vector of zero, leaving only small self-leakage. With the broadcast sink, the same stripe writes one shared vector across the sequence, so the output is essentially rank-1.
My extra sweep
I added a phase sweep the paper does not show in this form: vary the sink attention strength and the sink value scale, then measure when the output becomes rank-1.
At sink strength near 0.95, the output became rank-1-like once the sink value scale reached about 0.095. That is the useful edge case: a sink does not need a full-size payload to dominate if attention is already saturated. The boundary is set by a competition between the broadcast payload and the residual self-leakage left by 1 - attention_to_sink.
The geometric view shows the same thing. The no-op case barely moves tokens; the broadcast case adds a shared component, pulling token directions into alignment.
Takeaway
The result is small but sharp: a sink is not a mechanism. It is a routing pattern.
If the sink value norm is zero, the head is using softmax attention as a conditional off switch. If the sink value is nonzero and the update is rank-1, the head is using the sink as a broadcast channel. Those call for different interventions, which is the point of the paper's gating-versus-register framing.
The honest limitation is that this repro constructs the two regimes rather than training a transformer to discover them. It verifies the diagnostic logic, not the paper's large-model prevalence claims.
cd /nvme0n1-disk/code/papersreplicate/attention-sinks-two-algorithms
uv sync
uv run python -m attention_sinks_two_algorithms.run --out-dir figures
uv run pytest