Complete · weights public
SalienceFormer
A transformer with two additions borrowed from the hippocampus.
- Role
- Sole researcher
- Stack
- PyTorch · HuggingFace Hub
- Code
- Repository →
11.83
perplexity on WikiText-2
35%
better than the Gemma-2B baseline
15
ablation variants, significance tested
How it works
- 01 Token stream ordinary embeddings in
- 02 Salience gate scores what is worth encoding
- 03 Selective encode low-salience tokens pass thin
- 04 Consolidation rehearses gated states
- 05 Decoder same parameter budget
Nothing is added to the parameter count. The gate only changes where capacity goes.
WikiText-2 perplexity, lower is better
Equal parameters. 15 ablations confirm the gate, not the seed, is responsible.
The problem
Standard transformers spend the same capacity on every token they see. Biological memory does not: a salience signal decides what is worth encoding, and consolidation rehearses it afterwards. If that asymmetry matters, a same-size model should beat a baseline purely by being selective.
The approach
Two layers added to an otherwise ordinary architecture — a salience gate that scores what to encode, and a consolidation layer that rehearses gated representations across steps. Parameter budget held constant, so any gain has to come from the mechanism rather than scale.
What broke
The first gate collapsed: it learned to pass everything, which is the safe solution under a plain language-modelling loss. Recovering selectivity needed explicit pressure on the gate, and then fifteen ablation variants with significance testing to show the result was the mechanism and not a lucky seed.
What worked
11.83 perplexity on WikiText-2 against a Gemma-2B baseline — a 35% improvement at equal parameter count. Weights and configs are on the HuggingFace Hub so the number can be checked rather than believed.