Summary
- RetroSTC-Net implements synaptic tagging and capture in backprop-trained networks, so that a salient event after learning decides which weight changes become permanent.
- Each weight is split into a consolidated and a labile component, with decaying per-weight tags and shared plasticity-related products, and every consolidated change is logged with the task and event that produced it.
- On binary MNIST tasks an event at the end of learning keeps 91% of a task's learning and holds accuracy at about 0.95 instead of 0.61, and the onset of a later task can rescue an earlier one retroactively.
- In a deletion test the consolidation log separates two memories more clearly than Fisher importance, and used as importance on Split-MNIST it reaches EWC level (0.636 against 0.637).
Overview
The synaptic tagging and capture (STC) hypothesis (Frey and Morris, 1997) holds that learning leaves a short-lived change and a temporary tag, and the change becomes permanent only if plasticity-related products, which a separate salient event can trigger, arrive while the tag is present. Behaviourally, an unfamiliar experience shortly before or after weak learning makes the weak memory last.
This project implements the mechanism in backprop-trained networks, tests for the behavioural-tagging time window and uses the consolidation record as an explanation of what the network retains. Continual learning methods such as Elastic Weight Consolidation (EWC) and Synaptic Intelligence (SI) fix a weight's importance at learning time, whereas importance often becomes clear only later, for instance when a fraud label arrives.
Building on related work on tag-and-capture and two-component weights by Leimer, Herzog and Senn (2019), Li and van Rossum (2020) and Lehr, Luboeinski and Tetzlaff (2022), this project adds:
- retroactive consolidation in a backprop-trained network, decided by a salience event after learning.
- a behavioural-tagging time-window test in network weights, including rescue of one task by another.
- a consolidation log as an explanation of what was retained, validated with a faithfulness test.
Model
Two-component weights
Each parameter is split into a consolidated part L and a labile part E, with effective weight w = L + E. The optimiser updates w as usual, so standard training is unchanged. After each step the change is moved into E and the STC dynamics are applied:
| Biology | Network |
|---|---|
| Early-phase LTP decays | E ← E · exp(−1/τE) |
| Synaptic tag | h ← h · exp(−1/τh) + |Δw| |
| Plasticity-related products, shared per neuron | p ← p · exp(−1/τp) + α·s(t) |
| Capture | c = clamp(κ · p · [h > θ], 0, 1), then L += c·E, E −= c·E |
| Salient event | s(t): a pulse, or a sudden jump in training loss |
The default time constants are τE = 1000, τh = 150 and τp = 80 training steps, with capture rate κ = 0.05 and tag threshold θ = 10⁻³.
Tags, products and capture
Capture requires a tag and products at the same step, which creates a time window around the event. Without an event every change decays. In the default mode every unit receives the same pulse, so the products act as one global signal and the tags decide where capture happens. Two variants serve as ablations: products made only at synapses tagged at event time, and a separate tag for each learning episode's change instead of one per weight, which stops a later task from re-tagging weights it shares with an earlier one.
Salience events
An event is either an explicit pulse at a chosen step or the output of a surprise detector that needs no task labels. The detector fires when the training loss exceeds its running mean by three running standard deviations, with a refractory period of 150 steps. Its settings were fixed in advance rather than tuned.
Consolidation log and measures
Labile change is tracked per learning episode, so every consolidated change can be traced to its task and capturing event. The attribution is exact when events do not overlap within the products' lifetime. From this record the network can report, for each event, how much of each task it consolidated and which hidden units carry it, ranked by the consolidated change of their incoming weights. The main measure is the consolidated share of a task's learning, Σ(consolidated · change) / Σ(change²), the consolidated change projected onto the task's total change, since accuracy saturates quickly on these tasks.
Experimental setup
An MLP with two hidden layers of 256 units is trained with SGD (learning rate 0.05, batch size 64) on two binary MNIST tasks from harder digit pairs (A: 3 vs 5, B: 4 vs 9), over five seeds. Each run ends with 4000 steps without learning, after which the labile part has decayed to under 2% of its value, so only consolidated memory remains. Multi-task experiments use a separate output head per task, so interference between tasks runs through the shared hidden layers.
Results
Event timing determines what is kept
Task A is trained for 60 steps and a single salient pulse is placed at a variable time relative to the end of training.
| Event time (steps) | None | −300 | −150 | 0 | +150 | +300 | +600 |
|---|---|---|---|---|---|---|---|
| Share of A kept | 0.00 | 0.13 | 0.58 | 0.91 | 0.71 | 0.42 | 0.03 |
Without an event, accuracy on A falls from 0.95 to 0.61. Near the event it stays at about 0.95. As STC predicts, tag lifetime mainly shifts the after-side of the window and product lifetime the before-side: at +300 steps the share kept rises from 0.01 to 0.63 as τh goes from 75 to 300, while at −150 steps it rises from 0.11 to 0.91 as τp goes from 40 to 160. Since the shape follows from the time constants, this confirms that the mechanism works inside gradient training rather than showing that a window emerges on its own. Without tags, an event at +600 steps still keeps 51%, and size-triggered capture keeps 62% with no event, leaving later events no influence.
Cross-task rescue
Task A is trained for 60 steps, followed by an idle gap and 400 steps on task B, with a salience pulse at B's onset.
A salient onset of task B rescues task A retroactively. Without salience neither is kept, and B itself falls to chance after the delay. With per-weight tags, B re-tags shared weights and still captures 24% of A at a 900-step gap. With per-change tags the window is clean: the share of A kept drops from 0.91 at no gap to 0.43 at 300 steps and to zero at 900.
The consolidation log as an explanation
Faithfulness is evaluated with a deletion test: parameters are ranked by the log's attribution, by empirical Fisher importance computed on 1,000 training images, by change size or at random, and the top 5% are reset to pre-learning values. Removing the log's task A parameters drops A from 0.87 to 0.80 while sparing B. Removing B's drops B from 0.97 to 0.69 while sparing A. Fisher importance separates the memories far less clearly. Change size matches the log for A (0.796 against 0.797). The log does better for B (0.691 against 0.764), which is where consolidation adds to the explanation. The record is also consistent at the level of memory sources: evaluating only the consolidated weights at the end of task B gives higher accuracy on A (0.873) than the full weights (0.846), indicating that A is held in consolidated memory while B's labile changes interfere with it.
Forgetting on Split-MNIST
Five binary tasks (0/1, 2/3, 4/5, 6/7, 8/9) are learned in sequence, 500 steps each, in the task-incremental and domain-incremental settings. Surprise-gated variants fire an event when training loss jumps, without task labels. A further variant uses the consolidation log as importance in an EWC-style penalty that pulls each parameter towards its consolidated value, weighted by its normalised consolidated change. Baselines are EWC and SI, with penalty strengths tuned per method on a separate seed.
| Method | Task-incremental | Domain-incremental |
|---|---|---|
| Plain SGD | 0.980 | 0.616 |
| EWC | 0.987 | 0.637 |
| Synaptic Intelligence | 0.987 | 0.657 |
| STC, surprise-gated | 0.980 | 0.621 |
| STC log as importance, surprise-gated | 0.980 | 0.636 |
STC alone does not measurably reduce forgetting, since consolidated weights are not protected from later learning. Using the consolidation log as importance reaches EWC level, with importance selected by surprising events after learning rather than fixed during it. It does not reach Synaptic Intelligence.
Key takeaways
- Parameter importance can be determined after learning, as a salient event determined which changes the network retained.
- In a deletion test, the consolidation log accounted for retention more clearly than Fisher importance.
- Consolidation alone did not reduce forgetting. A benefit appeared only when the log was used as importance in a regularisation penalty.