DeepMath 2026 · Ohio State · 29–30 Oct

Self-Repair as a Bounded Disturbance: Input-to-State Stability of Neural Network Training Dynamics

Rokib Al Dhin Raadh · Independent researcher · high-school senior, Dhaka, Bangladesh

From the author

Thanks for scanning. I couldn't be here in person.

I'm Rokib Al Dhin Raadh, an 18-year-old independent researcher in Dhaka, Bangladesh. I wrote this paper on my own, outside any lab, and it was accepted after double-blind review. I couldn't travel to Ohio State for the conference, so the poster is here without me, but you can still hear the work from me and ask me anything about it.

Invited talk · Cohere Labs

When the Update Is Not a Gradient

Monday 2 November 2026, 12:00 ET (17:00 UTC), online

Cohere Labs invited me as a guest speaker to present this DeepMath work to its Open Science Community. If you're interested, register and join live, with time for questions.

Register on Luma

The poster

DeepMath 2026 poster: Self-Repair as a Bounded Disturbance. Tap to open the full A0 PDF.
Open full poster (PDF) Download

A0 landscape, vector PDF. Pinch to zoom on a phone.

The takeaway

Small repairs keep training stable. Only a jump can revive a dead layer.

We prove both, which makes self-repair a switched system: bounded flicks between isolated jumps.

Abstract

Methods that repair a network while it trains, such as Continual Backprop, ReDo, Shrink & Perturb and plasticity injection, apply updates that are not the gradient of any loss, so the usual convergence proofs do not cover them. We model each repair as an input to the training dynamics and ask for input-to-state stability (ISS). Proposition 1 shows that near a strongly convex minimum, bounded repairs keep training within a radius γ = 2βM/α + ε̄/(ηα) of the minimum, where β ≤ 0.3 weights each layer's local loss, M bounds that loss's gradient, and ε̄ is the asymptotic repair size. Theorem 2 gives the matching limit: in a Conv–BN–ReLU layer, a channel with a zero filter and non-positive BatchNorm shift is invariant under AdamW and under every per-channel-bounded repair, so only a reset can revive it. Self-repairing training is therefore a switched system: bounded corrections between isolated jumps. We test both predictions in a 383-run evaluation. After a shock, the largest correction per epoch falls by 93% (5.07 → 0.38); a CIFAR-10 CNN with two zeroed conv layers stays at exactly 10.00% under standard AdamW and recovers to 80.3% with self-repair; on ResNet-18 the skip path makes repair unnecessary.

Keywords: input-to-state stability · switched systems · loss of plasticity · self-repairing training · BatchNorm

Three numbers

  • 10.00%without repair, after two conv layers are zeroed: exactly chance, 99 epochs, 3/3 seeds.
  • 39 493the training step where the first reset fires, identical in every seed.
  • 80.3%with self-repair (85% before the lesion).

Two results

Proposition 1

Bounded repair is safe (practical ISS)

Near a minimum θ∗\theta^*, let LL be ℓ\ell-smooth and α\alpha-strongly convex, η≤1/ℓ\eta \le 1/\ell, repairs ∥ut∥≤εt\lVert u_t \rVert \le \varepsilon_t, and each layer's local-loss gradient, weighted by β≤0.3\beta \le 0.3, bounded by MM. While θt\theta_t stays in that neighbourhood:

lim sup⁡t→∞ ∥θt−θ∗∥≤γ=2βMα+εˉη α\limsup_{t \to \infty}\, \lVert \theta_t - \theta^* \rVert \le \gamma = \frac{2\beta M}{\alpha} + \frac{\bar\varepsilon}{\eta\,\alpha}

where εˉ=lim sup⁡tεt\bar\varepsilon = \limsup_t \varepsilon_t.

Repairs push training around, but never further than γ\gamma from a good solution.

Theorem 2

Bounded repair cannot revive (absorbing dead set)

In a Conv–BN–ReLU layer (BatchNorm in train mode), let channel cc have zero filter WcW_c and BN shift βc≤0\beta_c \le 0. This dead set DD is invariant under AdamW (fresh state) and under every repair bounded per channel, ∥R(W)c−Wc∥≤f ∥Wc∥\lVert R(W)_c - W_c \rVert \le f\, \lVert W_c \rVert:

θ0∈D  ⟹  θt∈D∀t\theta_0 \in D \implies \theta_t \in D \quad \forall t

Gradient descent and small fixes are both stuck, exactly. Only a jump leaves DD.

Notation: β\beta in Proposition 1 is the local-loss weight; βc\beta_c in Theorem 2 is a BatchNorm shift. The poster uses the same letter for both.

Figures

Test accuracy over 200 epochs. Two conv layers are zeroed at epoch 100. Standard AdamW stays at exactly 10.00% in all three seeds; with self-repair, a reset at step 39 493 is followed by recovery to 80.3 ± 1.5%. Below, every repair event per seed.
Figure 1. Theorem 2, in data. CIFAR-10 CNN; two conv layers zeroed after epoch 100. Small corrections cannot revive them, so accuracy sits at chance until a reset (a jump) redraws the dead channels at step 39 493 in every seed; small corrections then finish the recovery to 80.3 ± 1.5%. Standard AdamW stays at exactly 10.00% in 3/3 seeds. Bottom: every repair event (blue: small corrections; red: resets).
Largest correction per epoch on a log scale over 200 epochs after a shock: the mean over the first 20 epochs is 5.07 and over the last 20 epochs is 0.38.
Figure 2. The falsifiable prediction. Proposition 1's bound is informative only if repairs shrink. After a shock (σ = 1), the largest correction per epoch, εk, falls from 5.07 (first 20 epochs) to 0.38 (last 20), a 93% reduction.

Open problem

How often may a network jump?

Proposition 1 holds between jumps; Theorem 2 says jumps are unavoidable. Find a dwell time τD\tau_D such that jumps at least τD\tau_D apart keep training stable (cf. Hespanha & Morse, 1999).

Also open: strong convexity cannot hold exactly for BatchNorm layers (scale invariance); Cui et al. (2025) give ISS under a generalised PL condition, but whether BN networks satisfy it is open.

If you have ideas, I'd love to hear them.

Questions I'd expect at the poster

Isn't Theorem 2 just the dead-ReLU problem?

The mechanism is simple: a zero filter gives a constant pre-activation, train-mode BatchNorm maps it to βc ≤ 0, and ReLU blocks every gradient. What the theorem adds is the exact conditions: fresh optimiser state, and invariance under every per-channel-bounded repair, not only gradient steps. That makes it checkable, which is what ormas-monitor certifies, and it is the lower bound that makes self-repair a switched system.

You zeroed the layers yourself. Does this happen in real models?

The lesion is a controlled test, so exact zeros are by construction. Scanning 22 released torchvision ImageNet models, I find channels decayed to about zero (filter or BatchNorm scale below 10−5 of the layer median, shift ≤ 0) in 19 of the 21 that use BatchNorm → ReLU. They are not exactly zero, so the theorem does not certify them. On 64 real images (CIFAR-10 test images upscaled to 224 px) they output exactly 0 in ResNet-18, VGG-19-BN and MnasNet, and removing them from VGG-19-BN cuts compute by 18.6% with identical top-1 answers. Shrinking BatchNorm scales is also a known pruning technique (network slimming); the point is that one command finds them. Full results

With no repair, γ is still 2βM/α. Why doesn't training converge to θ*?

Because training also optimises each layer's local loss with weight β ≤ 0.3, which pulls against the global loss. That offset is why this is practical ISS; it becomes strict ISS only if β(t) → 0. (β here is the local-loss weight, not the BatchNorm shift βc in Theorem 2.)

Why model a repair as an input rather than as part of the optimiser?

Because it is not a gradient step. A repair fires when a diagnostic crosses a threshold, depends on the current state and is discontinuous. Writing training as θt+1 = θt − η∇L(θt) + ut lets input-to-state stability bound the effect of ut without requiring it to descend any objective.

How does this relate to Cui, Jiang, Sontag & Braatz (2025)?

They prove that perturbed gradient descent is small-disturbance ISS under a generalised Polyak–Łojasiewicz condition. Modelling repair as the input lets bounds of that kind apply to self-repair. What this work adds is the repair-specific side: the diagnostic-triggered input, the measurable condition the bound needs (repairs must shrink), and Theorem 2, which shows that bounded repair alone cannot leave a dead set.

Strong convexity cannot hold with BatchNorm. Doesn't that break Proposition 1?

It makes Proposition 1 a local, idealised statement, and the poster says so: BatchNorm's scale invariance rules out exact strong convexity. The natural replacement is a PL-type condition, under which Cui et al. (2025) already have ISS for gradient descent; whether BN networks satisfy it is the open step. Theorem 2 does not use convexity at all.

Why exactly 10.00%, with zero variance across seeds?

Once the layers are zeroed, the network's output no longer depends on the input, so it predicts the same class for every image. The CIFAR-10 test set is balanced, 1,000 images per class, so accuracy is exactly 10.00%. Theorem 2 says no gradient step or bounded repair can change that, and none did in 99 epochs across 3 seeds.

Does Theorem 2 depend on the optimiser state?

Yes. Leftover momentum could move a zero filter, so the theorem is stated for AdamW with fresh state for the channel. Every gradient into the channel is then exactly zero, and with decoupled weight decay the BatchNorm shift evolves as βc(t) = (1 − ηλ)t βc(0), which stays non-positive; the run logs match this to the bit. The ormas-monitor certificate checks these conditions before it certifies a channel as dead.

Why doesn't ResNet-18 benefit from repair?

The skip path routes around the same lesion, so the network barely notices it: 92.3% without repair against 91.6% with (3-seed means). Repair matters when there is no redundancy to fall back on.

Is the bound informative in practice?

Only if ε̄ is small, which is what makes it a prediction rather than a formality: a self-repairing network should need progressively smaller repairs. It does: after a shock the largest correction per epoch falls from 5.07 to 0.38 (Figure 2).

Reproduce it

Theorem 2 is implemented as a check you can run on your own PyTorch model: ormas-monitor watches training, certifies dead channels when the theorem's conditions hold, and can optionally repair them.

# the lesion experiment, small scale
pip install "ormas-monitor[demo]"
ormas-monitor demo

The demo trains a CIFAR-10 CNN, zeroes two of its three conv layers and clears the optimiser state, then trains on twice from that same state: once only watching, once repairing. It downloads CIFAR-10 (about 170 MB) once.

Cite

@misc{raadh2026selfrepair,
  author       = {Raadh, Rokib Al Dhin},
  title        = {Self-Repair as a Bounded Disturbance: Input-to-State
                  Stability of Neural Network Training Dynamics},
  howpublished = {Poster, DeepMath 2026: Conference on the Mathematical
                  Theory of Deep Neural Networks, The Ohio State University},
  year         = {2026},
  month        = oct,
  url          = {https://openreview.net/forum?id=kowltBZEIv}
}

Key references

  1. S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood and R. S. Sutton. Loss of plasticity in deep continual learning. Nature 632, 768–774, 2024. doi
  2. G. Sokar, R. Agarwal, P. S. Castro and U. Evci. The dormant neuron phenomenon in deep reinforcement learning (ReDo). ICML, 2023.
  3. J. Ash and R. P. Adams. On warm-starting neural network training (Shrink & Perturb). NeurIPS, 2020.
  4. E. Nikishin, J. Oh, G. Ostrovski, C. Lyle, R. Pascanu, W. Dabney and A. Barreto. Deep reinforcement learning with plasticity injection. NeurIPS, 2023.
  5. E. D. Sontag. Smooth stabilization implies coprime factorization. IEEE Trans. Automatic Control 34(4), 435–443, 1989.
  6. J. P. Hespanha and A. S. Morse. Stability of switched systems with average dwell-time. Proc. 38th IEEE Conference on Decision and Control, 1999.
  7. L. Cui, Z.-P. Jiang, E. D. Sontag and R. D. Braatz. Perturbed gradient descent algorithms are small-disturbance input-to-state stable. 2025. arXiv:2507.02131

Ask a question

Send me a question or a counter-example. I read everything.

Email your question