Research · ORMAS
ORMAS: the self-repairing neural network.
ORMAS, invented by Rokib Al Dhin Raadh, recovers to 80.3% accuracy after a trained layer is destroyed mid-training, while a parameter-matched baseline stays at chance (10.0%). Raadh built it alone, at 18, on one RTX 3090. Its stability result, the first formal local stability characterisation of a self-correcting neural architecture, was accepted after double-blind review at DeepMath 2026 (poster). This page covers the method, all 383 reproducible experiments, the Input-to-State Stability derivation and ablations, then Project Cherry, the next architecture.
ORMAS: a self-repairing neural network that rebuilds a destroyed layer to 80.3% accuracy while a standard network stays at chance.
ORMAS, invented by Rokib Al Dhin Raadh, is a neural network architecture that detects its own failing components during training, repairs them without stopping, and logs every correction with its cause. Existing interpretability methods (SHAP, LIME, integrated gradients, attention maps, probing classifiers, circuit discovery) analyse a model from the outside, after training, and reconstruct what probably happened; those reconstructions cannot be verified. In ORMAS the record is produced by the same computation that performs the learning, so it is measured rather than inferred.
Peer-reviewed: the first formal local stability characterisation of a self-correcting neural architecture. Raadh's paper “Self-Repair as a Bounded Disturbance: Input-to-State Stability of Neural Network Training Dynamics” was accepted after double-blind review for poster presentation at DeepMath 2026, the Conference on the Mathematical Theory of Deep Neural Networks (Ohio State University, Columbus, 29–30 October 2026). The complete ORMAS preprint is public on Zenodo (DOI 10.5281/zenodo.21730363).
Inventor and principal researcher, independent | 2024 – present | PyTorch · 16,316 lines · 85 files383 controlled experiments across four architectures (FC-DAG, CNN, Fat CNN, ResNet-18), all run on a single RTX 3090 and every one reproducible from seed. The stability result is local. The paper, the codebase that reproduces all 383 runs and the full results archive are linked below.
Explaining ORMAS: Transparent Neural Networks That Self-Heal
A presentation of the full research record: the three learning signals, the 383 experiments, the ISS derivation and the complete results, in under eight minutes.
- Autonomous recovery from catastrophic mid-training failure
- Compositional memory that emerges without replay buffers
- Why transparency is an engineering problem, not a philosophy question
Key results
A destroyed layer, rebuilt: 80.3% versus 10.0% (+70.3pp)
After a converged convolutional layer is zeroed at epoch 100, ORMAS recovers to 80.3% accuracy while the standard CNN stays at chance (10.0% ± 0.0%) on all three seeds: a +70.3pp gap. ORMAS gets there with 85 targeted corrections between epochs 100 and 110. A checkpoint rollback can also restore accuracy, but it discards all learning since the checkpoint and gives no account of what failed or why.
All three layers destroyed: 70.8% versus 10.0% (+60.8pp)
With every convolutional layer zeroed at epoch 100, removing every learned representation, ORMAS recovers to 70.8% ± 2.2% across three seeds while the standard CNN remains at 10.0% ± 0.0%: a +60.8pp gap. One run logged 72 corrections (54 oscillating, 18 dead) and reached 72.9%. Recovery is lower than in the single-layer case, as expected when every stage must be rebuilt at once.
Label noise: 2.5pp decay versus 7.8pp, with no second network
Under 40% symmetric label noise over 200 epochs, ORMAS decays 2.5pp from its peak and settles at 75.6% ± 0.8%, while the standard CNN decays 7.8pp, with no co-training, second network or specialised objective. Heavy dropout (p = 0.5) matches ORMAS on label noise but collapses to chance when a layer is lesioned: regularisation can mask noise, but it cannot diagnose and repair structural damage.
Sequential learning: 94.6% retention versus 47.3% for ResNet-18, and 58.8% zero-shot
Taught shape first and colour second, ORMAS retains 94.6% on shape while a standard ResNet-18 retains 47.3%. ORMAS keeps both tasks (94.6% shape, 96.5% colour) and scores 58.8% zero-shot on unseen shape–colour pairings, +33.8pp above chance. The PCGrad ablation reaches 59.1% ± 3.6%, statistically indistinguishable, isolating self-correction as the driver. The behaviour was not designed; it emerged from the health gate.
Why Raadh built ORMAS: agents learning from corrupted production data
ORMAS grew out of OXIMO, Raadh's multi-agent system for autonomous business operations (codebase). Its agents had to learn from real production data, with corrupted labels, adversarial inputs and contradictory signals. Every noise-robust method he tested was an external filter applied to a network that remained blind to its own internal state.
The requirement was a network that could detect corruption during training, repair it without stopping or human intervention, and come with a formal account of when that repair remains stable. No such architecture existed. ORMAS was designed to meet that requirement.
Standard backpropagation versus ORMAS: the same node failure, side by side
The central claim of ORMAS is not accuracy but attribution: every failure is named, diagnosed and logged with its cause. The same node failure is shown in both architectures; the telemetry on the right is quoted verbatim from the supplementary material.
A Node Dies Silently
One error signal, shared by everything. Nothing can report on itself.
epoch 42 loss 1.842 → 2.910
epoch 43 loss 2.910 → 3.221
# no node id. no cause. no timestamp.
# diagnosis is post-hoc, on a model
# that has already finished dying.The paper states it plainly: standard backprop provides no architectural mechanism for per-node attribution during training. Rollback or full retrain are the only recourse.
The Same Node Dies Loudly
Every node carries a second, bounded path. It can report on itself.
[EPOCH 42 | STEP 16384] --- DIAGNOSTICS ROUND ---
[node_12] Goodness: 0.88 -> HEALTHY
[node_13] Goodness: 0.04 -> CRITICAL
> Diagnostics: g_mag = 0.01 (DEAD),
g_grad = 0.11 (STAGNANT)
> Action: AUTONOMOUS NEUROGENESIS
> Constraint: zero-sum mean-subtraction,
||DeltaW|| <= 10%
--- VERIFICATION FORWARD PASS ---
> Global Loss Before: 1.842
> Global Loss After: 1.791
> Result: ACCEPTED — node 13 re-integratedName, score, diagnosis, treatment, bound, and a verification pass that can reject its own repair. One DAG run emitted 23,227 such events. That ledger is the product; the accuracy is the evidence it works.
ORMAS corrections decline to zero by epoch 50
A regulariser such as dropout acts at a constant rate. ORMAS corrections instead peak early, decay and reach exactly zero: the network adapts structurally rather than relying on continued intervention, as the ISS analysis predicts.
How ORMAS works: three simultaneous learning signals
ORMAS replaces the single-signal paradigm with three simultaneous learning signals:
- Signal 1 — Global Backpropagation: The standard cross-entropy loss backpropagated through the entire network. This is what every neural network already does.
- Signal 2 — Per-Node Local Loss (The Intrinsic Anchor): Each node gets its own independent assessment computed through a shared bottleneck readout (4,416 parameters, ) that classifies the input using only that node's local features. It cannot memorise noise, so it is forced to learn structural features.
- Signal 3 — Health-Gated Autonomous Self-Correction: Each node continuously computes a Multiplicative Bottleneck Goodness Score. When it drops below its own historical average by 2.5 standard deviations, the system diagnoses the specific pathology and applies the targeted treatment. A 0.3 gate floor prevents the dead-ReLU trap.
Seven failure modes ORMAS detects and repairs during training
ORMAS detects and treats seven distinct node pathologies in real time during training. Defence in depth: PCGrad operates at the gradient level (before weight updates); self-correction operates at the weight level (after updates). Any unmodelled pathology ultimately manifests as gradient death, triggering the fallback Kaiming reinitialisation: graceful degradation without requiring learned policies.
| Pathology | Symptom | Treatment |
|---|---|---|
| Dead Neuron | Output permanently near zero | Kaiming reinitialisation |
| Exploded | Activations > 500.0 | 90% Anti-Hebbian scale reduction |
| Saturated | Outputs stuck at max | Self-referential magnitude dampening |
| Oscillating | Weights swinging wildly | Momentum-based directional fixing |
| Loss Stagnant | Stuck on a plateau | Escalated noise perturbation |
| Gradient Dead | Zero gradient flow | Weight scaling to restart flow |
| Low Confidence | Uncertain output | Targeted noise perturbation |
GlassBox: five layers of per-node telemetry, emitted during training
Traditional neural networks fail silently. ORMAS reverses this with the Loud Failure Paradigm. GlassBox emits a 5-layer causal audit trail per node: (1) health status and goodness score, (2) correction traces with pathology diagnosis, (3) gradient conflict measurement (cosine similarity between global and local gradients), (4) topology census (active/suppressed routing per input region), (5) training pulse timeline.
A single ORMAS DAG training run emitted up to 23,227 autonomous correction events over 200 epochs. On the 50-node fully-connected DAG under 30% continuous noise — a maximally dense graph where uncorrected architectures collapse to NaN — ORMAS maintained numerical stability across 200 epochs via 22,014 autonomous corrections per run (mean of 3 seeds). Every correction is tagged with its pathology, node ID, and diagnostic evidence.
- What it tells you
- Per-node health, correction trigger, EMA baseline, cosine gradient conflict, spatial routing map
- Next extension
- Mapping each failing node to the visual concept it encoded, before and after repair
- Why that matters
- Structural telemetry is the foundation semantic interpretability builds on. Bridging the two is the primary extension direction.
383 controlled experiments against compute- and parameter-matched baselines
All baselines are strictly compute- and parameter-matched. 383 experiments on a single RTX 3090. 4 architectures (FC-DAG, CNN, Fat CNN 11.24M, ResNet-18). 6 noise regimes. 10 baselines. All trained for 200 epochs.
40% label noise: ORMAS ends at 75.6%, the standard CNN at 69.6%
Best accuracy vs. final accuracy after 200 epochs of training under label noise.
Asymmetric noise: 80.6% (Mixup: 50.0%). Real-world CIFAR-10N: 83.9%.
All Noise Regimes, Including Adverse Results
| Noise Regime | ORMAS | Standard CNN | Mixup |
|---|---|---|---|
| Sym 40% (final)Peak decay: ORMAS 2.5pp vs CNN 7.8pp | 75.6% ±0.8% | 69.6% ±0.9% | 36.5% |
| Asym 20% | 84.3% ±0.3% | 83.3% ±0.4% | 53.6% |
| Asym 30% | 83.1% ±0.1% | 81.5% ±0.6% | 52.0% |
| Asym 40%Mixup collapses — ORMAS +2.6pp | 80.6% ±0.3% | 78.0% ±0.2% | 50.0% |
| CIFAR-10N (Real)Real human annotator noise. Decay: ORMAS 2.7pp, CNN 3.7pp (1.4× more) | 83.9% | 83.8% | — |
All results CIFAR-10 unless noted. Equal-compute (200 epochs, same hardware). Framing note: each accuracy result shows that the mechanism works; none is offered as a competitive accuracy benchmark.
Gaussian weight shock: the ORMAS advantage grows with the damage, to +23.9pp
A σ-Gaussian weight perturbation is injected at training step 1,000, destroying all learned representations. ORMAS recovers autonomously; Mixup collapses permanently to 33.3%.
| Perturbation σ | ORMAS Final | Standard Final | Recovery Gap |
|---|---|---|---|
| σ = 0.1 | 80.2% | 82.1% | −1.9pp |
| σ = 0.5 | 78.6% | 74.8% | +3.8pp |
| σ = 1.0 | 75.4% | 59.0% | +16.4pp |
| σ = 2.0 | 62.1% | 38.2% | +23.9pp |
The gap widens monotonically with perturbation scale. At σ=2.0, ORMAS recovers 52.1pp vs Standard CNN's 28.2pp — nearly 2× the net recovery from the same catastrophic shock.
Scaling: The 11.24M-parameter Fat CNN also recovers from catastrophic weight explosion: 88.9% → crash to 22.7% → recovery to 69.3%. The three-signal mechanism scales from 637K to 11M parameters.
ResNet-18 Scale: Stages 2+3 killed at epoch 100. ORMAS-ResNet: 264 corrections → 91.7%. Vanilla ResNet-18: 92.6% (blind recovery). The accuracy gap is −0.9pp. But the blind baseline has no audit trail — it cannot tell the operator which nodes failed, when, why, or with what pathway. An accuracy number tells you a network recovered. The telemetry tells you how, which makes the recovery auditable.
The Baldwin Effect: Under 40% noise, ORMAS triggers an average of 67.6 surgical self-corrections per run. Epochs 0–50: a peak of 4.2 corrections/epoch, 1.35 on average. Epochs 50–200: zero. The network learns to not need correction — architectural immunity as an emergent property.
| Epoch Range | Corrections / Epoch |
|---|---|
| Epochs 0–50 | 1.35 / epoch (peak 4.2) |
| Epochs 50–100 | 0 / epoch |
| Epochs 100–200 | 0 / epoch |
Stress tests: five extreme failure scenarios, every result reported
With all three layers destroyed at once, ORMAS recovers to 70.8% while the standard CNN stays at 10.0%; the compound and 100-class scenarios mark where recovery becomes seed-dependent or partial. The bifurcated results are not measurement error — they are a physical phenomenon. Under compounded structural-noise perturbation, the network sits at the edge of a topological bifurcation: small initialisation differences determine whether the self-correction mechanism achieves stable recovery or collapses. This is the honest edge of the capability.
| Scenario | Standard Final | ORMAS Final | Gap |
|---|---|---|---|
| Simultaneous Full-Hierarchy LesionAll 3 conv layers zeroed at epoch 100 | 10.0% ± 0.0% | 70.8% ± 2.2% | +60.8pp |
| Compounded Structural + 40% NoiseDual attack: noise + lesion simultaneously | 10.0% ± 0.0% | Per seed: 31.8 / 76.9 / 69.4% | Edge-of-chaos |
| High-Cardinality (CIFAR-100)100-class partial failure — scope limit | 1.0% ± 0.0% | Per seed: 7.4 / 25.8 / 27.1% | Boundary found |
| Adversarial Weight InjectionExpected scope boundary — crafted to evade diagnostics | 84.1% ± 0.3% | 83.1% ± 0.3% | −1.0pp |
| Weight Explosion (100×)System does not overcorrect on mild damage | 86.0% ± 0.1% | 85.1% ± 0.4% | −0.9pp |
Zero-shot compositional generalisation: 94.6% retention and 58.8% on unseen combinations
ORMAS holds both tasks in sequential training and generalises to shape–colour combinations it was never shown; a standard ResNet-18 catastrophically forgets shape while learning colour. Phase 1: shape classification. Phase 2: colour classification. Test: accuracy on novel shape + colour combinations.
| Model | Phase 1 Shape | Phase 2 Colour | Shape Retention | Zero-Shot 4-way | vs. Chance |
|---|---|---|---|---|---|
| Standard ResNet-18 | 83.6% | 91.4% | 47.3% | 51.1% | +26.1pp |
| ORMAS Three-Signal | 94.6% | 96.5% | 94.6% | 58.8% | +33.8pp |
The ceiling is 91.3% (statistical independence: 0.946 × 0.965). ORMAS reaches 58.8% — strong but incomplete spatial separation. An emergent structural bias, not strict orthogonality. PCGrad ablation (remove Signal 2, retain Signal 3): 59.1% ± 3.6% — statistically indistinguishable. Self-correction is the necessary and sufficient driver.
The stability result: Input-to-State Stability for a self-correcting network, accepted at DeepMath 2026
Where the Mathematics Came From
In 1989 the mathematician Eduardo Sontag introduced Input-to-State Stability, a control-theory framework used to establish that a perturbed system, such as a rocket knocked off course, returns to a stable trajectory.
A self-correcting network is a perturbed system: by corrupted data, by structural damage and by its own corrections. Whether its correction mechanism settles or oscillates is therefore a control-theory question, one the field had already answered for other classes of system.
Raadh adapted the framework by treating each correction as a bounded input disturbance and the weight trajectory as the state, then characterising the conditions under which the state remains bounded. The result is the first formal local stability characterisation of a self-correcting neural architecture. Derived at seventeen, it was accepted at DeepMath 2026 after double-blind review.
Local stability characterisation, the first for a self-correcting neural architecture. Under standard regularity assumptions, the conservation constraint () bounds each correction's L² norm via mean-subtraction (weight magnitude is redistributed, not created). Empirical validation: late-stage parameter variance reduces from 0.86 (Standard CNN) to 0.19 (ORMAS) — a 4.5× reduction.
Ablations: on the DAG, removing either mechanism collapses training
On the DAG architecture under 30% noise:
| Configuration | Final Accuracy | Change |
|---|---|---|
| Full ORMAS | 40.1% (best 49.9%) | — |
| Remove Self-Correction (Signal 3) | NaN (best 14.2%) | Training collapsed |
| Remove PCGrad | NaN (best 12.1%) | Training collapsed |
On the DAG, removing either mechanism collapses training outright. On the CNN, removing PCGrad leaves accuracy unchanged (78.1% final) but forces the self-correction mechanism to work far harder — disabling one defence layer triggers proportional activation of the other. This is the intended behaviour: two independent defence layers.
Reproducibility: all 383 experiments regenerate from seed on one RTX 3090
- Tiered experiment infrastructure: 383+ configurations with automated seed sweeps.
- Shipped an interactive
reproduce.sh— one command reproduces every experiment. Core claims reproducible in under one hour. - 10-step training pipeline (
ORMASTrainer), multi-round forward loop with selective rollback tracking. After each correction, a verification forward pass runs; if both global and node-local loss worsen by more than 2%, that node's weights selectively roll back — typically affecting fewer than 30% of corrected nodes. - All 383 experiments on a single RTX 3090 (24 GB VRAM, 30 GB RAM, 8 vCPU). Consumer hardware. 16,316 lines of research code.
Hyperparameter robustness: under 0.8% accuracy variance across a 16× sweep
ORMAS's accuracy is nearly insensitive to its own hyperparameters — a strong indicator of mechanistic robustness, not overfitting to a narrow configuration:
- β_max ∈ [0.1, 0.5]
- Accuracy bounded 80.0%–80.1%
- <0.1% variation
- Gate threshold τ ∈ [0.2, 0.6]
- Accuracy 79.8%–80.0%
- <0.2% variation
- Bottleneck d_b ∈ {8, 32, 64, 128}
- Accuracy variance <0.8%
- 16× parameter range
- Health threshold σ_self 2.5→3.5
- 65→41 corrections (37% fewer), accuracy unchanged
- 79.7% vs 80.0%
Safeguards: healthy components are never touched
- Expert Immunity
- Nodes achieving EMA confidence ≥ 0.55 OR local loss < 0.20 become "experts" — immune from convergence penalties. Prevents disruption of already-converged features.
- High-Distress Override
- Expert immunity is overridden when goodness drops to ≤ 0.30. Forces immediate diagnosis regardless of expert status. No node hides catastrophic failure behind past performance.
- Adaptive Cooldown
- Post-correction cooldown C ∈ [10, 200] steps. Shrinks 20% on >60% success rate; expands 50% on <30% success rate. Prevents destructive oscillation while allowing rapid iterative repair.
- Selective Rollback
- After each correction, a verification forward pass runs. If global AND local loss both worsen by >2%, the node rolls back. Typically affects <30% of corrected nodes per training run.
Research roadmap: the next four experiments
Preemptive Immune Filtering
V1 is reactive: damage happens, then it gets repaired. V2 uses local loss disagreement as a per-sample gate, stopping corrupted samples before they ever reach the gradient path. Repair becomes immunity.
Transformer Architecture Extension
Carry three-signal learning onto attention. Each head gets an independent local loss anchored through a shared key-query bottleneck, which makes per-head health monitoring and surgical correction possible. Scale brings its own pathology classes to catalogue: attention collapse, entropy death, feature saturation.
Bridging Structural to Semantic Telemetry
GlassBox reports structure — which nodes failed, when, and how. It cannot tell you what concept a failing node encoded. The disentanglement result hints at the bridge: nodes that lock during Phase 2 are the ones preserving Phase 1 shape representations. Formalising that through CAVs or Grad-CAM trajectories across the correction lifecycle is the central open question.
Cherry — Self-Correcting Language Model
A language model trained from scratch on three-signal learning, rather than fine-tuned. It would be the first LLM able to catch and repair its own training pathologies while they happen.
One architectural decision, three results: repair, noise robustness and zero-shot generalisation
Structural recovery, noise robustness, zero-shot generalisation — these are not three findings. They are one structural property showing up three times. Bounding the local gradient chain produces an attribution signal. The attribution signal makes the health gate possible. The health gate delivers both the repair and the spatial separation of competing representations.
Which points at something larger: continuous autonomous correction may be sufficient on its own to make modular internal structure emerge — no explicit modularity constraints, no replay buffers. If that holds, it is a research direction rather than an engineering result.
Project Cherry: the next architecture after ORMAS, a network that grows its own components.
Project Cherry is Raadh's fully specified next architecture: a network, built on ORMAS, that grows new components on demand and retires those that no longer contribute. It is designed to remove the fixed-capacity ceiling that bounds every ORMAS result above. Development begins once multi-node compute is in place.
ORMAS already provides the difficult part: a network that knows which components are failing, and by how much, can be instructed to grow replacements. New components are introduced with zero net effect on current behaviour, so existing capabilities are undisturbed while they learn. This is how the fixed-capacity ceiling is removed.
How Project Cherry is designed to work
The Standard Model
One large model; every parameter handles every query.
The model is unchanged after every query. Teaching it anything new requires stopping, retraining and freezing it again.
A Mesh That Learns While It Works
Many small specialists; only the relevant ones activate.
Only a handful of nodes wake for any one question. The answer loops back and the mesh is different afterwards — and when it meets work nothing can handle, it grows a node rather than overwriting one.
How it differs from fine-tuning, LoRA and Mixture-of-Experts
- Not fine-tuning
- Fine-tuning is a discrete event against a frozen base, and it forgets. There is no frozen base here and no discrete event.
- Not LoRA or adapters
- Adapters need a task ID at inference — you must already know which one to load. Here, which parts activate is the result of routing, not an input to it.
- Not Mixture-of-Experts
- The closest existing approach. MoE fixes the number of experts before training and does not monitor them individually. Cherry grows components on demand, and each is independently monitored and repairable.