Home / Work / 005

An explainability framework for wet neural networks

Detecting spikes tells you that a biocomputer is active. It does not tell you which parts of the culture are doing the work, whether activity is spatially organised, or what would have to change for the system to behave differently. This framework asks those questions.

PhD research Python SHAP NetworkX ANNIGMA Moran's I
5Complementary methods
2Importance measures
Spatial+ topological analysis

The problem

A Multi-Electrode Array holds a living culture whose behaviour is the object of study. Unlike an artificial network, you cannot read out its weights — the parameters are chemistry and morphology, not numbers in a file.

What you can observe is activity at each electrode, and what you can build on top of that is a model that predicts something about the culture. The explainability question is then: can the model be interrogated in ways that produce claims about the biology, not just about the model?

A single method will not do this. Electrode importance tells you where, rule extraction tells you under what conditions, counterfactuals tell you what would have to change, and spatial statistics tell you whether any of it is organised or coincidental. The framework exists because those are four different questions.

The five methods

1. Electrode importance — ANNIGMA

Ranks electrodes by the gain they contribute through a trained network's weights. Cheap to compute and stable, which makes it a good first pass over a 60-electrode array.

2. Electrode importance — permutation

Measures the performance drop when an electrode's signal is shuffled. Model-agnostic and grounded in the actual predictive contribution rather than the architecture. Running both matters: where the two rankings disagree is where something interesting is happening — typically correlated electrodes whose importance one method attributes and the other distributes.

3. Fuzzy rule extraction

Extracts human-readable IF-THEN rules describing the conditions under which the model predicts activity. This is the direct link to the spike detection work, where the rule base is the classifier rather than a summary of one.

4. Counterfactual explanations

Answers the question an importance score cannot: what minimal change to the input would flip the prediction? For a biocomputer this is the closest available analogue to an intervention — a hypothesis about which activity pattern is doing the causal work, stated precisely enough to be tested.

5. Spatial autocorrelation and hub detection

Moran's I quantifies whether activity is spatially clustered across the electrode grid or effectively random. Combined with NetworkX-based functional connectivity analysis, it identifies hub electrodes — sites that are disproportionately connected within the culture's functional network.

This is where the framework stops describing a model and starts describing biology. A hub electrode is a claim about the culture's organisation, and it is falsifiable.

Explainability method triangulationFive explainability methods each answer a different question about a living neural biocomputer; agreement across methods is treated as evidence, whereas a signal from a single method is treated only as a lead.FIVE METHODS, FOUR QUESTIONSWHEREANNIGMAwhich electrodes carry gainWHEREPermutationwhich electrodes the modelneedsWHENFuzzy rulesunder what conditionsWHAT IFCounterfactualswhat would have to changeHOWMoran's I + hubshow activity is organisedConvergent finding = evidence about the biologysingle-method signal = a lead, not a result
Fig. 1 — How the five methods divide the explanatory question, and what agreement between them means.
Figure 2 — functional connectivity graph
hub electrodes highlighted, Moran's I annotated
Fig. 2 — Functional network topology and hub detection.

Why combine them

Each method has a characteristic blind spot, and the blind spots do not overlap.

MethodAnswersBlind to
ANNIGMAWhich electrodes carry gainCorrelated-feature redundancy
PermutationWhich electrodes the model needsInteraction effects
Fuzzy rulesUnder what conditionsContinuous magnitude
CounterfactualsWhat would have to changeGlobal structure
Moran's I / hubsHow activity is organisedPredictive relevance

Read together they triangulate. An electrode that ranks highly on both importance measures, appears in high-weight rules, and sits at a network hub is a genuine finding. One that ranks highly on a single measure is a lead.

View the repository