Skip to main content

​
The problem conformance solves

A spiking neural network trained in Python (f32 arithmetic, behavioral simulator) will produce different outputs when run on hardware (fixed-point arithmetic, physical circuits). The question is: how different? And: different in what way? Without a conformance standard, every hardware port is an ad-hoc validation exercise. With one, you can state precisely: “this backend’s outputs agree with the reference simulator within these bounds, on this fixture set, for this class of models.”

​
The conformance metric

THRINDEX uses the spike-equivalence metric: given a reference spike raster S_ref (from the simulator) and a backend spike raster S_hw (from hardware or a fixed-point model), compute: Normalised spike-time distance (T_spike) Tspike(n)=∣kn(ref)−kn(hw)∣max⁡(kn(ref),kn(hw),1)T_{\text{spike}}(n) = \frac{|k_n^{(\text{ref})} - k_n^{(\text{hw})}|}{\max(k_n^{(\text{ref})}, k_n^{(\text{hw})}, 1)} Where k_n is the spike count of output neuron n. Over all neurons and samples:
  • T_mean = mean of T_spike across all neurons and samples
  • T_max = 99th-percentile of T_spike
Prediction agreement (P_pred) Ppred=∣{s:arg⁡max⁡nkn(ref)(s)=arg⁡max⁡nkn(hw)(s)}∣∣S∣P_{\text{pred}} = \frac{|\{s : \arg\max_n k_n^{(\text{ref})}(s) = \arg\max_n k_n^{(\text{hw})}(s)\}|}{|S|} Fraction of samples where the argmax prediction matches between reference and hardware.

​
The V0 envelope

CONFORMANCE_ENVELOPE_V0 is the ratified conformance threshold for THRINDEX Certified backends. A backend must satisfy all three simultaneously to earn the badge on a given model class. Failing any one is a fail. What these numbers mean:
  • T_mean ≤ 0.020: on average, a neuron’s spike count differs by at most 2% of the larger count.
  • T_max ≤ 0.130: even the 99th-percentile worst-case neuron differs by at most 13%.
  • P_pred ≥ 0.900: at least 90% of classification decisions agree with the simulator.

​
How the envelope was derived

The V0 thresholds were derived empirically from a fixture set of 120 SHD samples using the ratification binary (thrindex-conformance ratify_envelope):
  1. The simulator (float reference) was measured against int8 per-channel quantization. This is the softest quantization that should pass — it represents hardware-class precision.
  2. The simulator was measured against int4 per-channel quantization. This is the hardest quantization that should fail — it represents a precision degradation that loses too much information.
  3. Thresholds were placed in the gap between the int8 and int4 distributions, at a point where int8 passes with margin and int4 fails decisively.
Fixture fingerprint: CRC32 e2ebd845, 120 SHD test samples, generated by freeze_shd_fixtures.py. See conformance/envelope-v0 for the full derivation including distributions.

​
Running conformance

Output:
The reference simulator trivially passes its own envelope (T_mean = 0, T_max = 0, P_pred = 1.0) — it is the reference.

​
Adding a backend

See conformance/adding-a-backend.