> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thrindex.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Event-Camera Vision

> Train a spiking network on N-MNIST — the canonical neuromorphic vision benchmark.

## What event cameras produce

A conventional camera captures a full frame at a fixed rate. An event camera (Dynamic Vision Sensor, DVS) outputs asynchronous events — each pixel fires independently the moment its brightness changes, producing a stream of `(timestamp, x, y, polarity)` tuples.

This is intrinsically spiking data: sparse, asynchronous, and event-driven. An SNN processes it natively — no frame construction, no idle computation between events.

## Dataset: N-MNIST

[Neuromorphic MNIST (N-MNIST)](https://www.garrickorchard.com/datasets/n-mnist) is the MNIST handwritten digit dataset re-recorded with a DVS camera executing three saccade movements over each printed digit.

| Property           | Value                        |
| ------------------ | ---------------------------- |
| Classes            | 10 (digits 0–9)              |
| Training samples   | 60,000                       |
| Test samples       | 10,000                       |
| Sensor resolution  | 34 × 34 pixels, 2 polarities |
| Recording duration | \~40 ms per sample           |
| License            | Creative Commons             |

**Reference:** Orchard et al. 2015 — "Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades". *Frontiers in Neuroscience*, 9, 437. [DOI: 10.3389/fnins.2015.00437](https://doi.org/10.3389/fnins.2015.00437)

## Preprocessing

Events are binned into **T = 20** equal-duration frames using [tonic](https://tonic.readthedocs.io). Each frame is a `[2, 34, 34]` binary tensor (ON polarity / OFF polarity). The spatial dimensions are flattened before the SNN, giving a **2312-feature** spike vector per timestep:

```python theme={null}
import tonic
import tonic.transforms as transforms
import torch

sensor_size = tonic.datasets.NMNIST.sensor_size  # (34, 34, 2)
frame_transform = transforms.ToFrame(sensor_size=sensor_size, n_time_bins=20)

dataset = tonic.datasets.NMNIST(
    save_to="/tmp/nmnist",
    train=True,
    transform=frame_transform,
)

frames, label = dataset[0]                      # frames: [T=20, 2, 34, 34]
frames_t = torch.from_numpy(frames).float()
frames_flat = frames_t.reshape(20, 2 * 34 * 34) # [T=20, 2312]
```

Multiple events in the same bin and polarity are clamped to 1 — no count accumulation.

## Architecture

```
Dense(2312 → 1024) → LIF(τ_mem=20ms, threshold=0.5)
Dense(1024 →  256) → LIF(τ_mem=20ms, threshold=0.5)
Dense( 256 →   10) → LIF(τ_mem=20ms, threshold=0.5)
```

Three fully-connected LIF layers. Spatial feature extraction is done implicitly — the Dense weights learn polarity- and position-selective filters. This pipeline compiles end-to-end through the current thrindex compiler.

```python theme={null}
import thrindex.snn as snn

model = snn.Sequential(
    snn.Dense(2312, 1024),
    snn.LIF(tau_mem=20.0, threshold=0.5),
    snn.Dense(1024, 256),
    snn.LIF(tau_mem=20.0, threshold=0.5),
    snn.Dense(256, 10),
    snn.LIF(tau_mem=20.0, threshold=0.5),
)
```

**Why threshold=0.5?** N-MNIST event frames are already binary `{0, 1}`. A threshold of 1.0 requires the membrane to accumulate across multiple timesteps before firing, which works but slows learning. 0.5 places the threshold at one event above resting potential, giving adequate gradient flow from the first epoch.

## Training

```bash theme={null}
pip install thrindex tonic
python templates/event-camera/train.py \
    --data-dir /tmp/nmnist \
    --epochs 30
```

`tonic` downloads N-MNIST (\~315 MB) automatically on first run. Training for 30 epochs on a GPU takes roughly 8–12 minutes.

| Parameter     | Value                              |
| ------------- | ---------------------------------- |
| Optimizer     | Adam                               |
| Learning rate | 5×10⁻⁴                             |
| Gradient clip | norm ≤ 5.0                         |
| Batch size    | 128                                |
| Epochs        | 30                                 |
| Loss          | Cross-entropy on mean firing rates |

## Results

**Committed accuracy floor: ≥ 95.0%**

| Reference                                                             | Accuracy | Architecture                |
| --------------------------------------------------------------------- | -------- | --------------------------- |
| Orchard et al. 2015 ([doi](https://doi.org/10.3389/fnins.2015.00437)) | 97.8%    | feedforward SNN, rate-coded |
| Lee et al. 2016                                                       | 98.7%    | backprop SNN, 2 layers      |

Floor is set conservatively below the feedforward baseline (97.8%). STDP-based and recurrent results (≥ 99%) are excluded as non-comparable.

## Compile and run

```python theme={null}
import thrindex as thx

thx.compile(model, "templates/event-camera/model.thx")
```

```bash theme={null}
thrindex run templates/event-camera/model.thx
```

Three N-MNIST-shaped spike-train samples are bundled in `templates/event-camera/samples/` for fully offline inference. Each encodes a `[T=20, 2312]` binary spike tensor.

## Why this template is different from keyword spotting

| Dimension      | Keyword spotting (SHD)       | Event camera (N-MNIST)          |
| -------------- | ---------------------------- | ------------------------------- |
| Input source   | Cochlear model → spike train | DVS camera → ON/OFF events      |
| Input format   | `[T=100, 700]`               | `[T=20, 2312]`                  |
| Input sparsity | 0.3–0.7%                     | 1–3% per polarity               |
| Encoding       | Pre-encoded (SHD native)     | Spatial flatten of event frames |
| Task           | 20-class spoken digit        | 10-class visual digit           |

The event-camera pipeline demonstrates that the same training loop, the same encoder interface, and the same `thx.compile` call handle both auditory and visual event-native data. The `.thx` format and the simulator are dataset-agnostic.