Event-Camera Vision

What event cameras produce

A conventional camera captures a full frame at a fixed rate. An event camera (Dynamic Vision Sensor, DVS) outputs asynchronous events — each pixel fires independently the moment its brightness changes, producing a stream of (timestamp, x, y, polarity) tuples. This is intrinsically spiking data: sparse, asynchronous, and event-driven. An SNN processes it natively — no frame construction, no idle computation between events.

Dataset: N-MNIST

Neuromorphic MNIST (N-MNIST) is the MNIST handwritten digit dataset re-recorded with a DVS camera executing three saccade movements over each printed digit.

Property	Value
Classes	10 (digits 0–9)
Training samples	60,000
Test samples	10,000
Sensor resolution	34 × 34 pixels, 2 polarities
Recording duration	~40 ms per sample
License	Creative Commons

Reference: Orchard et al. 2015 — “Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades”. Frontiers in Neuroscience, 9, 437. DOI: 10.3389/fnins.2015.00437

Preprocessing

Events are binned into T = 20 equal-duration frames using tonic. Each frame is a [2, 34, 34] binary tensor (ON polarity / OFF polarity). The spatial dimensions are flattened before the SNN, giving a 2312-feature spike vector per timestep:

import tonic
import tonic.transforms as transforms
import torch

sensor_size = tonic.datasets.NMNIST.sensor_size  # (34, 34, 2)
frame_transform = transforms.ToFrame(sensor_size=sensor_size, n_time_bins=20)

dataset = tonic.datasets.NMNIST(
    save_to="/tmp/nmnist",
    train=True,
    transform=frame_transform,
)

frames, label = dataset[0]                      # frames: [T=20, 2, 34, 34]
frames_t = torch.from_numpy(frames).float()
frames_flat = frames_t.reshape(20, 2 * 34 * 34) # [T=20, 2312]

Multiple events in the same bin and polarity are clamped to 1 — no count accumulation.

Architecture

Dense(2312 → 1024) → LIF(τ_mem=20ms, threshold=0.5)
Dense(1024 →  256) → LIF(τ_mem=20ms, threshold=0.5)
Dense( 256 →   10) → LIF(τ_mem=20ms, threshold=0.5)

Three fully-connected LIF layers. Spatial feature extraction is done implicitly — the Dense weights learn polarity- and position-selective filters. This pipeline compiles end-to-end through the current thrindex compiler.

import thrindex.snn as snn

model = snn.Sequential(
    snn.Dense(2312, 1024),
    snn.LIF(tau_mem=20.0, threshold=0.5),
    snn.Dense(1024, 256),
    snn.LIF(tau_mem=20.0, threshold=0.5),
    snn.Dense(256, 10),
    snn.LIF(tau_mem=20.0, threshold=0.5),
)

Why threshold=0.5? N-MNIST event frames are already binary {0, 1}. A threshold of 1.0 requires the membrane to accumulate across multiple timesteps before firing, which works but slows learning. 0.5 places the threshold at one event above resting potential, giving adequate gradient flow from the first epoch.

Training

pip install thrindex tonic
python templates/event-camera/train.py \
    --data-dir /tmp/nmnist \
    --epochs 30

tonic downloads N-MNIST (~315 MB) automatically on first run. Training for 30 epochs on a GPU takes roughly 8–12 minutes.

Parameter	Value
Optimizer	Adam
Learning rate	5×10⁻⁴
Gradient clip	norm ≤ 5.0
Batch size	128
Epochs	30
Loss	Cross-entropy on mean firing rates

Results

Committed accuracy floor: ≥ 95.0%

Reference	Accuracy	Architecture
Orchard et al. 2015 (doi)	97.8%	feedforward SNN, rate-coded
Lee et al. 2016	98.7%	backprop SNN, 2 layers

Floor is set conservatively below the feedforward baseline (97.8%). STDP-based and recurrent results (≥ 99%) are excluded as non-comparable.

Compile and run

import thrindex as thx

thx.compile(model, "templates/event-camera/model.thx")

thrindex run templates/event-camera/model.thx

Three N-MNIST-shaped spike-train samples are bundled in templates/event-camera/samples/ for fully offline inference. Each encodes a [T=20, 2312] binary spike tensor.

Why this template is different from keyword spotting

Dimension	Keyword spotting (SHD)	Event camera (N-MNIST)
Input source	Cochlear model → spike train	DVS camera → ON/OFF events
Input format	`[T=100, 700]`	`[T=20, 2312]`
Input sparsity	0.3–0.7%	1–3% per polarity
Encoding	Pre-encoded (SHD native)	Spatial flatten of event frames
Task	20-class spoken digit	10-class visual digit

The event-camera pipeline demonstrates that the same training loop, the same encoder interface, and the same thx.compile call handle both auditory and visual event-native data. The .thx format and the simulator are dataset-agnostic.

Getting Started

Tutorials

Concepts

Systems

API Reference

Errors

Benchmarks

Conformance

Event-Camera Vision

What event cameras produce

Dataset: N-MNIST

Preprocessing

Architecture

Training

Results

Compile and run

Why this template is different from keyword spotting

​What event cameras produce

​Dataset: N-MNIST

​Preprocessing

​Architecture

​Training

​Results

​Compile and run

​Why this template is different from keyword spotting

What event cameras produce

Dataset: N-MNIST

Preprocessing

Architecture

Training

Results

Compile and run

Why this template is different from keyword spotting