Running a Neural Network on a Microcontroller
A microcontroller with 256KB of RAM can run a useful neural network. It cannot run the one you trained on a workstation, and the gap between those two sentences is where most embedded ML projects quietly die.
The constraint is memory, not arithmetic
People new to embedded inference worry about clock speed. A Cortex-M4 at 80MHz has a single-cycle MAC, so it will sustain on the order of tens of millions of multiply-accumulates per second, and for keyword spotting or anomaly detection on a vibration sensor that is enough. What stops you is memory, and it stops you in two separate places.
Flash holds the weights. It is usually the number people quote, and it is the easy one — weights are static, and quantisation shrinks them predictably.
RAM holds the activations: the intermediate tensors that exist only while a layer is being computed. This is the number that kills projects, because it is invisible in the model summary and it does not scale with parameter count. A model with 40KB of weights can need 180KB of working memory if one early layer produces a wide feature map.
The practical consequence: you size the tensor arena by the peak, so one fat early layer taxes the entire model. Narrowing that layer often buys more headroom than pruning everything after it.
What quantisation actually costs
Converting float32 weights to int8 divides the weight memory by four and, on a core with DSP extensions, makes the arithmetic considerably faster. That part is well advertised. Two things are less well advertised.
First, quantisation is not uniformly lossy. Layers differ enormously in how much they tolerate it, and accuracy loss is usually concentrated in a couple of sensitive layers rather than spread evenly. Per-channel quantisation and a representative calibration dataset recover most of what naive conversion throws away.
Second, a “quantised” model often is not fully quantised. If one operator has no integer kernel, the runtime inserts conversions around it, and you pay for float storage, the conversion, and the lost speed. Check the operator report rather than the headline size — a model that is 95 percent int8 can behave like a float model at runtime.
The signal processing is most of the work
On an audio or vibration project, the network is frequently the smallest part of the pipeline. Before inference you window the signal, apply an FFT, warp to a mel scale, take logs, and stack frames into the shape the model expects. That front end runs on every single frame, in the interrupt path, whether or not the model fires.
Get it wrong and nothing downstream can save you. Common faults: a window hop that does not match training, a mel filterbank built with different edge conventions than the training pipeline used, or normalisation statistics baked from the training set and never applied on device. Each produces a model that scores beautifully offline and behaves randomly in the field, because it is being fed features it has never seen.
The most useful debugging tool here is boring: dump the on-device feature vectors over a serial link and compare them numerically against the host pipeline for the same input. Nine times out of ten the divergence is visible in the first frame.
Measure on the device
Host benchmarks lie in both directions. They understate performance by ignoring DSP instructions your target has, and they wildly overstate it by ignoring cache behaviour, flash wait states, and the fact that your inference is competing with an interrupt-driven sampler.
Instrument the real thing. Toggle a GPIO at the start and end of inference and put a scope on it — you get true wall-clock timing with essentially no overhead and no instrumentation bias. Track worst case, not mean: on a device that must not miss an audio buffer, the tail is the only number that matters.
What to take away
Embedded inference is a memory-budgeting exercise wearing a machine learning costume. Decide the RAM arena first, choose an architecture that fits it, quantise with real calibration data, and make absolutely certain the feature pipeline on the device is bit-for-bit the pipeline you trained against. The model is rarely the hard part.