EchoVision
Hybrid NPU-CPU Assistive Navigation
An auditory navigation aid for the visually impaired that runs entirely on a Raspberry Pi 5 with a Hailo-8 NPU — no cloud, no tether. Transformer attention won't compile to INT8, so we split the model at the CNN–attention boundary.
2.5 FPS is not real time
Assistive navigation has to classify obstacles in milliseconds. On the Raspberry Pi 5 CPU, YOLOv8n manages 2.5 FPS at 404ms.
- Offloading to an NPU is the obvious fix — but transformer attention is incompatible with INT8 fixed-point arithmetic, and existing toolchains fail to compile it.
- EchoVision's contribution is a systematic way around that incompatibility without sacrificing segmentation quality.
System Architecture
Two compiled models share one accelerator: detection stays on the NPU, attention falls back to the CPU.
- YOLOv8n runs entirely on the NPU — scanning at 16.5–19.7 FPS, triggering segmentation only on high-confidence detections.
- EfficientViT-SAM is split — CNN backbone (Stages 0–3) on the NPU; Stage 4 attention, the feature pyramid neck, and the mask decoder on the CPU in FP32.
- Both models load simultaneously via multi-context allocation.
Key Contributions
Hybrid CNN-Transformer Partitioning
First systematic method for running transformer vision models on INT8 NPUs, split at the CNN–attention boundary.
Quantization Failure Taxonomy
Five documented quantization failure modes on the Hailo-8 — a reference for anyone hitting NPU compiler limits.
Calibration-Based Dequantization
Linear regression from NPU uint8 to CPU float32, holding >0.97 cosine similarity across the boundary.
Hardware Platform
The Hailo-8 mounts over M.2 HAT+; the speaker runs from a custom transistor driver.
Why the speaker needs its own driver circuit
Standard GPIO pins are limited to ~16mA — insufficient for the 8Ω speaker. A 1kΩ base resistor controls the transistor gate, sourcing speaker power directly from the 5V rail, with a flyback diode for inductive spike protection.
Results
- 30× latency reduction — detection moved from 2.5 FPS on the CPU baseline to 19.7 FPS at 13.4ms.
- Bimodal operation — continuous scanning at 16.5 FPS, dropping to ~1.7 FPS while SAM segmentation runs.
- 100% proximity-classification accuracy at 2m and beyond.
Publication
Su Ho Lim†, Artemis Shaw, Huy Dinh, Nam Bui