EchoVision
Hybrid NPU-CPU Assistive Navigation
A fully edge-deployed auditory navigation aid for the visually impaired, running YOLOv8n and EfficientViT-SAM on a Raspberry Pi 5 with a Hailo-8 NPU — no cloud, no tether. The key challenge: transformer attention blocks are incompatible with INT8 fixed-point NPUs, so we partition at the CNN-attention boundary and bridge the gap with calibrated dequantization.
The Problem
Assistive navigation requires real-time scene understanding — obstacles must be classified in milliseconds. On a Raspberry Pi 5 CPU, YOLOv8n alone achieves just 2.5 FPS at 404ms. The obvious fix is to offload to a hardware NPU, but transformer attention blocks are fundamentally incompatible with INT8 fixed-point arithmetic: existing toolchains simply fail to compile them. EchoVision's contribution is a systematic strategy for working around that incompatibility without sacrificing segmentation quality.
System Architecture
EchoVision runs on a Raspberry Pi 5 (Arm Cortex-A76) paired with a Hailo-8 NPU (26 TOPS) connected via PCIe Gen 3, with an IMX219 camera for continuous visual input and a custom BJT driver circuit for proximity-coded audio alerts.
Two compiled models are loaded onto the Hailo-8 simultaneously via multi-context allocation. YOLOv8n runs entirely on the NPU, scanning at 16.5–19.7 FPS and triggering segmentation only for high-confidence detections. EfficientViT-SAM is split: the pure-CNN backbone (Stages 0–3) runs on the NPU, while Stage 4 attention, the feature pyramid neck, and the mask decoder run on the CPU in FP32.
Pipeline Breakdown
Object Detection
Runs continuously at 16.5–19.7 FPS, 13.4ms latency. Built-in NMS outputs per-class bounding boxes directly from the accelerator.
CNN Feature Extraction
EfficientViT-SAM's convolutional backbone extracts multi-scale features (~20ms). GELU replaced with HardSwish for INT8 compatibility.
Attention & Mask Decoding
LiteMLA attention, SamNeck, and mask decoder run in FP32. Dequantized NPU features bridge the boundary at >0.97 cosine similarity.
Key Contributions
Hybrid CNN-Transformer Partitioning
First systematic methodology for deploying transformer-based vision models on INT8 NPU hardware, partitioning at the CNN-attention boundary and maintaining semantic fidelity across the split.
Quantization Failure Taxonomy
Documented five transformer quantization failure categories on the Hailo-8 — a practical reference for edge AI practitioners navigating NPU compiler limitations.
Calibration-Based Dequantization
Linear regression mapping from NPU uint8 outputs to CPU float32 inputs, preserving >0.97 cosine similarity in feature embeddings across the hardware boundary.
Hardware Platform
The physical system is a Raspberry Pi 5 with the Hailo-8 accelerator mounted via M.2 HAT+, an IMX219 camera module, and a custom BJT transistor driver circuit for the audio speaker. Standard GPIO pins are limited to ~16mA — insufficient for the 8Ω speaker — so a 1kΩ base resistor controls the transistor gate, sourcing speaker power directly from the 5V rail with a flyback diode for inductive spike protection.
Results
Offloading to the Hailo-8 brought detection from 2.5 FPS (CPU baseline) to 19.7 FPS at 13.4ms — a 30× latency reduction. The full pipeline operates bimodally: continuous scanning at 16.5 FPS, dropping to ~1.7 FPS when SAM segmentation is triggered for a high-confidence detection. Proximity classification reaches 100% accuracy at 2m and beyond.
Authors
Su Ho Lim†,
Artemis Shaw,
Huy Dinh,
Nam Bui
University of Colorado Denver · University of Colorado Boulder · †Corresponding author
Publication
Published · EIFCOM 2026
EchoVision: Hybrid NPU-CPU Deployment of EfficientViT-SAM and YOLO for Real-Time Assistive Navigation
Lim, S.H., Shaw, A., Dinh, H., Bui, N.
EIFCOM 2026 (co-located with ACM MobiSys 2026) · Cambridge, UK · April 2026
DOI: 10.1145/3812836.3814756