EdgeFuse

Multi-modal sensing is becoming the norm on edge devices such as robots. Even for something as simple as two sensors — a visible-light camera and a thermal sensor — a CPU pipeline is already too slow just for image resizing. The frames arrive faster than the CPU can scale and align them, so the two streams fall out of sync.

Now imagine adding LiDAR, radar, or SAR. The preprocessing burden grows with every sensor added, and the CPU falls further behind. This is a structural problem, not a tuning problem.

Instead of sending raw frames to the CPU, we deploy a DODA overlay in the FPGA fabric between the sensors and the application core. As pixels arrive from the visible-light camera (45 FPS, 1280×960) and the thermal sensor (9 FPS, 160×120), DODA resizes and time-aligns the two streams on the fly — before the first byte reaches the ARM core.

The preprocessed, fused stream lands on the CPU already synchronized. The application starts the moment the last pixel of each frame arrives — 19× faster preprocessing than the CPU-only baseline.

Side-by-side on the same hardware. With CPU-only processing (left), the visible and thermal frames drift apart: the thermal overlay lags behind the person's actual position. With DODA preprocessing (right), the streams stay tightly locked.

CPU-onlyWith DODA

Beyond synchronization, EdgeFuse adjusts the fusion weight between EO and IR continuously based on ambient brightness — no fixed threshold, just a smooth blend that shifts as the lighting changes. When light is good, the visible view dominates with thermal picture-in-picture; as light falls, the fused thermal view takes over with the visible as PIP.

The switch is immediate: because the decision is made close to the sensors on the FPGA, the application sees the already-switched stream with no extra latency.

Everything runs on one FPGA SoC board with an ARM Cortex-A53. The thermal (IR) sensor and visible-light (EO) camera connect directly to the board; the DODA prototype lives in the FPGA fabric and the application runs on the A53.

This prototype was built on a tight timeline. The current 19× speedup is only the floor — the architecture has much more headroom. Two near-term priorities:

• Kalman filter for sensor fusion, implemented as a DODA operator on the FPGA.
• EO/IR geometric correction to compensate lens distortion before frames reach the CPU.
• Expected outcome with both improvements: ~100× speedup over the CPU-only baseline.