STM32N657 Neural-ART: int8 residual Add computes wrong (low-biased) result on target - both operand verified correct

[Jibril]

Board: NUCLEO-N657X0-Q (MB1940), STM32N657X0H (Cortex-M55 + Neural-ART NPU), Device ID 0x486 Rev Z.

Toolchain: ST Edge AI Core v4.0.1-20581 (STM32Cube.AI 12.0.1-RC2), atonn 1.1.3-275, NetworkRuntime1201_CM55_GCC, arm-none-eabi-gcc 14.3.1 (STM32CubeCLT 1.21.0).

 

SUMMARY
A small int8 QDQ model (TCN: Conv2D x7 + Gemm x2, macc ~2.37M, int8 weights ~20 KB) generated with "stedgeai generate --target stm32n6 --st-neural-art" runs to completion on the NPU but produces wrong outputs: high-input forecasts collapse to baseline. Independent cross-check: onnxruntime on the same int8 ONNX gives 9.664 (ppm, log-domain output dequantized) on our golden window; the NPU returns ~0.26. Confirmed on hardware with ST's own "stedgeai validate --mode target" (regression head rmse 8.78, cos 0.589 vs the reference).

 

LOCALIZED ON-TARGET TO A SINGLE OP

Per-layer on-target validation (backbone stage outputs exposed as ONNX outputs):
- input projection Conv: cos 0.999 (correct)
- block 0 (dilated Conv + Slice + residual Add + Relu): cos 0.606 (first divergence; error accumulates through blocks 1-5)
Isolating block-0 internals on target:
- dilated Conv output: cos 0.9996 (correct)
- causal Slice: cos 0.99999 (correct)
- residual Add + Relu: cos 0.606 (WRONG, biased low)
Both Add operands are computed correctly on the NPU; the int8 eltwise combine of two tensors with DIFFERENT quant scales is what diverges. Operand scales 0.052 / 0.099 -> output scale 0.049 (only ~2x ratio, non-pathological).

 

RULED OUT (output byte-identical across all of these): weight location (external flash / npuRAM3 / npuRAM5), M55 D-cache on/off, NPU CACHEAXI on/off/invalidate, RISAF12, NPU-master RIF secure vs non-secure, clocks/sleep settings, --Ocache-opt, EpochController encryption, input layout. Weights and input verified correct in NPU memory at run entry (SWD reads).

 

MODEL-SIDE WORKAROUNDS TRIED ON HARDWARE (all failed):
1. Concat + 1x1 Conv replacing the Add (numerically exact in float, max delta = 0): no help (cos 0.610) -> the defect is the different-scale int8 combine, not the Add operator itself (Concat requantizes the same way).
2. int16 activations: partial improvement (cos 0.589 -> 0.887) but a magnitude bias remains.
3. Shared-scale Add operands: not cleanly viable (the residual stream is a chain, forcing one GLOBAL scale ~7x coarser than the relu scales).
4. Residual folded into the conv (x + conv(x) = (conv+I)(x), exact on host, max delta 9.5e-07): fails on target (the int8 conv appears to lose the small delta next to the ~1.0 identity weight).

 

CURRENT STATUS: we ship the same int8 model on the Cortex-M55 CPU instead (bit-exact vs onnxruntime there, so the model itself is sound) at ~15.7 ms/inference vs the NPU's measured 1.124 ms. A fix would let us return to the NPU with no retrain.

 

QUESTION
Is the int8 eltwise Add of two different-scale operands a known Neural-ART/ATON issue in v4.0.1? Is there a generate option, or a recommended residual-connection pattern, that the NPU computes correctly?

A full evidence package exists (per-layer validate reports, ONNX cuts isolating the op, network_val_io.npz with on-target vs reference tensors) - happy to attach or share on request.

[Jibril]

Following up on the int8 residual-Add divergence reported above, we have found what appears to be a second, distinct mis-execution on the same setup, affecting a class of networks that contains no elementwise Add at all.

 

SETUP (unchanged from the original report): NUCLEO-N657X0-Q (STM32N657X0H, rev Z), ST Edge AI Core v4.0.1-20581, bare-metal LL_ATON runtime (BARE_METAL OSAL, SW fallback enabled), networks generated with "--st-neural-art" and a RAM memory-pool profile (weights in AXISRAM, staged by the application before each run - as before, since the ATON does not read our OCTOSPI2 mapping).

 

NETWORKS: four int8 (QDQ, per-channel symmetric weights) dilated causal TCNs - conv-only after BatchNorm folding: Conv1d stacks with dilations 1..32, causal padding trimmed by Slice, two small Gemm heads. No eltwise Add anywhere in the ONNX graph. The compiled epoch schedule is 57 epochs: 32 pure HW, 24 hybrid (Slice / SpaceToDepth / DepthToSpace), 1 SW (Gather). Four independently trained instances (different weights, different prediction targets, one from June and three new).

 

HOST REFERENCES: for each network, golden outputs were produced with onnxruntime on the int8 QDQ model, and cross-checked against the operator-equivalent model emitted by stedgeai itself (*_OE_3_3_1.onnx): host-vs-host agreement is 0-5 LSB. So the goldens represent the graph as compiled.

 

ON-TARGET RESULT: all four networks diverge far beyond any quantization tolerance. Max |NPU - golden| over a 40-window sweep per network, both outputs (int8 LSB): net A (June instance) 29 / 111; net B 241 / 140; net C 236 / 154; net D 247 / 137. The outputs are input-dependent (they vary across windows) but saturate toward a fixed byte: many windows, and even different networks, pin at the same output value (e.g. -113), which looks like an internal saturation/lowering artifact rather than accumulated rounding.

 

INTEGRATION RULED OUT (same discipline as the original report; 14+ variations there were output-byte-identical): weight blobs byte-verified in AXISRAM after staging (M55 read-back matches the generated atonbuf exactly); input verified to reach the network's input buffer (source/destination byte capture); init order permuted (one network initialized at a time vs all up-front): identical results; per-channel weight-scale magnitudes and max/min scale ratios are comparable to a network whose convolutions the ATON computes correctly on this same bench (our residual TCN: operands of its Add were verified correct at cos 0.999 in the original report); the bench itself validated in the same session: the June network re-ran and reproduced its earlier deviations byte-for-byte.

 

THE STRUCTURAL DIFFERENCE between these four failing networks and the network whose convolutions compute correctly is the conv-only causal/dilated form: dilation up to 32 with Slice-trimmed causal padding, which the compiler lowers through SpaceToDepth/DepthToSpace hybrid epochs.

 

QUESTIONS FOR ST:

(1) Is there a known correctness issue in the lowering of dilated and/or causal (Slice-trimmed) convolutions - in particular the SpaceToDepth/DepthToSpace decomposition - for Neural-ART on ST Edge AI Core 4.0.1? (Your operator documentation flags large dilation factors for performance; we are seeing what looks like a correctness effect.)

(2) A possibly-related prior report: thread 158215 ("STM32N6 onboard inference returns clipped values", Oct 2025, unresolved) describes quantized-only, systematically clipped/distorted on-target outputs vs the ONNX reference. Could these share a root cause?

(3) Does the fix planned for the int8 eltwise/Add issue (this thread) also touch conv lowering, and in which release?

 

A full reproduction package is available on request: the four int8 ONNX models, the stedgeai OE models, golden vectors, the memory-pool profile, and the firmware harness.