LL_ATON_LIB_Concat falls back to 4× memcpy when the concat axis is not leftmost — 32% of a 4 ms real-time budget on STM32N6

[Michael_K]

LL_ATON_LIB_Concat falls back to 4× memcpy when the concat axis is not leftmost — 32% of a 4 ms real-time budget on STM32N6

Hardware: NUCLEO-N657X0-Q (STM32N657X0H3Q), Cortex-M55 @ 800 MHz, Neural-ART. Tools: ST Edge AI Core 4.0, STM32Cube_FW_N6 V1.4.0, STM32CubeIDE 2.1.1. Application: real-time audio, hop = 192 samples @ 48 kHz = 4 ms = 3.2 M cycles, hard deadline, underrun must stay 0.

1. Concat falls back to a byte loop, and it is one third of the budget

Our graph is a streaming TCN (int8, L=12, C=88, T=48) run per audio hop with two voices, so the tensor layout has the voice axis leftmost: [C][V][T]. The graph contains 12 Concat blocks (channel-wise taps).

Epoch profiling (LL_ATON_EB_DBG_INFO, DWT cycle counter on the M55) showed those 12 blocks costing 1 019 501 cycles per inference — 32% of our entire 4 ms budget. No compiler option changed the number.

The cause is in the runtime source: LL_ATON_LIB_Concat takes its fast DMA path only when the concatenation axis is the leftmost significant axis. With V=2 sitting to the left, that test fails and the block is executed by the generic branch — four memcpy calls per block.

The arithmetic confirms it exactly. The 12 blocks move 145 728 bytes per hop (this equals the compiler's own cost estimate, which counts Concat as one element per cycle):

1 019 501 cycles / 145 728 bytes = 6.997 cycles per byte

and that figure is identical across all six different block sizes — the signature of a copy loop, not of a DMA setup cost.

Workaround, and why it is unsatisfying

We overrode memcpy with a strong symbol: an MVE implementation issuing four 16-byte loads before the first store, tail handled by predication. The strong symbol displaces the newlib one and reaches all code including ST files we must not modify. Result:

Concat: 1 019 501 -> 343 459 cycles   (6.997 -> 2.36 cycles/byte)
hop (silence): 2 826 500 -> 2 104 000 cycles

That is a 676 kcycle saving — 21% of the hop budget — recovered by replacing a libc function under the vendor runtime. It works, but overriding memcpy globally is a heavy hammer for an application to swing, and it silently changes behaviour for every ST driver linked into the image.

What we would ask for

1. Extend the fast path so it applies when the concat axis is contiguous in memory, regardless of whether it is leftmost.

2. Failing that, document the gating condition in the ST Edge AI docs, so that graph authors know the layout choice has a 30%-of-budget consequence.

3. Optionally: have the runtime use an internal optimized copy instead of libc memcpy, so applications do not have to override a standard function to get reasonable throughput.

Point 2 alone would have saved us several days: nothing in the documentation suggests that the position of an axis changes Concat cost by 3×.

2. dilations > 1 unsupported in ONNX import — pyramid emulation is ~9× over budget

Same setup. A causal TCN naturally uses dilated convolutions (1..32 across two stacks). atonn 4.0 rejects dilations > 1 on Conv and emulates dilation with a pyramid of reshapes. On our shape the resulting graph executed in 37.36 ms against a 4 ms hop — about nine times over budget.

We worked around it by expressing dilation explicitly: Slice taps + channel Concat + 1×1 Conv (we call this the "gather2" form). Mathematically the same operations; roughly 10× faster on this target.

This is presumably a known limitation, but the size of the gap seems worth reporting: the naive export path is not slightly slower, it is unusable, and the working form is not obvious. A note in the documentation — "express dilation as explicit slices; the pyramid emulation is not intended for real-time" — would be valuable.

Reproduction

Both numbers come from on-target measurement with the DWT cycle counter and per-epoch profiling, not from estimates. I can share the graph shape, the epoch profile dump, and the memcpy implementation if that helps.

[hamitiya]

Hello ​@Michael_K 

Thank you for all the information you have provided. It helps us a lot to improve our tools.

As you mentioned in your last sentence, would it be possible to provide extra information, such as:

• model used
• what is the override you have done with `memcpy`
• code generated by ST Edge AI Core, including settings
• or the project itself if it is not sensible

It would be easier for us to spot the differences you have made in your project.

 

Best regards,

Yanis

[Michael_K]

Hi Yanis, thanks for the quick reply. Everything is below, and I have to correct myself on one point first.

In my original post I described the twelve expensive Concat blocks as "channel-wise taps". That was wrong, and it would have sent you looking at the wrong nodes. The expensive ones are the state-ring concatenations along the width axis. Our graph does contain channel-axis concatenations as well — the dilation taps — and those are fine, they take the fast path and cost nothing worth mentioning. Both kinds sit in the same graph, same build, same image, which is really the cleanest evidence I can offer.

The condition is axis_is_leftmost in LL_ATON_LIB_Concat (ll_aton_lib.c): every dimension to the left of the concat axis has to be 1. On device our tensors are laid out [1, V=2, T, C=88] and we concatenate along the width axis, so the V=2 sits to the left, the check fails, and the block drops into the generic branch at the bottom of the function — four memcpy calls per block, two per input, one per V row. The channel-axis concats have only [1] to their left, so they pass.

The measurement is what convinced me it is a copy loop and not DMA setup. Twelve blocks, widths 50/52/56/64/80/112, which is exactly 2*88*(48+2*d) for dilations 1 to 32. They move 145 728 bytes per inference — the same number as the sum of the compiler's own est, since Concat is estimated at one element per cycle. On the M55, with the DWT counter and per-epoch profiling (LL_ATON_EB_DBG_INFO):

total    1 019 501 cycles / 145 728 bytes = 6.997 cycles/byte
by size  7.04  7.01  6.95  6.96  6.91  6.87

A constant price per byte with no fixed offset. If this were DMA setup there would be an offset, and the small blocks would look relatively worse. They don't.

I have attached repro_concat.py rather than the project itself. It needs onnx and numpy and writes a graph containing only the offending pattern — six Concat nodes on [1, 88, 2, W] along axis=3, no arithmetic at all — so whatever the generated code spends on it is the copy. The real graph has twelve such nodes because the stack is doubled; change DILATIONS in the script if you want the full count. I'd rather not dump the whole project on you: it is a trained audio model, and its topology would only make the Concat harder to find. If the actual .onnx and the quantization JSON would help, tell me and I'll send them privately.

About the model. It is a streaming causal TCN, int8 via quantize_static, 12 layers, 88 channels, dilations 1..32 in two stacks, input x[1, 8, V, T] and output y[1, 4, V, T] with V=2 and T=48. One layer looks like this:

cat_i       = Concat(state_in_i[1,C,V,2d], h_i[1,C,V,T], axis=3)   <- slow one
d == 1: r_i = Conv(cat_i, W[C,C,1,3], B)
d  > 1: tap_j = Slice(cat_i, j*d, j*d+T, axis=3), j = 0,1,2
        g_i   = Concat(tap_0, tap_1, tap_2, axis=1)                <- fast one
        r_i   = Conv(g_i, W[C,3C,1,1], B)
state_out_i = Slice(cat_i, T, T+2d, axis=3)
h_{i+1}     = Add(h_i, Relu(r_i))

The explicit slices are our workaround for the dilations > 1 limitation I mentioned in the first post.

The memcpy override is a strong symbol in the application, so it displaces the newlib-nano one and reaches the runtime too. MVE, four 16-byte loads issued before the first store, tail by predication:

c

void *memcpy(void *__restrict dst, const void *__restrict src, size_t n)
{
    uint8_t *d = (uint8_t *)dst;
    const uint8_t *s = (const uint8_t *)src;

    if ((((uintptr_t)d | (uintptr_t)s) & 3u) == 0u) {
        while (n >= 64u) {
            uint32x4_t a = vldrwq_u32((const uint32_t *)(s +  0));
            uint32x4_t b = vldrwq_u32((const uint32_t *)(s + 16));
            uint32x4_t c = vldrwq_u32((const uint32_t *)(s + 32));
            uint32x4_t e = vldrwq_u32((const uint32_t *)(s + 48));
            vstrwq_u32((uint32_t *)(d +  0), a);
            vstrwq_u32((uint32_t *)(d + 16), b);
            vstrwq_u32((uint32_t *)(d + 32), c);
            vstrwq_u32((uint32_t *)(d + 48), e);
            s += 64; d += 64; n -= 64u;
        }
        while (n >= 16u) {
            vstrwq_u32((uint32_t *)d, vldrwq_u32((const uint32_t *)s));
            s += 16; d += 16; n -= 16u;
        }
    } else {
        while (n >= 16u) {
            vstrbq_u8(d, vldrbq_u8(s));
            s += 16; d += 16; n -= 16u;
        }
    }
    if (n) {
        mve_pred16_t p = vctp8q((uint32_t)n);
        vstrbq_p_u8(d, vldrbq_z_u8(s, p), p);
    }
    return dst;
}

That took Concat from 1 019 501 to 343 459 cycles (6.997 to 2.36 per byte) and the whole hop in silence from 2 826 500 to 2 104 000.

I want to be precise about why it helps, because "we wrote a faster memcpy" is misleading. The newlib-nano one is not byte-wise — for aligned pointers it already copies words, we checked the disassembly. What changes is the number of outstanding misses on the path from the M55 to npuRAM, where a single transaction costs roughly 66 cycles. One miss at a time gives you seven cycles per byte no matter how the loop is written; four in flight give you 1.4 in a plain copy and 2.36 through the runtime's per-row calls. So this is a memory latency effect, and I don't think an application should have to redefine a libc function to get at it.

We check the override at start-up against a byte-wise reference over lengths 0..300 on eight alignments. The reference needs volatile pointers, otherwise GCC recognises the loop and calls memcpy — the very function under test.

Settings. The backend line is taken verbatim from network_generate_report.txt:

atonn -i n6_gather2_qdq_OE_3_3_1.onnx \
      --json-quant-file n6_gather2_qdq_OE_3_3_1_Q.json \
      -g network.c \
      --load-mdesc stm32n6.mdesc \
      --load-mpool stm32n6_nucleo_app_safe.mpool \
      --load-cdesc cortex-m55.cdesc \
      --optimization 3 --all-buffers-info --cache-maintenance --Oauto-sched \
      --native-float --enable-virtual-mem-pools --Omax-ca-pipe 4 \
      --Ocache-opt --Os --enable-epoch-controller --generate-stai

On the frontend, stedgeai generate from ST Edge AI Core 4.0 with user-allocated IO (--no-inputs-allocation, --no-outputs-allocation) and channel positions left at their defaults. The memory pool is stm32n6_nucleo_full_onchip with cpuRAM1/2 trimmed so they don't overlap our own RAM; weights (273 kB) and activations are all on-chip.

For completeness, compiler options that did not move Concat at all: eliminate_concat_split, fuse_consecutive_concats_new, --ec-optimize, -S. And --Ox was clearly worse — 172 blocks instead of 70, estimate 2 725 403 against 1 263 056.

As for what would help us: ideally the fast path would apply to any concatenation that is contiguous in the destination, not only when the axis is leftmost. Failing that, having the runtime call its own optimized copy instead of libc memcpy would remove the need for applications to do what we did. And at the very least, documenting the axis_is_leftmost condition would be worth a lot — nothing in the docs hints that a size-2 axis sitting to the left of the concat axis triples its cost, and that note alone would have saved us several days.

If it's useful I'm happy to run experiments on our board — the epoch profiler is wired up and I can turn a build around quickly.

[Michael_K]

Follow-up with links — you asked for the project, so I've made it public.

https://github.com/mikekss/tymbal

Everything from my last post lives in one file, fw/src/npu_neuralart.c: the memcpy override, the epoch-profiling hooks (LL_ATON_EB_DBG_INFO and LL_ATON_RT_SetEpochCallback) and the per-block counters. The Concat reproducer I attached earlier is tools/repro_concat.py, and docs/chip_findings.md is the raw bring-up log. One caveat: that repo is a snapshot with no history — MIRROR.md explains why.

The findings that are about package templates rather than the runtime went to the Cube repo instead, since that's a different team: https://github.com/STMicroelectronics/STM32CubeN6/issues/22 — dummy cycles 6 against the 20 the BSP actually programs, XSPI kernel clock at 266 MHz where this flash wants 50, and AXISRAM3..6 left disabled after the bootloader hands over.

The offer stands on the Concat side: I can turn a build around quickly if you want a specific experiment run on the board.

[jerry65]

LL_ATON_LIB_Concat falls back to 4× memcpy when the concat axis is not leftmost — 32% of a 4 ms real-time budget on STM32N6

Hardware: NUCLEO-N657X0-Q (STM32N657X0H3Q), Cortex-M55 @ 800 MHz, Neural-ART. Tools: ST Edge AI Core 4.0, STM32Cube_FW_N6 V1.4.0, STM32CubeIDE 2.1.1. Application: real-time audio, hop = 192 samples @ 48 kHz = 4 ms = 3.2 M cycles, hard deadline, underrun must stay 0.

1. Concat falls back to a byte loop, and it is one third of the budget

Our graph is a streaming TCN (int8, L=12, C=88, T=48) run per audio hop with two voices, so the tensor layout has the voice axis leftmost: [C][V][T]. The graph contains 12 Concat blocks (channel-wise taps).

Epoch profiling (LL_ATON_EB_DBG_INFO, DWT cycle counter on the M55) showed those 12 blocks costing 1 019 501 cycles per inference — 32% of our entire 4 ms budget. No compiler option changed the number.

The cause is in the runtime source: LL_ATON_LIB_Concat takes its fast DMA path only when the concatenation axis is the leftmost significant axis. With V=2 sitting to the left, that test fails and the block is executed by the generic branch — four memcpy calls per block.

The arithmetic confirms it exactly. The 12 blocks move 145 728 bytes per hop (this equals the compiler's own cost estimate, which counts Concat as one element per cycle):

1 019 501 cycles / 145 728 bytes = 6.997 cycles per byte

and that figure is identical across all six different block sizes — the signature of a copy loop, not of a DMA setup cost.

Workaround, and why it is unsatisfying

We overrode memcpy with a strong symbol: an MVE implementation issuing four 16-byte loads before the first store, tail handled by predication. The strong symbol displaces the newlib one and reaches all code including ST files we must not modify. Result:

Concat: 1 019 501 -> 343 459 cycles (6.997 -> 2.36 cycles/byte)

hop (silence): 2 826 500 -> 2 104 000 cycles

That is a 676 kcycle saving — 21% of the hop budget — recovered by replacing a libc function under the vendor runtime. It works, but overriding memcpy globally is a heavy hammer for an application to swing, and it silently changes behaviour for every ST driver linked into the image.

What we would ask for

1. Extend the fast path so it applies when the concat axis is contiguous in memory, regardless of whether it is leftmost. Check minecraft app.

2. Failing that, document the gating condition in the ST Edge AI docs, so that graph authors know the layout choice has a 30%-of-budget consequence.

3. Optionally: have the runtime use an internal optimized copy instead of libc memcpy, so applications do not have to override a standard function to get reasonable throughput.

Point 2 alone would have saved us several days: nothing in the documentation suggests that the position of an axis changes Concat cost by 3×.

2. dilations > 1 unsupported in ONNX import — pyramid emulation is ~9× over budget

Same setup. A causal TCN naturally uses dilated convolutions (1..32 across two stacks). atonn 4.0 rejects dilations > 1 on Conv and emulates dilation with a pyramid of reshapes. On our shape the resulting graph executed in 37.36 ms against a 4 ms hop — about nine times over budget.

 

Thanks for sharing, this post helps me a lot.

[hamitiya]

Hello,

We are still investigating performance gaps.

For small models (with already “low” inference time), it could be also interesting to execute without ST Neural-ART Accelerator, to compare overheads.

On my side, I was able to reduce from 2.8ms to 2.2ms with your memcpy patch, to 0.8ms without NPU, and 0.6ms by fine-tuning memory pools and optimization level (option from ST Edge AI Core).

Since your model already was not using acceleration that much, you may consider this option.

 

Best regards,

Yanis

[Mancy34]

This is a very useful finding. The 3× Concat penalty and ~9× dilation overhead are significant for real-time STM32N6 workloads. It would be great if ST could document these limitations and, ideally, optimize the Concat fast path and provide a supported way to handle dilations >1 without requiring application-level workarounds.