Skip to main content
Associate
August 18, 2026
Question

STM32N6 Neural-ART optimization fails with “Oauto did not find valid compile options” after successful U-Net quantization

  • August 18, 2026
  • 2 replies
  • 45 views

Environment

I am trying to deploy a U-Net segmentation model on STM32N6 using ST Edge AI Cloud.

ST Edge AI Core:

ST Edge AI Core v4.0.1-20581

Target:

stm32n6

Current Status

The model conversion and quantization steps complete successfully.

The workflow is:

PyTorch U-Net

ONNX FP32 export
✓ successful


Per-channel quantization
✓ successful


ST Edge AI optimization / Neural-ART compilation
✗ failed

Therefore, the issue does not occur during ONNX export or quantization.

The failure happens specifically during the Neural-ART optimization/compiler stage.

Model Architecture

Input:

[1, 3, 320, 320]

Output:

[1, 1, 320, 320]

The network is a U-Net-like segmentation architecture.

Encoder channels:

64 → 128 → 256 → 512

Bottleneck:

1024

Decoder channels:

512 → 256 → 128 → 64

Each convolution block consists of:

Conv2D
→ BatchNorm
→ ReLU
→ Conv2D
→ BatchNorm
→ ReLU

Downsampling:

MaxPool2D
kernel_size = 2
stride = 2

Upsampling:

ConvTranspose2D
kernel_size = 2
stride = 2

The encoder feature maps are concatenated with the corresponding decoder feature maps using U-Net skip connections.

Final layer:

Conv2D
kernel_size = 1
output_channels = 1

Simplified PyTorch definition:

class DoubleConv(nn.Module):
def __init__(self, in_channels, out_channels):
super().__init__()

self.double_conv = nn.Sequential(
nn.Conv2d(
in_channels,
out_channels,
kernel_size=3,
padding=1,
bias=False
),
nn.BatchNorm2d(out_channels),
nn.ReLU(inplace=True),

nn.Conv2d(
out_channels,
out_channels,
kernel_size=3,
padding=1,
bias=False
),
nn.BatchNorm2d(out_channels),
nn.ReLU(inplace=True),
)

def forward(self, x):
return self.double_conv(x)


class UNet(nn.Module):
def __init__(
self,
in_channels=3,
out_channels=1,
features=[64, 128, 256, 512]
):
super().__init__()

self.downs = nn.ModuleList()
self.ups = nn.ModuleList()

for feature in features:
self.downs.append(
DoubleConv(in_channels, feature)
)
in_channels = feature

self.bottleneck = DoubleConv(
features[-1],
features[-1] * 2
)

for feature in reversed(features):
self.ups.append(
nn.ConvTranspose2d(
feature * 2,
feature,
kernel_size=2,
stride=2
)
)

self.ups.append(
DoubleConv(feature * 2, feature)
)

self.final_conv = nn.Conv2d(
features[0],
out_channels,
kernel_size=1
)

def forward(self, x):
skip_connections = []

for down in self.downs:
x = down(x)
skip_connections.append(x)

x = F.max_pool2d(
x,
kernel_size=2,
stride=2
)

x = self.bottleneck(x)

skip_connections = skip_connections[::-1]

for idx in range(0, len(self.ups), 2):
x = self.ups[idx](x)

skip_connection = (
skip_connections[idx // 2]
)

x = torch.cat(
(skip_connection, x),
dim=1
)

x = self.ups[idx + 1](x)

return self.final_conv(x)

Failure During Optimization

The quantized ONNX model is generated successfully.

However, when running ST Edge AI optimization for STM32N6, the Neural-ART compiler fails during atonn.

Relevant output:

>>>> EXECUTING NEURAL ART COMPILER

Warning: Oauto did not find valid compile options: aborting

Configuration 0: alt-scheduler(false):
total bytes left unallocated=212825984

Configuration 1: alt-scheduler(true):
total bytes left unallocated=218560384

E103(CliRuntimeError):
Error calling the Neural Art compiler

The compiler is invoked with optimization enabled, including:

--Oauto-sched
--optimization 3
--Omax-ca-pipe 4
--enable-virtual-mem-pools

Questions

Since ONNX export and quantization both complete successfully, I would like to understand why the model fails specifically during Neural-ART optimization/compilation.

  1. What does the following message specifically indicate?

Oauto did not find valid compile options
  1. Does this mean that all operators are supported, but the Neural-ART scheduler cannot find a valid execution/memory schedule?

  2. Could the large activation tensors and U-Net skip connections / Concat operations be responsible for the optimization failure?

  3. Could ConvTranspose2D become problematic specifically during Neural-ART scheduling, even though the model can be successfully imported and quantized?

  4. What does:

total bytes left unallocated

represent in this context?

Does it indicate insufficient memory, or tensors that the scheduler could not place into a valid memory configuration?

  1. Is there a way to obtain more detailed diagnostics showing which layer, tensor, epoch, or operator causes Oauto-sched to fail?

  2. Are there recommended architectural changes for U-Net models targeting STM32N6 Neural-ART, particularly regarding:

    • skip connections,

    • concatenation,

    • feature-map size,

    • ConvTranspose2D,

    • decoder structure?

The main point I would like to understand is why a model that can be successfully exported and quantized cannot be successfully optimized by the Neural-ART compiler.

 

2 replies

hamitiya
ST Technical Moderator
August 18, 2026

Hello ​@Jason32 

During analysis, ST Edge AI Core was not able to allocate your model based on memory contrains for STM32N6570-DK

  • total bytes left unallocated=218560384

Your model seems quite large (218MB worth of activations or weights combined) which does not fit in this microcontroller.

Without Automatic mode, is ST Edge AI Core able to generate an output for your model? If yes, consider not using Automatic mode, and if you can share your model with us in order to improve generator behavior it can be great.

 

Best regards,

Yanis

​In order to give better visibility on the answered topics, please click on 'Best answer' on the reply which solved your issue or answered your question.
Jason32Author
Associate
August 22, 2026

Hi Yanis,

Thank you for the clarification and explanation.

After reviewing the model and the analysis results, I now understand that the failure under the Automatic configuration is caused by the model’s peak memory requirements exceeding what can be successfully allocated on the target.

This model was not originally designed by me, and I am still getting familiar with its architecture and memory characteristics. After further investigation, I found that the U-Net architecture, especially the large feature maps and skip connections, results in a relatively high peak activation memory requirement.

I have attached the quantized ONNX model that reproduces the issue in case it is useful for your analysis or for improving the generator behavior.

For now, I will discuss this with my teammates and consider possible modifications to the model architecture or deployment configuration to reduce the memory requirements.

If we encounter any further issues during this process, I will follow up with additional questions.

Thank you again for your help and support.



Best regards,

Jason32