STM32N6570-DK VENC H.264: FUSE_ERROR (-17) on first H264EncStrmEncode, then -7/-14/-3 on later frames

[vedantgg]

Hi all,

I am using the VENC H.264 encoder (Middlewares/Third_Party/VideoEncoder from the STM32 model zoo object detection application) on the STM32N6570-DK Discovery Kit with the IMX335 camera. The encoder is fed RGB565 frames from DCMIPP (1280x720 buffer in external PSRAM), and the output goes to a PSRAM buffer.

Problem:
H264EncInit, H264EncSetPreProcessing, H264EncSetRateCtrl and H264EncStrmStart all return OK. The first H264EncStrmEncode call (intra frame) then returns -17 (H264ENC_FUSE_ERROR). In the driver, this is raised when the VENC status register has ASIC_STATUS_FUSE (0x200) set. The register description in encswhwregisters.h says: "IRQ stream format fuse status bit. Format disabled by fuse."

After that, later frames often fail with -7 (INVALID_STATUS), then -14 (INSTANCE_ERROR), sometimes -3 (INVALID_ARGUMENT) or -16 (HW_RESET). At times the encoder recovers after frame 0 and produces a stream; at other times it never does. The behaviour is not consistent between runs, even at 640x480.

What I tried (no fix):

• Sizes 640x480, 960x540, 1024x576, 1280x720
• Levels 2.2, 3.1, 4, 4.2
• Heap for EWL: 2.25 MB up to 6 MB (this fixed -6 EWL_MEMORY_ERROR)
• Making the heap, output buffer and slice buffer non-cacheable through the MPU
• H264ENC_REFERENCE vs H264ENC_NO_REFERENCE_NO_REFRESH for ltrf
• Retrying the frame 0 encode
• Cache invalidate on the input buffer before each encode
• Bitrate 4.8 to 8 Mbps, pictureRc on

Setup details:

• Board: STM32N6570-DK Discovery Kit (32 MB external PSRAM confirmed working)
• Input type H264ENC_RGB565, origWidth/origHeight 1280x720, offsets 0 (or centered crop for 640x480)
• refFrameAmount = 1, 30 fps, byte stream
• LL_VENC_Init() called after VENC clock reset
• Fuse_Programming() from the application only sets the VDDIO2/VDDIO3 HSLV bits

Questions:

1. Which "format" does the VENC fuse check refer to? Is it the RGB565 input format, a resolution, a profile or level, or a specific H.264 feature disabled on some parts?
2. Is H.264 encoding at 720p30 supported on the STM32N657 in the Discovery Kit? What is the maximum supported resolution and frame rate?
3. Is there a required setup step I am missing (for example fuse or OTP configuration, clock settings, or a different input format such as YUV that is known to work)?
4. Could the intermittent behaviour point to a clock, bus or memory access problem for VENC reading from external PSRAM?