STM32N6 VENC at higher resolutions

[therealnerd2]

Hi,

We have STM32N6570-DK and are testing camera input, video encode and write to SD card

We have not been able to make the "VENC_SDCard_Appli" example work, the SD card just contains zeros.

The "VENC_SDCard_ThreadX" example does work however, resulting in an 800x480 video.

Are there any examples on encoding at 1080p resolution? Changing the encoder parameters and level results in memory allocation failures in EncAsicMemAlloc_V2.

Many thanks

[therealnerd2]

Do you have any (estimated or actual) power consumption numbers for the device when doing a 1080p encode? Including the external memory ideally. 

Thanks

[DanielS]

Hello,

We have published power consumption measures here: https://wiki.st.com/stm32mcu/wiki/Introduction_to_Hardware_Video_Encoding_with_STM32#Power_consumption

Regards,

[exarian]

Thank you @DanielS, that is a great reference!

Will this example soon be available in the STM32CubeN6 github?

Alternatively, where can the reference example be found that was used to take the power measurements?

Thank you in Advance.

[Will_Robertson]

Salut @DanielS @RomainR. and @therealnerd2 

Thank you very much - this is a very valuable discussion.

How are things going with the 'ready-to-use' 1080p example?

We're working on moving the reference frame and buffers needed by the VENC for h.264 encoding from the two 1024 K blocks of internal SRAM to external PSRAM - it looks like ST may intend us to do this by implementing EWL_USER_MM by overriding EWLMallocLinear and EWLFreeLinear but we weren't able to find any official ST documentation on this and any guidance from ST on this would be very valuable.

We're developing an open source wildlife camera for conservation of endangered European dormouse and squirrel species:

https://github.com/William-Robert-Robertson/WildCamera/tree/main

We've documented more about why and how we plan to move the VENC reference frame and buffers here - any guidance from ST on this would be very valuable:

https://github.com/William-Robert-Robertson/WildCamera/blob/main/Candidate%20Technologies/STM32N6x7_MCU/VENC_Video_Encoder.md

https://github.com/William-Robert-Robertson/WildCamera/blob/main/Candidate%20Technologies/STM32N6x7_MCU/Thanks_For_The_Memory.md

Merci beaucoup pour tes conseils !

Will

[DanielS]

Hello @Will_Robertson,

You have to look at latest  STM32CubeN6 (v1.3.0)

The configurability of the VENC_RTSP_Server and VENC_USB applications is enhanced. 

=> You can change encoding settings in the venc_h264_config.c file.

#if defined(FULL_1080P_SLICE)

#...

#elif defined (FULL_1080P_FRAME )

...

For information about encoder allocation hooks, refer to the integration guide, which describes them in detail:

./Middlewares/Third_Party/VideoEncoder/doc/Hantro.VC8000NanoE.V50x.SW.Integration.Guide-v1.02-20200708.pdf (see EWLMalloc, EWLMallocRefFrm, and EWLMallocLinear).

One important point is that EWLMallocLinear returns contiguous, aligned, and uncached memory.

In the examples above, the memory hooks use ThreadX memory pools.

There are three main buffers:

uint8_t input_frame[...]: raw frame as captured by the camera or DCMIPP

uint8_t ewl_pool[...]: H.264 encoder internal memory

uint8_t output_block_buffer[...]: encoded bitstream

Hope this helps,

Daniel

[Will_Robertson]

Hi @DanielS 

Thank you very much. Because the ST example project using the Neural-ART accelerator - STM32N6-GettingStarted-ObjectDetection - is bare metal rather than ThreadX we were focusing on integrating the bare metal VENC_SD example - it's very helpful to know that configuration in the VENC_RTSP_Server and VENC_USB applications is handled differently.

Would it be a good way forward for us to take this way of handling configuration from VENC_RTSP_Server and VENC_USB and attempt to port it over to the STM32N6-GettingStarted-ObjectDetection example?

Does this configuration in the VENC_RTSP_Server and VENC_USB applications also put the reference frame used by the VENC in PSRAM? I wasn't sure from the source code how the reference frame is handled in these applications and whether the reference frame was treated as part of "VENC internal buffer" or not?

/* Buffer placement ---------------------------------------------------------*/
/** Location macro for VENC internal buffer */
#define VENC_BUFFER_LOCATION    IN_PSRAM
/** Location macro for input frame buffer */
#define INPUT_FRAME_LOCATION    IN_PSRAM

I'd guessed that the reference frame is treated as part of "VENC internal buffer" from this but I wasn't sure:

    refPic     = picBuffer->refPic;        /* Reference frame store */
    cur_pic    = picBuffer->cur_pic;    /* Reconstructed picture */

Many thanks!

Will

[DanielS]

Hello @will Robertson,

In our ThreadX examples, there is no distinction between EWLFreeRefFrm (reference frame allocator) and EWLMallocLinear (encoder software/hardware shared buffer allocator).
Both use the same ThreadX pool allocator mapped to the same physical memory block: uint8_t ewl_pool[VENC_POOL_SIZE] ALIGN_32 VENC_BUFFER_LOCATION;

Therefore, when VENC_BUFFER_LOCATION is set to "IN_PSRAM", all memory used by the encoder resides in external PSRAM.

Regards,
Daniel

[Will_Robertson]

Hi @DanielS 

Thank you very much!

Is this memory configuration mechanism for the EWL intended for both ThreadX and bare metal implementations or is this memory configuration mechanism intended for ThreadX only with the older configuration mechanism in the VENC_SD example intended for bare metal?

Thank you very much for your help - it's an enormous help to us in understanding this.

Will

[DanielS]

Hello @Will_Robertson,

The EWL layer is available in three variants: ThreadX, FreeRTOS, and No OS.
The variant depends on the EWL_ALLOC_API compile flags.
The memory allocators are also defined as __weak, which allows you to adapt memory placement to specific requirements.

Best regards,
Daniel

[Will_Robertson]

Hi Daniel,

Thank you very much!

Fundamentally, our choice of ThreadX, FreeRTOS or bare metal determines which of the three EWL layer variants we use and we set the EWL_ALLOC_API compile flag to choose the appropriate EWL variant?

Thank you very much!

Will