Support for JPEG Encoding / SD Card storage Integration in STM32N6 Object Detection Firmware

[pikud]

Dear ST Community,

I have successfully run the Object Detection Application on the STM32N6 Discovery Kit using the reference firmware from the STM32AI ModelZoo.

As part of extending its functionality, I am looking to implement the following two features:

1. Convert the captured RGB frame to JPEG format
2. Store the JPEG image onto the SD card available on the Discovery Kit

For the RGB-to-JPEG conversion, I have referred to this JPEF Encoder example and integrated this JPEG encoding logic into the object detection firmware. Specifically, I am feeding the nn_in buffer (used as neural network input) to the JPEG encoder. However, during execution, the program gets stuck at this line:

jpeg_encode_processing_end = JPEG_EncodeOutputHandler(&hjpeg);

The jpeg_encode_processing_end flag is not being set to 1, indicating that the encoding process does not complete. I suspect either the JPEG input handling is not progressing, or the DMA isn't properly triggering the required callbacks.

Could you kindly review the approach and suggest the correct way to integrate JPEG encoding into the object detection pipeline?

I have attached my modified main.c file for your reference.

Appreciate your guidance on this.

[Saket_Om]

Hello @pikud 

Could you try to put as input the image in the JPEG encoder example. 

[pikud]

Hi @Saket_Om 
Following your suggestion, I tried using the static test image from the JPEG encoder example (image_320_240_rgb.h) as the input to the JPEG encoder of object detection firmware. However, upon doing so, the build fails with a linker error due to .rodata exceeding the available memory in the AXISRAM1_S region:

.section `.rodata' will not fit in region `AXISRAM1_S'
region `AXISRAM1_S' overflowed by 660280 bytes

To work around this, I moved large .rodata arrays (such as the test image or other large const buffers) into PSRAM using custom section attributes and appropriate linker script modifications. But by doing that I am unable to Run the program, but can build it.

Can you suggest what else needs to be done here?

[Saket_Om]

Hello @pikud 

---

@pikud wrote:

 But by doing that I am unable to Run the program, but can build it.

Can you suggest what else needs to be done here?

---

What is the problem that appear when you run the program?

[pikud]

Hi @Saket_Om 
1.) I’ve attached the problem log image below. Could you help identify what this might be related to?

2.) Also, the earlier issue of the program getting stuck at jpeg_encode_processing_end = JPEG_EncodeOutputHandler(&hjpeg) is resolved now 

It seems the JPEG encoding was getting stuck because the required interrupt handlers for JPEG and DMA were missing. I added them in stm32n6xx_it.x and it was fixed

3.)Currently, JPEG encoding is running, but I’m seeing corrupted or unexpected byte data in the encoded jpeg output. I've attached the updated main.c file for reference.

My question is: since nn_in is the pointer to the camera's output buffer (used as the NN input), is it safe and valid to pass it directly into the JPEG encoder like this?

JPEG_Encode_DMA(&hjpeg, (uint32_t)nn_in, RGB_IMAGE_SIZE_1, jpgBuffer);

Would appreciate any guidance on whether the nn_in buffer requires any format conversion or cache handling before passing it to the JPEG encoder.

[fisy]

Hi @pikud 

Were you able to solve your problems with the corrupted jpegs?
I have been trying to write images to my sd card via FileX based on these examples:
https://github.com/STMicroelectronics/STM32CubeN6/tree/main/Projects/STM32N6570-DK/Examples/JPEG/JPEG_EncodingFromOSPI_DMA
and https://github.com/STMicroelectronics/STM32CubeH7/tree/master/Projects/STM32H743I-EVAL/Examples/JPEG/JPEG_EncodingUsingFs_DMA

I have tried all possible combinations. My current approach is based on the second example, where the encode_dma was adapted to write to the file directly instead of going via a buffer.

In the end no matter what I try, i get images that look like this:
Sometimes also only the upper part of the image contains colored chunks and the rest is grey.

My current code is structured like this:

UINT SaveFrameJPEG(uint32_t frameIndex)
{
    FX_FILE fxFile;
    UINT status;
    CHAR filename[32];

    sprintf(filename, "frame%03lu.jpeg", frameIndex);

    /* Create the file */
    status = fx_file_create(&sdio_disk, filename);
    if(status != FX_SUCCESS && status != FX_ALREADY_CREATED)
    {
        printf("File create error: %u\r\n", status);
        return 1;
    }

    /* Open file for write */
    status = fx_file_open(&sdio_disk, &fxFile, filename, FX_OPEN_FOR_WRITE);
    if(status != FX_SUCCESS)
    {
        printf("File open error: %u\r\n", status);
        return 1;
    }

    /* Start JPEG encoding (DMA) directly to FileX file */
    JPEG_Encode_DMA(&hjpeg, (uint32_t)RGB_ImageAddress, ImageSizeBytes, &fxFile);

    /* Wait for DMA encoding to finish */
    do
    {
        JPEG_EncodeInputHandler(&hjpeg);
    } while(JPEG_EncodeOutputHandler(&hjpeg) == 0);

    /* Close FileX file */
    fx_file_close(&fxFile);

    printf("Saved %s\r\n", filename);

    return 0;
}

 

I have updated the encode_dma from the STM32N6 example to a style as seen in the H7 example:

FX_FILE *pFxFile;

/* Private function prototypes -----------------------------------------------*/
/* Private functions ---------------------------------------------------------*/

/**
  * @brief  Encode_DMA
  * @param hjpeg: JPEG handle pointer
  * @param  FileName    : jpg file path for decode.
  * @param  DestAddress : ARGB destination Frame Buffer Address.
  * @retval None
  */
uint32_t JPEG_Encode_DMA(JPEG_HandleTypeDef *hjpeg,
                         uint32_t RGBImageBufferAddress,
                         uint32_t RGBImageSize_Bytes,
                         FX_FILE *fxFile)
{
    pFxFile = fxFile;  // Save pointer to global

    uint32_t DataBufferSize = 0;

    /* Reset all global variables */
    MCU_TotalNb      = 0;
    MCU_BlockIndex   = 0;
    Jpeg_HWEncodingEnd = 0;
    Output_Is_Paused = 0;
    Input_Is_Paused  = 0;

    /* Get RGB Info */
    RGB_GetInfo(&Conf);
    JPEG_GetEncodeColorConvertFunc(&Conf, &pRGBToYCbCr_Convert_Function, &MCU_TotalNb);

    /* Clear Output Buffer */
    Jpeg_OUT_BufferTab.DataBufferSize = 0;
    Jpeg_OUT_BufferTab.State = JPEG_BUFFER_EMPTY;

    /* Fill input buffer */
    RGB_InputImageIndex   = 0;
    RGB_InputImageAddress = RGBImageBufferAddress;
    RGB_InputImageSize_Bytes = RGBImageSize_Bytes;

    DataBufferSize = Conf.ImageWidth * MAX_INPUT_LINES * BYTES_PER_PIXEL;

    if(RGB_InputImageIndex < RGB_InputImageSize_Bytes)
    {
        MCU_BlockIndex += pRGBToYCbCr_Convert_Function(
            (uint8_t *)(RGB_InputImageAddress + RGB_InputImageIndex),
            Jpeg_IN_BufferTab.DataBuffer,
            0,
            DataBufferSize,
            (uint32_t*)(&Jpeg_IN_BufferTab.DataBufferSize)
        );

        Jpeg_IN_BufferTab.State = JPEG_BUFFER_FULL;
        RGB_InputImageIndex += DataBufferSize;
    }

    /* Fill Encoding Params */
    HAL_JPEG_ConfigEncoding(hjpeg, &Conf);

    /* Start JPEG encoding with DMA method */
    HAL_JPEG_Encode_DMA(
        hjpeg,
        Jpeg_IN_BufferTab.DataBuffer,
        Jpeg_IN_BufferTab.DataBufferSize,
        Jpeg_OUT_BufferTab.DataBuffer,
        CHUNK_SIZE_OUT
    );

    return 0;
}

/**
  * @brief JPEG Output Data BackGround processing .
  * @param hjpeg: JPEG handle pointer
  * @retval 1 : if JPEG processing has finiched, 0 : if JPEG processing still ongoing
  */
uint32_t JPEG_EncodeOutputHandler(JPEG_HandleTypeDef *hjpeg)
{
    if(Jpeg_OUT_BufferTab.State == JPEG_BUFFER_FULL)
    {
        /* Write encoded chunk to FileX */
        UINT fxStatus;
        fxStatus = fx_file_write(pFxFile, Jpeg_OUT_BufferTab.DataBuffer, Jpeg_OUT_BufferTab.DataBufferSize);
        if(fxStatus != FX_SUCCESS)
        {
            Error_Handler();
        }

        Jpeg_OUT_BufferTab.State = JPEG_BUFFER_EMPTY;
        Jpeg_OUT_BufferTab.DataBufferSize = 0;

        if(Jpeg_HWEncodingEnd != 0)
        {
            return 1; // finished
        }
        else if((Output_Is_Paused == 1) && (Jpeg_OUT_BufferTab.State == JPEG_BUFFER_EMPTY))
        {
            Output_Is_Paused = 0;
            HAL_JPEG_Resume(hjpeg, JPEG_PAUSE_RESUME_OUTPUT);
        }
    }

    return 0;
}

The images above show the output when using the examples "Image_rgb_565.c" but before i was using data from IMX335 with similar output results. The same happens when using the approach seen in the STM32N6 example.

Has anyone seen similar results from the JPEG peripheral?

Best regards!