Custom bootloader: USB DFU bootloader fails after application initializes USB before system reset

[ARagn.1]

Problem

I am working on an STM32H743 and I have implemented a custom bootloader that should be able to jump to the STM32 internal system memory USB DFU bootloader.

The application can request a firmware update by setting a flag (in the BKPRAM) and performing a system reset. After the reset, my custom bootloader detects the flag and tries to jump directly to the STM32 system memory DFU bootloader.

The relevant part of my bootloader is:

int main(void)
{
    if (Bootloader_IsUpdateRequested())  //check if update is requested
    {
        Bootloader_ClearUpdateRequest();
        EnterUpdateMode();               //go dfu mode
    }

    HAL_Init();
    SystemClock_Config();

    MX_GPIO_Init();
    MX_USART3_UART_Init();

    HAL_Delay(100);


    int err = Bootloader_is_app_valid(&crc);  //check if app is valid


    if (err != 0)
    {
        // ...

        Bootloader_SetUpdateRequest();
        NVIC_SystemReset();
    }

    JumpToApplication();  //start application

    while (1)
    {
    }
}

The function used to jump to the system memory bootloader is:

void EnterUpdateMode(void)
{
    /* Disable all interrupts */
    __disable_irq();

    /* Disable SysTick timer */
    SysTick->CTRL = 0;

    /* Clear Interrupt Enable Register & Interrupt Pending Register */
    for (uint8_t i = 0; i < (MCU_IRQS + 31u) / 32; i++)
    {
        NVIC->ICER[i] = 0xFFFFFFFF;
        NVIC->ICPR[i] = 0xFFFFFFFF;
    }

    /* Re-enable all interrupts */
    __enable_irq();

    /* Set the MSP */
    __set_MSP(BOOTVTAB->Initial_SP);

    /* Jump to system memory bootloader */
    BOOTVTAB->Reset_Handler();
}

BOOTVTAB points to the STM32H743 system memory bootloader address according to AN2606.

In the application I use:

void Go_DFU_Mode(void)
{
    Bootloader_SetUpdateRequest();
    NVIC_SystemReset();
}

The important observation

If I do not initialize USB in the application, everything works correctly.

For example:

int main(void)
{
   ...
   Go_DFU_Mode();

   MX_USB_DEVICE_Init();
   ...
}

or, alternatively:

int main(void)
{
   ...
   MX_USB_DEVICE_Init();
   MX_USB_DEVICE_DeInit();

   Go_DFU_Mode();
   ...
}

In these cases, the MCU resets and the custom bootloader enters the STM32 system memory bootloader. Windows correctly detects the device as STM32 DFU.

However, if I initialize USB and then immediately request DFU:

int main(void)
{
   ...
   MX_USB_DEVICE_Init();
   Go_DFU_Mode();
   ...
}

the DFU bootloader does not work. The MCU resets, but Windows does not detect the STM32 DFU device.

So the only difference is that in the failing case the application has initialized the USB peripheral immediately before NVIC_SystemReset().

Question

What is the correct procedure for jumping from a custom bootloader to the STM32H743 System Memory USB DFU bootloader? I would like the DFU bootloader to work reliably even when USB has been initialized by the application and has not been deinitialized before the system reset.

I have already tried disabling interrupts, SysTick, and clearing the NVIC interrupt enable/pending registers before jumping to system memory, but this does not solve the problem when the application has initialized USB.

Any suggestions would be greatly appreciated.
Thanks!

[TDK]

The system bootloader assumes an as-reset state. You should deinitialize USB before jumping there. This is the supported method.

You can no-doubt reverse engineer the bootloader to jump directly into DFU with the appropriate setup in your application first but this isn’t supported, may change with new bootloader versions, and is a lot of work for not much benefit. To do this, you would need to initialize the USB with the same descriptor as the system bootloader.