When using the HAL libraries, the source of this problem can be a too small heap configured in the project. The solution is to either increase the heap size, or configure the USB library to use static allocation.
The heap configuration depends on the used IDE/Toolchain:
- CubeMX: In project options (see picture below). If project is generated with CubeMX, it will then apply this configuration for the selected toolchain (IAR, Keil, SW4STM32 etc.).
- IAR: In project setting or ICF file
- Keil: In startup_stm32xxx.s file
- SW4STM32/Eclipse: In the linker script file (e.g.: STM32F746NGHx_FLASH.ld)
The USB library configuration requires enough space to allocate the structure related to the implemented USB class (e.g. USBD_CDC_HandleTypeDef in case of CDC class implementation). A value of 0x1000 should be enough for most cases, but it might also depends on other software components used in the project.
If you want to use static allocation, you can rewrite the USBD_malloc and USBD_free macros in the usbd_conf.h file. The USBD_malloc is called only once when the USB class is initialized. The static variant can return always the same pointer to static allocated memory, and USBD_free can be left empty. The size of the static allocated memory will depend on the used class (e.g. size of USBD_CDC_HandleTypeDef).