FreeRTOS users: Dave's helpers now on GitHub (fix for newlib memory management bugs in ST tools)

[Dave Nadler]

This is a follow-up for: https://community.st.com/s/question/0D50X0000BB1eL7SQJ/bug-cubemx-freertos-projects-corrupt-memory?t=1594600862291

Due to popular demand, I’ve updated and placed a couple of components on Github: https://github.com/DRNadler/FreeRTOS_helpers, including:

• heap_useNewlib (stop those nasty memory-management crashes), and
• a port.c for Cortex M4-7 that adds MSP (ISR) stack checking

I’ve also updated and expanded the web page http://www.nadler.com/embedded/newlibAndFreeRTOS.html.

Hope you find this helpful!

Best Regards, Dave

[SSolo.2]

Thank U very much! I implemented your improvements in my code. Then I went on and rebuilt the toolchain with unstripped libraries.

I can dive into newlib whith gdb if someting goes wrong now. The source was downloades from arm site

(https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloads)

Unfortunatly distributed toolchain/newlib configuration assumes single thread model and implements file and stream locks as stubs. So we can not use stdin,stdout and stderr in totally thread-safe manner in spite of reentrancy mechanism. To overcome this restriction I also added 'weak' attribute to stubs in the lock.c which lays in src/newlib/newlib/libc/. I also redefined 'unused' field in struct __lock from chat to void* ( that garantees 4-byte placeholder for a pointer to free-rtos mutex for example. The final thing to do is to implement own wrappers.

Here below is patched "lock.c"

#include <sys/lock.h>
 
struct __lock {
    void* unused;
//  char unused;
};
 
struct __lock __lock___sinit_recursive_mutex;
struct __lock __lock___sfp_recursive_mutex;
struct __lock __lock___atexit_recursive_mutex;
struct __lock __lock___at_quick_exit_mutex;
struct __lock __lock___malloc_recursive_mutex;
struct __lock __lock___env_recursive_mutex;
struct __lock __lock___tz_mutex;
struct __lock __lock___dd_hash_mutex;
struct __lock __lock___arc4random_mutex;
 
void __attribute__((weak))
__retarget_lock_init (_LOCK_T *lock)
{
}
 
void __attribute__((weak))
__retarget_lock_init_recursive(_LOCK_T *lock)
{
}
 
void __attribute__((weak))
__retarget_lock_close(_LOCK_T lock)
{
}
 
void __attribute__((weak))
__retarget_lock_close_recursive(_LOCK_T lock)
{
}
 
void __attribute__((weak))
__retarget_lock_acquire (_LOCK_T lock)
{
}
 
void __attribute__((weak))
__retarget_lock_acquire_recursive (_LOCK_T lock)
{
}
 
int __attribute__((weak))
__retarget_lock_try_acquire(_LOCK_T lock)
{
  return 1;
}
 
int __attribute__((weak))
__retarget_lock_try_acquire_recursive(_LOCK_T lock)
{
  return 1;
}
 
void __attribute__((weak))
__retarget_lock_release (_LOCK_T lock)
{
}
 
void __attribute__((weak))
__retarget_lock_release_recursive (_LOCK_T lock)
{
}

 I built the toolchain on and for Ubuntu with the following options:

./build-prerequisites.sh --skip_steps=howto,md5_checksum,mingw

./build-toolchain.sh --build_type=native --with-multilib-list=rmprofile,aprofile --skip_steps=howto,manual,md5_checksum,mingw,strip

[Dave Nadler]

@SSolo.2​ - A few notes:

1. The distributed newlib does not assume single thread model, otherwise heap_useNewlib would not work, right? It carefully uses stubs to make it easy to adapt.
2. You don't need to add 'weak' attribute in the library, as your code (in object files) is linked prior pulling in library functions and takes precedence over the library functions.
3. As lock.h has taken pains to make the _LOCK_T lock argument an opaque pointer, you can use the argument as a FreeRTOS semaphore pointer with no structure definition.

Hope that helps!

Best Regards, Dave

PS: It is certainly helpful to rebuild the library with debug information!

[SSolo.2]

Thanks! I was simply confused with the debugger showing lock as normal structure :) Yes, sure _LOCK_T is opaque. Opaque structures is usual practice to emulate OO features in C.

I was faced a problem when I built without "weak". Linker failed with duplicated symbols error. Probably the reason was that I did not define ALL!! symbols from lock.c. Yes U are right, there is no reason to patch lock.c at all.

When I said "single-thread" I meant "single-thread" out-of-box.

[JSoll.1]

Im trying to apply the files provided by @Dave Nadler​ but my project simply hangs while initializing the usb when the patches are applied.

My scenario is as follows:

1. I have a STM32F407G-DISC1 (A discovery board)
2. Im using CUBEMX 6.0.1 and STM32CubeIDE 1.4.0
3. Using CUBEMX I create a simple project using FreeRTOS + USB CDC Device.
4. Only modification is to add basic code to blink the board leds on the StartDefaultTask loop
5. If I compile and flash the project as it is. It simply works. (Leds are blinking, and the board is recognized as a ttyACM1 device on my Ubuntu)

Now, starting from that point I apply Dave's patches as follows:

1. Copy heap_useNewlib_ST.c to <my_project>/Middlewares/Third_Party/FreeRTOS/Source/portable/MemMang/heap_4.c
2. Copy port_DRN.c to <my_project>/Middlewares/Third_Party/FreeRTOS/Source/portable/GCC/ARM_CM4F/port.c
3. Edit FreeRTOSConfig.inc and add the followig snippet at the end:

// DRN ISR (MSP) stack initialization and checking
#if !defined(EXTERNC)
  #if defined(__cplusplus)
    #define EXTERNC extern "C"
  #else
    #define EXTERNC extern
  #endif
#endif
#define configISR_STACK_SIZE_WORDS (0x100) // in WORDS, must be valid constant for GCC assembler
#define configSUPPORT_ISR_STACK_CHECK  1   // DRN initialize and check ISR stack
EXTERNC unsigned long /*UBaseType_t*/ xUnusedISRstackWords( void );  // check unused amount at runtime

After applying the patches, the build just fails because the _sbrk symbol is duplicated, so as stated in Dave's instructions I remove from the project the default-included sysmem.c and syscalls.c files and at this point the project builds just fine. But once flashed it just hangs after MX_USB_DEVICE_Init();,

If I comment the MX_USB_Device_Init() the leds just blink fine so it seems the problem is related to this call.

Also, if I start the board without the USB connected to the PC the LEDS are blinking, but it hangs as soon as I connect the USB cable to the PC.

Can someone confirm Dave's patches working in a similar setup? Could be compiler version related?

[Dave Nadler]

@JSoll.1​ - Most likely you are asserting with malloc called inside ISR.

Fix your project so you can catch such assertions!

You can pause the program to see where its sitting (inside heap_useNewlib),

and the call stack that caused the problem (likely USB ISR).

I'm assuming ST has not yet fixed the craptastic USB stack,

despite the gorgeous PowerPoint they made for us showing

how they were going to fix all these problems.

Replace the malloc inside USB ISR to use a static structure to avoid all this mess...

Hope that helps!

Best Regards, Dave

PS: And complain to your ST salesperson about this crap STILL not getting fixed!

[SSolo.2]

Also people advise not to use ST stack but use  https://github.com/hathach/tinyusb.

[JSoll.1]

@Dave Nadler​ yes!!! that was the issue.

Now I have it working with USB cdc using your patches! Thank you very much!!

Btw I discovered when CubeMX generates the same project with Toolchain/IDE option set to "Makefile" instead of "STM32CubeIDE" the following things occur:

• sysmem.c and syscalls.c files are not included in the project by CubeMX (don't know if this is intentional or not)
• Even applying heap/port/freertosconfig patches + replacing USB malloc by a static declaration the uC just hangs when connected to USB and works (leds blinking) if not connected.

So in short: same (patched) code doesn't work when using Makefile. But I'm using a newer toolchain and I still need to try with exactly the same toolchain version used by STM32CubeIDE.

I wanted to use makefiles to easily include c++ code, but anyway I can start playing around with STM32CubeIDE :)