When is going malloc() / dynamic allocation to be fixed?

[DavidAlfa]

Edit: It seems there's a bug that will randomly delete sysmem.c and syscalls.c.

If it happens, cubeMX won't regenerate them.

Also no warnings at compile time, so you won't suspect anything was deleted.

So the solution is to make a new empty project and copy these files back.

___________________________________________________________________________________ 

I don't know if it's a bug, wrong initialization, or lacking code.

I remember once in 2019 where it worked, I could run malloc(X) and the pointer would be null ​if wanting too much.

But since then, none of the Cube IDE versions have worked correctly.

I can run a 100KB malloc on a ​16KB stm32, yet it will return a valid pointer.

Of course a hard fault will happen when you try to access the memory.

I had to modify​ every project to use static allocation, which is sometimes a problem because I could save a lot of RAM when certain parts of the code is not being used.​

The minimum heap allocation​ just reserves the heap at compile time, doesn't fix anything.

​

Also, free() doesn't seem to work correctly.​

I can malloc 64KB on a 128KB device and use it.

But if I malloc 1KB and 16KB, then free both, I can't malloc 64KB (it does, but outside the memory limits).

I don't understand why, as these pointers where freed up,​ so memory fragmentation shouldn't be happening?​

​

I'm spending​ more time in looking over these bugs than on my own ones...

​

Don't tell me​ that I have to increase heap. Because I could put 16TB of heap, and still get hard faults when it goes too far.

Why doesn't malloc check the remaining space?

​

[Tesla DeLorean]

Hard to say but the _sbrk() implementation has been a train-wreck for a long while.

Thing is with these tool chains, you get what you paid for, and them some..

[DavidAlfa]

Tell that to Microchip or TI.

MplabX was a hell of an IDE when it first came out back in 2009 or so, but became a solid IDE.

And TI support is amazing.

If you are selling millions and getting a big chunk of the market​ share, how the hell can you let the software getting so bad? That is your main target.

Stm32 are powerful and cheap MCUs, if the software is also good, developers​ will feel comfortable and will develop more with them, then just sit and fill your pockets!

​

With 100+MHz​, 2MB flash, 512KB RAM, you should be very comfortable.

This beast can run anything!

​

But my actual feel when starting a project is "Will I get it to run? Or some weird bug will appear and trash all my work?".​

[Pavel A.]

ST sells chips. They don't want to compete with software partners in their ecosystem (Keil, IAR and others).

[DavidAlfa]

Very reasonable to buy​ Atollic and make a new half-crippled ide, announcing new MCUs but not fixing such important issues.

Keil and IAR have terrible interfaces, also Keil is a pita to set up. When you have to add a project and discover you can't just import a folder, but you must add every file manually!

​

[DavidAlfa]

So, I returned to IDE v1.4.0.

Not only that malloc works perfectly, but also the speed increased dramatically! I can ensure it's not placebo effect.

I cleanly installed 1.6.1, CubeMX wizard was 5x slower.

The MCU list being slowly drawed line by line like in an old DOS computer, took 2-3 seconds.

Deleted everything, cleanly installed V1.4.0, MCU listing drawed on 1/2 second.

Still slow as hell, considering I'm running it from a computer with SSD, 16GB RAM, i7 3770K at 4.4GHz.

I assigned more ram in stm32cubeide.ini, but didn't make any difference at all:

-Xms1024m

-Xmx8192m

Not everything is ST's fault, Java is usually slower and resource-eater, also eclipse has it's own issues. 

But clearly CubeIDE is getting worse on every new release.

Instead enhacing the IDE, you guys seem to be getting it worse intentionally. Almost 1 year with broken malloc, and just ignoring it.

Not to mention the dropping of Error_handler(__FILE__, __LINE__) 3 years ago.

Thankfully the users came for help, I've been using this macro for a while:

https://community.st.com/s/question/0D50X00009XkffVSAR/stm32cubemx-v421-errorhandler-definition-issues-in-mainh

It seems we must keep a collection of dirty hacks to fix everything that gets broken on every new release.

I don't see any sense on that.

This works now:

// MCU: STM32F411CEU6 (128KB RAM)
// Test allocating 8-256KB, increasing by 8KB on every itineration
 char* data;
 for (uint16_t t=8;t<257;t+=8){
	 data = malloc(t*1024);
	 if(!data){ // In 1.4.0, it correctly returns 0 when trying to allocate 128KB
		 asm("nop"); // But on later versions it will return valid pointer and cause a Hard fault
	 }
	 else{
		 free(data);
		 data = NULL;
	 }
 }

[DavidAlfa]

In v1.4.0, _sbrk is declared in sysmem.c. 

But could't find it in v1.6.1, neither is sysmem.c created. Where is it?

[PMath.4]

"Where is it?"

syscalls.c

[DavidAlfa]

Anyways, checking the disassembly, the difference is huge!

Clearly, the latter does absolutely nothing about checking limits.

Cube IDE v1.4.0

_sbrk:
08000a20: push {r7, lr}
08000a22: sub sp, #24
08000a24: add r7, sp, #0
08000a26: str r0, [r7, #4]
59 const uint32_t stack_limit = (uint32_t)&_estack - (uint32_t)&_Min_Stack_Size;
08000a28: ldr r2, [pc, #80] ; (0x8000a7c <_sbrk+92>)
08000a2a: ldr r3, [pc, #84] ; (0x8000a80 <_sbrk+96>)
08000a2c: subs r3, r2, r3
08000a2e: str r3, [r7, #20]
60 const uint8_t *max_heap = (uint8_t *)stack_limit;
08000a30: ldr r3, [r7, #20]
08000a32: str r3, [r7, #16]
64 if (NULL == __sbrk_heap_end)
08000a34: ldr r3, [pc, #76] ; (0x8000a84 <_sbrk+100>)
08000a36: ldr r3, [r3, #0]
08000a38: cmp r3, #0
08000a3a: bne.n 0x8000a42 <_sbrk+34>
66 __sbrk_heap_end = &_end;
08000a3c: ldr r3, [pc, #68] ; (0x8000a84 <_sbrk+100>)
08000a3e: ldr r2, [pc, #72] ; (0x8000a88 <_sbrk+104>)
08000a40: str r2, [r3, #0]
70 if (__sbrk_heap_end + incr > max_heap)
08000a42: ldr r3, [pc, #64] ; (0x8000a84 <_sbrk+100>)
08000a44: ldr r2, [r3, #0]
08000a46: ldr r3, [r7, #4]
08000a48: add r3, r2
08000a4a: ldr r2, [r7, #16]
08000a4c: cmp r2, r3
08000a4e: bcs.n 0x8000a60 <_sbrk+64>
72 errno = ENOMEM;
08000a50: bl 0x8008b0c <__errno>
08000a54: mov r3, r0
08000a56: movs r2, #12
08000a58: str r2, [r3, #0]
73 return (void *)-1;
08000a5a: mov.w r3, #4294967295
08000a5e: b.n 0x8000a74 <_sbrk+84>
76 prev_heap_end = __sbrk_heap_end;
08000a60: ldr r3, [pc, #32] ; (0x8000a84 <_sbrk+100>)
08000a62: ldr r3, [r3, #0]
08000a64: str r3, [r7, #12]
77 __sbrk_heap_end += incr;
08000a66: ldr r3, [pc, #28] ; (0x8000a84 <_sbrk+100>)
08000a68: ldr r2, [r3, #0]
08000a6a: ldr r3, [r7, #4]
08000a6c: add r3, r2
08000a6e: ldr r2, [pc, #20] ; (0x8000a84 <_sbrk+100>)
08000a70: str r3, [r2, #0]
79 return (void *)prev_heap_end;
08000a72: ldr r3, [r7, #12]
80 }
08000a74: mov r0, r3
08000a76: adds r7, #24
08000a78: mov sp, r7
08000a7a: pop {r7, pc}
08000a7c: movs r0, r0
08000a7e: movs r0, #2
08000a80: lsls r0, r0, #16
08000a82: movs r0, r0
08000a84: lsls r4, r3, #5
08000a86: movs r0, #0
08000a88: lsrs r0, r6, #32
08000a8a: movs r0, #0

Cube IDE v1.6.1  

_sbrk:
0801e64c: ldr r2, [pc, #16] ; (0x801e660 <_sbrk+20>)
0801e64e: ldr r1, [pc, #20] ; (0x801e664 <_sbrk+24>)
0801e650: ldr r3, [r2, #0]
0801e652: cmp r3, #0
0801e654: it eq
0801e656: moveq r3, r1
0801e658: add r0, r3
0801e65a: str r0, [r2, #0]
0801e65c: mov r0, r3
0801e65e: bx lr
0801e660: ldrb r4, [r3, #4]
0801e662: movs r0, #0
0801e664: ldrb r0, [r4, #4]
0801e666: movs r0, #0

[DavidAlfa]

It seems there's a bug somewhere.

When creating a new project, sysmem.c and syscalls.c are generated correctly.

However, if these files are deleted, cubeMX doesn't regenerate them.

At some point, sysmem.c and syscalls.c dissapeared. I'm still trying to find the culprit.

A moment ago it happened after reopening cubeMX and enabling FastFS, but I couldn't reproduce the problem again. The gremlins...

With them, malloc works again in v1.6.1.. 

But the thing is that without these files, it compiles ok, without any warning.

[Pavel A.]

> At some point, sysmem.c and syscalls.c dissapeared. I'm still trying to find the culprit.

Interesting...

Use version control for your projects. It will notice any missing files.

> But the thing is that without these files, it compiles ok, without any warning.

The default library likely has an empty _sbrk routine that does nothing.

[DavidAlfa]

Yeah, I usually do. Probably these files were gone from the very beginning (before pushing Head for the first time) , and since I never used dynamic allocation in this project... never noticed.

Now, I made a few new projects. Sometimes the files are gone, sometimes not.

Didn't find the way to always reproduce the problem, it's very random.

The problem now is that it seems I'm having internal fragmentation. Not external. Let me explain:

From malloc.h:

struct mallinfo {
 size_t arena; /* total space allocated from system */
 size_t ordblks; /* number of non-inuse chunks */
 size_t smblks; /* unused -- always zero */
 size_t hblks; /* number of mmapped regions */
 size_t hblkhd; /* total space in mmapped regions */
 size_t usmblks; /* unused -- always zero */
 size_t fsmblks; /* unused -- always zero */
 size_t uordblks; /* total allocated space */
 size_t fordblks; /* total non-inuse space */
 size_t keepcost; /* top-most, releasable (via malloc_trim) space */
};

 Test:

#include <malloc.h>
 
struct mallinfo info;		// To gather current mem info
uint8_t *a, *b, *c, *d;		// Test ptr
 
a = malloc(8192);		// Allocate 8KB
b = malloc(1024);		// Allocate 1KB
c = malloc(810);		// Allocate 810B
d = malloc(1492); 		// Allocate 1492B
info = mallinfo();		// arena=11552 , uordblks=11552
free(a);
free(b);
free(c);
free(d);
info = mallinfo();		// arena=11552, fordblks=11552
 
a = malloc(10240); 		// Allocate 12KB
info = mallinfo();		// arena=23848, uordblks=12296, fordblks=11552
free(a);
info = mallinfo();		// arena=23848, fordblks=23848

After freeing up all the memory, it will not reuse it if the next allocation is bigger than the total previous.

So it gets fragmented anyway, although the heap is completely free.

After few runs the heap will fail.

After 1st malloc: 
Heap: [8192][1024][810][1492] (11552 due alignment, usage variables)
 (a) (b) (c) (d)
 
After free:
Heap: [11552]
 (free)
 
Then, when allocating 12KB, this happens;
Heap: [11552] [12296]
 (free) (new alloc)
 
Instead:
Heap: [12296]
 (new alloc)

If the free region is smaller, it works:

a = malloc(106736);
 info = mallinfo(); // Total used = 106744
 free(a);
 a = malloc(100000); // Works

But if you request only 1 byte more, it try to allocate at the end of the free space, and fail:

a = malloc(106736);
 info = mallinfo(); // Total used = 106744
 free(a);
 a = malloc(106745); // Doesn't work

A fix is to first allocate all the heap:

a = malloc(121*1024); // Allocate 121KB (Max heap)
free(a);
// Now any allocation will work and not cause any fragmentation when freed up

[Pavel A.]

Try this: https://github.com/MaJerle/lwmem

[DavidAlfa]

Looks nice. Will try it. Thanks!