Hard Fault on function start - but cannot debug

[AScha.3]

I have a problem, but dont know, how to solve...

when this function ( mp3dec_decode_frame(..) ) is used, get hard fault. ok, but cannot debug , because no step into function is possible, hard fault is always next step.

Now the prerequisites : this function i used to decode MP3 in my H743 project, no problems.

then copied to this H563 project, but now using rtos Azure (because forced to use this by STM)…

and now : not working anymore, even cannot debug, what happens there, because first is : jump to hard fault.

So maybe the stack is in nirwana, but i cannot see (understand ARM assembler) anything, that leads me to solve the problem.

See , here is the debug at starting the function:

disassm shows here:

(i see no problem...) stack is in mem:

and this seem ok:

so: how to find the reason for this hard fault ?

next step ...hard fault :

cpu then:

 

 

[waclawek.jan]

Is the floating-point coprocessor enabled?

JW

[AScha.3]

should be...

 

[HTD]

I have similar experience with Hard Faults. In my case I just allocated a bigger memory pool for the component and it worked. So the library I used (FILEX) probably had a bug with some out of bounds writes. In a perfect world it should crash into a nice spin loop showing you exactly where it could not write data. And when you have the stack frame overwritten - no debugging for you. That's probably the kind of bug where the data goes where it should not go. The common symptom of this is debugging showing nothing or just some random garbage that makes no sense.

[AScha.3]

@HTD , right, i also had suspicions against Azure - stack, memory management - and increased values:

8K stack, 100K mem

but not in filex, because this seems (!!!) working fine.

still have 300K ram free, so i try with more mem for filex thread - as you suggest.

 

[Tesla DeLorean]

Looks like it might depend on the double precision FPU-D of the H7. Perhaps use library from CM4F part or F74x core with FPU-S

[AScha.3]

...there was a problem with : on what cpu we are here ?

float seem used .

but at first the mp3-lib had some ARM64 setting found in defines...i dont know why.

i just put a "!" (=not) on this define, to get 

#define HAVE_ARMV6 0

before it had aarch64 active.  Crazy, this was not in my H743 project (but i dont remember, whether i set some define there or modified a global define...afaik i didnt; it was set right and worked. )

now armv6->0 and so should be ok, in this respect.

optimizer..on -> no errors + warnings

paths seem ok:

 

[LCE]

> ...there was a problem with : on what cpu we are here ?

Is it working now?

[AScha.3]

no. 

thats why i asked - i thought, now all should be ok.  and no idea, what still wrong. :frowning_face:

[waclawek.jan]

As @Tesla DeLorean said above, from disasm (vpush of double registers) it appears that the library uses double-precision FPU which is not available in Cortex M33/'H5. I'm not versed enough to tell if vpush of double is indeed dependent on presence of the double-precision FPU, but, just another question to the many: do you compile that library from sources, or does it come as a precompiled .a?

Also, I'm not sure the -fpu setting guarantees the FPU to be switched on. IIRC in the Cube infrastructure, compilation of the couple of instruction needed to switch it on depends on some specific preprocessor symbols to be present - see startup code or functions called from there for relevant details.

JW

[AScha.3]

@waclawek.jan  , ok, so i just set it to "software fpu " --- but still same: not even one step in the function possible.

+ give filex more space, 20K -> 120K ...but changes nothing.

+ i forgot to tell: at first i had it on a F411 (black pill) board, runs without problems. Just slower and not enough ram, to use other decoder (for FLAC) also, so i moved it to a H743 board, all decoders working here fine. 

But to get an audio host - big problem. No example by STM , just one USB audio-host, for a disco-xxx board. But i tried to use this, got it working...and then found: hey, it just can enumerate a headset ! nothing else !!

Just good for advertising : Oh, we have audio-host also !

so i looked for a REAL audio-host, but found only one , that looked promising: USBX - but is part of  Azure rtos.

So took new nucleo-H563 board, can import Azure, usbx, filex... and got it working, 

can enumerate a sound card and play on it.  good, so far. now add the decoders...flac seem ok, but mp3 makes this crazy hard fault. 

[LCE]

The above assembler, starting with STMDB and using the stack pointer, that's where it instantly fails?

I guess you already checked stack size and location in the linker file? Especially after increasing stack and heap?

[AScha.3]

>I guess you already checked stack size and location in the linker file? 

no - to be honest, i never ever looked at a linker file (what should i look for there?)

Just i think, i see it in debug anyway, at runtime (or not ??)

here (now) at decoder start:

thread info:

cpu:

I am at a loss . looks all ok for me...

UNTIL THE NEXT STEP ! then hard fault - not even in assembler one step to the function...

 

 

[AScha.3]

just a funny idea : can i jump to other position in this function?

ok, i moved cursor down 3 lines and set (menu)run -> move to line , then single step ... no problem !

so the problem is the first instruction - right? 

from ARM ...:  >>

STMDB / STMFD

Store Multiple Decrement Before (Store Multiple Full Descending) stores multiple registers to sequential memory locations using an address from a base register. The sequential memory locations end just below this address, and the address of the first of those locations can optionally be written back to the base register.

<<

how can this generate a hard fault, if SP is middle in the "byte pool" ???

[Piranha]

Tesla and Jan already told you that the VPUSH instruction is pushing double floating point registers and therefore the code is compiled for double sized FPU devices. Jan also asked whether you are compiling that module yourself or using an already compiled library.

[AScha.3]

sorry, i forgot to write: compile from source.

compiler setting should be ok:

and vpush - can be with single or double floats , as i read this:

<sreglist>

Is the list of consecutively numbered 32-bit SIMD&FP registers to be transferred. The first register in the list is encoded in "Vd:D", and "imm8" is set to the number of registers in the list. The list must contain at least one register.

<dreglist>

Is the list of consecutively numbered 64-bit SIMD&FP registers to be transferred. The first register in the list is encoded in "D:Vd", and "imm8" is set to twice the number of registers in the list. The list must contain at least one register, and must not contain more than 16 registers.

but : i tried -as i wrote- also with software float - but same : hard fault at stmdb instruction.

here is the code for this function:

 

int mp3dec_decode_frame(mp3dec_t *dec, const uint8_t *mp3, int mp3_bytes, mp3d_sample_t *pcm, mp3dec_frame_info_t *info)
{	// inp.buf. > ; decodiert 1 frame ca. 418 B./128kb = 1152 sampels = 4608 Byte
 int i = 0, igr, frame_size = 0, success = 1;
 const uint8_t *hdr;
 bs_t bs_frame[1];
 mp3dec_scratch_t scratch;

 if (mp3_bytes > 4 && dec->header[0] == 0xff && hdr_compare(dec->header, mp3))
 {
 frame_size = hdr_frame_bytes(mp3, dec->free_format_bytes) + hdr_padding(mp3);
 if (frame_size != mp3_bytes && (frame_size + HDR_SIZE > mp3_bytes || !hdr_compare(mp3, mp3 + frame_size)))
 {
 frame_size = 0;
 }
 }
 if (!frame_size)
 {
 memset(dec, 0, sizeof(mp3dec_t));
 i = mp3d_find_frame(mp3, mp3_bytes, &dec->free_format_bytes, &frame_size);
 if (!frame_size || i + frame_size > mp3_bytes)
 {
 info->frame_bytes = i;
 return 0;
 }
 }

 

 

 

 

[HTD]

I'm not sure if I haven't got lost here, but is this that FILEX related thing? Because if you run in on M33, I run it on M33 too. STM32U599 to be specific. I tested it with several SD cards and several pendrives. Do you use version 6.2.0 of FILEX? You will find the version in `fx_api.h`. I tested it with 6.2.0.

[AScha.3]

@HTD , yes is 6.2.0  too. 

seems to work fine now. (with short wires to sd-card-holder).

see -if you want -

https://community.st.com/t5/stm32cubeide-mcus/sd-card-not-working-with-azure-filex-anybody-got-it-working/m-p/617597#M22578

 

the problem here ... is difficult. Maybe the compiler/toolchain is the problem. 

[Piranha]

Indeed even your initial code allocates ~17 KB of stack space in that function:

sub.w sp, sp, #17280 ; 0x4380

 

If it is not done directly in the code of that function, it is a result of function inlining done by compiler optimization.

[AScha.3]

@Piranha  , your right . :)

but still i dont understand: on single step, who generates hard fault at the first instruction , where just some pointers are pushed to the stack ?

and even the stack gets too big, it just would "kill" some other memory (for usb stack..) - so how or who calls hard fault here? is it the rtos ?

now:

and threadX stack statistics:

 

[Piranha]

You are stepping just one of the RTOS threads. As the SysTick is still running and it's interrupt is already pending, as soon as the CPU starts running, the RTOS kernel kicks in and switches your thread out. After that all active threads of a higher and same priority gets a chance to run until the RTOS comes back to your thread. To test this, you can disable all interrupts globally around that call and see what difference it makes.

[HTD]

Hey, have you made it? Meaning solving the FILEX hard fault problems?

Because I got into them recently. Hard faults all the way. Weird, because my simple test passed on all devices, so I made a more elaborate test that created, renamed and deleted more files, also longer transfers and it started to crash and misbehave. What I figured out it's not (currently) related to a specific media, I get similar issues on both USB and SD cards. So I create file, read and write. It seems to work. But I do more file operations and it crashes or misbehaves. Recent weird behavior - all works except deleting files. It worked before.

What I noticed about THREADX - it's super sensitive about the memory. They don't say how big the memory pools should be, nor the thread stacks. However - tuning the pools and stack for correct threads made the hard faults go away. Yet I'm not entirely sure it's totally fixed, without MPU used the middleware can still overwrite things, it just doesn't crash the app on spot. If anyone asks - USB host thread for MSC needs 64KB of the pool and 44KB for the stack. However, the actual thread that uses the file system needs probably bigger stack. I moved all file operations to my main application thread (I mean RTOS thread, not the initial thread). And I allocated all available memory to it, to not have to worry about it anymore. I took me like 48 hours to figure out all middlewares minimal configurations. If you use STM32U5x9 - I can share my memory map for it. It might be not perfect, but it seems to work and leaves enough RAM for a big app with a GUI.

As we loosely discuss the quality of this stuff - it's not the best design when it crashes instead just returning a civilized error code. But I remember having crashes with FATFS too. There's one more thing: the worst bug ever in FATFS. When you open a file, you provide a file handle to it. When it's not ZEROED - the FATFS will not crash, it will start to express various undefined behaviors, a real hell to debug, diagnose, test and fix. Well, FILEX seem to initialize the file handles properly. Yet I zero them to be extra sure to avoid mines ;)

The worst thing when coding in C and C++ is to assume things. Like the memory is zeroed, or the index is within range, the pointer is not dangling and so on... Our favorite FS tools have a lot of such assumptions in them, unfortunately. 

[AScha.3]

Yes, seems working now . 8)

Problem was (probably) stack overgrown in the area of the usbX stack area...

The mp3 decoder eating up 23KB stack, so i was just lucky to make a test with set to 24KB, then no hard fault.

Other problem with sometimes error was the hardware: lines to sdcard too long (15cm) , curiously the fatfs i tried had less problems , or no problem at first. But after some days - also errors sometimes. Then i guessed, maybe its not the software and give fileX another try. With short wires (25mm) now reads at 200Mbit, about 20MB/s .

I work on a nucleo-H563 with this, and agree: >> THREADX - it's super sensitive about the memory.

+ it should tell about, not just crash. :grimacing_face:

I thing there is a stack monitoring - but dont know, what it is doing...can be set in tx_user.h

#ifndef TX_USER_H
#define TX_USER_H

/* USER CODE BEGIN 1 */
#define TX_ENABLE_STACK_CHECKING
/* USER CODE END 1 */

+

>  I moved all file operations to my main application thread 

I had same idea , named it "tx_app_main" , to know where i am  :)

 

just my settings:

FileX   20KB pool   , 4K stack. exfat enable , MAX_CACHE_SIZE_NB_BIT 9 ; map_size 128;

MAX_FAT_CACHE_NB_BIT 4 ; MAX_SECTOR_CACHE_NB_BIT 4 ; 

usbX   32K pool  , 24K sys stack, 4K app stack.

threadX 60K pool , 24K app stack .

and added small test thread : 1K stack . (blink LED , to "see" multitasking is running )

if you know better settings, tell... :)

[HTD]

Holly cow, I'm surprised it works wit so little stack for your file access thread. I gave my file access thread 128KB and it still crashes depending on the target file system. I guess there's something terribly wrong with my code. It's peculiar that the media and file handles are pretty low in the memory, but my local variables are on the stack limit when it crashes. It seems like something is pumping the stack there. I probably gave it more than enough memory, something else must be wrong.

That's the first time the function is called here.

The offending code seems to start here:

Maybe it's related to media statistics. And AFAIK it's free space evaluation, number of files and that sort of thing. Maybe it's broken? IDK, I'll just try to disable and see what will happen. I noticed that my tests use to pass when performed on empty media, and crash when I add some files to them (I just copy a bunch of my sources for testing).

BTW, my crash occur on `fileExists` function, so it shouldn't even need much memory to complete, just a few sectors for directory entries. I'm sure it uses more than one. One is needed for the parent directory, one for the candidate (it must load an entry to test if it matches). Then it returns the found entry, so at least 3 are needed to perform the search. But it's just theory.