> how to catch it when we do not even have a hardfault occurring!?
Oh, bugs are a live thing and not all of them fail in a neat reproducible way clearly indicating the point of failure. The "random bug" is the worthy one to catch, and how to hunt them down, is the art of this trade.
So you have to have at hand a whole repertoaire of tools - hardware like probes, oscilloscope you have a good grip at, logic analyzer, even the humble multimeter. "But I am a software engineer" is a whine to be left out of the door. Of course you have to have documentation at hand, and consult it often. And software - have a good grip at the toolchain and its darker corners, know thy mapfile. It's good to have a sleeveful of tricks in the mcu's software, too - like knowing how to toggle pins to be observed by LA, how to output or store relevant debug info/files without relying on printf() and/or any "semihosting" automagic, how to use otherwise unused resources of the mcu like memories embedded in the peripherals, etc. But, when it comes to bugs, disassembler is your biggest friend, together with the on-chp-debugging utilities (but those have to be known and understood well, too). And remember - software, or the box on the table, may be called debugger, but the real debugger sits on your chair.
So be innovative. The first thing is to get the bug reproduce, at least in some way. Enhance reproduction by stressing the application - churn on it communication at maximum speed, connect pushbuttons to random pulse generators, feed the audio processor with noise or some 10-hour youtube ***, deliberately overdriving the input. Carefully observe symptoms - even if they may be unrelated to your initial complaint or any theory you might have formulated, deviation of any form from what is "normal" *must* be explained.
Try to catch the bug. It may be that the symptoms indicate the bug has happened far before the symptoms occur, but try to halt execution as soon as observable symptoms occur. For example, if stack overflow is suspected to be the raw cause, then using the DEADBEEF method this theory can be safely disproved even at a late catch if some DEADBEEF remains (it can't be safely confirmed but suspicion certainly increases in that direction if there's no DEADBEEF left).
Then try carefully crafted changes, observe the behaviour. Understand, what the changes really mean. Try to make the changes very local - this is hard to do I know. Make sure you can always return to the reproducible state. It may help to take notes at this point.
Formulate theories, then devise experiments to prove or disprove them. You suspect stack overflow? How could overflow be catched? It's a write to an address beyond the last address allocated to the stack, so what about using the on-chip-debugger's data breakpoint facility; or, if desperate, the MPU? Or, maybe a simpler
You may also try the divide et impera method I mentioned - omitting whole blocks of program. Sometimes it gets obvious what's wrong. Not often; but hey, this is the Real Bug, so there's little to lose. Still make sure you can return to the reproducible state.
Sleep well. Discuss the problem with your colleagues, with the cleaning lady, with your partner at home, with your teddy bear.
Be persistent. When the Real Bug creeps in, deadlines are void. Getting a product out of the door knowing there's a bug - well, that gives "deadline" the real meaning. If a manager pops in demanding progress, explain him the problem in great technical detail (and don't let him slip out), and then ask him to hold an oscilloscope probe - yes, ON the PIN 57 of LQFP176, NOT on the pad! - while making your software experiments finally with some comfort. An hour or two should suffice.
All this takes years to get good at. And then, one day, comes a Real Real Bug, and I - with all those years and knowledge and experience - then look as a complete a****ole anyway... ;)
JW
