Sometimes a crash in the main-loop occurs: STM32F407 with CAN, USART1 and TIMER2

[torsten]

Posted on January 19, 2015 at 20:18

Dear All,

I am working on a project where I want to control a couple of motors via CAN (CANopen).

The STM32F407 is connected via an USB-Serial converter (FT232RL, FTDI) to a PC.

(USART1 with

USART_InitStructure.

USART_BaudRate

= 3072000;

)

On the PC a GUI application is running (configuration, demand value propagation for the STM32, monitoring of demand and actual values etc.).

TIMER2 is configured with a cycle time of 1 millisecond: Every millisecond a flag is set to trigger an action in the main-loop.

The following 3 interrupts are active overall:

1. Receive commands from PC

void

USART1_IRQHandler

(

void

)

{

if

(USART_GetITStatus(USART1, USART_IT_RXNE) ==

SET

)

{

rxBuf[p_rxBuf] = USART_ReceiveData(USART1);

…

2. Receive CAN messages (three each millisecond from three motor controller overall)

void

CAN1_RX0_IRQHandler

(

void

)

{

CAN_Receive(CAN1, CAN_FIFO0, &CAN1_RxMessage);

…

3. Trigger flag. If the flag is set some jobs (calculations, message processing etc.) will be done in the main-loop.

void

UB_TIMER2_ISR_CallBack

(

void

)

All interrupts are set up with

NVIC_InitStructure.

NVIC_IRQChannelPreemptionPriority

= 0;

NVIC_InitStructure.

NVIC_IRQChannelSubPriority

= 0;

So they shouldn't interrupt each other.

The information flow from STM32 to PC is done according to the following:

void

UART1_SendByte

(uint16_t val)

{

while

(USART_GetFlagStatus(USART1, USART_FLAG_TXE) ==

RESET

);

USART_SendData(USART1, val);

}

This happens in the main-loop during the trigger flag is set.

Every millisecond up to 80 Bytes are transferred from the STM32 to the PC (monitoring data).

Every millisecond up to 3 CAN messages are sent to the motor controllers:

…

timeout = 0;

while

(!(CAN1->

TSR

& CAN_TSR_TME0 || CAN1->

TSR

& CAN_TSR_TME1 ||

CAN1->

TSR

& CAN_TSR_TME2))

{

timeout++;

if

(timeout > CAN1_TX_TIMEOUT)

break

;

}

CAN_Transmit(CAN1, &CAN1_TxMessage);

…

This also happens in the main-loop during the trigger flag is set.

If I don't send data from the GUI to the STM32, everything is fine.

The motor-control-loop with cycle time of 1 millisecond (3 Tx-CAN-Msg, 3 Rx-CAN-Msg) works fine.

Also the information flow from STM32 to PC via USART1 is fine.

The blocking implementation of Tx-USART1 consumes CPU time, but even if I sent 80 bytes per millisecondm, the calculations and jobs in the main-loop according to the TIMER2 trigger signal consume only roundabout 40% of the available CPU time.

The Problem occurs if I send data from the PC to the STM32.

At first everything is fine, but somewhere along the line the crash happens.

In the main-loop nothing happens any longer, but the TIMER2 ISR still works, even sending data via USART1 works in the TIMER2 ISR.

Variables which are accessed in the main-loop and by interrupts are secured by bit-banding flags like the following:

#define

UART_RX_ISR_FLAG *((

volatile

unsigned

long

*) (0x22000000))

#define

UART_RX_MAIN_FLAG *((

volatile

unsigned

long

*) (0x22000004))

For data transfer from variables to CAN or USART1 Tx-buffers and vice versa I use “memcpy�?:

txBuf2[0] = 10;

txBuf2[1] = 76;

memcpy

(&txBuf2[2], &t, 4);

memcpy

(&txBuf2[6], &curDem[0], 12);

memcpy

(&txBuf2[18], &curAct[0], 12);

…

It would be really great if somebody has an idea what is the reason for that crashes

I seems so as if the USART1 Rx Interrupt causes the problem, but I have no idea how to work around.

Perhaps USART in DMA mode could be a workaround!???

Thanks in advance,

Torsten

[Tesla DeLorean]

Posted on January 19, 2015 at 20:56

Crashes how exactly? Ends up in Hard Fault Handler? Does that output any useful information, or a while(1)? Stack utilization?

[torsten]

Posted on January 20, 2015 at 08:44

Hello Clive,

at first thanks for your quick response.

I am not an expert in microcontroller programming, especially if it is done near to the hardware.

How can I check the hard fault handler?

I assume there is a special interrupt which will be activated and a register what I have to check, but what interrupt and what register?

The last tested programm structure (main.c) is the following

…

volatile

uint32_t tR = 0;

volatile

uint32_t t = 0;

…

int

main(

void

)

{

//

Initialization etc.

…

while

(1)

{

if

(t < tR)

{

GPIOD->

BSRRL

= GPIO_Pin_15;

//

T

o HIGH (for Monitoring with oscilloscope)

// Start of jobs

t = tR;

…

…

// End of jobs

GPIOD->

BSRRH

= GPIO_Pin_15;

//

To

LO

W setzen

(for Monitoring with oscilloscope

)

}

}

}

void

UB_TIMER2_ISR_CallBack

(

void

)

{

…

tR++;

//

Update every millisecond

…

tS++;

if

(tS > 999)

// Send every second ...

{

i =

sprintf

(strBuf,

''tR=%d, t=%d''

, (

int

) tR, (

int

) t);

UART1_SendString(strBuf, i);

tS = 0;

}

}

In most cases of a „crash“ tR will still be updated, but not t.

I also observed on the oscilloscope that in case of a crash PD15 could remain on HIGH level as well as on LOW level, but in most cases it is the LOW level, what indicates for me, that there is no deadlock in the while(1) loop.

Best regards,Torsten

[AvaTar]

Posted on January 20, 2015 at 09:15

How can I check the hard fault handler?

 

For instance by setting a breakpoint there, or add some pin toggle code in this handler to watch by scope.

By a brief review of your issue, I can imagine two reasons for the crash.

First, an incomplete interrupt handling. Because of your ''round-robin'' like interrupt handling, you might get overflow interrupts, which are never cleared (causing infinite interrupt loops).

Second, the stack might be too small. Do you use the FPU in your code ? That may add massively to the interrupt stack size, if the FPU registers are saved/restored.

If you can refrain from using the FPU in interrupt code, enable 'lazy stacking'.

[carl2399]

Posted on January 20, 2015 at 09:23

Hi Torsten,

Nice high speed comms :) although that may be the problem :( I think you should be able to get this working, but I suspect that having the comms receiver inside of the interrupt handler, and comms transmitter outside may be causing some problems. You could try making some FIFOs to handle the comms in both directions, then both transmitter and receiver can be handled in the context of the ISR. I mostly use C++, but the following demonstrates the idea:

extern
''C''
void
USART1_IRQHandler(
void
)
{
if
(U1.UART->SR & USART_FLAG_RXNE) 
// Receiver not empty?
U1.RxToFifo();
if
(U1.UART->SR & USART_FLAG_TXE) 
// Space in transmitter?
U1.TxFromFifo();
}

Just and idea . . . Regards, Carl.

[torsten]

Posted on January 20, 2015 at 11:58

I am using CoIDE 1.7.6. The stack size seems to be defined in „startup_stm32f4xx.c“:

/*----------Stack Configuration-----------------------------------------------*/

#define

STACK_SIZE 0x00001000

/*!< Stack size (in Words) */

Incrementing the stack size has no effect to the fault.

I think the guess of „Argentum“ is most likely:

„... an incomplete interrupt handling. Because of your ''round-robin'' like interrupt handling, you might get overflow interrupts, which are never cleared (causing infinite interrupt loops).“

But how to handle this problem best???

[waclawek.jan]

Posted on January 20, 2015 at 12:14

> void

UB_TIMER2_ISR_CallBack

(

void

)

You call this from the Timer ISR? Then don't

> i =

sprintf

(strBuf,

''tR=%d, t=%d''

, (

int

) tR, (

int

) t);

> UART1_SendString(strBuf, i);

Set only a flag in the

ISR, and then do this in main().

JW

[AvaTar]

Posted on January 20, 2015 at 13:10

My suggestion: keep the interrupt routines as short as possible.

Especially avoid calling routines that involve other interrupts ( as jan suggested in his post - calling UART send function from within will cause trouble).

Check for any possible error/overflow condition in interrupts. Those flag are usually not self-clearing (like RXNE on read).

There is a known issue if clearing the interrupt flag is the very last thing in the handler. However, I expect that not to be your problem, as it would be persistent, not sporadic.

Not sure if you had done this already, but you can guard your interrupt routines with GPIO toggles, and measure the interrupt load. This would also give an indication if you got caught in an infinite interrupt loop.

[torsten]

Posted on January 20, 2015 at 15:07

To find out where in the code the crash occurs I used the following structure:

volatile

uint32_t tR = 0;

volatile

uint32_t t = 0;

//---------------------------------------------

int

main

(

void

)

{

while

(1)

{

execFlag = 0;

if

(t < tR)

{

t = tR;

execFlag = 1;

// Job 1

execFlag = 2;

// Job 2

…

execFlag = 13;

}

}

}

//---------------------------------------------

void

UB_TIMER2_ISR_CallBack

(

void

)

// Every millisecond

{

tR++;

tS++;

if

(tS > 999)

// Every second

{

i =

sprintf

(strBuf,

''tR=%d, t=%d, execFlag=%d''

, (

int

) tR, (

int

) t, (

int

) execFlag);

UART1_SendString(strBuf, i);

tS = 0;

}

}

//---------------------------------------------

Remark: It has no influence to the occurrence of the fault, if „UART1_SendString“ is used in the TIMER2 ISR or not. Besides „UART1_SendString“ isn't directly linked to an interrupt. USART1 TX is realized by polling ...

Here is what the PC receives:

Msg-ID 70: 12 of 12 Bytes transfered!

Msg-ID 70: Pose processed!

Msg-ID 70: 12 of 12 Bytes transfered!

Msg-ID 70: Pose processed!

Msg-ID 70: 12 of 12 Bytes transfered!

Msg-ID 15 with length of 29 Bytes: tR=18000, t=17964, execFlag=0

Msg-ID 15 with length of 29 Bytes: tR=19000, t=17964, execFlag=0

Msg-ID 15 with length of 29 Bytes: tR=20000, t=17964, execFlag=0

Msg-ID 15 with length of 29 Bytes: tR=21000, t=17964, execFlag=0

Msg-ID 15 with length of 29 Bytes: tR=22000, t=17964, execFlag=0

Msg-ID 15 with length of 29 Bytes: tR=23000, t=17964, execFlag=0

stateOpened = 0

Port succesfully closed ...

All Jobs in the “

if

(t < tR)

“-condition seem to be processed in a proper way.

At some time the reentry in the “

if

(t < tR)

“-condition doesn't work any longer and a few instances later the whole while(1) loop is hanging while the TIMER 2 ISR still works proper.

It's crazy, I don't know what's going on here …

[torsten]

Posted on January 20, 2015 at 15:40

In addition:

The ISR

USART1_IRQHandler

(

void

)

is in an infinite loop:

800 nanoseconds active

200 nanoseconds inactive (TIMER 2 ISR !?)

Is it possible to observe and control this interrupt within another, e.g. TIMER 2 ISR???

[AvaTar]

Posted on January 20, 2015 at 16:35

Is it possible to observe and control this interrupt within another, e.g. TIMER 2 ISR???

Not a good idea. Better find what happens in the UART interrupt. How about debugging, i.e. attaching to the MCU when the problem actually occurs ? (Not sure if  your IDE supports this).

BTW, I think it is not a crash, but just an endless loop. The cause might be some kind of race condition, i.e. a certain sequence of interrupts exposing a bug in your UART interrupt handling.