FDCAN receive errors with STM32G491 on long run.

[trojanobelix]

Hello,

We are experiencing an intermittent CAN-FD issue in our system and would appreciate any advice from the community.

 

We have developed seven new I/O modules based on the STM32G491VETx with an external 20 MHz crystal.
All have the same CAN configuration using ISO1042BDWVR.

 

The CANopenNode stack with the CANopenNodeSTM32 driver is implemented on all electronic units.

See https://github.com/CANopenNode/CanOpenSTM32 

 

The CAN bus operates at 250 and 500 kbit. All electronic units occasionally experience the problem that the NodeGuarding or heartbeat (set to 3s!!!) stops responding after 24 hours to 5 days. SDOs do not respond either, whilst TPDOs continue to operate.

 

We have set up over 10 networks, each with 4 different nodes (all G491), and all exhibit the same behaviour sporadically. We can rule out hardware faults.

The electronic runs for about 48 hours with no Problem on 500k an low bus load (about 2%, Nodeguarding and some TPDO/RPDO).

Suddenly FIXFDCAN RF0 FIFO FULL and RX NEW MESSAGE bit but NO CALLBACK RX
00:39:13:960| ERR: RX NEW MESSAGE: NO CALLBACK RX -> FIXERR: RX NEW MESSAGE: NO CALLBACK RX -> FIXERR: RX NEW MESSAGE: NO CALLBACK RX -> FIXFDCAN RF0 FIFO FULL
00:39:14:961| FDCAN RF0 FIFO FULL

This is in main loop 1s bevor the Nodeguarding (3000 ms runout)
Try to Clear IR Errors ELO, BO,  EP,  EW,  PEA, PED, ARA, MRAF, WDI, TOO, RF0L,RF1L,

00:39:15:477| NMT state; CANnormal; CANerrorStatus; RXF0 fill; RXF0 lost; RXF0 get_idx; RXF0 put_idx; FDCAN IE;   FDCAN ILE;  FDCAN IR;   FDCAN PSR;  FDCAN ECR;  FIFO FULL
00:39:15:489| 5;         1;         0x0000;         0;         0;         2;            2;            0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:503| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:515| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:529| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:542| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:554| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:567| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:581| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:594| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:607| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:621| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:634| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:648| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:661| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:675| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:688| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:703| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:717| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:732| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:747| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:762| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:777| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:792| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:807| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:822| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:836| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:850| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:865| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:880| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:895| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:910| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:925| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:940| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:955| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:971| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:15:974| FDCAN RF0 FIFO FULL
00:39:15:988| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 1; 1; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 0
00:39:16:540| BO; EP; WARN; TEC; REC; LEC; Err; ACK; BIT; CRC; STF; FRM; CO_ErrorRegister; FDCAM!.IR 
00:39:16:548| 0;  0;  0;    0;      0;   0;   2;     2;   0;   0;   0;   0;   0x10;             0x00000000
07:18:26:325| ERR: RX NEW MESSAGE: NO CALLBACK RX -> FIXBO; EP; WARN; TEC; REC; LEC; Err total; ACK; BIT; CRC; STF; FRM; CO_ErrorRegister; FDCAM!.IR 
07:18:26:333| 0; 0; 0; 0; 0; 0; 2; 2; 0; 0; 0; 0; 0x10; 0x00000000
07:18:27:355| BO; EP; WARN; TEC; REC; LEC; Err total; ACK; BIT; CRC; STF; FRM; CO_ErrorRegister; FDCAM!.IR 
07:18:27:363| 0; 0; 0; 0; 0; 0; 2; 2; 0; 0; 0; 0; 0x10; 0x00000000
 

Our interpretation:

What definitely does NOT happen: The CAN bus itself has no problem:
CANnormal      = 1
CANerrorStatus = 0x0000
PSR = 0x0000070F
ECR = 0x00000000
BO   = 0
EP   = 0
WARN = 0
TEC = 0
REC = 0

So:
✅ no bus-off
✅ no passive error
✅ no warning error
✅ no ACK errors on the bus
✅ no bit, CRC, stuff or form errors
The CAN controller considers the bus to be completely healthy.

What happens? 
Firstly: ERR: RX NEW MESSAGE: NO CALLBACK RX, multiple times.
This means: RF0N = set but RX callback is not running
Then: FDCAN RF0 FIFO FULL, shoud never happen, due to read out FIFO completly  in RX callback. 
Sequence: 
    Message arrives
    ↓ RF0N is set
    ↓ ISR is not running
    ↓ FIFO is not emptied
    ↓ further messages
    ↓ FIFO full
    ↓ RF0F

What’s particularly interesting is that just 500 ms later in the log:
RXF0 fill = 0
RXF0 lost = 0
get_idx   = 2
put_idx   = 2
and: IR = 0

This means:
FIFO empty
no lost messages
no pending interrupt
So the recovery apparently worked.

FGI == FPI means: FIFO empty
So the FIFO is not stuck.

Even more importantly: Only RF0F but never RF0L
i.e. FIFO FULL but no MESSAGE LOST: The fault was detected before any telegrams were actually lost.

The error counters
Later:
Err total = 2
ACK       = 2
so:
2 ACK errors
but at the same time:
TEC = 0 
REC = 0
BO = 0 
EP = 0
These are presumably historical CANopen counters.
They are not related to the RF0N problem.

The most striking thing. The same error reappears:
00:39:13...    ERR: RX NEW MESSAGE: NO CALLBACK RX
and later:
07:18:26    ERR: RX NEW MESSAGE: NO CALLBACK RX
Almost 7 hours later.

So, not a sporadic bus error, not a start-up problem, not a one-off overflow, but a recurring condition.

Assessment 
Very high probability:
The FDCAN is receiving correctly,
RF0N is set, the message is in the FIFO, but the interrupt path is not being processed at times:
ISR is not running

Probability: medium:
Interrupts are being temporarily blocked somewhere:
C__disable_irq() or CBASEPRI ≠ 0 and not reset properly.
That is precisely why your new log shows:

Probability: low:
Fault in the CAN bus.
There are no signs of this: TEC, REC, BO, EP, WARN, CRC, BIT and FORM are all 0.

From the current log, it looks to me as though it is not the CAN controller that is hanging, but rather that the FDCAN interrupt is occasionally not being processed, even though messages are being received correctly. 

 

 Here ist the HAL_FDCAN_RxFifo0Callback code, it puts the message in a SW FIFO :

volatile uint32_t isrFifo0Cnt = 0;
void FDCAN_GetMessageFIFO0(void){
	isrFifo0Cnt++;
	FDCAN_HandleTypeDef *hfdcan =
		((CANopenNodeSTM32*) CANModule_local->CANptr)->CANHandle;

    /* ✅ HW FIFO komplett leerziehen */
    while (HAL_FDCAN_GetRxFifoFillLevel(hfdcan, FDCAN_RX_FIFO0) > 0)
    {
        FDCAN_RxHeaderTypeDef rx_hdr;
        uint32_t tmpData[2];              // ✅ aligned buffer
        CO_CANrxMsg_t rcvMsg = {0};       // ✅ wichtig: jedes Frame neu!

        /* ✅ Nachricht lesen (immer FIFO0!) */
        if (HAL_FDCAN_GetRxMessage(hfdcan, FDCAN_RX_FIFO0, &rx_hdr, (uint8_t*)tmpData) != HAL_OK){
            return;
        }

        /* ✅ Identifier korrekt (11-bit!) */
        rcvMsg.ident =
            (rx_hdr.Identifier & CANID_MASK) |
            (rx_hdr.RxFrameType == FDCAN_REMOTE_FRAME ? FLAG_RTR : 0);

	    /* DLC */
	    switch (rx_hdr.DataLength) {
	    case FDCAN_DLC_BYTES_0: rcvMsg.dlc = 0; break;
	    case FDCAN_DLC_BYTES_1: rcvMsg.dlc = 1; break;
	    case FDCAN_DLC_BYTES_2: rcvMsg.dlc = 2; break;
	    case FDCAN_DLC_BYTES_3: rcvMsg.dlc = 3; break;
	    case FDCAN_DLC_BYTES_4: rcvMsg.dlc = 4; break;
	    case FDCAN_DLC_BYTES_5: rcvMsg.dlc = 5; break;
	    case FDCAN_DLC_BYTES_6: rcvMsg.dlc = 6; break;
	    case FDCAN_DLC_BYTES_7: rcvMsg.dlc = 7; break;
	    case FDCAN_DLC_BYTES_8: rcvMsg.dlc = 8; break;
	    default: rcvMsg.dlc = 0; break;
	    }

        /* ✅ DATA IMMER ALS LETZTES kopieren! */
        memcpy(rcvMsg.data, tmpData, 8);

        /* ✅ nur gültige Frames reinlassen */
        if ((rcvMsg.ident & ~(CANID_MASK | FLAG_RTR)) == 0 && rcvMsg.dlc <= 8){
            rb_push(&rcvMsg);
        }
    }
}

Hardware 

 Clocks

We’re just about to start mass production and urgently need help. Any idea what might be causing this?
What tests could we carry out to get closer to identifying the cause?

[mƎALLEm]

Hello,

Unfortunately the post is too verbose and the log is not readable. But will try to provide some hints

1- You didn’t show the FDCAN clock from the CubeMx screenshot. What FDCAN clock you have set? There is a constraint on the FDCAN clock setting. Please refer to this knowledge base article: FAQ: Fixing STM32 FDCAN communication disruptions - APB bus, kernel, and time quanta clocks

This is the constraint:

And from the reference manual: RM0440 Rev 9 /Page 1957:

2- Otherwise, create a basic CAN example where nodes communicate together i.e. without CANOpen stack, activate all possible errors interrupts available and check if you face an issue/error. This will at least verify if you have a hardware issue or software issue.

3- I don’t see an issue in the schematic you shared.

[mƎALLEm]

3- I don’t see an issue in the schematic you shared.

Forget to ask about the 2x terminating resistors of 120 ohm. I don’t see them in the schematic:

Did you put them directly on CAN the bus

[trojanobelix]

We set up the bus with multiple electronic units via network cables of at least Category 5e and terminate the network using a 120-ohm resistor encapsulated within an RJ45 plug. X21 / X22 are the RJ45 sockets for Can IN/OUT  

NMT Master (PLC, terminated) -------- -  X21 1st electronic unit X22 -------- X21 2nd electronic unit X22 -------- (...) ---------  X21 nth electronic unit X22  (terminator)

[trojanobelix]

 

 

Thank you for your feedback.

 FDCAN is 16MHz as the others.  

As regards the restriction on the FDCAN clock setting, it seems to me that even within ST there isn’t complete agreement on this:
The article clearly states: “The FDCAN requires that the CAN time quanta clock is always below the APB clock.” So it must be (fdcan_tq_ck < fdcan_pclk)

Mr Bowman32 says in the comment: “The FDCAN specification has been updated. The FDCAN requires that the CAN time quanta clock is always below or equal to the APB clock (fdcan_tq_ck ≤ fdcan_pclk). For reference, please see RM0517 Rev 1, section ‘Bit timing’.”  

He refers to the Reference Manual for STM32H5E4/5F4 and STM32H5E5/5F5” 

The STM32G491 Reference Manual RM0440 states: “The FDCAN requires that the CAN time quanta clock is always below or equal to the APB clock (fdcan_tq_ck < fdcan_pclk).   

https://community.st.com/stm32-mcus-60/faq-fixing-stm32-fdcan-communication-disruptions-apb-bus-kernel-and-time-quanta-clocks-141123

All in all, this raises more questions than it answers.

[mƎALLEm]

The article clearly states: “The FDCAN requires that the CAN time quanta clock is always below the APB clock.” So it must be (fdcan_tq_ck < fdcan_pclk)

Mr Bowman32 says in the comment: “The FDCAN specification has been updated. The FDCAN requires that the CAN time quanta clock is always below or equal to the APB clock (fdcan_tq_ck ≤ fdcan_pclk). For reference, please see RM0517 Rev 1, section ‘Bit timing’.”  

He refers to the Reference Manual for STM32H5E4/5F4 and STM32H5E5/5F5” 

The STM32G491 Reference Manual RM0440 states: “The FDCAN requires that the CAN time quanta clock is always below or equal to the APB clock (fdcan_tq_ck < fdcan_pclk).   

https://community.st.com/stm32-mcus-60/faq-fixing-stm32-fdcan-communication-disruptions-apb-bus-kernel-and-time-quanta-clocks-141123

 

I’ve already noticed that. I’ve already escalated that for investigation and fix (internal ticket for follow-up: CDM0064161).

But you didn’t state on my second point. Did you try to do a test with a very simple project? so you can validate the hardware?

[trojanobelix]

The interesting question is: Which is correct? Can the FDCAN frequency be the same as the APB clock, or is it worth reducing it? 

 

One more point regarding the documentation:
The STM32G491 Reference Manual RM0440 states: “The FDCAN requires that the CAN time quanta clock is always less than or equal to the APB clock (fdcan_tq_ck < fdcan_pclk).  

This statement does not match the formula – it’s easy to overlook this!

[trojanobelix]

I will set up a simplified project to demonstrate the fault. However, I will need to have 30 further electronic units, including control systems (PLCs), set up in order to reproduce the fault within a reasonable timeframe.
This may take a few days.

Regarding the probability of occurrence:
When working from home, a load test is always run at the weekend with a 30 per cent bus load and approximately one message per second without any errors. I’ve calculated this to be around 250,000,000 error-free frames.  That is why we have set up 36 control systems in parallel in our test laboratory, so as to achieve a rate of approximately one failure per day.

 

BTW: Is there something wrong with your servers? I’m having trouble posting here without getting an error message. (502, 503, 504...)

[mƎALLEm]

“BTW: Is there something wrong with your servers? I’m having trouble posting here without getting an error message. (502, 503, 504...)”

Indeed there was an issue. I think it was solved..