FDCAN receive errors with STM32G491 on long run.

Question

Hello,We are experiencing an intermittent CAN-FD issue in our system and would appreciate any advice from the community. We have developed seven new I/O modules based on the STM32G491VETx with an external 20 MHz crystal.All have the same CAN configuration using ISO1042BDWVR. The CANopenNode stack with the CANopenNodeSTM32 driver is implemented on all electronic units.See https://github.com/CANopenNode/CanOpenSTM32  The CAN bus operates at 250 and 500 kbit. All electronic units occasionally experience the problem that the NodeGuarding or heartbeat (set to 3s!!!) stops responding after 24 hours to 5 days. SDOs do not respond either, whilst TPDOs continue to operate. We have set up over 10 networks, each with 4 different nodes (all G491), and all exhibit the same behaviour sporadically. We can rule out hardware faults.The electronic runs for about 48 hours with no Problem on 500k an low bus load (about 2%, Nodeguarding and some TPDO/RPDO).Suddenly FIXFDCAN RF0 FIFO FULL and RX NEW MESSAGE bit but NO CALLBACK RX00:39:13:960| ERR: RX NEW MESSAGE: NO CALLBACK RX -> FIXERR: RX NEW MESSAGE: NO CALLBACK RX -> FIXERR: RX NEW MESSAGE: NO CALLBACK RX -> FIXFDCAN RF0 FIFO FULL00:39:14:961| FDCAN RF0 FIFO FULLThis is in main loop 1s bevor the Nodeguarding (3000 ms runout)Try to Clear IR Errors ELO, BO,  EP,  EW,  PEA, PED, ARA, MRAF, WDI, TOO, RF0L,RF1L,00:39:15:477| NMT state; CANnormal; CANerrorStatus; RXF0 fill; RXF0 lost; RXF0 get_idx; RXF0 put_idx; FDCAN IE;   FDCAN ILE;  FDCAN IR;   FDCAN PSR;  FDCAN ECR;  FIFO FULL00:39:15:489| 5;         1;         0x0000;         0;         0;         2;            2;            0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:503| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:515| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:529| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:542| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:554| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:567| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:581| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:594| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:607| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:621| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:634| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:648| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:661| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:675| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:688| 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:703| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:717| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:732| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:747| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:762| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:777| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:792| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:807| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:822| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:836| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:850| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:865| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:880| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:895| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:910| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:925| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:940| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:955| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:971| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 2; 2; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:15:974| FDCAN RF0 FIFO FULL00:39:15:988| ERRORS CLEARD! 5; 1; 0x0000; 0; 0; 1; 1; 0x006E0289; 0x00000003; 0x00000000; 0x0000070F; 0x00000000; 000:39:16:540| BO; EP; WARN; TEC; REC; LEC; Err; ACK; BIT; CRC; STF; FRM; CO_ErrorRegister; FDCAM!.IR 00:39:16:548| 0;  0;  0;    0;      0;   0;   2;     2;   0;   0;   0;   0;   0x10;             0x0000000007:18:26:325| ERR: RX NEW MESSAGE: NO CALLBACK RX -> FIXBO; EP; WARN; TEC; REC; LEC; Err total; ACK; BIT; CRC; STF; FRM; CO_ErrorRegister; FDCAM!.IR 07:18:26:333| 0; 0; 0; 0; 0; 0; 2; 2; 0; 0; 0; 0; 0x10; 0x0000000007:18:27:355| BO; EP; WARN; TEC; REC; LEC; Err total; ACK; BIT; CRC; STF; FRM; CO_ErrorRegister; FDCAM!.IR 07:18:27:363| 0; 0; 0; 0; 0; 0; 2; 2; 0; 0; 0; 0; 0x10; 0x00000000 Our interpretation:What definitely does NOT happen: The CAN bus itself has no problem:CANnormal      = 1CANerrorStatus = 0x0000PSR = 0x0000070FECR = 0x00000000BO   = 0EP   = 0WARN = 0TEC = 0REC = 0So:✅ no bus-off✅ no passive error✅ no warning error✅ no ACK errors on the bus✅ no bit, CRC, stuff or form errorsThe CAN controller considers the bus to be completely healthy.What happens? Firstly: ERR: RX NEW MESSAGE: NO CALLBACK RX, multiple times.This means: RF0N = set but RX callback is not runningThen: FDCAN RF0 FIFO FULL, shoud never happen, due to read out FIFO completly  in RX callback. Sequence:     Message arrives    ↓ RF0N is set    ↓ ISR is not running    ↓ FIFO is not emptied    ↓ further messages    ↓ FIFO full    ↓ RF0FWhat’s particularly interesting is that just 500 ms later in the log:RXF0 fill = 0RXF0 lost = 0get_idx   = 2put_idx   = 2and: IR = 0This means:FIFO emptyno lost messagesno pending interruptSo the recovery apparently worked.FGI == FPI means: FIFO emptySo the FIFO is not stuck.Even more importantly: Only RF0F but never RF0Li.e. FIFO FULL but no MESSAGE LOST: The fault was detected before any telegrams were actually lost.The error countersLater:Err total = 2ACK       = 2so:2 ACK errorsbut at the same time:TEC = 0 REC = 0BO = 0 EP = 0These are presumably historical CANopen counters.They are not related to the RF0N problem.The most striking thing. The same error reappears:00:39:13...    ERR: RX NEW MESSAGE: NO CALLBACK RXand later:07:18:26    ERR: RX NEW MESSAGE: NO CALLBACK RXAlmost 7 hours later.So, not a sporadic bus error, not a start-up problem, not a one-off overflow, but a recurring condition.Assessment Very high probability:The FDCAN is receiving correctly,RF0N is set, the message is in the FIFO, but the interrupt path is not being processed at times:ISR is not runningProbability: medium:Interrupts are being temporarily blocked somewhere:C__disable_irq() or CBASEPRI ≠ 0 and not reset properly.That is precisely why your new log shows:Probability: low:Fault in the CAN bus.There are no signs of this: TEC, REC, BO, EP, WARN, CRC, BIT and FORM are all 0.From the current log, it looks to me as though it is not the CAN controller that is hanging, but rather that the FDCAN interrupt is occasionally not being processed, even though messages are being received correctly.   Here ist the HAL_FDCAN_RxFifo0Callback code, it puts the message in a SW FIFO :volatile uint32_t isrFifo0Cnt = 0;void FDCAN_GetMessageFIFO0(void){	isrFifo0Cnt++;	FDCAN_HandleTypeDef *hfdcan =		((CANopenNodeSTM32*) CANModule_local->CANptr)->CANHandle;    /* ✅ HW FIFO komplett leerziehen */    while (HAL_FDCAN_GetRxFifoFillLevel(hfdcan, FDCAN_RX_FIFO0) > 0)    {        FDCAN_RxHeaderTypeDef rx_hdr;        uint32_t tmpData[2];              // ✅ aligned buffer        CO_CANrxMsg_t rcvMsg = {0};       // ✅ wichtig: jedes Frame neu!        /* ✅ Nachricht lesen (immer FIFO0!) */        if (HAL_FDCAN_GetRxMessage(hfdcan, FDCAN_RX_FIFO0, &rx_hdr, (uint8_t*)tmpData) != HAL_OK){            return;        }        /* ✅ Identifier korrekt (11-bit!) */        rcvMsg.ident =            (rx_hdr.Identifier & CANID_MASK) |            (rx_hdr.RxFrameType == FDCAN_REMOTE_FRAME ? FLAG_RTR : 0);	    /* DLC */	    switch (rx_hdr.DataLength) {	    case FDCAN_DLC_BYTES_0: rcvMsg.dlc = 0; break;	    case FDCAN_DLC_BYTES_1: rcvMsg.dlc = 1; break;	    case FDCAN_DLC_BYTES_2: rcvMsg.dlc = 2; break;	    case FDCAN_DLC_BYTES_3: rcvMsg.dlc = 3; break;	    case FDCAN_DLC_BYTES_4: rcvMsg.dlc = 4; break;	    case FDCAN_DLC_BYTES_5: rcvMsg.dlc = 5; break;	    case FDCAN_DLC_BYTES_6: rcvMsg.dlc = 6; break;	    case FDCAN_DLC_BYTES_7: rcvMsg.dlc = 7; break;	    case FDCAN_DLC_BYTES_8: rcvMsg.dlc = 8; break;	    default: rcvMsg.dlc = 0; break;	    }        /* ✅ DATA IMMER ALS LETZTES kopieren! */        memcpy(rcvMsg.data, tmpData, 8);        /* ✅ nur gültige Frames reinlassen */        if ((rcvMsg.ident & ~(CANID_MASK | FLAG_RTR)) == 0 && rcvMsg.dlc <= 8){            rb_push(&rcvMsg);        }    }}Hardware  ClocksWe’re just about to start mass production and urgently need help. Any idea what might be causing this?What tests could we carry out to get closer to identifying the cause?

mƎALLEm · Accepted Answer

We don’t list all the unexpected behaviors. Unexpected behavior = any behavior is not inline what we expect from the periphjeral.

As suggested previously, you need to do the simple test so we can progress in the case, otherwise, it’s not possible to go further.

mƎALLEm · Answer

Hello,Unfortunately the post is too verbose and the log is not readable. But will try to provide some hints1- You didn’t show the FDCAN clock from the CubeMx screenshot. What FDCAN clock you have set?There is a constraint on the FDCAN clock setting. Please refer to this knowledge base article:FAQ: Fixing STM32 FDCAN communication disruptions - APB bus, kernel, and time quanta clocksThis is the constraint:And from the reference manual:RM0440 Rev 9 /Page 1957:2- Otherwise, create a basic CAN example wherenodes communicate together i.e. without CANOpen stack, activate all possible errors interrupts available and check if you face an issue/error. This will at least verify if you have a hardware issue or software issue.3- I don’t see an issue in the schematic you shared.

Sign up

Login with SSO

Login to the community

Login with SSO