CAN Bus: Massive Noise on the bus when a power converter starts

[KMew]

Hello,

We have been using CANbus as our form of communication for a system with multiple nodes. Each node was individually tested with CAN and all of them work without issue. However, when we started system integration testing, we noticed significant noise on the CANbus channel that caused data to stop. 

To give more details, we are running 2 battery packs, a power electronics DC/AC converters (to create a 120Vac signal) and a central supervisory controller, which are all using STM32 MCUs to communicate to each other via CANbus. We are using an STM32H7AGI6, STM32F401RE, STM32F303RE. When we send a CANbus command to start the DC/AC converter, that is when the noise appears and we cannot communicate with the MCUs until the DC/AC converter is turned off.

We have termination resistors at the end of the bus and I have verified that the 60 Ohm resistance is seen on the bus. I've also attached a copy of our schematic for the CANbus, which all of the nodes use (except that the 120 Ohm resistor is only populated in the two connected at the ends)

I've attached a picture of the waveform too, where the two "messy" segments are when we try to start the DC/AC converter. The picture below is the CANH line, with the probe grounded to the GND pin in the CANbus node.

Can anyone give me some tips to improve this or prevent this from happening?

[LCE]

So better cabling already ("real" twisted pair) improved things?

I would try shielded twisted pair next.

And maybe even this:

use shielded twisted pair cable with at least 2 twisted pairs,
1 twisted pair (CAN+ & CAN-), the other pair for the GND connection, 
and with the cable shield only connected on one side, preferably with the shield not connected on the DCAC side.

[KMew]

Hello Karl,

The DC/AC converter is turned on and off via a CANbus message from another board, but communication is lost while it is running, so I have to manually turn it off (it's powered by a removable battery, so we just remove said battery to stop operation).

[KMew]

Hello LCE,

We are preparing to try shielded, twisted pair right now.

However, we continued testing and noticed that, once we put the unit back into our enclosure, even with the "real" twisted pair, the communication continued to freeze with no improvements from when I first posted the question. So now I'm questioning the twisted pair at all.

I am having a very hard time isolating this problem right now.

[Tesla DeLorean]

>>smaller AWG wiring

Smaller AWG, thicker the wire

[Karl Yamashita]

---

@KMew wrote:

Hello Karl,

The DC/AC converter is turned on and off via a CANbus message from another board, but communication is lost while it is running, so I have to manually turn it off (it's powered by a removable battery, so we just remove said battery to stop operation).

---

You still didn't answer my question if after you powered off the DC/AC converter, does communication commence or do you have to power cycle the other devices to get communication working again? 

Have you tried debugging the devices that stop communication and checked if the TEC counter is greater than 127 or 255? Is ABOM enabled? You may want to check the REC count as well.

[KMew]

Hello Karl,

I apologize, I misunderstood your question.

So we have a removable battery pack that powers this DC/AC. The removable battery pack has a CAN node in it, as does the DC/AC and our supervisory control board. 

Communication is lost when the DC/AC is activated. When the battery is removed to turn off the DC/AC, then we put the battery back in, the battery pack starts communicating with the supervisory control board immediately. The DC/AC and the supervisory control board, however, lose their power (since the battery pack powered them), so the MCU is reset and, of course, restores operation because of it. 

So the only node that is never powered down during this sequence is the battery pack, which *does* start to work again when the DC/AC is "turned off."

Now, a note that I think leads into your second question, the F series MCU's have an "automatic bus off management" configuration in its Init, which I had *disabled* when I first posted this question. At this point, the CAN communication was never restored after the DC/AC turned on. When we enabled the automatic bus off management, the communication was lost *while* the DC/AC was running, but restored *after* the DC/AC was turned off. So this does lead me to believe we are indeed seeing an error in the battery pack node. The only thing that has stopped me from confirming that is that, when the DC/AC is on, the debugger loses communication with the MCU. I'm unsure how to solve that problem either. 

[Karl Yamashita]

---

The only thing that has stopped me from confirming that is that, when the DC/AC is on, the debugger loses communication with the MCU. I'm unsure how to solve that problem either. 

---

Is it possible that the device is going into BOR detection when the DC/AC converter is on?  

[KMew]

So my reasoning may be wrong, but if a BOR was occurring, would the MCU go into an error handler or hard fault or something similar that would stop all functions from working?
The reason I ask is that we are using an LCD display (LTDC interface) on this MCU and the display continues to update, even when the DC/AC is on. So if it was in a hard fault or error handler, then that would not work.

Is there a way to confirm if a BOR occurred?

[Karl Yamashita]

Get the data sheet for your particular device and read up on the Reset/Clock section. The RCC_CSR register will hold the reset status. If you have an LED that you can use for a status, then use it for testing the reset status.

So upon POR, read the RCC_CSR status register and test for the reset bits, particularly the BORRSTF bit . If any of them besides the SFTRSTF bit is set, then turn on the LED. 

[KMew]

So a general update of my testing. We still haven't solved the problem, but we have identified something curious. 

Our supervisory control board (which uses the same CAN hardware as the original post) is powered by the removable battery pack, which is the same one that powers the DC/AC. We tried a test where the supervisory control board is powered by a separate DC power supply, but kept everything else the same (DC/AC, battery pack, supervisory controller all connected via CAN, DC/AC powered by the battery, twisted pair cable, ground wire still connected) and the communication is never lost in this scenario. 

Does this help explain it at all? Does anyone have any ideas based on that?