There was an manufacturing defect in my board. The D and R pins were shorted while the MCU was soldered IG... Anyways, I checked with the other board which did not have an short and it worked very well.
I don't see anything obvious. I don't have much experience with the STM32 CAN interface, although my company uses all sorts of CAN devices.
Perhaps you can try a lower baudrate, and play with the bit timing parameters (what the CAN spec calls "quanta"). The latter e.g. define at what point during a signal the IP samples the ACK.
There was an manufacturing defect in my board. The D and R pins were shorted while the MCU was soldered IG... Anyways, I checked with the other board which did not have an short and it worked very well.