"Homologation issues" sounds like a late stage when you don't want to redesign things substantially so you don't want to hear this, but you're not likely to fix this just by throwing some code at it. Crosstalk between digital and analog stems from the design details - layout, decoupling details, decoupling of digital and analog power supply and ground domains, high-speed traces length and relative position to other traces especially the analog, high speed traces termination, and quite often , improper return/grounding of high speed signal. To a certain amount it's inevitable using a heavily-digitalized mixed-signal chip, even inside it to a certain amount, especially when it's primarily digital and the analog part is more-less a compromise.
What the compensation cell does is no miracle - it changes the output drive (probably switches on more or less transistors in parallel) as the supply voltage and chip temperature vary.
So, review your PCB layout, especially grounding; try to add flying wires to it returning directly the high-speed traces where the return on board appears to be not straight enough; consider decoupling of analog ground/supply/reference/any-other-analog-related-pin (cut traces+air-mount chokes/filters], add terminations to high-speed traces, change OSPEEDR to lower settings (yes it may still work, the "50MHz" doesn't mean the pin stop working below 50MHz, it just changes the drive/slew rate and it depends on the loading capacitance and traces length/layout/termination and the PHY, whether this causes functional issues or not).
And, this is the last thing you want to hear, but you might ultimately need resort to using an external DAC.
JW
PS. Change your username to a normal nick.
