PROBLEM SOLVED
Appears to be an issue of the Nucleo-144 demo-board itself (NUCLEO-H743ZI specifically), NOT AN STM32H7 FAULT!
HOW TO FIX?
- Desolder/remove L2 ferrite bead to disable Ethernet PHY chip (see Nucleo-144 schematic, it is located close to that IC nearby Ethernet port).
- Or use NUCLEO board without Ethernet hardware (OOOPS ... not available for H7)
- Alternatively you may remove SB2, but that will disable USB-port circuitry as well
WHY?
- This chip designated to PAM-modulate and amplify the Ethernet signal before feeding it into long wires of CAT5/6 cable.
- Apparently it does not have any low-power/standby mode when not in use.
- So even if you not use it - its clock (25MHz) injected into board circuitry (my guess - over the ground). Observed at ADC with 5mV peak-to-peak swing, basically reducing ADC resolution to 10-bits!
Can you have Ethernet and clean ADC capture at the same time?
- Somewhat possible with oversampling or moving-average filter.
- 16-samples needed to get 12 effective bits of resolution, but limits your acquisition bandwidth to around 100kHz
And here is my story of how did I figured it out:
First of all - sorry for a delay since my last reply. To approach this problem I had to wait for a good weekend (free of other tasks and events) since I decided to go "full ballistic" to eliminate any guess work:
- The Nucleo-144 board has been put into 3D-printed enclosure which was painted with MG Chemicals Super Shield conductive paint to block any external EMI sources.
- I've wired ADC inputs to 1.5V AAA battery. The wires were as short as I could get and battery did fit into shielded enclosure of the Nucleo board.
- I've used external power supply 8..12V.
- Also the ADC analog block (AVDD) was powered form the same LDO as the MCU itself. So MCU digital blocks can interfere with the ADC. It is always recommeded to use separate supply and even dedicated Vref for ADC. So I've soldered a separate circuit with 3.3V LDO isolated from the main external supply by capacitance multiplier (see EEVBolg youtube video on what is that: https://youtu.be/wopmEyZKnYo ).
- So SB12 (see Nucleo schematic) was de-soldered to connect this new clean-output LDO to AVDD. SBL162 was de-soldered as well and AGND was wired directly to the LDO ground pin. In other words AVDD and AGND of the nucleo board were now wired directly to its dedicated LDO.
- Test DC-signal source , the negative tab of 1.5V AAA battery was wired to the LDO ground. Eliminating ground loops for signal source.
- Add more bypass-capacitors for analog supply pins (soldered few more MLCCs on top of C34)
And then I've got the same high-speed (2 MSPS) raw data captures again:

as you see it still shows this aliasing affect, but it looks now way cleaner and more "pronounced" than it was before:

If using moving average filter using 2 samples, it improves slightly:

Trying to run ADC at its maximum sampling speed (5MSPS) did not help to figure out interference frequency, so it is higher than that.
As I've already mentioned in my previous post - you can not identify that this is aliasing of higher-frequency interference IF you do slow sampling. As in this 13kSPS example it just looks like Gaussian noise, because clock drift (PLL jitter) introduces a lot of phase noise to notice any aliasing patterns of 13kHz vs 25MHz interference.

For these who don't now:
ADC aliasing happens when there is higher frequency at your ADC input than the Nyquist frequency (more than 1/2 of you sampling speed). You can not work with this kind of acquired data if you want to do any decent signal processing on it. In all good designs ADC inputs are guarded with good low-pass filters. Here is good example from the web (https://e2e.ti.com/blogs_/archives/b/precisionhub/archive/2015/09/04/aliasing-in-adcs-not-all-signals-are-what-they-appear-to-be)

As you see 6 and 8 Hz signals can not be reproduced properly when sampling at 7Hz, you see that as 1Hz signal on ADC output.
In our case - the parasitic input was way beyond my Nyquist frequency of 1Mhz. If we zoom-in time scale of the first picture and plot it in MATLAB:

The parasitic signal is apparently at 5mV peak to peak range (100 LSB out of 65536 for 3.3V full range).
Since all external sources were eliminated by shielding and ADC is powered by dedicated clean LDO there are few reasons left:
- Internal issue in STM32H7 - highly unlikely. In datasheet we have SINAID specified at 82dB for single-ended input. If my calculations are correct - it should be equivalent of 5LSB RMS-noise. In other words - if measuring a perfect DC signal (from AAA-battery) 68% of samples should be with-in 10LSB range, and 99.7% for Gaussian noise should be with-in 30LSB peak-to-peak range.
- Interfering signal source is some other component on the Nucleo-144 board, could be ST-Link as well getting something from USB-grouding.
To figure this out you need a spectrum analyser, LNA amplifier and H-field probe (https://youtu.be/crs_QLuUTyQ)
You can go cheaper way and get an SDR radio USB dongle with self-made H-field probe (https://www.stupid-projects.com/emc-probe-using-rtl-sdr)
I've found and ordered an MSi.SDR that has a perfect range for 10kHz to 2GHz for MCU board purposes.
Without having a patience to wait for SDR and semi-rigid coax delivery I've used an oscilloscope probe which came tip-to-BNC adapter. Using another female-to-feamale BNC-connector, piece of copper wire, aluminum foil and electric tape I've got an E/H-probe in like 20 min:


With this contraption I was simply observing waveform on the scope and sniffed the interference source when I've stick it into Ethernet PHY IC package. I was lucky it was powerful enough to get me 5mV spikes on the scope without the need of any LNA.
Here is the raw sampling after I've thrown away an L2 ferrite bead. It should actually serve as a filter connecting a +3V3V_PER peripherals supply line to Ethernet PHY circuitry, but apparently interference is so strong that it propagates through the ground, or EMI into the wires connecting my DC-signal source to ADC input(s).
Here is the RAW capture of DC-signal source (AAA battery) after the hack:

And here its overlay (red) with the RAW capture before disabling Ethernet PHY (blue):

As you see - I've got it all well within spec, major half of samples are somewhat in 10LSB peak to peak range, while all of them within 30LSB.