A count of 1 or 2 could just be a rounding error if you only set and/or display the time to the nearest second. For example when you set the clock the time is 12:57:03.77, but your setting rounds the time down to whole seconds i.e. 12:57:03 - so it starts 0.77 seconds slow.
One question that should be thought about is how your measured performance compares with the specification of your crystal. You don't say what your crystal is, or how you chose those capacitors.
Please confirm whether you are using a low-speed-external (LSE) or high-speed-external (HSE) crystal as the frequency source for your RT.
Looking at one of my regular component suppliers (Farnell in UK), I see 32768 Hz crystals come with tolerances between 10 ppm and 50 ppm (parts per million).
If I assume "a few hours" is 4 hours, then that amounts to 1-2 seconds in 4 x 3600 = 14400 seconds, or 70 - 139 ppm. So even with the worst crystals, that is out-of-spec. But given my mention of rounding errors if things are only set and measured to the nearest second, you will need to run your test for much longer to be sure your oscillator is running more slowly that expected.
How did you decide on 18 pF capacitors?
Looking at AN2867, I see ST suggest crystals designed for load capacitances between 6 and 12.5 pF. The external capacitors you use need to take into account stray capacitance and the pin-capacitance of the TSM32. ST's example suggests stray being about 5 pF. If I assume your crystal is designed for 12.5 pF, and take off the 5 pF stray, that means your two external capacitors in series need to provide 12.5 - 5 = 7.5 pF load. If you have equal-value capacitors, that would suggest 15 pF each as the right value.
Hope this helps,
Danish
