I will strengthen the requirement on PA11: It must be logic high when you enter the bootloader. On two different PCBs with STM32H743VIT6 rev V silicon, I can control whether the USART3 bootloader responds by keeping port A in reset and shorting PA11 to 3.3V (success) or 0.0V (no response from bootloader). PA12 is unconnected on one of the boards so I haven't spent as much time examining that pin.
That said, PA11, and PB15 aren't the only pins which will prevent the USART3 bootloader from working, but I don't know what the others are. I've had to deal with a nondeterministic problem with the bootloader while bisecting for deterministic failures from pin state, so debugging is rather slow. It literally ended up being faster to write my own ST-compatible bootloader.
I'll also note that the problem(s) with the USART3 bootloader appear to be more fundamental. I modified the startup vector table of my BIOS to jump directly to the address pointed at by 0x1FF09804:
const void *const kStartupInterruptVectors[] __attribute__((section(".startup_vectors"))) = {
[kInterruptInitialSpValue] = (void *)0x24003AF8, // Value taken from 0x1FF09800.
[kInterruptReset] = (void *)0x1FF09ABD, // Value taken from 0x1FF09804.
With this change, no user code is executed; the bootloader runs immediately (with the caveat that the interrupt vector table offset is wrong). As before, holding PA11 high enables bootloading and PA11 low prevents the bootloader from responding. ...on the custom PCBs that I'm developing on. If I use either a NUCLEO-H743ZI (rev Y silicon) or a NUCLEO-H743ZI2 (rev V silicon), with a POF transceiver hooked up to PB10 and PB11 for USART3, bootloading simply doesn't work after powering up until I reflash the part with the ST-LINK (at which point the USART3 bootloader always works). From this, I'm inclined to conclude that the USART3 bootloader is just broken.
