I changed the clock source from external resonator (HSE) to internal 64Mhz oscillator (HSI) and the VGA siganal is maybe not perfect, but much better than before
I have had the same problem, Nucleo H743ZI2 revision V same exact issue, it cleared up a bit when using an HSI (had to mod the dev board to do it...) HOWEVER i got a purely amazing VGA output when I set ALL the LTDC pin outs to LOW (not high speed) the VSYNC, HSYNC, DE, and CLK set these to VERY HIGH...... for me it fixed the issue right up!
I did notice before I DID this that the changes. the more detail on the screen the worse the noise and stability got. NOW my setup is RGB565 output pins, these pins are set to speed: LOW
the VSYNC, HSYNC, DE, and CLK speed: VERY HIGH.
I hope this helps you.
note my VGA cable is badly shielded and i did not use any resistors on the VSYNC, HSYNC. (i dont use DE or CLK)