> the temperature will rise(~50 degrees)
C or F degrees? Absolute or rise above the ambient temperature? Anyway even 50C with an ambient temperature 25C seems to be too much.
Double check the RJ45 socket pinout. Double check the PHY pinout, wiring, PCB routing, heat transfer design on the PCB. Check the solder quality, resolder the same chip, replace the chip.
> NRST is necessary to set IO in stm32 MX or not?
Either you just connect it to the NRST of MCU, or connect it to some GPIO and do a reset with a software. Unless you have some special requirement of PHY chip NOT being reset, when the MCU is reset, connecting PHY NRST to MCU NRST is easier to manage and does not require an additional GPIO pin.