Seems to me you keep asking this question in different ways, if I'm wrong and it is someone else, I apologize. What is the reasoning behind not using heat sinks? They are needed for something of this number of amps. You can't get around this, you either use an add on heat sink or have the same size area of exposed copper on the board, it all comes down to having the exposed area of metal to conduct the heat from the mosfet. Something like this mounts to the mosfet and gives heat removal from the mosfet.
Is this heatsink will be enough for the job?. I am asking as i have dc driver with a transistor NCE4080k without heat sink it is soldered to the pcb , so i want to know differences and compare.
thanks for your patience
You need to limit the MOSFET power dissipation to no more than a watt or so (for a TO-220 type case) if you don't want it on a heat sink.
The power is determined by the square of the current through the MOSFET when ON times the ON resistance times the duty cycle, plus the switching losses.
The switching losses are determine by how fast you can drive the MOSFET to turn it off and on.
This requires a high current driver to switch the large MOSFET gate capacitance.
I used Ir2112 as gate driver with Irf3205 but it got warm when i increased the current to 5 A at 20khz. I also calculated the dissipated power due to Rds on and switching losses it won't exceed 1.7 Watt. Is this small heat sink enough to dissipate 1.7 watt. without blowing up?
Yes in reasonable circumstances the pictured heatsink would easily be sufficient for 1.7W dissipation.
To drive down the switching losses further you could consider a driver that can source/sink Amps (rather than 100mAmps), eg. IR2110. If switching losses are the main problem then using IR2110 may mean you don't need a heatsink.