ok, maybe this will do, only problem is that capacitors are now 3mm away from the heatsink, it wont generate much heat, but I like to keep them away.

ok, environmental rules state that I minimize the removal of copper from the pcb, so, fill all that space with earth ground ?

If you have any more sugestions, please, I'm trying to learn

 

Like I said, I would move the capacitors to the lower edge of the board between the connectors. Right now you have high peak switch currents going from one side of the board to the other between the input and output caps.

 

hello again, I've built the board, and took it to a friends oscilloscope, the values were not bad(in my opinion), with DC 12vols @ 12amps in and 5.16volts@1Amp out, maybe 20~50mV oscillation(It's a analog display oscilloscope), but it increases, maybe up to 200mV with a load of near 3 amps, the heatsink is below 30ºC.
I've change the design again(reduced the heatsink size, and will fix it into the case).
It's not a big reduction in the ground plane, but the capacitors are near, anyway, I wounder if I can improve the output capacitors, right now, they are 2 aluminium electrolict Panasonic FC series rated at 50volts 150uF with 0.8Arms each one(half of the max load).
I'm not expecting a clear answer, but is the is the correct way to go forward to increase the Arms value of the capacitors ?

So far ,good or bad desing of the board, made me look into Mosfet voltage reverse protection, capacitor ratting, capacitor heat generation, inductor rattings and a lot of things, so, not that bad experience . Anyway the original board is way more cheap but it does not teach me anything.

Attached is the new design, please comment on the capacitor improvement.

 

That layout looks better to me. As for:

with DC 12vols @ 12amps in and 5.16volts@1Amp out, maybe 20~50mV oscillation(It's a analog display oscilloscope), but it increases, maybe up to 200mV with a load of near 3 amps,

Are you sure it's oscillation? It sounds like output ripple which is present on any switcher's output. If the frequency is the switch frequency, it's ripple.

anyway, I wounder if I can improve the output capacitors

Use what is recommended on the data sheet. I think there are some solid Tantalums listed that had lower impedance than the Panasonics.

but is the is the correct way to go forward to increase the Arms value of the capacitors

basically, yes. Or stack more caps in parallel. You want caps with lower impedance at the switching frequency.

 

This PCB layout is possible because the heatsing has been reduced, I think that the math is wrong, P = I^2*RDS(on); internal MOSFET RDS(on) is 0.12ohm so it would be P = 3^3 * 0.12; P = 1.08W, the switching part is missing in the math, because it was below 30ºC, room temperature of >25ºC(I live in Portugal and it's still summer)

Yes, the correct word is ripple.

I'll try to find some Tantalums at mouser, these panasonic are the ultra low impedance types. Thank you!