The voltage regulator I use in my project starts to make noise as soon as I put the tantalum capacitors at the output, if it is not installed at the output, the regulator does not make a sizzling sound

 

I am guessing that the buzz sound is from current pulses in the energy storage inductor, L3. If that is the actual case, then it needs mechanical damping to prevent future failure. OR replace it with a more solidly built inductor.
It might also be that the tantalum capacitors have excessive leakage current.