Why are the immobile charges immobile in the depletion region?

I understand that electrons move from the N-type region towards the P-type region and combine with holes. I also understand the localized positive and negative charges on each side of the depletion region, and the electric field that result from it.

What I don't understand is why don't the Boron atoms, which currently lack an electron (3 covalent bonds), 'steal' the electrons from the adjacent Boron ions which have an additional electron (4 covalent bonds). Why do these ions hold on to their electrons so dearly?

Thanks.