Not worried about destroying the battery as it is documented that the battery should be removed before using connecting external power.Yes, it's an issue. That will eventually overcharge a battery that isn't meant to be recharged. It may not be terribly unsafe, but it will destroy the battery. A blocking diode would solve the problem but you don't have much voltage to work with. A switch of some kind might be a better solution.
Thanks, did not know of that part.You could use an ideal diode such as this to prevent the battery from being charged