I have a APC by Schneider Electric Back-UPS BX - BX1400UI UPS. I have read how to upgrade the battery which seems straight forward, just disconnect the current batteries, extend the battery wires to the external battery or batteries and away you go.

One caveat seems to be you need a UPS with twice the wattage you need because they aren't designed to run for prolonged periods, which is exactly what upgrading the battery does.

Can I upgrade the inverter? I assume that's the weak point here. Is it even maybe a case of just swapping out the transformer?

I seen a YT video on a DIY 4kw inverter, that I think is something I could be doing with.

 

Are you honestly using the UPS at anything close to its rated VA capacity? (Use a power meter like the Kill-A-Watt to check.) If not, I wouldn't worry, or I'd just add a fan that's triggered by heatsink temperature (if that can be done safely). If you really did need to upgrade the inverter, look for a bigger used UPS that needs a new battery, or buy one refurbished. I wouldn't (and don't) hesitate to use a 20 year old APC UPS that's been updated with a fresh Panasonic battery.

 

It seems to me that the inverter is most of the UPS so upgrading it doesn't seem a lot different, nor easier, than just replacing the UPS with a larger one.

But it may not be necessary. At some point the system will be in thermal equilibrium and not get any hotter. If the existing battery gets you to that point, then extending the run-time won't make it get any hotter.