An audio power amplifier applies a positive voltage at a high current and a negative voltage at a high current to a low resistance speaker. So the upper transistor (NPN) in the totem pole pulls the speaker positive then the lower transistor )PNP pulls the speaker negative.
My first encounter with the term was with TTL. I lifted this off of Wikipedia.
I think the term has expanded somewhat, for any output using two transistors, usually (but not inclusive) being the same type. The LM555 for example has an output that looks a lot like this, and I've heard it called totem pole:
I think the term has gotten looser over time, but basically it is two transistors that connect the output to the power supply rails, but is not rail to rail. I would not class the boosters shown on the second half of the above picture as totem pole, though it could be argued.