Is this because (almost) no current flows into the input of the op-amps?That arrangement posted is how I would do it and recommend.
The output from the pot would connect directly to pin3 of the LM380.
The pots will allow the users to adjust there volume independently without affecting the other.