No, not necessarily. The term "ground" is a matter of definition, and is usually the common point for all power within a circuit. However, there can be many "grounds" in a circuit, for instance: digital ground, analog ground, video ground, and let's not leave out chassis ground. Many times circuits are created without any thought or regard to earth ground.Lets consider a simple circuit: A battery connected to a bulb in a flashlight. The voltage across the battery is known, but the voltage between any terminal of the battery and ground is undefined. Therefore the whole circuit is floating with respect to ground. Most technical people would say that the circuit is isolated from ground.
I could easily draw your battery & light bulb circuit with a circuit ground. I would simply use a ground symbol on one side of the battery (probably negative, but in this case it wouldn't matter), run a wire connection between the battery and the bulb, and then use a ground symbol on the other side of the bulb. This circuit now has a common point that will be known as GROUND. Although it has nothing to do with earth ground, it is merely syntaxical and you really shouldn't get hung up on it. However, in this case of grounding the (-) side of the battery, you would now say that the bulb is running on +V relative to ground. Although this would not matter with a filament light bulb, it would matter greatly with an LED which has a specific polarity.
A power supply with a floating output will generate either a + or - voltage depending upon which polarity output is connected to the circuit ground. The LED would only light if proper polarities are observed during design & construction. One of the easiest ways of determining polarity, is to establish a circuit ground.
In most modern designs, circuit ground is one or more layers of solid copper within a multi-layer board. This "solid plane" ground technique creates a much lower impedance and contributes greatly to alleviating ground bounce and EMI issues, along with improving signal integrity.