If you directly connect the inverting and non-inverting inputs together you will almost certainly still get a non-zero (albeit small) output, due to the internal input offset voltage of the opamp (arising from chip manufacturing asymmetries) being multiplied by the opamp gain. You can add a small bias voltage, of appropriate polarity, to an input to compensate for the offset.What would happen if the difference between the two terminals are 0V?