Probably because everything, importantly the thermistor, has a thermal time constant and can't respond instantly to a temperature change.the voltage is drifting
If your supply voltage drifts (have you confirmed it doesn't?) then so will the reference voltage.



