OK, I'm fed. For now.

Looking at the wiring diagrams on pp. 63-64 of the datasheet, and the internal ADC block diagram on p. 561 of the Family Reference Manual (http://www.ti.com/lit/ug/slau144j/slau144j.pdf), it looks like chip-to-board ground differentials could indeed be the source of your problem. It also looks like there might not be a darned thing you can do about it.

The way I interpret the block diagram, the 2.5V reference voltage probably is generated relative to internal AVss (it's hard to tell; they show the ref generator going to a ground symbol, without specifying which Vss it is). If you're using that internal reference, that means Vref+ is +2.5V relative to that point, not +2.5V relative to circuit board ground. So how stable the ADC reference is, is therefore partially dependent on how much that internal AVss bounces around. That might be only a little, or it could be a lot; I have no way of telling.

One investigation you could do would be to take a look at the Vref+ output from the chip using a good scope set for AC coupling and a vertical scale of 10mV/div or thereabouts, and see if you observe any fluctuation as the μC goes through its paces. That might yield a clue: if you see absolutely no bouncing around whatsoever, I'd say chip-to-board ground differentials are probably not what's causing the problem. If you do see Vref+ wiggling up and down, they probably are-- but in that case, I suspect you're out of luck because I can't see any way to correct the problem without resorting to an external, stable reference.

JohnInTx and nsaspook might have different angles on the problem, but that's my take.

I am definitely NOT a fan of voltage references on μC chips, because of exactly this problem. They're OK for crude, low-precision work, but when you're trying to squeeze a full 12 bits of accuracy out of an on-chip ADC, those built-in voltage references fall far short of the mark. (I noticed on the datasheet that the voltage reference has a tempco of up to ±100 ppm/°C; that means with a 12-bit ADC, it can drift as much as 1 LSB every two degrees. Not good!)

That's all I have for now...

 

Let me see if I'm understanding this right. It sounds like it's theoretically possible for the ADC ground to be permanently held high if there's steady current through the uc? In addition to any ripple on it, it might just generally run some number of mV above board ground?

I feel like we see little, if any evidence of noisy ADC readings, but mostly just steady, predictable offsets. I have to do a little math, but my hunch is that the RTD op amp circuit is generating a 0-2.516V signal which is being converted by an ADC reading it as -0.016 to 2.5V because of the difference in ground potentials (or some other difference, I just chose 16mV since my reference happens to be 16mV higher than expected.)

I'll have to see if that would even explain the numbers. Thought there was scaling, not just a steady voltage difference. Hmmm...

 

Let me see if I'm understanding this right. It sounds like it's theoretically possible for the ADC ground to be permanently held high if there's steady current through the uc? In addition to any ripple on it, it might just generally run some number of mV above board ground?

That's the way I see it, yes. The chip-to-package ground bonding wires have resistance, and so does the chip metallization, and any current through any of those resistances is going to cause a voltage drop which will lift the chip internal ground potential above circuit board ground potential. That resistance is probably on the order of an ohm to two ohms or so, if my measurements on PICs are any indication; and if so, a couple of dozen milliamps flowing through those wires could cause that many millivolts of differential.

I feel like we see little, if any evidence of noisy ADC readings, but mostly just steady, predictable offsets. I have to do a little math, but my hunch is that the RTD op amp circuit is generating a 0-2.516V signal which is being converted by an ADC reading it as -0.016 to 2.5V because of the difference in ground potentials (or some other difference, I just chose 16mV since my reference happens to be 16mV higher than expected.)

That makes sense, and those numbers are in the order of magnitude I'd expect.

 

Let me see if I'm understanding this right. It sounds like it's theoretically possible for the ADC ground to be permanently held high if there's steady current through the uc? In addition to any ripple on it, it might just generally run some number of mV above board ground?

I feel like we see little, if any evidence of noisy ADC readings, but mostly just steady, predictable offsets. I have to do a little math, but my hunch is that the RTD op amp circuit is generating a 0-2.516V signal which is being converted by an ADC reading it as -0.016 to 2.5V because of the difference in ground potentials (or some other difference, I just chose 16mV since my reference happens to be 16mV higher than expected.)

I'll have to see if that would even explain the numbers. Thought there was scaling, not just a steady voltage difference. Hmmm...

I'm no expert on this MCU, but I was interested to see if they had really implemented a 12 bit ADC that had an inbuilt error of the internal ground voltage drop. I don't think this is the case.

You haven't told us how you have configured the ADC lower reference. From the ref guide it is possible to configure the ADC lower reference to be either the internal chip AVss or an external Vref-. As OBWO549 has said, there could easily be an offset of 10mV or so on the internal AVss relative to the external pin voltage. For lowest offset you should use the external Vref- and connect it to the same gnd potential as throughout your analog input chain.




Then when you do a conversion you would set SREFx to 101 to use the internal Vref+ and the external Vref-.



These settings should remove the DC offset effect of the internal AVss, but you are still left with it on the Vref, which would give a small scale error. You could remove that with the fixed voltage you measure on one of your mux settings.

If you are already using the external Vref- then you could try (depending on what's possible with your hardware):

Get it working without the multiplexing first. Freeze the multiplexer onto your RTD and check all voltages with a DVM. Check for gnd errors by measuring voltages between the cold ends of the RTD, R33, R34 and the ADC Vref-. Run an ADC measurement and check it works within spec to the measurement across the pins.

If this works without offsets then look for dynamic effects. If your software does not let you vary the time between the mux switching and the ADC conversion, then hardwire the mux to your RTD but connect one address pin to a function generator. Monitor the waveform at the ADC input as you vary the switching frequency - this should let you see the settling time.

I'm suspicious of C30. Why is it there? It looks like a bad fix for some settling time issue.

 

There is another source of offset that I can see, it is significant, but it should work in the opposite direction to what you see.

Firstly, you say that you expect the INA331 to give you 0 to 2.5V (ish). However, looking at the datasheet, you can see that the INA331 can only swing to within about 5mV of the lower rail when loaded with 30k:


So this will give you a fixed offset, you can't actually get to 0V.

It's probably a bit worse that that, in the INA331,



You have set Vref = V- = 0V and V+ = +3.3V. When Vin- = Vin+ then the output of A2 is zero, i.e. you need it to go all the way to the lower rail, and I suspect it can't do that. The datasheet doesn't specify it, but whatever the saturated output voltage of A2 is gets multiplied by 5, in your circuit, to the output of A3. If A2 was as good as A3 then with a 40k load you get about 4mV which is multiplied by 5 to give an output voltage of 20mV.

Now the datasheet doesn't say that Vref can't go to V-, but the specs are for Vref = Vs/2, and none of the examples show it connected to V-. It's pretty clear from the analysis that your configuration is very dependent on the output characteristics of A2.

Note that this is the opposite of what you say you see - this will prevent the voltage getting down to zero, the ADC would see something like 0.02V - 2.5V.

You can see I was bored today....

 

There is another source of offset that I can see, it is significant, but it should work in the opposite direction to what you see.

Firstly, you say that you expect the INA331 to give you 0 to 2.5V (ish). However, looking at the datasheet, you can see that the INA331 can only swing to within about 5mV of the lower rail when loaded with 30k:

So this will give you a fixed offset, you can't actually get to 0V.

The Output Swing vs. Load Resistance chart only specifies how close the output can get to the rails; it doesn't specify an offset that applies throughout the output's active voltage range. Provided the output doesn't attempt to get closer to the rails than those values, there will be no effect.

As for the output getting down to 0V, this is only going to happen when the temperature gets down to about -25°C (-13°F), which I assume is probably outside the OP's intended measurement range (although he doesn't say).

 

The Output Swing vs. Load Resistance chart only specifies how close the output can get to the rails; it doesn't specify an offset that applies throughout the output's active voltage range. Provided the output doesn't attempt to get closer to the rails than those values, there will be no effect.

As for the output getting down to 0V, this is only going to happen when the temperature gets down to about -25°C (-13°F), which I assume is probably outside the OP's intended measurement range (although he doesn't say).

You're correct OBW0549, I shouldn't have called it an offset - the output simply saturates.

It is not clear what range the OP wants, he says his hunch is that the RTD op amp circuit is generating a 0-2.516V signal, and if that is what he really wants, he won't get it with this arrangement, it will saturate at the bottom.

Also note that the offset voltage spec on the INA331 is not that good - and the schematic specifies the low spec device (and at 3.3V), it could be of the order of 1mV RTI, so you could get 10mV or so from here, but it should be random not systematic.

 

Also note that the offset voltage spec on the INA331 is not that good - and the schematic specifies the low spec device (and at 3.3V), it could be of the order of 1mV RTI, so you could get 10mV or so from here, but it should be random not systematic.

Agreed. I couldn't see much in his design that could give 4-15 mV offsets, all in the same direction from unit to unit.

The MSP430's internal voltage reference is one possible culprit; though I'm not sure I yet understand all the subtleties of using the reference and configuring it, I still suspect internal/external ground differentials may be mucking things up.

Leakage through the ESD7V0D5 protection diodes was another possible suspect, but a glance at the datasheet shows the maximum leakage current (30 nA) to be far too low to be causing the offsets he's seeing.

Dunno. It's a stumper...

 

The MSP430's internal voltage reference is one possible culprit; though I'm not sure I yet understand all the subtleties of using the reference and configuring it, I still suspect internal/external ground differentials may be mucking things up.

I think that using the internal AVss for the ADC Vref- is the most likely culprit as it would give just the sort of systematic offset the OP describes. If it is configured as I suggested then there should be no systematic offset error. Also, as his Vref is simply buffered from the MPU then I don't think the AVss offset on the reference will even give a scaling error as it affects the input voltage to the ADC and the ADC scaling identically.

I'm still suspicious of C30, not that it is a source of trouble itself, just that it is a strange design that suggests to me that someone may have been attempting to squish some mux glitch issue. If the OP runs out of options, unless he is 100% convinced that the ADC is read 1ms after the mux is changed, I'd suggest changing the ADC delay, even a simple bodge like patching the code to read the same RTD twice in succession and seeing if there is any difference when the delay is doubled.

It was an interesting problem...

 

Agreed. I hope there's a follow-up to this if/when the problem gets solved.

 

I can't tell you both how much I appreciate all the thought you're giving this. Sorry I was crazy busy yesterday and unable to check in. Also sorry that some relevant details were left out earlier.

The RTDs are DIN spec 2000 ohm units. The theoretical range of the sensing circuit is around 132 to 294, but in practice the firmware only even shows numbers starting around 170 and nothing on the machine should EVER get above 265 or there's a serious problem! So issues like not getting the op amp to 0V won't hurt us, but large offsets or scaling issues would. The two set points that matter and where accuracy would be most useful are in the 195-205 range and in the 245-255 range.

I got a batch of 0.1% resistors in values spanning the range of interest (3 each at 5 different values) and took readings from them yesterday. The table below shows three sets of data. The left two columns are the temperature to resistance data for our RTDs, with interpolated points added in orange for the resistors I got for testing. Next are calculations of what the expected RTD voltage should be (based on actual measured V_REF of 2.517 on this board) along with what op-amp voltage, ADC value, and displayed temperature my calculations say should come from that RTD voltage. The next section shows the actual measured voltage at the RTDs and the actual displayed temperature.

Following those two numbers are my "apparent" ADC value and op-amp output voltage. I say apparent because I'm still struggling to interpret all the code. I can't find any mention in there of reference selections like you both mentioned above (although I did find the relevant portions of the MSP430 datasheet, so I think I know the sorts of codes I should be looking for.) Since I'm clearly still missing some portions of the code, either looking in the wrong spot or just reading right through important lines without realizing it, I can't be 100% sure about the relationship between displayed temperature and what's happening on the board after the mux. Also I don't have a compiler for this, so I can't try simple code changes on my own to experiment with longer pauses or even freezing the mux to do post-mux voltage measurements. Very frustrating. So it's still possible that the circuit is working perfectly and that the problem is all in the code... but I digress. Assuming the code is good, the next two columns represent the ADC value and op amp output voltage that correspond with the displayed temp.



As you can see, the error gets larger at higher temps/voltages. The data isn't super-smooth, but looks pretty linear to me, and seems to be mostly a gain error, with a very small offset, although I could be wildy misinterpreting the data (which is why I posted all the data, not just my interpretation of it!) If I understood the ground reference issues properly, these would lead to the largest errors being closest to ground, with full scale readings being pretty good. I seem to have the opposite here.

So I realized when looking at this that the two resistors which set gain on the op amp would seem like the most likely culprits for a significant gain error (unless it's in the code!) I further realized that I've checked the values of those two in our copy of BOM and on the schematic, but not on the actual board. As much as it pains me, we've had discrepancies in the past between schematic, BOM, and production before due to quick verbal or e-mail changes that never got documented. Usually it's just the schematic that remains out of date and the BOM matches production, but I wouldn't put anything past these guys at this point. So... I'll have to verify the physical reality of those two resistors on Monday. I'll also keep digging through the code and try to find the voltage reference selections as well as anything that could be scaling our readings.

Thank you both so much for all of your insights. If the new data brings any new ideas to mind, please let me know!

Cheers!

 

Oh yeah, the other thing I'm looking for in the code, but haven't found yet, is any mention of reading that fixed voltage reference that feeds into the mux on S7. Seems like it has to be there for a reason, but I never see it called in the code. As far as I can tell the code just scans through the five RTD inputs our machine uses and never checks the fixed voltage reference. But I'll keep looking - if it did the wrong thing with that reference, that would be a really easy to create a scaling error.

 

And on the mux to adc delay, I'm as confident as a rookie can be that we've got a 1ms delay between the two. It's one of the few parts of the RTD related code that I thought I was reading clearly. Next time our programmer stops by I'll see if he has time to take a closer look at all this. For now I'm doing the best I can to learn the code with only limited Arduino coding background to go on.

 

I've got to noodle on your numbers a bit... back later.

 

Sweet, thanks!

 

Hmmm... I strongly suspect you've got a software problem here. Looking at your tables, I note the following:

• This is not a case of a simple offset error.
• In fact, it looks like the major error is a scale factor error-- and a BIG one at that, like maybe around 16%.
• There may be an offset error in there as well, but if there is it's positive, not negative.
• I checked your RTD resistance values v. temperature and they match up with my own DIN tables, exactly.
• I also checked your calculations for the expected RTD voltage divider output v. temperature, and they're correct.
• I noted that your measured RTD voltage divider output voltages correspond very closely to the calculated values.
• I did NOT check any of your other calculations, such as calculated voltage into the ADC.

I suspect the software may simply be calculating the temperature wrong: using the wrong tables, or using the wrong polynomial, or failing to take into account that a voltage divider does not produce an output voltage that varies linearly with the resistance of the lower leg, or whatever. Who knows.

A couple more things might be worth doing, to try to get a handle on this:

• It would be good if we could have a few more rows in that table-- say, 3 or 4 more entries down toward room temperature (or even lower), plus 1 or 2 more entries above 250 °F.
• Unsolder the mux chip and replace it with a wire jumper between one of the input channels and the mux output, so you're always reading the same channel. (As an alternative, you could lift up the A0, A1 and A2 pins of the mux and hardwire them high/low to get a constant channel selection.) Then, step through your sequence of RTD resistance values on that channel, while measuring the actual ADC input voltage with a good, high-resolution DVM, and recording the measurements as another column in your table.

If those readings match your expected ADC input voltages, then you DEFINITELY have a software boo-boo.

 

I didn't go any further on the table because the display simply reads "high" or "low" outside of the useful range. Unfortunate, but not much I can do about it.

I like your idea of bypassing or manually controlling the mux. That would allow for useful downstream measurements without firmware hacks. Not sure if my bosses will let me modify one of our boards, but it's worth a shot.

I'll verify the gain-setting resistors on Monday. Until then I'll keep exploring the code. I tend to think you're right that this is more likely firmware than hardware.

The code uses a quadratic equation to approximate temperature directly from the ADC value. That formula is what I used to generate the "calc display temp" column on my table, so I know the formula is good (within a couple tenths, anyway.) But it's assuming that the ADC reports 0-4095 for a 0-V_REF range of voltages. If there is any scaling, trimming, calibration, etc. happening to the ADC value before it gets to that quadratic equation, it could be wreaking havoc. I've found the relevant formula, but I'm not positive that I've found the entire signal chain from raw ADC input to that formula. I fear there's a step in between.

I'll report back Monday with whatever I find.

 

Well, I think I understand the flow of the code a little better now, and I really can't find anything in between the raw ADC reading and the quadratic equation. If it's not bad gain-setting resistors on the op amp, I'm truly stumped now.

On the bright side, I did finally realize that the reference setting info was right in front of me the whole time:

// initializes the adc for temperature sensing
void adc_initTemperature(void)
{
  ADC12MCTL0 = SREF_1 + INCH_0;             // ref+=Vref+, channel = A0
  ADC12CTL0 |= ENC;                         // Conversion enabled
}

SREF of 1 would be "VR+ = VREF+ and VR- = AVSS" which is not ideal, but still not enough to explain the massive gain/scaling issues.

...and even as I'm writing this I decided to take one more look at the quadratic equation and I found something that strikes me as odd. Let me first say that I'm super-new with code and still regularly get lost on variable types, when I need to convert to and from floats, how the syntax of such modifications work, etc. With those disclaimers, it seems to me like the way floats are handled in the following code is inconsistent between the first and second terms (Ax^2 portion vs Bx portion). In the first portion, (float) is followed by (adc*adc) with A_TERM outside of the float parentheses, but in the second portion, (float) is followed by (adc*B_TERM) with B_TERM inside the parentheses along with adc. I don't know which one is right, but I assume one of them is wrong?

static const float A_TERM = 1.4841E-6;
static const float B_TERM = 3.3878E-2;
static const float C_TERM = 1.3228E2;

...

float _calcTemperature(uint16_t adc)
{
  return ((float)(adc*adc))*A_TERM + (float)(adc*B_TERM) + C_TERM;
}

Is part of this formula not handled correctly? Could we be truncating important numbers in the first and/or second portions of this formula? I'm embarrassed to have the formula right in front of me and still be so unsure, but it doesn't seem quite right to me.

 

Wow!!!! I think that was it. I tried dropping the A term portion of the formula entirely in my spreadsheet, making the formula just adc*B_TERM+C_TERM and got all the display readings to within a few tenths. I can't believe I found it. This is crazy. And all cause a few parentheses were in the wrong place. I still don't feel confident about which way the syntax should look to make it work, but I think I found the root of the problem.

 

Is part of this formula not handled correctly? Could we be truncating important numbers in the first and/or second portions of this formula? I'm embarrassed to have the formula right in front of me and still be so unsure, but it doesn't seem quite right to me.

I don't know; I'm at least as much a n00b as you when it comes to C, being still in the very early learning stages. It does look odd, but I'm not sure it makes any difference.