We've got a circuit board which uses the CTS 742C083103JP resistor array as pull up resistors for 4 microcontroller inputs. There are pin headers on the board so that you can use a shunt/jumper to pull an input to ground, and this is used to select one of four IDs for the board (there are up to a maximum of four these boards per machine, so this system provides each board with a unique address.)

The problem is that a small, but significant number of these boards fail to properly recognize the shunt position and instead go into their default "no address selected" test mode. It's usually an intermittent fault - very few of them have consistently failed all of the time. It's happened for years now, but I always failed to identify the cause of the problem. Just recently I found that two of the inputs are, for lack of a better term, "partially shorted" together. What I mean by that is that there is a substantial leakage path as shown below:

So far, I've only been able to detect and measure this fault on two failed boards, and one of them stopped having a measurable fault within hours of when I first discovered it. We sent it back to our board assembly house and they did x-rays to look for loose solder balls, etc. but didn't find anything wrong. The second one has tested consistently bad for three days in a row now, but I haven't gotten it back to the assembly shop yet.

There are only a few ways I can see this failure happening:

• solder bridges
• failure in PCB traces
• failure at header pin block
• failure at resistor pack
• failure in microcontroller

This isn't a terribly complicated board, so it's pretty easy for me to check that the PCB itself appears to be fine, there's no solder bridging, and the header pin spacing is wide enough that I'm pretty sure I'd be able to see any fault there. That only leaves internal faults in the MCU or resistor pack. Given that we have this particular problem with some regularity, but we essentially never have any other MCU failure, I'm leaning pretty hard towards blaming the resistor pack.

I'm just having a hard time accepting this. It seems like a resistor pack would be an excruciatingly simple thing to build well. If microcontrollers can be made reliably for incredibly low prices with really low failure rates, why would resistor packs be hard to get right?!?!?

Any thoughts on this? Does the resistor pack failure seem like a reasonable explanation? Has anyone else had problems with resistor packs? Any opinions on brands? Are we just dealing with a crummy manufacturer here?

 

When a failure occurs, can you measure the voltage with a DMM or oscilloscope on each of the four inputs, or resistor pins 1-4, or jumper pins, 1, 3, 5, 7?

 

Measuring resistance the way you are on either a powered or unpowered board should produce unpredictable results. You can draw any conclusion you want from a false premise, that there should be no low impedance path. IMHO resistor pack failures are exceedingly rare.

1. When you do this testing, is the board powered or unpowered?
2. What is the configuration of the uP pins in RESET?
3. What is the configuration of the uP pins after the firmware initializes the ports?

Numerous such problems have their root cause in sloppy initialization procedures for port pins. This is even more true as port pin configuration structures become more complex.

 

When a failure occurs, can you measure the voltage with a DMM or oscilloscope on each of the four inputs, or resistor pins 1-4, or jumper pins, 1, 3, 5, 7?

On a fully functional board, resistor pins 1-4 are all 3.3V when there's no jumper/shunt in place. If you connect a jumper (across J3 1-2, 3-4, etc.) then the corresponding pin drops to 0V.

On an earlier malfunctioning board that I no longer have here, the voltages measured as follows:

In other words, the pin that has a shunt attached drops to 0V as expected, but the adjacent pin drops to 0.21V instead of staying at 3.3V where it belongs. This reasonably well matches the predicted values, based on the ~620ohm leakage measured on that board (prediction was 0.19V, not 0.21, but given that I'm using a cheap-ish meter for all this, I'd say that's a match)


The board I have now measured 85 ohms leakage a few days ago. Today it read 41 ohms leakage from RP1 pins 3-4. With power applied and shunt in position 3 (header pins 3-4) RP1 pin 3 measured 0V as expected, and pin 4 measured 13.1mV, which again is a pretty good match for what the sim would predict (simulation predicted 13.47mV based on 41ohm leakage path.)

 

Measuring resistance the way you are on either a powered or unpowered board should produce unpredictable results. You can draw any conclusion you want from a false premise, that there should be no low impedance path. IMHO resistor pack failures are exceedingly rare.

Totally understand about in-circuit measurements. In this case, assuming the MCU isn't configured wrong as you've suggested, I can see all other paths clearly. The expected resistance between these two pins is 20k, passing through the (2) 10k pull-up resistors. On all normal boards, and on the unaffected pairs of pins on malfunctioning boards, this 20k resistance is spot-on.

When you do this testing, is the board powered or unpowered?

Resistance measurements are taken with the board un-powered, and voltage measurements are taken powered.

• What is the configuration of the uP pins in RESET?

• What is the configuration of the uP pins after the firmware initializes the ports?

Unfortunately, I have no idea. I believe I can get access to the source code, but it'll be way over my head. I might be able to ask the programmer your questions directly and have better luck.

Numerous such problems have their root cause in sloppy initialization procedures for port pins. This is even more true as port pin configuration structures become more complex.

That's a really interesting idea. Hadn't occurred to me, so thanks for that! The programmer who's done all of our modifications and updates for 3+ years now is very smart and very careful. I'm 99.9% confident that he would not have left anything undefined... however, he was building from existing code that was written by someone else who was quite a bit sloppier, so it's possible we inherited something bad there and hadn't caught this particular slip-up yet. I don't quite see how pin configuration errors would explain the voltages and resistances I've measured, but I'll give it some more thought.

 

What happens when you move the shunt to pin 4? Does the voltage on pin 3 drop?

 

What happens when you move the shunt to pin 4? Does the voltage on pin 3 drop?

Essentially the same - measured 13.0mV at pin 3 with pin 4 grounded.

 

Could you be installing the wrong part?

I wonder what would happen if the part you are installing is one with a common connection internally?

 

I've never encountered a processor where I/O configuration is non-volatile. I wouldn't be totally surprised if such a thing existed, but every processor I've ever used has the pins configured as inputs by default so resistance between the pins would be very high unless measured with a meter using a voltage high enough to make diodes conduct, and even cheap meters don't do that except on a diode test range.

Does the X-ray inspection have adequate resolution to show tin whiskers?

Does moving the boards into the lab change the humidity appreciably from where they were operating?

What kind of flux was used in board assembly?

As far as CTS goes, it has been making resistor arrays for many decades, though of course the SM types are more recent.

===
I once spent almost two days trying to fix a video terminal, finally discovering a small bit of metal that appeared to have come from a tapped hole for a case assembly screw was making a high resistance path between an address line and a data line.

 

Could you be installing the wrong part?

I wonder what would happen if the part you are installing is one with a common connection internally?

Are there resistor arrays in hybrid combinations (some pins connected, others not?) I've only seen arrays where all resistors are independent, or where one side of all resistors is common. What I'm getting in these rare instances is two out of the four resistors in the pack are "partially" connected, but the other two are still fully independent. Even if that were the case, I would expect the common sides to really be common, not have anywhere from 40 to 620 ohm resistance.

 

I've never encountered a processor where I/O configuration is non-volatile. I wouldn't be totally surprised if such a thing existed, but every processor I've ever used has the pins configured as inputs by default so resistance between the pins would be very high unless measured with a meter using a voltage high enough to make diodes conduct, and even cheap meters don't do that except on a diode test range.

Does the X-ray inspection have adequate resolution to show tin whiskers?

Does moving the boards into the lab change the humidity appreciably from where they were operating?

What kind of flux was used in board assembly?

As far as CTS goes, it has been making resistor arrays for many decades, though of course the SM types are more recent.

===
I once spent almost two days trying to fix a video terminal, finally discovering a small bit of metal that appeared to have come from a tapped hole for a case assembly screw was making a high resistance path between an address line and a data line.

I have no idea about the x-ray resolution. They seemed pretty confident that there wasn't any physical problem to be found, and I have a good enough relationship with them that I'm inclined to trust their assessment, but I don't actually know.

Humidity is a fair question. I never measure resistance on these things unless they fail when connected to their machine... and when they're on the machine there's significant humidity around them. When they're pulled off of the machine, they're in a somewhat drier environment. When I'm measuring resistance, it's not in a "lab," just on a workbench about 20 feet from where the live machines are running, so not completely isolated from the moisture coming off the machines, but at a significantly reduced level. The conditions at the assembly house are probably even drier. Given that many of the boards that failed in operational test later "started working" again on the workbench away from the moisture, some sort of moisture-based explanation seems plausible.

I'd already kind of wondered about this, but didn't know where or how the humidity would come into play. We're certainly nowhere near condensing humidity levels, so it would have to be something more subtle - parts swelling as they absorb airborne moisture? Residual flux or other contaminants on the board surface becoming more conductive at higher humidity levels? Blaming humidity in the general sense seems to fit, but I'm at a loss as to the actual mechanism that would get us from elevated humidity to these specific failures... much less what we could do about it if that were the case. I'd love to hear more if you can make sense of this, cause I certainly haven't!

I'm pretty sure the boards are assembled with a no-clean solder paste/flux.

 

Are there resistor arrays in hybrid combinations (some pins connected, others not?) I've only seen arrays where all resistors are independent, or where one side of all resistors is common. What I'm getting in these rare instances is two out of the four resistors in the pack are "partially" connected, but the other two are still fully independent. Even if that were the case, I would expect the common sides to really be common, not have anywhere from 40 to 620 ohm resistance.

Good reminder... I agree that the 40 to 620 ohms could not be a result of having an internal common in the resistor network. There is still the chance that you have the wrong resistor network installed. Not likely but I can't count how many times I have found the wrong part in a circuit. Once the wrong part is specified or ordered, the production depart will consistently make every unit the same (and wrong).

That resistance (40 to 620 ohms) is about what I would expect from measuring the network resistance in-circuit. This is caused by the meter forward biasing diodes in the input pin protection circuit.

Something else to think about... Coincidentally, the on resistance of an output port transistor is in the range of tens to hundreds of ohms. This could give some credence to the port initialization being wrong.

 

Something else to think about... Coincidentally, the on resistance of an output port transistor is in the range of tens to hundreds of ohms. This could give some credence to the port initialization being wrong.

The GPIO port configuration wouldn't stay active even when board has been un powered for days, would it? The no-power resistance readings I'm getting seem hard to explain with MCU port configuration.

It's an MSP430 series chip if that makes any difference to you guys.

 

Just found an interesting, if slightly cryptic note about precautions for resistor arrays. I'm not sure precisely what ionic residuals they're referring to, nor whether this would explain intermittent failures, or only persistent failures.

• Adhesive of ionic residuals on the resistor or board could cause migration between the neighboring terminations.

Source: http://www.koaglobal.com/product/caution/resistor?sc_lang=en

 

Just found an interesting, if slightly cryptic note about precautions for resistor arrays. I'm not sure precisely what ionic residuals they're referring to, nor whether this would explain intermittent failures, or only persistent failures.

Source: http://www.koaglobal.com/product/caution/resistor?sc_lang=en

So what on earth are "ionic residuals"? ions are charged atoms. Are they referring to static charge buildup? I'm not aware of boards having a static charge or at least significant net static charge between pins, typically. Excluding capacitive charge (both intentional and parasitic) of course

Perhaps they're referring to unclean boards (flux between pins).

It appears to be a Chinese company - perhaps something is being lost in translation?

 

The GPIO port configuration wouldn't stay active even when board has been un powered for days, would it? The no-power resistance readings I'm getting seem hard to explain with MCU port configuration.

It's an MSP430 series chip if that makes any difference to you guys.

Exactly which MSP430 chip?

 

Exactly which MSP430 chip?

Sorry, was on phone, away from office at the time... I shouldn't have said anything til I had the full answer:

TI MSP430F2232IDAR

https://store.ti.com/MSP430F2232IDAR.aspx
https://www.digikey.com/product-detail/en/texas-instruments/MSP430F2232IDAR/296-21444-1-ND/1300148

 

Does the X-ray inspection have adequate resolution to show tin whiskers?

Copper, not tin, dendrites are a known problem in thick film resistors, which these are. How about getting a few off the boards for testing?

 

Its not something silly like micro internal pull ups/downs turned on ?

 

Troubleshooting:
In my opinion, notions of incorrect I/O configuration or incorrect parts are inconsistent with the observations ebeowulf has described., at least as related to the unwanted electrical paths he's found. Configuration isn't going to change in an unpowered board sitting on the bench. Wrong parts would (I presume) have been revealed by problems apparent at intitial test, and certainly aren't going to change with time. In a new design or production batch, these would be valid concerns. I just don't see them as valid in board that have been in service for some time.
I can easily imagine firmware that has lost its mind reconfiguring input pins as outputs or turning on pullups, but turning off the power makes that go away fairly promptly (some processors will retain memory contents and some config things down to remarkably low voltage, so shorting the supply is sometimes worth doing if you don't want to wait for several minutes after removing power). Pull ups (or downs) are not going to be anywhere near as low as the resistances measured. Berserk firmware usually does other rude things, too.
Having said all that, it is not inconceivable that there are two or more things causing what appears to be the same problem.

Whiskers/dendrites - I'm thinking of tin whiskers on the circuit board between terminals of the resistor network. Because of the close spacing of the pads, there may not be a solder mask dam between them, which may make tin whiskers possible there while improbable elsewhere on the board. As far as I know, copper dendrites in thick film resistors are a possible problem with laser trimmed resistors, and not at all likely otherwise. But I know next to nothing about this. These won't be laser trimmed resistors.

Ionic residues - fluxes are "activated" with ionic compounds. Older fully-activated rosin flux, of the sort you would see in great quantity all over the surface of phenolic circuit boards in consumer goods, had a fairly high content of ionic activators. Newer "no-clean" fluxes have substantially less. Water soluble fluxes (more on these below) have very high ionic compound content, are corrosive, conductive and must be very promptly removed. Any cleaning process to remove fluxes is going to expose everything on the board to ionic compounds, fairly heavily early in the process and (hopefully) virtually none by the end of the process.

Humidity
This is where the type of flux and the cleaning process could be big contributors. No-clean fluxes seem to be by far the most popular in general use for SMD assembly. While they are not entirely free of ionic compounds, they are regarded as safe to leave on all but the most sensitive high-impedance boards - which is good, because they are really hard to remove. If boards need to be cleaned after soldering, water soluble flux, generally containing fairly large amounts of organic acids, makes cleaning comparatively easy. While cleaning requires only water and not organic solvents or saponifying agents (as would be used for no-clean or rosin fluxes), it is essential that cleaning be done very promptly after soldering - hours, at the most. If cleaning isn't prompt and thorough, corrosion products of metals can be formed and they are extremely difficult to remove.

Board washing is often done with deionized water in dishwasher-like machines. I have serious reservations about the ability of some of these machines to do an adequate job with surface mount components, especially when no surrfactant (detergent) is used. As soon as water hits the board, capillary action will draw water under the components. Of course there will be flux residue under the parts. Once that water is under there, it will dissolve some of the residue, but it will also have a strong tendency to stay put unless blasted out by water impingement of adequate force. Process control in board cleaning is often done by measuring the conductivity of the wash water. As the ionic constituents of the flux are diluted with subsequent rinses, the conductivity rises (again, deionized or distilled water is being used). My belief is that this is a poor means of assessing cleanliness with SMD boards because the volume of water in the washer relative to the volume that can be "trapped" under parts is extremely high. Maybe it works. I'm unconvinced. A friend told me of problems with cleaning of boards where he was working, but at the time I was so fed up with tech of any sort that what he said went in one ear and out the other.

On boards I've had cleaned this way, I've seen visible residue left behind after putting a very small amount of isopropanol on parts - it wicks under, dissolves crud which diffuses into the alcohol around the part and when the alcohol evaporates there is visible crud left behind. I regard that as gross contamination. By the time I discovered problems like this I was winding down doing electronics, so I never pursued it further. I will stipulate that this was an assembly house that gave me the impression that not a single person working there was really any sort of expert. I've cleaned lots of boards myself after assembly with organic acid flux using a multistage process of fairly high pressure water spray followed by compressed air blow-off (forcing water out from under parts), repeated & followed with distilled water rinse. My boards didn't leave telltale rings if I used the alcohol test. Anyway...

Humidity, the "important" part:
Resistor arrays require closely spaced pads. Depending on the copper thickness the bottom of the part will be extremely close to the surface of the solder mask, which means that anything in that tight space is going to be difficult to remove. If residue, either of water soluble flux or ionically-comtaminated wash water from the cleaning process for any flux type, remains in that space, there is risk of corrosion and production of conductive products. Tin oxide is quite conductive - it is what is used for the conductive pattern on assorted opto things and transparent resistive touch screens. If gunk is at all hygroscopic (water vapor absorbing), the conductivity might change substantially with humidity.


What would I do next?
I would try to remove the resistor array from a failing board using hot air and great care to lift it straight up, as best I was able. I would inspect the part and the PCB with my handy dandy stereo microscope, looking for anything out of the ordinary (actually, I'd do a visual inspection before removing the part, but I think that's already been done pretty thoroughly).

If I were reasonably convinced humidity might be a problem, I would condition a board that had problems in a simple humidity chamber - a plastic storage box (Rubbermaid, or the like) with some moist paper towel for several hours. I might also do the opposite, using desiccant. Warm air from a hair dryer could be useful, but I'd not want to raise the board temperature much, so lots of air and very little heat.

Stuff like this can be really horrible and time consuming to diagnose. Then you have to work out a fix.