Help determining if a circuit was tampered with

Thread Starter

Redux

Joined Feb 4, 2016
13
As someone who has also worked in the electronics repair service industry over the years I am quite familiar with what people will do for warranty work.

The common theme I found was those who want and claim it the most are those who are too cheap, lazy and dumb to find a better way to get new gear which ultimately makes finding deliberate sabotage or tampering fairly easy for anyone who is familiar with the equipment.
Typically the most common things I saw were based on the revers compatibility of updated circuit boards and the like where a new version of a circuit board will work in an older model machine but the older model machines circuit board wont work in the newer model machines. Certain customers would find out how much to was to replace a bad board on their old machine then say they don't want to spend the money on the repair so they would buy the newest version of the same model which just by coincidence would come back in a month to two for warranty work with the older models circuit boards in it.
I would let my boss know and we would take a few pictures and document the machines build dates Vs the build dates and production codes on the boards and whatnot then put in all new stuff and tell the customer they can have their new machine back for the cost to the new boards plus labor of which we always made substantially higher than what the repair of their old model was. :D

Unfortunately given what you have shown us and where the faults occurred and how I have very little reason to believe it was deliberate customer damage. The where and how of the damage looks like what I would see as common IC and socket failure that lead to component overloads and circuit board burning.
From what I see nothing looks like tampering, Intelligently done or otherwise. Anyone who was smart enough to tamper with it at that level would have the knowledge of how to disable a board like that without leaving any physical damage in place. :(

Static shock to one of the main processor IC's resulting in major board malfunction is easy to do with nothing more that shuffled feet and a well placed unfolded paper clip plus would be impossible to trace.
Thank you very much.
 

MrSoftware

Joined Oct 29, 2013
2,200
It strikes me as odd that they both failed at the same time, unless they were plugged into each other and one took the other out. Is it possible that he wet them again?
 

RichardO

Joined May 4, 2013
2,270
Did both units fail near connectors? Maybe there has been a power wiring/grounding problem between multiple units. This could explain why the 3D printer worked with an SD card but smoked with the USB connected.

This might have been caused by water damage to the power line wiring at the customers site. In an extreme case there could have been a loss of neutral on one phase.
 

Thread Starter

Redux

Joined Feb 4, 2016
13
Did both units fail near connectors? Maybe there has been a power wiring/grounding problem between multiple units. This could explain why the 3D printer worked with an SD card but smoked with the USB connected.

This might have been caused by water damage to the power line wiring at the customers site. In an extreme case there could have been a loss of neutral on one phase.
Yes however the switch was near a network port and no nics were plugged in.
Good call i didn't think about issues at customer site.
 

hp1729

Joined Nov 23, 2015
2,304
Hello All!
This is my first post here, so thank you in advance for any answers and I'm happy to be part of the community.

I work for an electronics restoration company. We clean, repair, test and restore electronics after smoke, fire or water.
I am the primary IT electronics tech. So when it comes to circuitry I am a bit above a novice but not much. I can identify components and solder easy things but I am in no way advanced or even a strong intermediate.

The issue
A client had an incident involving water. The item were being stored and were not powered on or connected to power at the time of the loss. At most these items had some dampening from the water and were no where near soaked. We see this all of the time. Items are cleaned with deionized water and solvents and placed in a static free dry room then they are tested or in this case items were cleaned using denatured alcohol and contact cleaner and dried. All common practice that yields 90% positive results.
After cleaning and drying the items are tested. The items in question are a Flashforge Dreamer 3D printer and a Cisco SG500X-48 Managed Switch. We cleaned and dried them and they tested fine. The switch was tested to power on and handle a little bit of network traffic. The 3D printer was tested with a 4hr print and a 14hr print both prints went well and the unit was on for about 48hrs.
Both the 3D printer and Switch were returned to the customer. A few days later he complained about the items not functioning properly when he tested them. The client is extremely tech savvy. I guarantee he knows more about circuitry than I do. His complaints were that one of the extruders on the 3D printer wasn't functioning properly and that the switch had a loud fan.
We picked up the items and brought them back to test. Upon return of the items the switch was plugged into power and the tech went to the bathroom. Literally 2min later one of the network jack arrays started smoking the board. It is a perfect circle under the first nic connection array. The 3D printer I had up printing via sd card. It was functioning well. Later I plugged in the usb to test other settings via the pc software and 2 min later more magic smoke and an IC behind the usb port fried and caused the unit to fail.

Now these issues are consistent with what water could be expected to do to a circuit. However they did this after we followed a very much practiced restoration process and after we tested them and the client tested them. This seems very fishy to me. It is highly uncommon as we do this all of the time. So my question is what could be done to these circuits or components to cause this? As in is there a way to tamper with the devices to yield the issues observed. The switch went first so I visually inspected the 3D printer prior to working on it. Please see attached photos. I saw nothing odd about the components of circuits involved upon visual inspection, I did not break out a meter prior.
I have attached pictures with the affected boards and components highlighted.
View attachment 100022 3d printer prior
View attachment 100023 3d printer prior
View attachment 100024 3d printer after
View attachment 100027 Switch prior
View attachment 100028 Switch prior
View attachment 100029 Switch prior
View attachment 100031 Switch after
View attachment 100032 Switch after
Why would the customer do such a thing? Do you guarantee your work to the point of replacing the equipment?
I can see sending a power supply or such thing in to a repair service stuffed with bad caps, but such is not your business is it?
Do you assume the device was working before the catastrophe? Do you guarantee it will work after you clean it?
 

KJ6EAD

Joined Apr 30, 2011
1,581
Liquid damage to electronics is pernicious. I would never have full confidence in a device so damaged nor would I warranty it despite the cleaning, drying and testing.
 
Fascinating story, but so tricky to pinpoint. I think the other responses have provided a level-headed spectrum of likelihoods.

For what's it worth, that failure in the NIC looks to be a partial short between the Power over Ethernet (PoE) pins of that Ethernet port. Assuming the CISCO supports PoE, may supply up to 20W once it sees a partial short of 25kOhm between the data pair pins. That may or may not explain why the problem was not detected at first - the damage may have been pernicious, to borrow KJ6EAD extremely apt term, and only manifested itself once the damage crossed that 25kOhm "signature".

The damage next to the USB socket could be many things, but similar situations I've seen are invariably related to ground loops. Typically a PC's USB port will have the DC- pin electrically connected to the earth of its power supply. This can sometimes cause a drama if the USB slave is also powered by a separate wall-wart. The earth reference of wall-warts can vary, which means the DC- pin might be at a potential different to earth. The electronic design of the printer will try to isolate the two sources, but if that isolation is compromised and breaks down over time, a ground loops forms and your circuit board becomes the unwitting conductor.

So while I much prefer the malicious tamperer story, both these failures could be explained by latent water damage. The coincidence is shocking, but Occam's Razor might be a guiding principle here.
 
Hello All!
This is my first post here, so thank you in advance for any answers and I'm happy to be part of the community.

I work for an electronics restoration company. We clean, repair, test and restore electronics after smoke, fire or water.
I am the primary IT electronics tech. So when it comes to circuitry I am a bit above a novice but not much. I can identify components and solder easy things but I am in no way advanced or even a strong intermediate.

The issue
A client had an incident involving water. The item were being stored and were not powered on or connected to power at the time of the loss. At most these items had some dampening from the water and were no where near soaked. We see this all of the time. Items are cleaned with deionized water and solvents and placed in a static free dry room then they are tested or in this case items were cleaned using denatured alcohol and contact cleaner and dried. All common practice that yields 90% positive results.
After cleaning and drying the items are tested. The items in question are a Flashforge Dreamer 3D printer and a Cisco SG500X-48 Managed Switch. We cleaned and dried them and they tested fine. The switch was tested to power on and handle a little bit of network traffic. The 3D printer was tested with a 4hr print and a 14hr print both prints went well and the unit was on for about 48hrs.
Both the 3D printer and Switch were returned to the customer. A few days later he complained about the items not functioning properly when he tested them. The client is extremely tech savvy. I guarantee he knows more about circuitry than I do. His complaints were that one of the extruders on the 3D printer wasn't functioning properly and that the switch had a loud fan.
We picked up the items and brought them back to test. Upon return of the items the switch was plugged into power and the tech went to the bathroom. Literally 2min later one of the network jack arrays started smoking the board. It is a perfect circle under the first nic connection array. The 3D printer I had up printing via sd card. It was functioning well. Later I plugged in the usb to test other settings via the pc software and 2 min later more magic smoke and an IC behind the usb port fried and caused the unit to fail.

Now these issues are consistent with what water could be expected to do to a circuit. However they did this after we followed a very much practiced restoration process and after we tested them and the client tested them. This seems very fishy to me. It is highly uncommon as we do this all of the time. So my question is what could be done to these circuits or components to cause this? As in is there a way to tamper with the devices to yield the issues observed. The switch went first so I visually inspected the 3D printer prior to working on it. Please see attached photos. I saw nothing odd about the components of circuits involved upon visual inspection, I did not break out a meter prior.
I have attached pictures with the affected boards and components highlighted.
View attachment 100022 3d printer prior
View attachment 100023 3d printer prior
View attachment 100024 3d printer after
View attachment 100027 Switch prior
View attachment 100028 Switch prior
View attachment 100029 Switch prior
View attachment 100031 Switch after
View attachment 100032 Switch after
You need to seal electronics, you can buy special printers and a tamper proof label cartridge (Seiko is one! But there are others), that can print a special label, with your Logo and saying "Warranty void if damaged". These labels have a special glue on the back that is white and it comes off in neat squares, if someone is screwing around you can see it immediately!! Look here for Tamper Proof Labeling:-

http://labelprinters.sii-thermalprinters.com/security-labels

Also add invisible marking on components. I use UV ink. I have a small LED UV torch, if someone has smudged the ink, you know someone was in there, both by the warranty label being tamper proof and the smudged UV ink. The pens are cheap on ebay!!

I hope this helps....
 

NCSailor

Joined Jun 15, 2013
33
The chip that burned in the Dreamer is an ST Micro USBLC6-2SC6 http://www.stm32circle.com/resources/Datasheets/USBLC6-2SC6.PDF
It is an ESD protection chip for the USB port. As it burned on your premises, I would be more concerned that maybe you USB cable was at a different potential with respect to mains ground than the Dreamer was, or that your VBus was way out of spec.
I think it might be easy to burn that chip (and the ARM processor) with a high voltage to the USB port but I don't see how anyone could set that up so that it happened in your shop.
That printer is very easy to kill on that processor board as the driver board has a bunch of MOSFETs switching 24V to ground with 3.3V gates directly tied to the processor GPIO pins on the board that failed. Cheap fans that short out, extruder heaters with exposed connections near aluminum mounted thermocouples, etc. all have been known to blow a MOSFET or the TC amp and subsequently kill the MCU.
Graham
 

Tonyr1084

Joined Sep 24, 2015
7,900
MMMmmmm; lets see; how would I cause a board to fail? Well, some flux's are conductive, but they leave evidence behind that can be seen and tested. So maybe a heat gun on one or more components and cause them to fail from excessive heat. But I don't know that would produce what you've seen. ESD can cause damage. So can testing by a technician using a scope probe and accidentally (or deliberately) touch a power pin with an output pin.

I did some failure analysis many years ago (as a micro section technician). Had a chip that tested fine but failed in the work environment. When retested on the bench - it worked. The suspect chip was a hybrid that had a metal lid. Under normal work conditions it was expected that the lid would deflect up to 0.005". Engineering ordered the chips with a 0.015" minimum clearance. The customer's C of C stated the mean average clearance was 0.030". Nobody could figure out what was going wrong, so the project was scrapped. I got ahold of several of those chips and did some micro section work on them. I determined - and was able to demonstrate that the chips I cut open had a clearance of 0.0028 to 0.0032". Below the anticipated metal can lid touching the tops of the wires that interconnected the pins to the circuitry. The president, standing in my lab, put his hands in his pocket and said "We'll re-make the chips." The project came out of the scrap pile and went on to be successful.

Sometimes you just get lucky and can prove something. But without proof of the manufacturers defect our company was willing to scrap a multi-million dollar project, all for the sake of 10 chips valued at $1,000 each.

ESD usually results in an open circuit condition. So for someone to deliberately cause the board to fail; well, they're pretty smart if they actually did that. And like so many before me have said, it IS somewhat suspect that both units failed at about the same time. Still, stranger things have happened via mere coincidence. If the product is insured, I'd say "Don't give yourself a headache trying to prove what may be unprovable."
 
Top