MaxHeadRoom
- Joined Jul 18, 2013
- 30,664
You're positive there was no spike that occurred at some time before the outage?
Max.
Max.
No I'm not positive of anything. But that's officially my position, as I just sent this to the owner of the company at the request of my boss (I think for the purposes of an insurance claim).You're positive there was no spike that occurred at some time before the outage?
Max.
Thanks for all the help!****,
Per ****’s request this is to document my assessment of the root cause of the failure of 7 out of 9 Allen Bradley PowerFlex 40 VFDs in the conveyor control panel on the PVC line at the **** facility. I had to do a fair amount of research to reach any conclusion at all, as a mass failure on this scale is unprecedented in my experience, absent any obvious smoking gun like a direct lightning strike to the supply transformer. The conditions leading up to the failure were as follows:
I previously knew of, or discovered in the course of my research, the following causes of failure of aluminum electrolytic capacitors like those in our VFDs:
- A shift ended and the equipment was left powered on as usual. The operators always leave equipment powered on because on most lines it causes machine bugs to arise when powered off and then back on, that have to be flushed out by maintenance.
- Temperature dropped, causing excess electricity demand and rolling blackouts.
- Power lines iced over, causing transformer explosions and prolonged blackouts in several areas.
- Temperature dropped to record lows (12˚F I believe, locally) as blackouts continued across the region
- After blackouts and freezing temperatures were over (about 4 days later) the next crew came in and found the panel inoperable.
- 6 out of 9 VFDs were dead, with capacitor electrolyte dripping from them.
- The 7th VFD failed to come back on after I cycled power in the course of troubleshooting.
- Desiccation of electrolyte from prolonged periods of disuse
- The electrolyte in the capacitors will “dry out” after prolonged periods without power applied.
- VFD manufacturers typically prescribe a re-forming procedure for drives that have sat on the shelf for more than 2 years
- Our drives were only powered down for 4 days so I do not think this is the cause.
- Extremes of temperature.
- The Nichicon capacitors in our VFDs are rated -40˚C to 105˚C.
- We did not go outside that range. I do not believe the cold to be the cause.
- Life expectancy exceeded.
- The capacitors in our drives are rated at 20,000hrs.
- The date stamped on the drives was 10/14/2010 and I believe the line was commissioned in 2011.
- Assuming an even 10 years at 75% utilization, we have put about 66,000 hours on the capacitors. However I still do not believe this to be the cause because:
- The 20k hrs life expectancy is the time in operation until the capacitance value drifts outside the tolerance specified by the manufacturer; not the time until the capacitor explodes and sprays electrolyte all over the place. Time to ultimate failure like we witnessed is not provided as a specification, but is many times longer than what the manufacturer specifies as its “life expectancy.”
- I have seen VFDs in service 2X-3X longer than these were in service before similar capacitor failure
- The timing for them all to all fail on the same day (or in the same 4-day span) after 10 years of service without issue, is too perfect. There would need to be a planned obsolescence timer in the VFD set to make the capacitors explode after X hrs.
- Over voltage condition
- Long periods of time spent slightly over the rated voltage will cause failure of the capacitors.
- Mains voltage in that panel is 489V, which is well within the ±5% steady state voltage tolerance for nominal 480V
- I do not believe mild/prolonged overvoltage is the cause
- Brief periods of time spent well in excess of the rated voltage (spikes/surges) will cause failure of the capacitors.
- By process of elimination, I believe this is the cause. It is the only option left.
- This is exactly the kind of carnage I have observed after nearby lightning strikes.
- We don’t know with certainty what happened to the line voltage throughout the time that the facility was unmanned as we don’t have any data logging equipment installed, but we know that there were rolling blackouts, downed power lines, and transformer explosions recorded all across the area, and one of those events is the only plausible cause.
- Most likely it was the direct result of the rolling blackouts, caused by the electric power provider. Here is an excerpt from the document linked below:
“A blackout starts a series of related electrical events that can significantly damage or degrade electrical and electronic equipment. When the utility company reapplies power to its systems after a blackout, harmful power fluctuations, including power surges, are created. Power surges assault circuit boards, control logic power boards and other components in electrical and electronic devices, causing them to malfunction or fail. [...] Depending on their source and magnitude, surges can cause immediate, catastrophic damage or the continuous degradation (latent damage) of electronic systems and components.”
That description best matches our observations in my opinion.- https://www.nemasurge.org/wp-content/uploads/2015/01/rollingblackouts.pdf