Regularly frying 48V POE powered ethernet switches with Fiber ports. How and why.

Thread Starter

rebuilder86

Joined Oct 20, 2021
9
I am an aussie working for a telco in Philipines, and on my weekends, my own project, I distribute interent to a poor community of villages in the remote jungle where my wifes family lives.

I distribute the internet over singel fiber optic drop cables over relatively short distances (anywhere between 100m to 3 km) and use cheap active reverse POE powered fiber optic devices which allow multiple houses within a 100m radius in a village to connet to the said switch by sending their own 48V POE ethernet cables to these cheap circuit board swithches. Apparently, each house supplies power to these things and the chips on the baord are able ot use bits of each hosueholds power to power the device fairly. This design is usedby Mikrotik in a reverse POE powered switch, which is unaffordable to me.

The circuit boards I use do the same thing but are insanley cheap form china, and look like this:
Type one, with 7 reverse 24-48V reverse (power in) POE ports and one POE out port, and 2 x single mode WDM fiber optic transceivers.

1634735052497.png
Type two, with 3 fiber ports and 2 ethernet ports only one of which is reverse POE.
1634735158844.png
And a bunch of other variants like that with minor differencess.

They are incredibly cheap, and thats how it needs to be, because these ppl make about 2.50 USD equivalent a day to live their lives.

Nearly every time there is a power outage, one , some or many of my switches at different locations get damaged in some way or another. Orignially I thought the problem had a pattern where after a lightning strike, I would loose connectivity on 1 or 2 of the fiber optic transceivers on the circuit board., but only those fiber ports that were pluged in and being used would fail. So that led me to beleive that the tranceivers were being burned out by overpowered light coming from the source. I figured that the power surge was overpowering the supply source (at my house) and burning out the receiver on the other end, due to the short distance and overpowered cheap china technolgoy. however viewing the trasnceivers with a cellphone camera, which can normally see the 1310nm light wave; after these failures, I could no longer see any light being transmitted from the trasnceiver, so it seems as if somehow a power surge manages to get into the circuitry, and burn out the transmitter part, but only on the fiber ports that had a cable plugged in. This makes no sense to me and i have no idea how that could be possible. How would the electrical surge only harm those ports that has a plastic SC fiber connector plugged in??? it just doesnt make sense. Somehow though, that pattern was 100% reccuring and i am adamant that the electricity was somehow only frying the fiber ports in use.


Fast forward to today, for some reason i keep installing new houses, knowing that its going to fail, ppl are just deseprate for connection to the ouside world, so i continue, and low and behold, a power outage strikes 3 days after i connect this new hosue, and a brand new 900 peso (20 USD) 3 port fiber switch is fried to the point that absolutely nothing on it turns on any more. Just a quick flash of the power LED and then dead. so this is the first time that one of these surges has managed to kill the device to the point it wont turn on..

I've had some other simple fiber devices fail while being powered by 5 - 12 volt power supply packs not only by the revesrse POE method. These are standard media converters and failed in the same manner, no light visible through the phone camera after the failure. Also, some of these power packs were damaged by the events, where the power supply pack has nearly electorcuted me to death after its y cap failed short-circuit and leaked 100% 230V through to the dc jack that plugs in to the device. this has happend to abotu 3 of my 50 purchased units. SO ive had to start instalign them in plastic boxes so that residents cant touch them if they failin such a dangerous way..

I intend to not jstu throw money at the problem. IM hoping someone here can guide me to decide hwo i can investigate whether the issues a re transient quick major surges from the lightning, or from the current rush on resumption of power after an outage.

Whichever it is, i am going to need to do some insane amount of engineering to fix the issue, because none of the hosues involved have any kind of earth pin on their power sockets! So if i need to buy frikn SPDs for every install, its going to have to be some kind of plug in 230V SPD with an auxilliary each connecton for a random hack job. Terrible. I cant start doing electrical works and rewiring their houses to install din rail SPDs, onto their bamboo distribtuion boards, to not fry my gear that i install practically for free for them. IT takes about 2 months of servcie revenue from the customer to pay back the instalation costs, I keep it affordable and use my international connections tobring the world to them, im that kind of guy. So in short, i cant be looking at automaticservo voltage regulators or UPSs for these ppl, they are dirt poor. whatever solution i come up with, it needs to be correct and well investigated. and i just dont know how. i do internet, not electrical fault analysis. hehe

Anyone that can help, all i can promise is good photos of happy people as a result! :)
 
Last edited:

ElectricSpidey

Joined Dec 2, 2017
2,758
Yea, power outages are notorious for damaging electronics, depending on the cause of the outage.

POE is not really in my wheelhouse, but a concept that comes to mind would be to place a module in between your devices and theirs that clamps the voltages.

I would also look into surge arresters for each power pack plugged into a mains supply.

Without proper grounding at each house, the problem could be real expensive to solve.
 
Last edited:

Thread Starter

rebuilder86

Joined Oct 20, 2021
9
so when you say "clamps the voltage", are you reffereing to one of these simple voltage cut off devices? ther are some semi affordable "voltage protectors" available here which state they do what i need, but im not convinced, becasue if the solution was this cheap, why would there still be business for the manufacturers of SPDs and AVRs? is this what you mean?
https://panther.ph/product/pvp-2500-panther-voltage-protector/

THey say "instanataneous cut off", but this country is known for hopeless prodcuts, and missleading advertising , and general nature of fooling the population into filling their pockets. So i jsut doubt everything really. If im getting a transient voltage surge, like that which occursas a spike on the sine wave, then i find it hard to beleive that somethign like this could do the job of what someonelike schneider charged 4 times to cost of this for, a metal oxide varistor.

THen agin, if its jsut a power fluctuation then i see thtat this would help. i just dont know how to investigate what is happening. IS it the kind of thing that industry experts jsut throw money at and protect by all means known to mankind? Or is it posible to test a circuit board, like test something on it to see if it has suffered a long period of over/under voltage or curent, or just a zap from a transient.??
 

Thread Starter

rebuilder86

Joined Oct 20, 2021
9
ok so that's what ive been thinknig too. i suspect, and imagine that the most damaging surges are those transient quick microsecond surges which require dissipation to earth via a MOV or gas tube... yeh?
So would anyone on earth have the expertise to analyse a failure like mine and work out if tis a quick transient problem or a long lasting voltage sag on resumption of power with associated current surge, or.. BOTH? do ppl get that smart? :p
Do people have that kind of expertise or is it something you need to throw money at? I hope that queestion makes sense. im tryign to work out whatis the extent of my lack of knowledge.
 

Thread Starter

rebuilder86

Joined Oct 20, 2021
9
is it any clue or hint, that when these failures happen, the POE injectors continue to function and its just the circuit board at the end which ends up with failed components? Can that maybe indicate that its either a transient or a prolonged surge?
And also, those few transformers that haveshocked and nearly electrocuted me due to failed Y caps. What would be most likely to cause that, to become so dangerous. A transient? or a long surge?
 

michael8

Joined Jan 11, 2015
410
I don't know what is really happening, but here's a view:

This view assumes that the fiber is non-conducting, that only light
travels via the fiber and there isn't anything conductive along the
fiber path like a wire to aid using a metal detector to find the fiber.

So this view assumes no spike travels via the fiber.

Wired non-POE ethernet has a transformer in the path at each end of each
link providing DC and low frequency AC isolation. This includes 50/60
cycle AC power isolation. These transformers can isolate voltages in
the KV range:

https://www.pulseelectronics.com/network-signal-transformers/ethernet-transformers/

A POE input port still has the ethernet transformer, however the POE
power is taken off from the non-isolated side of the transformer.

https://en.wikipedia.org/wiki/Power_over_Ethernet

So a voltage spike on a POE entry wire is directly hitting the circuitry
on the POE device.

In properly wired single building, a nearby lightening strike causes
the whole building "ground" voltage to rise, possibly to a KV or more.
Since it all rises together there aren't major voltage differences and
nothing gets hurt.

With your POE power coming from different buildings, any one could
rise to KV type levels relative to the others.

Other than lightening, the 230 VAC POE injection power supplys could be
arcing over inside and "providing" 230 volts (or spikes from "normal"
power lines) to the POE switches.

Perhaps having only a single POE feed to a switch would allow the
ethernet transformers to provide enough isolation to allow the
switch to live.

link to possible reverse POE switch:

https://www.aliexpress.com/item/4001291795976.html
 

Thread Starter

rebuilder86

Joined Oct 20, 2021
9
thanks michael , im now significantly more informed thankt to that reply! and yes , thats the exact switch, Alibaba owns a philippines e-commerce site called lazada, and thats where i get them!, so its that exact one, same sellers name even haha.

With our Reverse POE setups, the longest POE cable cable is about 50 m, all the houses on a single switch are within a stones throw of each other, you can stand on the customers doorsteps and talk at normal voice level to the other customers. And i geuss i figured that would mean that they should all be pretty close to having the same ground voltage. IT jsut blew up on a setup where there is only one customer on the switch, and its has done that for many of our non Reverse POE setups too, so it feels to me like that means its nto abotu differences between different buildings. but its possible that there are multiple failure modes going on here, so im not going to say its NOT that jsut because it failed this time with only one POE injector involved. IN fact, this time, wtih only the one POE injector involved, tghe failuirewas a bit different, it fried the entire switch, not jsut the fiber port, so thats an other intereting factoid. cant get my head around what that might mean.

SO yeh the fiber is indeed totally non conducting, only glass and ployethylene and plastic transceivers involved everywhere, no chance of the long fiber cable having any thing to do with the electricity. the steel messenger wire terminates a good 2 meteres away. I know sometimes ppl make errors of judgement, but im quite confident its related to the POE input. hell it could be as simple as the cheap china POEinjectors and 5 volt power transformers that have been involved so far, they could just be junk with junk Y caps leaking the current through after a brown out when the voltage drops.
And that combined with the fact that the inpuit port takes the DC POE power in form the unisolated side, means that it possibly is related to a stupidly high spike from nearby lightning.

Wiht my limited understanding, i definitely originaly thought it was the ground potential rising during a lightning strike adn zapping my gear. but then recently its become evident that its happening and ppl are reporting there was a power outage at the time. so i have no idea if its the storm, or the resulting power outage adnreturn of power hehe.

Thanks so much for those sources, im balls deep reading now!
 

michael8

Joined Jan 11, 2015
410
the longest POE cable cable is about 50 m, all the houses on a single switch are within a stones throw of each other,

50 m is too far, 1 m could be too far. Look up step voltage:
https://greymattersglobal.com/step-and-touch-potential/

I would definitely suspect the quality of the POE power supplies.
I had cable modem problems caused by a failing external switching supply.

Can you source a few equivalent "old style" power supplies which use real 230 VAC transformers to see
if it helps? So a single POE feed from a "good" power supply...

You might also check what the output of one of the POE supplies is with a (dummy) load matching
one of the switches and with an input voltage other than 230 VAC.

Lower input? HIgher input? Higher spike?

What does the POE supply do when not feed "real" 230 VAC?
 

Thread Starter

rebuilder86

Joined Oct 20, 2021
9
50m really? damn ok, back to the classroom for me. hah.
I like the way you add sources to your replies man! It means ur trying to teach and educate rather than trying to sound clever! Can you please learn everything else abotu everything ever and join every other forum i frequent lol.

Not sure if i would be able to find an old style linear power supply here, if thats what you mean? id have no idea what to ask for, and i dont think ppl at stores anywhere in the country would have a clue what im talking about. im really on my own out here on this archipelago.

Ive been doing a lot of reading, (because I also consult for this kind of thing in my line of work), and some sources sugget that for solid state equipment, surge suppressors (protectors that dissipate the surge as heat or similar) are better than surge arresters (MOV and other things that shunt it to ground) due to the fact that the total power that can be clamped is greater (longer duration) than an "arrester".
https://www.brighthubengineering.co...-surge-arresters-for-industrial-applications/
It sounds like major critical telecommunications facilities (fiber optic transmission facilities) would require both, an arrester at the main panel (type 2 to handle the microsecond lightning related surge), and a supressor/protector at the power strip (type 3 to handle the longer surges)! so for me and my personal project, im not going to buy both lol, these customers pay me about 2.50 usd equivalent a month.
 

michael8

Joined Jan 11, 2015
410
old style linear power supply

I was thinking of a supply which looks like the one which comes with and has the same output voltage but weights more
because it has a real 230VAC transformer in it. The thought was perhaps you could find an old one just to
try it, I wasn't expecting you could buy a new one.

I'd certainly test out both a new & used switching POE supply with under/over input voltages. I'm not sure how to test
for "momentary" overvoltage output events. A single event might kill the switch but leave the POE supply
output testing as working.

Oh, how old are the POE supplies driving the failed switches? Perhaps they just age out (capacitors drying out
or end of life?) and then the output goes bad. Bad might be excessive output ripple to too high a voltage...

This might be a problem which takes collecting statistics on which ones failed, environment (power failure?, lightening?)
what was damaged etc. There could be more than one problem...
 

Thread Starter

rebuilder86

Joined Oct 20, 2021
9
everything involved has been started this calendar year. purchased adn installed this year. gota huge box full of broken stuff and a bunch of local villagers who think i dont know what im doing. its there lcoal elecricity suppliers faault, and soehow ive pushed myself into a position where i have to keep spendinging money on switches to supply them with internet, because of their dirty power.

I think theres a problem with using a different power suppy though... the problme is that 48 volt poe injectors do, like a series of handshakes before supplying power, so a normal dc power supply of any variety wont do that. saying that, i guess i could jsut man up and forget the formalities and handshakes, and jsut give it 48 volts or thereabouts and tell it to "deal wtih it" :p
 

michael8

Joined Jan 11, 2015
410
Ouch, you're trying to help with limited resources and riding the
learning curve finding out the "real world" isn't plug and play.

Here's tonight's thoughts and questions (and the possiblity you might
gain some more information from your box of broken stuff).

The link I found to the "switch" doesn't describe the POE
power injectors and now I realize it isn't a reverse POE switch.

What does your POE power supply look like? link?

If you are going to run ethernet cables (wires) between houses you could
gain the most isolation from the ethernet transformers (minimum 1.5KV
isolation, might be more) at each end if you dropped the POE.

The switch would be in (near?) one house which would supply 5v power
(at up to 2A) via the switch power input connector. Possibly there could
be some sort of UPS on this power and/or power transient suppressor or ?

To avoid the reverse POE "feature" from bypassing the ethernet transformer
isolation requires switches without the reverse POE feature.

As an alternative to new switches I'd explore the possibility of cutting
the traces on the reverse POE switch which connect from the ethernet
cable side of the ethernet transformers to the reverse POE function.
(see below for the possibility that a "failed" switch might work?)

This cut has to be able to block the 1.5KV or more, not a minimal
cut in a trace -- I'd try to take out something like 10 mm (to get >
1.5KV isolation). Also I'd cut the traces for every port being used
(or just all).

search target: pcb creepage distance

https://www.smpspowersupply.com/ipc2221pcbclearance.html

Another thought is to do more of a postmortem on the power supplies and
switches which have failed in an attempt to determine more on what is
really happening.

What test equipment do you have available?
- digital multimeter (model?)
- lab type power supply (model?)
- misc power supplies (working), like 5V 2A? likely switching
- oscilloscope
- cable testing capability?
- pc / network monitoring? (record when home systems go up/down)
- other?

Possbily some of the failing switches have just had the "reverse POE"
internal power supply (likely a switching supply) fail. The fiber
and ethernet switch might still work? Have you tried one via the 5V
power input? I'd try a current limited lab type supply, the power
input might be shorted... If shorted perhaps the POE supply can
be disconnected (cut) from the 5v supply path? This is a different
cut than the ethernet isolation cut above and just needs to break
the connection, not isolate 1.5KV+.

Even if such a switch can't be used, knowing what is failing on the
switch might help determine what is causing the failures.

Other:

Possibly one bad power POE supply (or it's AC source?) of several
powering one switch kills a switch. When you move in a new switch it gets
killed too. Tracking this down would seem to require keeping track of
which supply & switch was where over time.

Some of the failed units might be just "normal" infant mortality or at
least not related to power events. Assembly problems: solder joints,
PCB through holes opening... Some might be driven by temperature
changes & thermal stress.

Changes for less impact on failure:

This could be more fiber links and/or fewer houses per switch.

Or use shared WiFi for a group of close houses for a backup path for
when the wired ethernet dies? (I know WiFi isn't the same but it's
likely better than nothing if the wired path fails.)
 

michael8

Joined Jan 11, 2015
410
I could no longer see any light being transmitted from the trasnceiver,
so it seems as if somehow a power surge manages to get into the circuitry,
and burn out the transmitter part, but only on the fiber ports that had
a cable plugged in.


Likely the non-active ports are powered down to save power. A connected
and powered at both end fiber link likely (best memory) is actively sending
and receiving some sort of idle traffic. If there's no other end there
wouldn't be traffic (or very minimal which you might not see? not sure).

A power spike might only take out active circuits.

It wouldn't take much to stop the laser from lighting. A problem in anything
including the on board power supply, the IC driving the laser, or the laser chip itself
could cause no visible emission.

It's interesting that diode lasers are very quick to die when overdriven (but this
may not be your problem):

https://www.newport.com/medias/sys_...49454622/AN03-Protecting-Your-Laser-Diode.pdf
 

Thread Starter

rebuilder86

Joined Oct 20, 2021
9
yeh they get overdriven quickly, and im not sure if thast the issue. it could be, but the way that the ltihgt no longer emits from it after the failure makes me think that its actually the transmitter.

OK so, i like the idea of getting rid of poe actually. im considering a cheap deep cycle battery and solar charger at each of the swtich locations, all 8 of them. These cheap circuit board switches can be purcahsed in a simple 12v dc opton, like the one you say you found, it is almost identical to the RPOE version in apearance. The reason i went POE was it was the cheapest capital expense, after my highly paid network engineer at work showed me the mikrotik expensive RPOE outdoor switch, where i did some googling and found the hilariously cheap circuitboards that do the exact same thing. haha mistake number uno, was perhaps not buying this expensive solution. https://mikrotik.com/product/netpower_lite_7r#fndtn-specifications
I also didnt want anyone to queastion why their house would be supplying power to a switch that supplies ther people, so this technology promised to blance the input so that i could say that everyone is paying their fair share of thetiny electricity cost. they are so poor, that one guy has been constantly unplugging his POE injector because someone told him how it works. they want it FREE. the 2 us cents for electricity is scary to them :(
But yeh its getting to the stage where im going to have to take action and change.
Each location has a more significant more important "user" conneted, a central wifi vending machine where all the lower income ppl put a piso coin in and get 5 minutes of low latency, 4 mbps internet. I could just bight the bullet and make these machines the power supply hub sites and just force the owners of those houses to be the sole power supply of the switches for a while and monitor the outcome of that. that would be a free test of the theory that it could be differences in ground voltages.
Ive got a feeling that its still going to fail, we installed a house last weekon its own, with its own reverse POE swithc on its wall, and it died after the first power outage 3 days later.

My POE power suplies, please dont laugh, remember im in a poor place. i get the US 48V 0.5 amp variant.
https://www.lazada.com.ph/products/...-us-eu-wall-plug-i2166716434-s9690945584.html

ok, so post mortem, id love to be taught what to do mate.

When the big switches break, its just the fiber ports that fail. the RPOE still turns on the switch, and the switch stil functions on copper ethernet.


What test equipment do you have available?
- digital multimeter (model?) UniT digital (UT33B)
- lab type power supply (model?) nope
- misc power supplies (working), like 5V 2A? likely switching heaps, but all terrible china quality
- oscilloscope nope, but want to buy for work reasons, would love suggestion on what to buy
- cable testing capability? no actually, and this is terrible, i crimp cables then visualy inspect, and they just work
- pc / network monitoring? (record when home systems go up/down) Pinginfoview, only tells me currently whos up adn down. nothing to show me timestamps of when they go down, just word of mouth
- other?
 

michael8

Joined Jan 11, 2015
410
A lot of thoughts, I'll break it into pieces.

the way that the ltihgt no longer emits from it after the failure makes me think that its actually the transmitter.

My mental model (could be wrong) of the reverse POE switch is that it takes in the "high" compared to "logic power"
voltage (48 volts or whatever) via reverse POE and the switch contains a switching power supply to switch this down
to a "logic power" voltage of 5 or 3.3 volts.

Possiblities are that this internal switching power supply gets overwhelmed and lets it's output raise and
the optical module/section tries to limit or use this extra and fails (possibly protecting the rest of the
switch or else the rest of the switch isn't a sensitive to the overvoltage).

Or perhaps the optical section needs a higher voltage and does it's own switching from the POE power -- it might
be possible to determine some of this by looking/tracing out some of the power supply wiring on the switch circuit
board.
 

michael8

Joined Jan 11, 2015
410
Ive got a feeling that its still going to fail, we installed a house last weekon its own, with its own reverse POE swithc on its wall, and it died after the first power outage 3 days later.

I'm assuming this was a reverse POE switch. Is this the "laser" failure & switch still works?

Will the same switches run on 24V instead of 48v if the POE ethernet cable losses aren't too high (short cable)? This might give
more margin somewhere if the problem is a small increase in the 48V (a small increase in 24v would still be < 48v).

Power quality: I found this searching for: Philippines power overvoltage

https://electricalengineerresources...Philippine-Distribution-Code-2017-Edition.pdf

Section 3 describes various ranges of voltages and times. It's talking about minutes of 400+ volt AC so varisistors
aren't going to deal with out (just cook or blow a fuse). I'd also be concerned with low voltage cases. When you have
a constant power load (the switch) being driven by a switching regulator the input current goes up as the input
voltage goes down. So a low input voltage is also a stress point for the power supply (POE injector on 230 VAC).
 

michael8

Joined Jan 11, 2015
410
A lot of thoughts, I'll break it into pieces.

the way that the ltihgt no longer emits from it after the failure makes me think that its actually the transmitter.

My mental model (could be wrong) of the reverse POE switch is that it takes in 24 or 48 volts via reverse POE
and the switch contains a switching power supply to switch this down to a "logic power" voltage of 5 or 3.3 volts.

Possiblities are that this internal switching power supply gets overwhelmed and lets it's output raise and
the optical module/section tries to limit or use this extra and fails ( possibly protecting the rest of the
switch or else the rest of the switch isn't a sensitive to the overvoltage).

Or perhaps the optical section needs a higher voltage and does it's own switching from the POE power -- it might
be possible to determine some of this by looking/tracing out some of the power w
be
 

michael8

Joined Jan 11, 2015
410
ok, so post mortem, id love to be taught what to do mate.

It's likely more learning then teaching. It's building a mental model of what the board is doing and comparing your
mental model with reality and fixing the differences. You could start by just looking over the board and
trying to figure out what most of the parts are or are used for. Some of this can perhaps be found by looking
them up onilne or by what they are connected to on the board.

It takes understanding what is supposed to be happening compared to what is happening to recognize a failed
section. What failed (and how it appears/tests after failure) might hint at what is causing it.

When the big switches break, its just the fiber ports that fail. the RPOE still turns on the switch, and the switch stil functions on copper ethernet.


That's a big clue. I'd try to figure out the power wiring/path on the switch board. On an unpowered board I'd try to follow
the traces from the external side of the ethernet transformers (visually and/or ohm meter on low ohms range: not enough to
pass a diode junction). There are likely diodes early in the path which won't pass an ohm meter so visual is a definite
part of this. As is thinking of what the parts do. I'd expect to find all the POE ethernet connectors going to ethernet
transformers and some tap on that winding(s) being commoned (before or after diodes?) to the input of the
POE to "logic supply" switching supply on the switch board.

I try to figure out how many power supplies were on the switch board. There's at least one for the POE input power,
perhaps more. (hint: count the inductors? each supply has at least one)

Once some of the power paths are known, including ones to the optical section, I'd measure the voltages at various
points (where I expected it to be stable like across capacitors, not switching supply switching points) with the switch
powered. Possibly the optical section doesn't have power...

Some of this might be hard if there are traces inside of the board...
 

Thread Starter

rebuilder86

Joined Oct 20, 2021
9
omfg, 24 Volts instead, thats so stupidly simple and yes it can run from 24 volts. in fact, I originaly powered one of these wiht a 24 V POE injector and, for that entire time, there was no problem. I jsut changed to 48 V because i thoguht that woudl make the POE injectors run at a lower temp, because less current. I think your onto it here, your points about the switches internals getting confused, thats the kind of impresion i got with it. overpowering the transceiver and burning it out, and at the same time, destroying the receiver at the other end by blinding it with overpowering light.
 
Top