Hi All-



I've designed my first embedded system which brings vacuum tubes and uC's together (a tube stereo receiver, part fun/challenge, part sick of not being able to get a good sounding stereo anymore for decent $$).



I was extremely happy at how well everything came together with only some minor bumps in the road. HOWEVER, after successful tests of every subsystem, a major problem reared its ugly little head and I can't hear my work: The system freezes when I engage the high voltage supply. Agh...So close!



I've been working for 2 weeks or so on the solution but nothing seems to be working. Worse, I don't have adequate test tools (100Mhz Analog scope, Fluke DMM, and a function gen).



What I think is happening is that the current surge from turn-on of the high voltage supply is causing a fluctuation on the digital supply. I think it may actually not be causing the AVR to freeze, but the I2C bus which the entire digital system depends on.



The system is rather complex, so I'll just try to cover the basics. There are 4 power supplies in the unit, and I know the offending unit is the HV supply from removing fuses. All of the power supplies are fed from an MLX power connector and the HV supply goes left, and the LV supplies go down ("L" shape to minimize problems like this...fail). The digital supply has a common mode choke for EMI/isolation, and is dual regulated (12V first, then 5V). All supplies have grounds connected together at 1 point on the board. The digital circuitry is almost always more than 8 inches away from the HV supply, so I assume the only good radiative coupling path is right at the connector.



I can both measure a small voltage drop (50mV or so)when the HV supply is activated on the digital supply, and see LED's dim briefly, but I have no way of capturing the event at the moment. I've tried using the min/max function on the DMM to see if it was fast enough to show a huge drop/spike but it's not.



I should also mention that the digital supply has very hefty 3900uF caps on the input and output of the regulators, and a 470uF between them so I'm amazed these caps can't hold the line steady.



I've already written a lot, so I'll wrap it up with the questions:



1. How do I track down the source of the EMI/spike with the equipment I have?

2. Are what are some things I can try around the uC that may be making it susceptible to this beyond the basic stuff (coupling caps...have them, B.O.D., Watchdog disabling, etc.)? For instance, maybe my unused uC pins are the issue?

3. I know software fixes can be the answer, but I'd like to solve the electrical first as software would be like putting a bandaid on things. For instance, I could likely setup the Watchdog to restart the system, but that's a poor solution.



I'm using the Fleury driver, but I can't seem to figure out how to modify it or my code so that if the bus hangs, to try what it was doing again once or twice. I suspect some of the while loops in that code are primed to cause lock-ups like this. Any ideas on that front? I believe there's a TI app note about this, but I can't seem to find it.



Thank you in advance! Sorry for the novel, I just want to provide the necessary information to anyone generous enough to attempt to help!0

 

Perhaps slowing down the high voltage supply start-up?

 

This is an interesting idea, one I haven't considered due to a lack of experience with Thermistors. One of the things I'm noticing right away is the load current on the HV supply is too low to be of use to the thermistor (~100mA) based on the percentage of rated current spec. If I'm reading that correctly, take the CL-210 for examples sake, at the load current, would that become essentially a PTC device?

 

Have ypu tried turning on the HV supply first then the low voltage power.

 

you can isolate the uC further with small resistor, a diode (for testing), and tantalum/ceramic capacitor close to it. Eventually a small RF choke just for the uC.

You may get stray noise into the I2C or the clock generator becomes disturbed.

For instance I have a digital thermometer- when I insert the tiny NTC into a cooling grid for a switching supply, at some point it will freeze (the internal controller crashing), in a way I must interrupt the battery supply.

Inserting a 330uH RF choke into the sensor line cured this problem.

Also another example, a PIC 16f59 with I2c temp sensor. It had trouble until I inserted a ringcore with magnet wire directly into the supply wire.

there are a lot of maybe's here but this is something you can try.

I had another SMPS prototype, a cranked up LM2576 (or 2675?), having a small control transformer, and in turn connected to a high voltage MOSFET.

When I'd fire a digital camera, the circuit would shutdown (under full load) just from the EMP!

Means I's say you have a section in your circuit which is EMI sensitive.

It can have other causes. Electrolytics are not so good, they are inductive and take a while, ceramic or tantalum can react much faster.

Or the I2C needs damping in some kind, can't really say without to see the circuit.

 

the load current on the HV supply is too low to be of use to the thermistor (~100mA)

Is that 100 ma at the high voltage output or is it more current in a lower voltage section of the circuit?

There are other ways to slow down or frankly delay the start-up, but we can't be specific without a schematic, and this is still a guessing game at this point, anyway.

 

@BReeves: This is a good idea but I intended to be able to switch the tubes on or off depending on my mood so the PCB's are hardwired for that. I have no doubts your suggestion would solve the problem however, I will keep this in mind!

@#12: Yes, the HV supply's load current is 100mA. One of the things I've tried is to solder a high value resistor (330k 2W) across the relay for the HVS to essentially "pre-load" it thinking the surge wouldn't be as bad with 50-60V already on the caps.

I agree this is a guessing game, especially not being able to see what's happening with my limited equipment. I'll work on a dumbed down PS schematic if necessary but I don't think there's any surprises to be found there, I'm not doing anything revolutionary, just a standard bridge-C-(electronic choke)-C supply with a cap multiplier feeding the tubes.

@takao21203: Thank you. I was considering getting some SMD ferrites pulling up the power pins of the uC and soldering them in place (it's an SMD uC) so that may be something to try. I also never thought about tants, maybe a couple of 1uF soldered to the decoupling ceramics may give a bit of extra stability to the uC if that's the problem.

I really think it is the I2C bus picking up the EMI, simply because I notice the SDA or SCL line stuck low along with some of the IO expander interrupt pins (which I was going to use to not have to poll the devices in order to detect a button/input but I will need to do some refining to get that to work). There's quite a few long runs in there from the micro to devices (3 x 12").

 

Hoping not to insult you with something so very basic, but a 10 uf aluminum cap and a .01 ceramic cap, right at the uc power pins is considered a necessity in most cases, even if you are not having start-up glitches. Have you installed these capacitors in your design?

 

No, not an insult! It's my first digital design (I know, I know, likely way too ambitious) so I'm still learning. But the shameful answer is, no. I don't have the electrolytics, but I do have 100nF ceramics right at the power pin to the GND plane. I just read about the 10-22uF electro requirements/recommendation an hour or so ago. Gonna try this right now.

 

#12: I had a bunch of 22uF 50V caps so I installed them....No change. I was really hoping that was it! I guess I keep digging...

 

A photo of your build might be helpful.

 

The oscillator circuit of the uC is high impedance so it can be easily effected by radiated interference. You could try a temporary shield to verify.

Possible sources of RF emissions:
1. the relay when it switches.
2. The tubes as they power up.
3. Relay contact switch bounce. Fix with RC snubber.

Does the uC hang the instant the relay switches?

Just guessing,
Ifixit

 

@gootee: would pics be better or a sort of layout diagram that cuts out a lot of the information overload potential?

I just got done playing around with the analog scope and found that I can observe a voltage fluctuation (not clearly however) when the switch is activated on both the digital power and I2C bus lines. The I2C bus looks to be about +/-0.5V!! I can also see that the I2C buss shuts down completely afterwards. Good news is that SCL and SDA waveforms look VERY good before the "Event".

So three questions come to mind:
1. In a mixed system such as this, especially in one where the power is fed through the same xfrmr (different windings obviously), is a surge inevitable? I assume the power surge is the typical 5-7 times the load current (500-700mA), so at ~430V that's between 225-300W for a brief instance. I feel like no matter how robust the design, that's going to cause some issues.

2.Since the I2C waveforms look so good normally, some filtering could be possible. Good idea?

3. If 1 is correct, then soft starting the HV supply seems like the only real solution. I'm envisioning something like this:

Where the relay was, replace with a Power MOSFET with drain to source in series with the + line between the bridge and either the filter cap or a blocking diode. A relay connects the drain to a resistor and the other end of this resistor connects to the MOSFET gate and a cap to ground. When the relay activates, the cap is slowly charged through the resistor, and the output voltage rises slowly along with the current.

My only gripe with this is that it's between the bridge and the filter cap, and that loop is the greatest source of noise in the system. I've worked really hard to make that loop area as small as possible.

Any thoughts on that?

 

Hi ifixit! Yes, it's instantaneous.

 

You should have posted pics and complete schematics on the second day. Most of those here would prefer too much information vs too little, in a situation like this.

It looks like a classic case of the need for decoupling. #12 already mentioned adding decoupling for the digital ICs. So there are three things that still need to be completed, for that approach:
1. Try beefing-up the digital decoupling caps.
A. Add a very large cap where power/gnd enter the digital system, hopefully close to the IC. Using several in parallel is better than one large one. Also add to or increase the 22 uF cap nearer to the IC. If there are any other ICs, they must also have two caps per power pin.
B. Actually calculate what value was needed right at the pins, using the rise times and max currents of a worst-case combination of outputs, and the spec for the supply voltage range, and use i = C dv/dt rearranged as C = i dt / dv, where i is the total current that might be demanded, dt is the minimum rise time, and dv is the allowable voltage dip. Also, to really do it right, you should calculate the maximum tolerable inductance, which gives you the maximum length of the dcap's connections plus lead spacing and if that can't be done then you'd need to start paralleling caps there. Just use v = L di/dt rearranged as L = v dt / di, where v is the allowable supply dip, di is the worst-case current, and dt is the rise time spec. If it's close, you should also account for the ESR.
2. If you haven't used proper construction techniques and wire dress practices, even the correct decoupling might not work yet. Inductance and enclosed loop area are the enemies. Inductance mostly equals conductor lengths, for this, especially in the power distribution system and any place where any fast-changing current needs to travel. Enclosed loop area means never separating flow and return paths, lest you make antennas that all time-varying magnetic fields will induce currents in, and which will also radiate time-varying magnetic fields whenever a time-varying current flows, with current and field magnitudes proportional to the geometric area of the gap between the flow and return conductors, all else being equal.

It sounds like you have a ground plane so that should help with the loop areas but depending on what stage you're at and the type of construction you might need to twist any wire pairs between boards, and any AC pairs in the supply. If you have any single conductors going anywhere by themselves, you're doing it wrong.

You will also want to double check your star grounding, to be sure that no conductor length is shared by the ground returns from the different subsystems. If all else fails you might want to temporarily use a battery or something for the digital supply, to verify. ALSO, any ground reference points for inputs or signals should have their own separate ground conductor, all the way to the star griund point.

3. Add decoupling caps for the other subsystems. You should probably roughly calculate their required values, as above, for both high and low frequencies. Or just go for massive overkill. It's usually easier to stop the problem at its source. Make sure that the decoupling paths' loop areas are small, also, so they radiate as little as possible.

 

Gootee, Thank you!!! This was a great post! I'm about to leave for Thanksgiving, so I'll be able to post pics and schematics then. A few things until then.

For 1A, the digital supply should be very robust. From the bridge, it looks like this: 3,900uF - 12V Reg - 470uF - 5V reg - 3,900/100nF ceramic and now every uC power pin and IC has 22uF/100nF ceramic. Additionally, I have two additional boards stacked on the main board, and each of those has a 470uF/100nF ceramic board entry caps.

1B: I have NOT however calculated as you described, so I'll do that ASAP.

2: I try to design everything according to the research I've done on EMI/EMC. Even the analog circuitry I try to use the same techniques because a good design at high frequencies should be a good design at low frequencies (to an extent). At least, I try to layout in terms of current/inductance.

In that regard, I always route power and ground traces either right on top of each other or right next to each other and make the traces as big as I possibly can. If possible I use the 3:1 rule (ground trace 3 times as large as the power run in parallel under/over).

One of the possible antennas though is how Phillips tells you to layout the I2C bus (although I can't think of a better way). They say to route as so: SDA, Vcc, GND, SCL. You'll see me doing this in the pics. This seems fine except for long runs on a PCB like I have.

ALSO, any ground reference points for inputs or signals should have their own separate ground conductor, all the way to the star griund point.

This is where I get foggy. I definitely do this after the last filter cap but I don't get the true star benefit over a series ground. If a power/signal source splits, it makes sense.

However, a power supply say from bridge to last filter cap, why is a star the best method? It would seem that the main cap will have two currents flowing through, one through the bridge, and one through the regulator/inductor/resistor and down through the next cap and so on. If we star that, we are forcing those currents to share a trace or wire for a longer distance than they would if we connected them in series. Series connection allows them to only share the lead of the component, which on a PCB can be very short. Am I missing something here?

Anyways, again thank you for taking the time out to help me. I hope you have a great Thanksgiving.

 

Maybe a high frequency dc/dc chip for the uC can deal with the EMI.

For instance there are some new 3 MHz chips, only 1uH inductor, and very small ceramic caps.

If you have a bridge/electrolytic for power supply, maybe the EMI from the high voltage supply can propagate through it easily.

I'd also try to supply clocking signals externally for testing. And see if anything is changing/improving.

Maybe the high voltage supply needs isolation/filtering itself.

I have built many uC circuits with switcher supplies onboard and always some modification/adjustment was neccessary.

You really need to isolate the problem, test everything, if the relay alone is not causing trouble, you don't need to change it.

If you insert a suitable coil into the supply path for the HV supply, it might give enough damping already and prevent a surge at the time of turning it on.
So it is handy to have various inductors, cores and coils around for testing.

Capacitors are needed but often only a few uF are enough, no need for large capacitors.

Example: Large LED matrix, 8 controllers, 768 LEDs, one small ready made LM2576 PCB for 24V -> 5V.

Each controller has one small 100uF electrolytic, and also a 47uF tantalum each is neccessary or the LEDs go crazy. Wires are very long, no common grounds or special care about layout.

There are also seperate crystal osc cans, routing high frequency over long distance is problematic.

 

liquidair,

Don't worry too much about the grounding in the PSU itself, yet. First, figure out how to place the star ground so that the voltages induced by the charging pulses, across the PSU conductors themselves, can't get into anything but the PSU. (ALSO, make sure that the rectifier/cap/transformer loop is as small as possible.)

That means that the star ground cannot be directly on a trace that is upstream of any capacitor. But it COULD be on a (short) stub trace, sticking out from the caps' trace, possibly. Or, it could be just downstream from the last reservoir cap.

[EDIT: I forgot that you have multiple regulators, too. Will need to think about whether they should have separate ground conductors to a star near the PSU or how that would best be laid out, and how that affects the star point location/topology for the downstream stuff.]

What I was mainly concerned about, besides the star point's location, was that you were fully aware of the option of running completely-separate grounds, for places like input signal reference grounds and anyplace else that might need a steady, quiet ground. (It might not even be at-all necessary, for your digital stuff. But I'll try to explain it, anyway.)

The input of an audio amplifier is a good example. If you just tied the "ground" end of the input resistor to the rest of the grounds, there, which would include things like the decoupling caps' grounds, and maybe the output/load ground if we were really inept, then, as they normally would, the time-varying currents in that ground conductor would induce voltages across the parasitic inductance (and resistance) of the ground-return conductor itself. Those induced time-varying voltages, in whole or in part, would then appear back at the NON-PSU end of every ground trace that shared any length of conductor with those other currents, on their way to/from the PSU. In the case of the "ground" end of an amplifier's input resistor, it would then be what is called a "bouncing ground", i.e. it would be a non-zero and time-varying voltage, and, since the amp's input is only "relative to ground", that voltage would effectively be arithmetically summed with the input signal of a high-gain power amplifier! Not so good.

If you did need to do that type of separate ground run, it might involve sectioning-off a part of your ground plane, if it was on a board with a ground plane. But IF it were needed, it could make quite an improvement in the ground voltage's stability, especially if the PSU was more than a few inches or cm away. But, typically, you would probably just use separate ground planes for the different types of grounds, and each ground plane would have its own conductor back to the star ground point.

Usually, people star the power, too, along with the grounds. If using star grounding, it seems almost required, to me, actually, to star the power too. With multilayer boards, it might mainly just mean that you should basically never "daisy chain" power OR ground between boards, or even between sub-circuits, often. i.e. Always bring a whole new set of power and ground conductors from the PSU, for any sub-system that will have any fast-changing, OR large, current demands, and for any that don't; and generally, separate power and grounds for separate functions, too.

 

Back from the holidays and I wanted to update those who had helped out, gootee, ifixit, takao21203, #12, and BReeves.

First, I "patched" the problem using a MOSFET soft start that I described in a previous post, it seems to be working well.

Second, I check my grounding and I did star ground everything to the filter cap that feeds whatever and the feed and return traces are always run right next to or under each other. Then each power supply is star grounded at the chassis, so we are good there.

Third, 3 a.m. last night I'm running through the circuitry in my head and I found the one spot where there's a loop and I didn't adhere to #2 above.

The unit has a mute circuit which is a P-JFET strapped across the volume control. It is controlled by the digital circuitry through a photo-transistor, so I ignored it. But to keep the gate high, it needs a positive voltage that I derived from the Aux supply. I should have done this from the HV supply (not directly, maybe a resistor-zener or something).

While that would normally only create a loop in the audio/aux supplies (still bad), I used an old transformer I had with only 2 windings for testing, so the digital supply is grounded to the Aux/Heater/Digital star which has to come from the first cap in each supply.

I don't know for sure if that is the culprit but it's a big enough error to make you go "hmm?".

So, again, thank you very much! There's still problems, but at least we are moving forward.