Just to make sure, can you post how the grounds are hooked up? Just a block diagram showing every time ground leaves a major component. For example: Start with frame ground, 5 volt ground, your circuit ground, your sensor ground and the ground for what your circuit drives.
We wouldn't want to just treat the symptoms and have the beer cooler down the hall create the same problem.
Is the sensor shielded wire?

 

Absolutely! Feeling kind of swamped today, so I'm not sure how soon I can get something uploaded, but I really appreciate the help. I definitely want to beat this problem at the source(s), and I wouldn't be surprised if there are multiple problems to fix (relay arcing, improper grounding, inadequate filtration on 5v boards, who knows what else!) Thanks!

 

Sorry my sketch got a little cluttered, and my dark green pen for ground wires scanned as basically black. Oops!

Basically the power cord enters the electronics enclosure and the ground immediately goes to a 1/4"-20 bolt with star washers binding it to the enclosure. The same bolt also carries an external wire to the chassis ground, a wire to the power board (5V ground and chassis ground are common,) and separate grounds for each pump motor.

The pump motors are external, connected with 16-3 SOO (or SJO, I always mix those up!) cord, and the motor bodies are grounded through the #16 wire back to the ground bolt.

Most of the 5VDC electronics and their power supply are on the main power board inside the electronics enclosure. The exceptions are the solenoid switching circuits, which are located throughout the machine right next to each solenoid valve (4-5 per machine.) Each of those picks up a ground via #22 wire and a ring terminal attached to the nearest available chassis point.

My experimental hall-effect sensing circuit is currently set up in the same way as the solenoid switching circuits, picking up a nearby chassis ground.

I've experimented with running independent ground wires all the way back to the main grounding bolt for one of the solenoid switches and for my hall effect circuit. Both circuits showed MUCH greater resistance to apparent noise pickup with grounds going back to a central point. I say "apparent noise pickup" because the only thing I have to go on for noise identification is tiny, dim blinks on the LED indicators of these circuits when a pump motor is switched on. Each circuit showed blinks upon motor activation ~30-40% of the time with the standard grounding arrangement, and dropped to ~5% with new grounding, and the few blinks that remained were even dimmer.

The hall effect sensor wires are not shielded, nor are the switched 5V signal wires which run from the power board to activate the solenoid switching circuits.

As of right now, the grounds have been put back to their standard chassis ground arrangement (more sensitive to noise) while we try the AC snubbers in parallel across each motor. The snubbers seem to deliver roughly the same improvement as the modified grounding scheme, dropping the LEDs to very low errant blink-rates, but not eliminating the blinks entirely.

 

Sounds like a big machine.
For discussion lets call the frame of the machine frame ground and the electronics enclosure DC ground.
For sure you want the hall effect board to get it's ground from DC ground - best if you can get it right where it goes into the pc board, but the screw is better than out on the frame.
It's hard to tell if it is picking up radiated noise, but using shielded cable or at least twisting the wires for signal, ground and voltage from the sensor might help, especially if they are a long ways away from the board. If this helps we can come up with some better filtering than just the .047 cap.
Is your circuit inside the box?
I'm not sure why they didn't return the motor grounds to frame ground, but may have had a reason.

 

Thanks for your insights. I'll definitely push for getting all of our dc circuits (solenoid switches and hall effect) grounded to dc ground instead of frame ground. Does each dc circuit need its own wire back to dc ground, or could multiple dc circuits share one ground wire back to dc ground, joined together at terminal blocks or soldered junctions? Running one extra ground wire is easy, but 8 extra (what our largest machine would need) is impractical.

Right now the experimental hall sensor is mounted to part of the frame and the comparator circuit is external, not in the frame nor the box. Its wires run alone in unique paths and I think they're relatively safe. Also, the solenoid switching circuits experience the same noise, and they don't rely on any analog signal like the hall sensor, so I don't think that sensor wire is adding much to our noise vulnerability. If we decide to use the hall effect system, tiny smt boards will be made with sensor and comparator circuit on one pcb, so the analog signal will never travel through wires anywhere, just 5v and ground in, and two 5v push/pull outputs going back to the power board.

I believe the motor grounds run that way for two reasons:
1) The grounding bolt in the box is the most direct path to the "real" ground coming from the power cord. Frame ground only gets to the power cord ground via the #14 wire connecting the frame and the electronics box.
2) The cords for each pump already enter the electronics enclosure to hook up to their respective relays, so it would take more wiring to run them to the frame.

 

So it sounds like the best we can do is to supply the sensor ground from the DC ground point and not the frame, some snubbers across the noise makers and good decoupling on the power on the sensor board.
I haven't looked real close at your circuit to see how much hysteresis it has but that might be another option if you can afford the dead band.

 

So it sounds like the best we can do is to supply the sensor ground from the DC ground point and not the frame, some snubbers across the noise makers and good decoupling on the power on the sensor board.
I haven't looked real close at your circuit to see how much hysteresis it has but that might be another option if you can afford the dead band.

Sounds spot on. Can't really afford any more hysteresis, but since each of the other three solutions seems to get me 100% functional and 95% noise-free, I suspect(hope?) that the combination of all three (assuming I can sell my boss and co-workers on the extra parts/labor to do it right) will be more than adequate.

It looks like the snubbers we already stock as spare parts for an earlier version of the machine work well to reduce noise generated when the pump motors start. And the two filter caps recommended above (10ufd and 0.1ufd) made a world of difference on the DC inputs to my hall effect comparator circuit.

So now I just have to fix the grounding as best I can. I finally finished reading the Siemens paper referenced earlier in this thread. It is dense reading, but has lots of great info in it. In order to get a better understanding of star-grounding and acceptable variations on it, I also read this grounding tutorial. So my big question now is whether DC grounds can tie together in a ground bus which shares one common line (insulated from frame) all the way back to the main DC ground point, or if they each need a separate wire all the way back.

The easiest grounding solution for us would be an extension of our current power-rail system. Right now we have 3-position barrier strips like these at roughly 1 foot intervals across the upper front of the machine's frame carrying L1, L2, and +5VDC lines so that we can pickup those voltages as needed for solenoid valves, reed switches (hopefully soon-to-be-replaced by my new hall-effect circuit) and a piezo switch. We could switch to 4-position barrier strips and run a common ground bus there, which would make running a non-frame ground to all these different items pretty easy. If we really need to run ground wires separately for each device/circuit all the way back, it would require a larger conduit (or a second conduit) and a lot of additional labor, and I would have a really hard time getting people convinced that it was worth the effort. Which leads me to ask, would it be worth the effort, or would a ground-bus solution work just as well, as long as there are no ground loops and it doesn't rely on frame-ground?

 

In an ideal world what you would like is only things like motor cases tied to frame ground and only one wire connecting frame ground to DC ground. It is good practice to avoid having something that draws high current or is noise prone (like solenoids) on the same ground as sensitive signals (hall sensor).

 

Cool, thanks! I'll upload a diagram of what I have in mind for shared 5V grounding paths sometime soon (too sleepy now, gotta work early tomorrow.) I think what I have in mind meets your criteria, but I may be misunderstanding something still.

 

So here's my idea for our new grounding scheme:

The solenoid valve bodies and the solenoid coil cases are already grounded to frame ground (by copper pipes and mounting hardware.) Hypothetically, if the "solenoid switch" circuits (essentially SSRs) keep solenoid noise from feeding back into the low voltage system through them, is it ok to have multiple circuit boards (solenoid switches and hall effect comparator circuits) sharing one common ground path? So far we've seen evidence of the solenoid switch circuits being a "victim" of motor-switching noise, but little (if any) evidence of the solenoids contributing noise to the system.

On a more basic level, I'm really asking if every single dc item is supposed to have its own ground wire all the way back to the grounding bolt (the point where dc ground and frame ground join with the incoming power-cord ground.) If we imagine that there are no hall effect circuits at all, does each solenoid switch need a separate wire back to the bolt? Or, conversely, if there were hall effect circuits and no solenoids at all, would each hall effect circuit need a separate ground wire?

Sorry to belabor this point, but I think I've got people at work convinced that we should improve our grounding schemes, even if we're not using my new hall effect circuit idea. Given this opportunity, I want to make sure I understand (as best I can with my limited electronics background) how far we need to go in modifying the grounding scheme: I want to know if some imperfect compromise grounding improvements are worth considering (if there are diminishing returns for the harder improvements) or if this is more of an all-or-nothing scenario to make the grounding system do what it should.

Thank you all for your patience and help!

 

That looks pretty good! The only thing I might worry about is running the signal wires - the wires that tell the solenoids to turn on and the signals coming back from the halls, outside the conduit that holds the AC wires. The low voltage wires probably don't need to be in conduit if they are under 48 volts (UL). I think the single feed of the 5 volts and ground are probably ok since they are low current.
Here is a little write up:
http://www.lh-electric.net/tutorials/gnd_loop.html
Good luck!
Let us know how it works.

 

Normally where possible, all earth ground references come back to a central star point, this often consists of a copper plate that is drilled and tapped accordingly to accept all the terminations.
This includes motor frame grounds etc, also for suppression using solenoids you also need either a diode for DC versions or a snubber for the AC types connected across the coil.
The Siemens PDF outlines equi-potential bonding, which is to make all metallic parts of the machine conform to earth ground and remove the possibility of ground loops etc.
Also keep in mind that although SSR's offer transition from one voltage/system to another, they do not always provide complete isolation, if the DC control is referenced to earth ground, then also the AC controlled by the SSR is also referenced to earth via the grounded neutral.
Max.

 

So, my boss is asking a question now that I can't answer. He says that in his experience working on cars, they almost always tie every single device to a frame ground AND run a ground wire back to the battery, and often run redundant ground wires between the various devices too.

It seems like this would create ground loops all over the place and it clearly isn't star grounding. Cars have all sorts of noise sources and lots of potential victim circuits, so how does this work?

I've read a tiny bit (mostly Wikipedia article) about multi-point grounding schemes and it sounded like they work, but are terribly difficult to design properly. I had wondered if cars are deliberately using multi-punt grounding, but what he described sounds pretty haphazard compared to what I read.

Any insights or clarification? Thanks!

 

As outlined in the Siemens PDF, it is called equi-potential bonding, the effort is to make all point at exactly the same potential, and avoid the often quoted 'Ground-Loops'.
Also as per Ch6 of the PDF it is now recommended to ground both ends of shielded cable, when the correct bonding is done.
Max.

 

Hey everybody, just wanted to give an update on this project. There have been many developments since I last posted, most of which I've described in response to my first thread, but I thought I should respond here to the situation regarding noise issues in particular.

I've added the two filter capacitors to the power input side of this circuit as recommended earlier in this thread, and they seem to be enough to prevent false triggering of the outputs, although the LED indicators still flicker occasionally when a motor is turned on. I've also done quite a bit of testing now on the use of snubbers right at the motor inputs and on the improved (albeit imperfect) grounding scheme laid out in post #30. Each change alone is pretty effective, and together they reduce the noise to a point where I can no longer detect it at all.

Unless something strange happens between now and then, we should be implementing the noise-reducing modifications at the same time we launch some other design changes, probably within the next few months.

As I said in my other thread, thanks again to everyone on this forum. I really couldn't have figured this all out without the help I found here, both in terms of reading other people's questions and answers, and in terms of the questions I asked directly, from which I got lots of great feedback. I wish I could share more details about the final design, but my employer doesn't want to make this too terribly easy for competitors to copy, so I'm including a few pics, but nothing super-detailed (otherwise I would have done a more complete write-up in the Completed Projects forum.)

 

So I found a used oscilloscope as-is, with no probes, looking like it fell off a truck, for $20. I was pleasantly surprised that it seems to work, so I used it to compare the motor induced electrical noise in several wiring configurations.