Wouldn't you need some capacitors between each phase and ground?
I'm not sure what you need exactly, but when I had some EMI problems, I used something from these guys (Corcom). They have circuit diagrams for their products, which had capacitors between the phases like you did, but also had capacitors between each phase and ground after the common mode choke.

Also - if you were going to copy one of the Corcom designs, on theirs, you would want your capacitor that ties the phases together on the other side of the common mode choke, and the capacitors to ground on the side of the choke that your phase-to-phase capacitor is now ...

Interesting observations. Of the things we've tried so far, the one that worked the best was a Corcom filter. The particular one we tried wasn't enough, but we're definitely going to try a few others with more suitable frequency response characteristics for what we need. Ideally we'd like to integrate a solution into our board, not use a separate filter module, but we may end up modeling our approach on one of theirs.

 

Dear @ebeowulf17,
The information provided is very useful.
Can I know how do you introduce Common Mode and Differential Mode noise to the filter circuit ? because as of my knowledge EMI noise enter to the circuit via parasitic components.
Please reply as soon as possible.

 

For good filtering, I would use both filters: both common mode and differential. Also, for a more accurate simulation, I would clarify the magnetic coupling coefficient K ~ 0.999. To do this, it is necessary to measure the inductance with the LLeak short-circuited one winding. K = 1-0.5 * LLeak / L. And use capacitors at the inlet and outlet of the filter.

 

Ferrite, which is used in such filters has a loss. This can be simulated using resistors parallel to the inductances. In LTspice this is Rpar. Let the cutoff frequency 1 MHz ==> Rpar = 2 * 3.1415 * 1000000 * 0.011 = 69 kOhm. But I think that the frequency is less than 1 MHz and Rpar<69kOhm

 

Dear @ebeowulf17,
The information provided is very useful.
Can I know how do you introduce Common Mode and Differential Mode noise to the filter circuit ? because as of my knowledge EMI noise enter to the circuit via parasitic components.
Please reply as soon as possible.

I'm honestly not sure. My attempts at simulating filter response in this context never did work properly - I couldn't get a simulation to match the manufacturer's claimed response, nor the actual response we found in lab testing, so obviously my attempts at simulation weren't done properly.

I defer to @Bordodynov, who is an expert at LTspice simulation.

 

See link:
https://www.murata.com/products/emc/emifil/case/household

 

In case anyone's interested, although I never did make useful simulations of the filter circuits, we did eventually solve our noise problem.

We made two separate trips to a local test lab. On the first trip, we tried 6 different filters with our existing power supply. Each filter had specs that indicated it should've solved our noise problems. During testing, each filter reduced the noise a little, but none did enough, and the differences between the filters were much, much smaller than expected. Finally the lab technician identified our real problem. The SMPS was radiating noise out through the air, which then got picked up by the mains wires on the far side of the filter. In other words, the filters were blocking noise CONDUCTED through the wires just as they should've done, the the RADIATED noise was injecting itself right back into the wires! We crafted a hand-made copper shield to contain the EMI, essentially putting a metal hat on the power supply, and that solved the problem. Unfortunately, hand made copper hats weren't going to be a viable solution for production quantities.

So on our second trip, we tried 5 different power supplies, and found several power supplies that were clean enough to work without any filters. Luckily, one was the right size and shape to drop right into our PCB without changing the layout, and it cost much less than the ones we'd been using, so we were able to solve our problem and save money!