The duty cycle varies with the output power.
Without slope compensation flyback converters go unstable above 50%.
You are getting very close to continuous conduction mode, which is to be avoided as it will result in reverse recovery losses in the secondary rectifier.
The overshoot on Vds seems rather large, which would explain the snubber losses

The slope compensation should be used if duty above 50%,
but when I design the converter, I had set the maximum duty should be 40%, so that I have enough margin for duty-cycle to handle the load change.

I did some modifications to this converter:
Replace main switch Q1 into IRF840
I re-design the transformer, reduce the primary turn number to 25 (the origin is 33 turns).
And the parameter of inductance is:
Lp=72uH Llk=12uH (*Air gap of my transformer is about 2mm(Is this too large?)
Add heatsink to snubber resistor R3, and change the value into 15kOhm/5W cement resistor
Replace C4 into 100nF/630V.
Add heatsink to D3, and replace R4 into 30Ohm/5W cement resistor.
Replace current shunt resistor Rs4 into 0.1Ohm/2W.
BTW, in the end, I'll fully re-layout the PCB, the whole thing is messy now.

The duty-cycle of 1A load seems the same, but when I increase the load current, the PWM seems not going to 45% so fast that I can increase the current up to 5A.
But the auxiliary rectifier diode D3 and resistor R4 is so hot so I stop adding the load current.

Please show how you made the transformer? It looks like the primary and secondary are not coupled well.

The transformer design step is as follow:

0. Set the converter specification
Input: 110VAC
Output:12V/10A
Switching frequency: 100kHz
Maximum duty-cycle: 40% (Full load)
Core: PQ35/35 Ae=1.538cm2
Maximum flux density: 1500Gauss
Expect efficiency η : 85%

*Maybe the efficiency is too efficient that the snubber resistor consumes most of the energy, my experiment shows that the real efficiency is about 65% when the output current is 2A.

1. Calculate the magnetizing inductor of the transformer:

Vin,min = 120.2VDC (85VAC)
fs,min = 100k
So the transformer inductance Lm<81.8uH
(I don't know why the inductance L of the equation should be "smaller" than the formula for working under DCM reason?)
The actual transformer magnetizing inductance Lm by measurement is 300uH.

2. Calculate primary peak current Ip,peak

So Ip,peak = 5.87A
Choose a suitable wire for the primary current.
I chose 0.32*4 Litz wire to provide 0.32mm2 cross-sectional area.
How to determine the wire size? Peak current or average current?

3. Calculate the primary turn number

So Np = 20.81 turns

4. Calculate the turns ratio

Let forward voltage of output diode Vf is assuming 1V
So n = 6.16

5. Calculate the secondary turn umber

Ns= 3.37 turns
Choose 4 turns for secondary

5. re-calculate the primary turn number
Np= 4 * 6.16 = 24.64
Choose 25
(How to determine the upper number or lower number?)

The experiment result under 1A load is like below:
CH1 PWM, CH2 Vds, CH3 Vaux-Vcc (D3 and R4 voltage)


I'll check the maximum duty-cycle this converter and get.
And there is a new problem with the auxiliary winding rectifier diode and the main switch Q1.
I'll update the result.

 

The reason I choose IRF840 is because the origin P20N60 blow up and I don’t know why.
I think is the leakage inductance of the transformer too large, so the spike voltage destroyed the component.
But I’m wondering why the auxiliary diode is getting so hot (about 140C), the 100kHz switching frequency should not be problem for STPS30150 Schottky diode.

 

I know you have nothing to do but this is a good read. power
Page 12 has good information on how they made their transformer. They have a inductance vs leakage inductance of 450uh vs 5uh. while you have 42:12uH. See how little time it takes to dump the energy in the leakage.
Page 25 shows wave forms.

 

Please show how you made the transformer? It looks like the primary and secondary are not coupled well.

Just as important as how you calculated the number of turns, is where you placed the windings, and physically how you wound it. It’s behaving as though your primary and secondary aren’t even on the same bobbin!

 

I know you have nothing to do but this is a good read. power
Page 12 has good information on how they made their transformer. They have a inductance vs leakage inductance of 450uh vs 5uh. while you have 42:12uH. See how little time it takes to dump the energy in the leakage.
Page 25 shows wave forms.

Thanks for the helpful information, but I can't really understand the Transformer build diagram really means.

Is this a side view of transformer or... something?
As the document said the primary side is 31 turns, and should be seperate into even part,
I can't really understand how to wire the transformer as it said...
Is anyone can help?

 

I did not go back and read the paper again. This what I think and might be wrong.
1) wind the control winding. That powers the IC. four wires of #32, 2 turns spread out over the entire window.
2) tape.
3) Primary starting with the MOSFET-Drain. Wind two wires #25. Wind 8 turns then 8 turns. End on pin 2. (center tap of primary)
4) Do not do the copper foil. Too complicated for now. Do 3 turns of tape.
5) Power secondary. 4 wires of #24, 3 turns. Fill window.
6 Do not do the copper foil. Too complicated for now. Do 3 turns of tape.
7) Start primary on pin 2 (center tap) End at supply. Wind two wires #25. Wind 8 turns then 8 turns.
8) Do 3 turns of tape.

Things to remember.
Try to fill the winding window with wire in each layer.
Primary to Secondary tape is important.
In this case they split the secondary into two pieces with secondary in-between. This reduces the LL.
In this case the primary is actually in 4 layers. I would wind 8. One layer of tape wind 8. The wire will be smother winding on tape than winding on wire. This will also decrease the interwinding capacitance.

 

I wouldn't say "Try to fill the winding window with wire in each layer" - I'd say it was mandatory. Gaps = leakage inductance; and, in a flyback, leakage inductance = inefficiency, because the energy goes into the clamp.
Adjust the wire gauge and the number of parallel windings to make sure that the bobbin is full across its width (provided that the wire is thick enough), no gaps and no quarter-filled extra layers.
If your primary has an odd number of turns, put the odd extra one on the first winding.

 

Here is the update after a week.
I had re-winding the transformer, and choose PQ32/30 core.
Using the method that recommends by @Ian0 and @ronsimpson to reduce leakage inductance.
I realize that I should choose the right "window" area according to my turn number, so I pick the shorter bobbin and core.
The image of my transformer is as below:

The magnetic inductance and leakage inductance of the final transformer is
Lm = 297.6uH
Llk = 2.7uH
And the turn ratio N1:N2:Naux = 31:5:6 (with air gap).



Also, I add P6KE2000 TVS (Transient Voltage Suppressor) between the snubber to reduce the heat on the resistor.
The final layout and schematic attach here, too.
Though there are still some problems that the transformer is noisy (sometimes there is sound, but sometimes not),
the performance is acceptable (77% efficiency), and the PWM duty cycle matches my design value (10A, 40% duty).
The waveform is as below:


As you can see, the duty cycle at full-load is about 40%, the converter is operated in CCM mode.
The ringing effect is reduced because the leakage inductor had decreased to 2uH.

There is still some small question:
When attaching the light load (3A), the PWM is "discontinuous".
The waveform is like below:

As the picture shows, some times the PWM only output "Nothing", so Vds of a switch start resonant.
this causes additional power loss on the snubber resistor.
And the input current and output diode current sometimes continuous, sometimes not.
I assume that this is because the energy output from my converter is enough, so IC omit the PWM to save power,
Is this correct?

Hope the experience of this post can help more people who also interesting in Power Electronics!
Thanks everyone here for help, and Merry Christmas

 

It is not usual to operate a flyback in continuous-current mode, because of the extra reverse-recovery losses of the secondary diode.
Try slightly more gap. That will increase the primary current but decrease the inductance. As energy stored is I^2L/2, the same energy will be stored in a shorter time, and it should come out of continuous conduction mode and efficiency should go up.
Alternatively, slow down the clock; but in both cases, make sure the core doesn't saturate.