What is the gate voltage relative to source - ie red lead on gate, black on source pin (not ground)? -12v, ground end of resistor, gnd of DAC, opamp and Arduino should all be connected to a single point - no daisy chains or loops. If you have a scope monitor output of opamp (to ground) on A and voltage at source (to ground) on B channel and set it to A-B - it shouldn't be more than 2 -3v, maybe 4 max and stable DC. If you see any oscillations at high frequency then you need to add compensation to your loop as already mentioned.... your rats nest isn't helping here.

 

i agree with Irwing.

btw. when you expect 100mA load and it is not there, you need to find out why. do not keep increasing current limit because you can damage things.

for example set PSU to 12V 100mA.
with Arduino set output current to something less than 100mA so that PSU limit does not kick in.
suppose you choose 50mA.

Since Rsense =1 Ohm. DAC need to output 50mV and that should be 41 count.
Vout = 5V* 41/4096 = 0.050V

then you measure DAC output to confirm that it is really that voltage. and you measure voltage across 1 Ohm resistor - it should be the same.

if that is not the case, measure OpAmp output voltage.you get Vgs by subtracting previous measurement from this one.

if this part is working, you can crank the current limit to 2.2Amp abd try other current values (counts 0-1830)

 

For some reason, I did not observed this little detail earlier. But I do it now.
So the opamp is doing something as long as these readings are in synchronization.
Observe the direct relation between Vout-Gnd DVM reading and varPSU2 Amp display:

Another detail:
- This is varPSU1 which is powering the opamp that is powering the Gate of the mosfet. This 12V is constant, not fluctuating. Nor the Amperage from it. As it should be.
...with this relatively constant temp on the mosfet, when set on the max 10W ; <1min.

also the rats nest is bigger now !

 

i cannot deduce specs of that product from the picture. link to it would be nice.

its my PSU2 30V 10A 300W, almost virgin, I use it a few times.

My PSU1 is half wattage than this, at 30V 5A 150W, but I used it all the time for all my projects. It stand the test of time and from my opinion, it is a -very good- PSU.

 

This is my secondary variable PSU.
I set it to 12V and 100mA initially.When I connect it to the cct, on the 12V rail, it dropped to 4.29V, and switched to CC !

Why is this surprising to you?

What is the value of the resistor, and what voltage and current are you are trying to get?

If you use a power supply set to 12V and limited to 100mA, what voltage do you expect when the load is a 42.9Ω resistor?

 

with this relatively constant temp on the mosfet, when set on the max 10W ; <1min.

Where are you measuring that temperature? If on the MOSFET front casing that's meaningless. You need to be measuring the tab temperature as near to the case as possible. Also you should turn your heatsink so the fins are vertical. What is your room ambient?