Your schematic is so beautiful that I have missed several "bugs" but in this case, I missed D1, which makes what I thought was a problem -not a problem. I would worry about the repetitive peak current through D1, being a small signal diode, but I still worry more that your source can only go about 11 volts while the drain is at 24 volts, meaning that your source follower switch will be less than 50% efficient. Much less.
To repeat in paraphrase, it is ok to use a source follower for the high side switch if you can provide gate drive that is a sufficiently higher voltage than the drain.
Do you have an oscilloscope?