Facebook
Google
GitHub
Linkedin
Hello there,
I had recently rediscovered this trick which is kind of neat.
When we open a file for writing, a lot of languages lock the file automatically so that another process cannot open the file for any reason (reading or writing). This allows even a blank file to be used as a mutex.
Let's say we have three processes, Moe, Larry, and Curly. They all want to do something that requires some exclusive access, like reading a file with an index number, then incrementing it, then writing it back to the file. Let's say Moe is first to open the Lock file, then Larry, then Curly, but it can be any order.
Moe opens the Lock file for writing, and if Larry and Curly try to open it, they can't get access, so they have to keep trying.
Meanwhile, Moe opens the data file for reading, reads the integer like 1, then increments it to 2, stores it back in the file, then closes it. Lastly, it closes the Lock file.
Only after the Lock file is closed, then either Larry or Curly can open the Lock file (for writing). If Curly gets there before Larry, then Curly gets access and now Larry and Moe are locked out as they can't access the Lock file.
Next, Curly reads the integer 2, increments it to 3, then writes it back to the data file, then finally closes the Lock file.
Now Moe or Larry can access the Lock file, and either one of them does the same thing with the open, read, write, close, etc. In this way only one process can have access at a time.
Also important is if Moe and Larry are still busy, Curly can access the file a second time, increment the number, store it back, and close again. This means all three processes are always working on something. Also, since the file is opened, read written to, and then closed again in a short time, it's a small overhead to the system considering the normal tasks are always longer (and of course assuming they actually are longer).
This is slightly simplified because the processes actually do other things too AFTER they get the Lock and then release it again. For example, run a test on the bytes. While any or all of them are running the test, the others can have access to the lock.
This behavior is fairly typical but the interesting part is that when the Lock file is opened in the WRITE mode, no other process can access it, so it maintains exclusive access to some resource.
In many languages this will work without having to specify some exclusive rights to the file. In pure DOS however, I think you have to specify something else to get it to work right, maybe SHARE.
Of course if you are going though the Windows API for file creation and access you can restrict the access to a file open in the READ mode also, but then you have to want to use that. In another language it could be possible that they have a built in access control, but with some they do not have that for read access, only write access.
Another use for this (or any mutex) is a scheduler, which assigns a task to several processes that have to be able to handle tasks one by one, which allows for parallel processing.
For example, say we want to check 100 files. We store that integer in one file and only allow read access once the process is able to obtain write access to the Lock file. It reads the integer, and it knows what file to work on. In this way, 8 processes could process 8 files at the same time (parallel) if the CPU supports it. This means Moe might be working on file 3, while Larry is working on file 7, and Curly is working on file 12, etc., and if Larry gets done before Moe or Curly, Larry can start working on file 13.
Testing this on a real system, it shows that on average there is only about 1 contention per 100 files. That means for only 1 out of 100 files Larry (or one of the others) can't open the Lock file until the second try, which does not waste much time. Also, this means that all 100 files get done almost N times faster (in theory but depends on the CPU cores and thread count).
Currently I need this parallel functionality and up to now I was using a striping technique where with 8 processes each process would get their file share assigned at startup. Thus Moe would get 1,9,17 and Larry gets 2, 10, 18, etc. The problem with that is if Curly gets files that are much bigger than the others get, that process does not complete in almost the same time as the others complete, which means the entire task could take longer because once Moe and Larry are done, the CPU defaults to using only one thread again for Curly while the others just have to sit there doing nothing for all that time.
Using a scheduling technique this should improve.
[Edit: added that Curly can access the file a second time if the other two are still busy and Curly finished early.]
I had recently rediscovered this trick which is kind of neat.
When we open a file for writing, a lot of languages lock the file automatically so that another process cannot open the file for any reason (reading or writing). This allows even a blank file to be used as a mutex.
Let's say we have three processes, Moe, Larry, and Curly. They all want to do something that requires some exclusive access, like reading a file with an index number, then incrementing it, then writing it back to the file. Let's say Moe is first to open the Lock file, then Larry, then Curly, but it can be any order.
Moe opens the Lock file for writing, and if Larry and Curly try to open it, they can't get access, so they have to keep trying.
Meanwhile, Moe opens the data file for reading, reads the integer like 1, then increments it to 2, stores it back in the file, then closes it. Lastly, it closes the Lock file.
Only after the Lock file is closed, then either Larry or Curly can open the Lock file (for writing). If Curly gets there before Larry, then Curly gets access and now Larry and Moe are locked out as they can't access the Lock file.
Next, Curly reads the integer 2, increments it to 3, then writes it back to the data file, then finally closes the Lock file.
Now Moe or Larry can access the Lock file, and either one of them does the same thing with the open, read, write, close, etc. In this way only one process can have access at a time.
Also important is if Moe and Larry are still busy, Curly can access the file a second time, increment the number, store it back, and close again. This means all three processes are always working on something. Also, since the file is opened, read written to, and then closed again in a short time, it's a small overhead to the system considering the normal tasks are always longer (and of course assuming they actually are longer).
This is slightly simplified because the processes actually do other things too AFTER they get the Lock and then release it again. For example, run a test on the bytes. While any or all of them are running the test, the others can have access to the lock.
This behavior is fairly typical but the interesting part is that when the Lock file is opened in the WRITE mode, no other process can access it, so it maintains exclusive access to some resource.
In many languages this will work without having to specify some exclusive rights to the file. In pure DOS however, I think you have to specify something else to get it to work right, maybe SHARE.
Of course if you are going though the Windows API for file creation and access you can restrict the access to a file open in the READ mode also, but then you have to want to use that. In another language it could be possible that they have a built in access control, but with some they do not have that for read access, only write access.
Another use for this (or any mutex) is a scheduler, which assigns a task to several processes that have to be able to handle tasks one by one, which allows for parallel processing.
For example, say we want to check 100 files. We store that integer in one file and only allow read access once the process is able to obtain write access to the Lock file. It reads the integer, and it knows what file to work on. In this way, 8 processes could process 8 files at the same time (parallel) if the CPU supports it. This means Moe might be working on file 3, while Larry is working on file 7, and Curly is working on file 12, etc., and if Larry gets done before Moe or Curly, Larry can start working on file 13.
Testing this on a real system, it shows that on average there is only about 1 contention per 100 files. That means for only 1 out of 100 files Larry (or one of the others) can't open the Lock file until the second try, which does not waste much time. Also, this means that all 100 files get done almost N times faster (in theory but depends on the CPU cores and thread count).
Currently I need this parallel functionality and up to now I was using a striping technique where with 8 processes each process would get their file share assigned at startup. Thus Moe would get 1,9,17 and Larry gets 2, 10, 18, etc. The problem with that is if Curly gets files that are much bigger than the others get, that process does not complete in almost the same time as the others complete, which means the entire task could take longer because once Moe and Larry are done, the CPU defaults to using only one thread again for Curly while the others just have to sit there doing nothing for all that time.
Using a scheduling technique this should improve.
[Edit: added that Curly can access the file a second time if the other two are still busy and Curly finished early.]
Last edited:
