Hello there,

I had recently rediscovered this trick which is kind of neat.

When we open a file for writing, a lot of languages lock the file automatically so that another process cannot open the file for any reason (reading or writing). This allows even a blank file to be used as a mutex.

Let's say we have three processes, Moe, Larry, and Curly. They all want to do something that requires some exclusive access, like reading a file with an index number, then incrementing it, then writing it back to the file. Let's say Moe is first to open the Lock file, then Larry, then Curly, but it can be any order.
Moe opens the Lock file for writing, and if Larry and Curly try to open it, they can't get access, so they have to keep trying.
Meanwhile, Moe opens the data file for reading, reads the integer like 1, then increments it to 2, stores it back in the file, then closes it. Lastly, it closes the Lock file.
Only after the Lock file is closed, then either Larry or Curly can open the Lock file (for writing). If Curly gets there before Larry, then Curly gets access and now Larry and Moe are locked out as they can't access the Lock file.
Next, Curly reads the integer 2, increments it to 3, then writes it back to the data file, then finally closes the Lock file.
Now Moe or Larry can access the Lock file, and either one of them does the same thing with the open, read, write, close, etc. In this way only one process can have access at a time.

Also important is if Moe and Larry are still busy, Curly can access the file a second time, increment the number, store it back, and close again. This means all three processes are always working on something. Also, since the file is opened, read written to, and then closed again in a short time, it's a small overhead to the system considering the normal tasks are always longer (and of course assuming they actually are longer).

This is slightly simplified because the processes actually do other things too AFTER they get the Lock and then release it again. For example, run a test on the bytes. While any or all of them are running the test, the others can have access to the lock.

This behavior is fairly typical but the interesting part is that when the Lock file is opened in the WRITE mode, no other process can access it, so it maintains exclusive access to some resource.
In many languages this will work without having to specify some exclusive rights to the file. In pure DOS however, I think you have to specify something else to get it to work right, maybe SHARE.
Of course if you are going though the Windows API for file creation and access you can restrict the access to a file open in the READ mode also, but then you have to want to use that. In another language it could be possible that they have a built in access control, but with some they do not have that for read access, only write access.

Another use for this (or any mutex) is a scheduler, which assigns a task to several processes that have to be able to handle tasks one by one, which allows for parallel processing.
For example, say we want to check 100 files. We store that integer in one file and only allow read access once the process is able to obtain write access to the Lock file. It reads the integer, and it knows what file to work on. In this way, 8 processes could process 8 files at the same time (parallel) if the CPU supports it. This means Moe might be working on file 3, while Larry is working on file 7, and Curly is working on file 12, etc., and if Larry gets done before Moe or Curly, Larry can start working on file 13.
Testing this on a real system, it shows that on average there is only about 1 contention per 100 files. That means for only 1 out of 100 files Larry (or one of the others) can't open the Lock file until the second try, which does not waste much time. Also, this means that all 100 files get done almost N times faster (in theory but depends on the CPU cores and thread count).

Currently I need this parallel functionality and up to now I was using a striping technique where with 8 processes each process would get their file share assigned at startup. Thus Moe would get 1,9,17 and Larry gets 2, 10, 18, etc. The problem with that is if Curly gets files that are much bigger than the others get, that process does not complete in almost the same time as the others complete, which means the entire task could take longer because once Moe and Larry are done, the CPU defaults to using only one thread again for Curly while the others just have to sit there doing nothing for all that time.
Using a scheduling technique this should improve.

[Edit: added that Curly can access the file a second time if the other two are still busy and Curly finished early.]

 

The dynamic scheduling part makes sense, especially when the files vary a lot in size. It should avoid the situation where one process gets stuck with all the large files while the others finish early.

One thing I would be careful about is assuming that opening a file for writing always gives an exclusive lock. That depends on the operating system and the language or runtime being used. On some systems, another process may still be able to open the same file unless an explicit locking function is used.

I would probably use the platform's normal mutex or file-locking function if one is available. If a lock file is used, its creation should be atomic, and there also needs to be a way to recover if a process crashes while holding the lock.

Also, I would add a short delay or backoff between retries rather than checking continuously. Apart from those details, using a shared “next file” counter looks better than fixed striping for this job.

 

Hello there,

I had recently rediscovered this trick which is kind of neat.

When we open a file for writing, a lot of languages lock the file automatically so that another process cannot open the file for any reason (reading or writing). This allows even a blank file to be used as a mutex.

Let's say we have three processes, Moe, Larry, and Curly. They all want to do something that requires some exclusive access, like reading a file with an index number, then incrementing it, then writing it back to the file. Let's say Moe is first to open the Lock file, then Larry, then Curly, but it can be any order.
Moe opens the Lock file for writing, and if Larry and Curly try to open it, they can't get access, so they have to keep trying.
Meanwhile, Moe opens the data file for reading, reads the integer like 1, then increments it to 2, stores it back in the file, then closes it. Lastly, it closes the Lock file.
Only after the Lock file is closed, then either Larry or Curly can open the Lock file (for writing). If Curly gets there before Larry, then Curly gets access and now Larry and Moe are locked out as they can't access the Lock file.
Next, Curly reads the integer 2, increments it to 3, then writes it back to the data file, then finally closes the Lock file.
Now Moe or Larry can access the Lock file, and either one of them does the same thing with the open, read, write, close, etc. In this way only one process can have access at a time.

Also important is if Moe and Larry are still busy, Curly can access the file a second time, increment the number, store it back, and close again. This means all three processes are always working on something. Also, since the file is opened, read written to, and then closed again in a short time, it's a small overhead to the system considering the normal tasks are always longer (and of course assuming they actually are longer).

This is slightly simplified because the processes actually do other things too AFTER they get the Lock and then release it again. For example, run a test on the bytes. While any or all of them are running the test, the others can have access to the lock.

This behavior is fairly typical but the interesting part is that when the Lock file is opened in the WRITE mode, no other process can access it, so it maintains exclusive access to some resource.
In many languages this will work without having to specify some exclusive rights to the file. In pure DOS however, I think you have to specify something else to get it to work right, maybe SHARE.
Of course if you are going though the Windows API for file creation and access you can restrict the access to a file open in the READ mode also, but then you have to want to use that. In another language it could be possible that they have a built in access control, but with some they do not have that for read access, only write access.

Another use for this (or any mutex) is a scheduler, which assigns a task to several processes that have to be able to handle tasks one by one, which allows for parallel processing.
For example, say we want to check 100 files. We store that integer in one file and only allow read access once the process is able to obtain write access to the Lock file. It reads the integer, and it knows what file to work on. In this way, 8 processes could process 8 files at the same time (parallel) if the CPU supports it. This means Moe might be working on file 3, while Larry is working on file 7, and Curly is working on file 12, etc., and if Larry gets done before Moe or Curly, Larry can start working on file 13.
Testing this on a real system, it shows that on average there is only about 1 contention per 100 files. That means for only 1 out of 100 files Larry (or one of the others) can't open the Lock file until the second try, which does not waste much time. Also, this means that all 100 files get done almost N times faster (in theory but depends on the CPU cores and thread count).

Currently I need this parallel functionality and up to now I was using a striping technique where with 8 processes each process would get their file share assigned at startup. Thus Moe would get 1,9,17 and Larry gets 2, 10, 18, etc. The problem with that is if Curly gets files that are much bigger than the others get, that process does not complete in almost the same time as the others complete, which means the entire task could take longer because once Moe and Larry are done, the CPU defaults to using only one thread again for Curly while the others just have to sit there doing nothing for all that time.
Using a scheduling technique this should improve.

[Edit: added that Curly can access the file a second time if the other two are still busy and Curly finished early.]

What are you actually talking about? What OS are you talking about? All of this stuff is documented, the OS itself relies on mutual exclusion, its not part of the file system, its a native OS capability upon which the file system rests.

 

The dynamic scheduling part makes sense, especially when the files vary a lot in size. It should avoid the situation where one process gets stuck with all the large files while the others finish early.

One thing I would be careful about is assuming that opening a file for writing always gives an exclusive lock. That depends on the operating system and the language or runtime being used. On some systems, another process may still be able to open the same file unless an explicit locking function is used.

I would probably use the platform's normal mutex or file-locking function if one is available. If a lock file is used, its creation should be atomic, and there also needs to be a way to recover if a process crashes while holding the lock.

Also, I would add a short delay or backoff between retries rather than checking continuously. Apart from those details, using a shared “next file” counter looks better than fixed striping for this job.

Yes I mentioned DOS will not automatically work like this. From what I understand though, the Windows op sys strongly enforces exclusive access and I have tested this with long running programs just to test that.

With that in mind, I might also add a checkup mechanism to make sure no process can ever get the lock while it is already locked by another process. In my case though, it would extremely rare due to the random nature of the other files involved, but even if it did the worst would be that the same file would have been processed twice. That seems weird, but would not break the overall system.

I like your idea about the process crashes. From what I understand, if the process exits normally the lock is dropped by that process. I would have to check if that is the same for a crash on the Windows op sys. This should be easy to test I'll try it later today hopefully.
I guess an idea would be to monitor the execution time of each process and start a new lock if one process takes too long. If each process keeps track of it's last index, it could start all over from there maybe, even if that means doing 4 to 8 files over again.
In the testing there were no crashes, and because the code is so simple it should never crash, but we all know we can't depend on that forever. I never like to depend on anything when it comes to the Windows op sys.

 

What are you actually talking about? What OS are you talking about? All of this stuff is documented, the OS itself relies on mutual exclusion, its not part of the file system, its a native OS capability upon which the file system rests.

I mentioned Windows by name, and partly excluded DOS because that has a special extra requirement.

About the use of a more normal method for a mutex, that goes without saying, but is worth mentioning here because once you invoke a normal mutex in Windows that means you have to bring in some Windows libraries. This is not impossible, but it's extra code bulk and extra code typing to get it going. Using a simpler coding only needs certain simpler functions like the file open function.
This is a little hard to explain unless you use the Euphoria language, which is nice but requires Windows DLL's to work with the system functions that would include a mutex object. I've done that countless times, but the file open trick is really much faster and requires a lot less coding.

If you don't feel comfortable with this then I would say definitely don't use it. You need to trust your own code that's basic common sense.
In my case it may be a little different and because I did so much testing with this I can 'almost' trust it completely. As in the other post, I did not test what happens if one of the processes crashes while it holds the lock, which means I will test that idea today to see what happens. If it holds up the other threads, I will have to hopefully find a simple mechanism to get around it like some sort of timing. I already incorporate timing into these programs for progress reporting so it shouldn't be too hard to code, but I'll see what happens.

If you would like to know absolutely what systems are being used and tested, it is Windows 11 and Euphoria (either interpreted or compiled/bound into .exe form).
I am looking into Python again to see what that can offer too but am not using that for these tests, yet, and may never do that.

 

I mentioned Windows by name, and partly excluded DOS because that has a special extra requirement.

About the use of a more normal method for a mutex, that goes without saying, but is worth mentioning here because once you invoke a normal mutex in Windows that means you have to bring in some Windows libraries. This is not impossible, but it's extra code bulk and extra code typing to get it going. Using a simpler coding only needs certain simpler functions like the file open function.
This is a little hard to explain unless you use the Euphoria language, which is nice but requires Windows DLL's to work with the system functions that would include a mutex object. I've done that countless times, but the file open trick is really much faster and requires a lot less coding.

If you don't feel comfortable with this then I would say definitely don't use it. You need to trust your own code that's basic common sense.
In my case it may be a little different and because I did so much testing with this I can 'almost' trust it completely. As in the other post, I did not test what happens if one of the processes crashes while it holds the lock, which means I will test that idea today to see what happens. If it holds up the other threads, I will have to hopefully find a simple mechanism to get around it like some sort of timing. I already incorporate timing into these programs for progress reporting so it shouldn't be too hard to code, but I'll see what happens.

If you would like to know absolutely what systems are being used and tested, it is Windows 11 and Euphoria (either interpreted or compiled/bound into .exe form).
I am looking into Python again to see what that can offer too but am not using that for these tests, yet, and may never do that.

I have no idea what you are talking about. File IO on Windows is part of the Win32 API as are all of the IPC functions.

This library DLL contains all of these: KERNEL32.DLL, so you need to use that for Mutexes or File IO.

Also what are you talking about when you say "feel comfortable with"? All Windows apps use Win32, its the only way to interact with the OS unless you are writing kernel mode device drivers. Whether you work with Python, C++, C# - any language - they will all leverage Win32.

 

I have no idea what you are talking about. File IO on Windows is part of the Win32 API as are all of the IPC functions.

This library DLL contains all of these: KERNEL32.DLL, so you need to use that for Mutexes or File IO.

Also what are you talking about when you say "feel comfortable with"? All Windows apps use Win32, its the only way to interact with the OS unless you are writing kernel mode device drivers. Whether you work with Python, C++, C# - any language - they will all leverage Win32.

Hello again,

Yes but if you use a higher level language you may not have access to the Win API directly, without loading some DLL's.
If you don't want to load DLL's and go with the extra code base required for that, you might use the file locking method (file "write" only).

I can show you the code with and without the DLL's but it's quite simple: either you link to the DLL's for that (such as the Kernel) or you don't.
If you don't, you rely on the higher level language to do that as it does it already and it's built in. It may not give you the file locking mechanism
though so you have to either link to the DLL's or use the file write lock method.

This may not make sense to you until you use a high level language that does not have a file locking method built in.
One such language is called "Euphoria" and it's free and on the web. It's an interpreted language and fairly simple, sort of like Pascal.
Official development has ended so file locking may have not been added yet, but there are other unofficial versions made by others that could possibly have file locking added now.

In the last official release, this is all you get:
fni=open(MyFilePath,"r")
close(fni)
or:
fno=open(MyFilePath,"w")
close(fno)

There is no:
fni=open(MyFilePath,"r",lock=1)

or similar, just those two functions.
The only other possibilies are open append "a" or binary "rb" or "wb" and the like, no lock option.

The default for the "w" (write) option is LOCKED though, so that's the only way you can get a lock without going to the DLL API's directly.

It's not like it is really 'hard' to load the DLL's either, it's just interesting that this other method works too, and takes a lot less code, the .exe file is smaller, load time is reduced, etc., etc. Maybe not a huge advantage but still interesting.

If you still don't understand this, it's because you've never used a language like this one. I have not checked out Python yet for a locking mechanism, it may have one already.
I just looked this up, it apparently has:
" msvcrt.locking() "
which supposedly allows you to lock a file.
Some of the locking function appear to work only on Linux, but that one apparently works on Windows.

 

Yes I mentioned DOS will not automatically work like this. From what I understand though, the Windows op sys strongly enforces exclusive access and I have tested this with long running programs just to test that.

It all depends on the program that opened the file. For instance Notepad++ doesn't lock files when you open them, which is very handy because I can open a file that another program writes to and then, when I go back to Notepad++, it pops up a dialog telling me that the file has changed and do I want to reload it. This allows me to monitor the output without having to constantly close and reopen the file. I can also open files in Notepad++ that are open in other programs, such as Excel, that lock files. I don't know what would happen if I try to save one of those files, however (I'd have to do a test).

 

Hello again,

Yes but if you use a higher level language you may not have access to the Win API directly,

Stop right there...

Whatever language you use, KERNEL32.DLL will always have to be loaded, explicitly or implicitly.

You CANNOT do any OS interaction without that library being loaded into the process address space, period.

Please tell me too, exactly which programming language you are talking about here, you keep making very general statements that are untrue for C, C++, C#, Python, Rust - so what language are you talking about?

Once again you cannot do any kind of file operations without KERNEL32.DLL being mapped into the process's address space.

All user mode interaction with the OS is implemented by entry points in that DLL.

If you disagree with anything I've said in this post then please clearly say what it is you disagree with.

https://en.wikipedia.org/wiki/Microsoft_Windows_library_files#KERNEL32.DLL

 

Stop right there...

Whatever language you use, KERNEL32.DLL will always have to be loaded, explicitly or implicitly.

You CANNOT do any OS interaction without that library being loaded into the process address space, period.

Please tell me too, exactly which programming language you are talking about here, you keep making very general statements that are untrue for C, C++, C#, Python, Rust - so what language are you talking about?

Once again you cannot do any kind of file operations without KERNEL32.DLL being mapped into the process's address space.

All user mode interaction with the OS is implemented by entry points in that DLL.

If you disagree with anything I've said in this post then please clearly say what it is you disagree with.

https://en.wikipedia.org/wiki/Microsoft_Windows_library_files#KERNEL32.DLL

Hello,

"All user mode interaction with the OS is implemented by entry points in that DLL. "
But it depends what the higher level language exposes to the programmer using that language. That's the main point. Some languages do not even allow linking to any DLL's at all.


You still do not get the point and as i said that is because you must have not used a language like these. Higher level languages gives you access to things like files without have to resort to using Kernel functions directly. The higher level language does it for you, and provides you with a much simpler wrapper.
Haven't you ever used QBASIC or something like that? QBASIC is a higher level language while calling the Kernel functions directly is lower level.

When I say that the language has this function:
fn_out=open(MyFile,"w")
that is ALL you get.

When I say that language has this function:
fn_in=open(MyFile,"r")
that is ALL you get, and that "open" function is a built in function that is a higher level wrapper for the API file functions in the Kernel32.dll.

The Kernel32.dll is what has most of the file functions, but that is, at that point, completely invisible to the higher level language, and since the higher level language does not expose CreateFile(...) directly, you never get the Kernel functions, only the higher up wrapper functions. That means you can not do:
fn_in=open(MyFile,"r",NO_SHARE,...)
or whatever it is in the Kernel (like CreateFile).

In the code for the higher language, somewhere we could find this:
function open(FilePath,FileMode)
Handle=CreateFile(FilePath,FileMode, MoreParams)
return Handle
end function
and although that wrapper has 'MoreParams' that COULD include NO_SHARE, it only does that for the "w" write file mode not for the "r" read file mode.

The higher level language wraps the Windows API functions so we never have to supply the entire list of parameters that the CreateFile function requires.
Unfortunately, that means we don't get to specify "NO_SHARE" or whatever it is in the Windows kernel function.

Now if I wanted to, I could include some code like:
Hdll=open_dll("Kernel32.dll")
and then link to CreateFile and CloseHandle and all that, and call CreateFile directly as needed, which then includes NO_SHARE.
This gives me access to:
HANDLE CreateFile(
LPCTSTR lpFileName, // address of name of the file
DWORD dwDesiredAccess, // access (read-write) mode
DWORD dwShareMode, // share mode
LPSECURITY_ATTRIBUTES lpSecurityAttributes, // address of security descriptor
DWORD dwCreationDistribution, // how to create
DWORD dwFlagsAndAttributes, // file attributes
HANDLE hTemplateFile // handle of file with attributes to copy
);
and then I can set dwShareMode=0 (I have been using "NO_SHARE" to represent zero, as in: constant NO_SHARE=0).

This also requires the code or similar (after you decide if you want ASCII or Unicode or whatever):
link(Hdll,"CreateFile",atom lpFileName, atom dwDesiredAccess, atom ..., C_POINTER)
atom lpFileName
lpFileName=allocate_text("MyFile.txt")
as well as some other stuff.

So the choice is to use this for a lock:
fn_out=open(MyFile,"w")

or do all that other stuff above that, plus more constants and whatnot.
There is a list of constants you have to also have so you can set some flags and other stuff. There is also a constant for an invalid handle that you have to use to check the return value.

You should be able to see the difference now. It's not about who or what links to the Kernel, it is what has to do it. Either you do it yourself, or you let the higher level language do it automatically which is a bit cleaner if you don't need special modes. It's one line of code vs probably 50 lines of code all of has to be perfect.
I've done it many times before using my own higher level library for Windows that wraps API functions in ways that makes calling easier, but none of that is as easy as just opening a single file in "w" write mode.

Does this make more sense to you now?

 

It all depends on the program that opened the file. For instance Notepad++ doesn't lock files when you open them, which is very handy because I can open a file that another program writes to and then, when I go back to Notepad++, it pops up a dialog telling me that the file has changed and do I want to reload it. This allows me to monitor the output without having to constantly close and reopen the file. I can also open files in Notepad++ that are open in other programs, such as Excel, that lock files. I don't know what would happen if I try to save one of those files, however (I'd have to do a test).

Hi,

Oh yes, well Notepad is an 'application' which is at a higher level than the 'operating system'.
The operating system is what enforces a lock, not the application, unless the application is written to enforce the lock.
In other words, if Notepad itself 'enforced' the lock then it would relegate that functionality to the operating system which is built to enforce the lock.
If Notepad did not wish to enforce the lock, then it can relieve the operating system of having to make sure the lock actually acts like a perfect lock even for tiny time periods where two try to open the file "at the same time".
You or I can write a program like this that either uses locks, does not use locks, or allows the user to decide whether to use locks or not. Locks are enforced through the operating system but of course only if the application wishes to use one. If the app does use one, then the op sys is what makes sure the lock actually works as intended.

This sounds strange because if we use the Win API to open a file it seems like "WE" decide to lock it or not, but actually if we do lock it the operating system takes care of that when we tell it to do so. So we aren't really providing the locking mechanism, we just tell the op sys to lock it and it locks it. If we do not tell it to lock it, then it makes it easier for the system I guess because it does not have to make sure nothing else opens it.

The chain would go like this:
App(Let's lock it) ---> OpSys(ok, I'll lock it and make sure it stays locked until you say otherwise; no other calls can open it until then).
App(Let's not lock it) ---> OpSys(ok, it will not be locked).
If a User is involved, then of course it would be: User(lock it) ---> App(Lock it) ---> OpSys(ok i'll lock it and make sure it stays locked).

 

Hello,

"All user mode interaction with the OS is implemented by entry points in that DLL. "
But it depends what the higher level language exposes to the programmer using that language. That's the main point. Some languages do not even allow linking to any DLL's at all.


You still do not get the point and as i said that is because you must have not used a language like these. Higher level languages gives you access to things like files without have to resort to using Kernel functions directly. The higher level language does it for you, and provides you with a much simpler wrapper.
Haven't you ever used QBASIC or something like that? QBASIC is a higher level language while calling the Kernel functions directly is lower level.

When I say that the language has this function:
fn_out=open(MyFile,"w")
that is ALL you get.

When I say that language has this function:
fn_in=open(MyFile,"r")
that is ALL you get, and that "open" function is a built in function that is a higher level wrapper for the API file functions in the Kernel32.dll.

Exactly so whether one is aware or not KERNEL32 will get loaded somehow. The C stdio (if you look at the src) calls Win32 API.

The Kernel32.dll is what has most of the file functions, but that is, at that point, completely invisible to the higher level language, and since the higher level language does not expose CreateFile(...) directly, you never get the Kernel functions, only the higher up wrapper functions. That means you can not do:
fn_in=open(MyFile,"r",NO_SHARE,...)
or whatever it is in the Kernel (like CreateFile).

In the code for the higher language, somewhere we could find this:
function open(FilePath,FileMode)
Handle=CreateFile(FilePath,FileMode, MoreParams)
return Handle
end function
and although that wrapper has 'MoreParams' that COULD include NO_SHARE, it only does that for the "w" write file mode not for the "r" read file mode.

The higher level language wraps the Windows API functions so we never have to supply the entire list of parameters that the CreateFile function requires.
Unfortunately, that means we don't get to specify "NO_SHARE" or whatever it is in the Windows kernel function.

Now if I wanted to, I could include some code like:
Hdll=open_dll("Kernel32.dll")
and then link to CreateFile and CloseHandle and all that, and call CreateFile directly as needed, which then includes NO_SHARE.
This gives me access to:
HANDLE CreateFile(
LPCTSTR lpFileName, // address of name of the file
DWORD dwDesiredAccess, // access (read-write) mode
DWORD dwShareMode, // share mode
LPSECURITY_ATTRIBUTES lpSecurityAttributes, // address of security descriptor
DWORD dwCreationDistribution, // how to create
DWORD dwFlagsAndAttributes, // file attributes
HANDLE hTemplateFile // handle of file with attributes to copy
);
and then I can set dwShareMode=0 (I have been using "NO_SHARE" to represent zero, as in: constant NO_SHARE=0).

This also requires the code or similar (after you decide if you want ASCII or Unicode or whatever):
link(Hdll,"CreateFile",atom lpFileName, atom dwDesiredAccess, atom ..., C_POINTER)
atom lpFileName
lpFileName=allocate_text("MyFile.txt")
as well as some other stuff.

So the choice is to use this for a lock:
fn_out=open(MyFile,"w")

or do all that other stuff above that, plus more constants and whatnot.
There is a list of constants you have to also have so you can set some flags and other stuff. There is also a constant for an invalid handle that you have to use to check the return value.

You should be able to see the difference now. It's not about who or what links to the Kernel, it is what has to do it. Either you do it yourself, or you let the higher level language do it automatically which is a bit cleaner if you don't need special modes. It's one line of code vs probably 50 lines of code all of has to be perfect.
I've done it many times before using my own higher level library for Windows that wraps API functions in ways that makes calling easier, but none of that is as easy as just opening a single file in "w" write mode.

Does this make more sense to you now?

You originally claimed that a contrived file solution can get you mutex behavior without needing to load the DLL for mutexes. But the same DLL contains all mutex AND all file API calls.

Perhaps I misunderstood, are you saying that some languages only expose file calls and offer no mutex calls nor direct API access, and so in those cases you. can use the supported file calls to simulate a mutex?

if so, my apologies, I understand you now, but what is such a language?

 

Exactly so whether one is aware or not KERNEL32 will get loaded somehow. The C stdio (if you look at the src) calls Win32 API.




You originally claimed that a contrived file solution can get you mutex behavior without needing to load the DLL for mutexes. But the same DLL contains all mutex AND all file API calls.

Perhaps I misunderstood, are you saying that some languages only expose file calls and offer no mutex calls nor direct API access, and so in those cases you. can use the supported file calls to simulate a mutex?

if so, my apologies, I understand you now, but what is such a language?

Hi,

No problem, when we have discussions we expect to not understand each other right away. That's fairly normal.

Yes, I was saying that with such a language it is interesting that we can do a mutex for file READS by using a separate file where we WRITE to the file only, and never have to read it. We just try to write to the file and if we can't write to it, then another process must have the limited access. Once that other processes closes the file, the second process can write to it which tells it that it also now has access to the file we want to read.

So the flow goes like this (and we can use any type of file):
Process A opens "FileM.txt" in write mode. It then reads "FileData.txt" and then maybe gets an integer from that file like 5. It then can increment that to 6, then write it back to "FileData.txt", then closes "FileM.txt".
However, while Process A has "FileM.txt" open, Process B may be trying to open it too, also in write mode. Process B does not get access though, so it has to wait and try again after some delay. Once Process A closes "FileM.txt" (its last step in the sequence) then Process B opens "FileM.txt" and then opens "FileData.txt" and reads that '6' integer, then increments it to 7, then writes it back to "FileData.txt", then closes "FileM.txt".
If Process A tried to write to "FileM.txt" a second time while Process B has it open, it won't be able to until Process B closes it. Eventually Process B closes it and then Process A will read that updated '7' integer.

The integers are just an enumeration of files for example, where each file has to be processed and we need a way to divide the group of files up between processes so all processes are busy most of the time.

The particular language is called Euphoria. It's a Pascale like language that can be used to create console apps and Windows apps. There are user donated libraries that wrap the Windows API function calls into more easily managed functions.
The main feature for the Official version is it has simple syntax and structure. It uses sequences instead of arrays which can be a little confusing at first, but for the most part they are simple. I'll provide a little insight into this language.

For example, to store text we would do something like:
sequence s
s="My current line of text"

but it can get more elaborate for example:
s={"FirstLine", "SecondLine","ThirdLine"}

which is a two level sequence.
Items are accessed like arrays:
s[1], s[2], etc.

Sequences do not have to have the same structure throughout though:
s=
{
"Name",
{1,2,3,4,5},
{"Text", 3.1415926, 2.7, "dingdong", {a,b,c}}
}
That sequence has three subsequences, and that third subsequence has subsequences in it also. a and b and c are variables that can be numerical or strings or even subsequences themselves.
To access "dingdong" we would do
subseq=s[3][4]
and now the variable subseq would contain "dingdong".

So it is very simple but the sequences can be very complex. Usually we work with simpler sequences so it does not get too strange, except for special applications. The sequence structure is allowed to be very variable though as the programmer decides how complex they need it to be.
Since the first level is accessed with s[1], s[2], etc., we can use a sequence to store data that varies a lot. For that one example, we might have just "Name" in s[1] and then the entire thing as s[2].

It's also interesting that we can create another sequence that contains that 's' sequence above:
ss=s
meaning that ss[1] now equals 's' above.
We can then add to ss with just about anything:
ss=append(ss,"MyName")
so the entire ss sequence looks like this:
{
{
"Name",
{1,2,3,4,5},
{"Text", 3.1415926, 2.7, "dingdong", {a,b,c}}
},
"MyName"
}

I know it looks strange at first. It's very useful, but the processing is generally slower once we get very large sequences with lots and lots of elements.
I know it is useful because I've created a huge number of Windows programs with this language as well as console programs.

From what I remember, the official version was going to provide a locking mechanism for files opened in the 'read' mode also, but I guess that never happened before the official work on it came to an end. It is a free download though, and it also has a method for binding the programs into .exe format to create stand-alone programs.

 

Hi,

No problem, when we have discussions we expect to not understand each other right away. That's fairly normal.

Yes, I was saying that with such a language it is interesting that we can do a mutex for file READS by using a separate file where we WRITE to the file only, and never have to read it. We just try to write to the file and if we can't write to it, then another process must have the limited access. Once that other processes closes the file, the second process can write to it which tells it that it also now has access to the file we want to read.

So the flow goes like this (and we can use any type of file):
Process A opens "FileM.txt" in write mode. It then reads "FileData.txt" and then maybe gets an integer from that file like 5. It then can increment that to 6, then write it back to "FileData.txt", then closes "FileM.txt".
However, while Process A has "FileM.txt" open, Process B may be trying to open it too, also in write mode. Process B does not get access though, so it has to wait and try again after some delay. Once Process A closes "FileM.txt" (its last step in the sequence) then Process B opens "FileM.txt" and then opens "FileData.txt" and reads that '6' integer, then increments it to 7, then writes it back to "FileData.txt", then closes "FileM.txt".
If Process A tried to write to "FileM.txt" a second time while Process B has it open, it won't be able to until Process B closes it. Eventually Process B closes it and then Process A will read that updated '7' integer.

The integers are just an enumeration of files for example, where each file has to be processed and we need a way to divide the group of files up between processes so all processes are busy most of the time.

The particular language is called Euphoria. It's a Pascale like language that can be used to create console apps and Windows apps. There are user donated libraries that wrap the Windows API function calls into more easily managed functions.
The main feature for the Official version is it has simple syntax and structure. It uses sequences instead of arrays which can be a little confusing at first, but for the most part they are simple. I'll provide a little insight into this language.

For example, to store text we would do something like:
sequence s
s="My current line of text"

but it can get more elaborate for example:
s={"FirstLine", "SecondLine","ThirdLine"}

which is a two level sequence.
Items are accessed like arrays:
s[1], s[2], etc.

Sequences do not have to have the same structure throughout though:
s=
{
"Name",
{1,2,3,4,5},
{"Text", 3.1415926, 2.7, "dingdong", {a,b,c}}
}
That sequence has three subsequences, and that third subsequence has subsequences in it also. a and b and c are variables that can be numerical or strings or even subsequences themselves.
To access "dingdong" we would do
subseq=s[3][4]
and now the variable subseq would contain "dingdong".

So it is very simple but the sequences can be very complex. Usually we work with simpler sequences so it does not get too strange, except for special applications. The sequence structure is allowed to be very variable though as the programmer decides how complex they need it to be.
Since the first level is accessed with s[1], s[2], etc., we can use a sequence to store data that varies a lot. For that one example, we might have just "Name" in s[1] and then the entire thing as s[2].

It's also interesting that we can create another sequence that contains that 's' sequence above:
ss=s
meaning that ss[1] now equals 's' above.
We can then add to ss with just about anything:
ss=append(ss,"MyName")
so the entire ss sequence looks like this:
{
{
"Name",
{1,2,3,4,5},
{"Text", 3.1415926, 2.7, "dingdong", {a,b,c}}
},
"MyName"
}

I know it looks strange at first. It's very useful, but the processing is generally slower once we get very large sequences with lots and lots of elements.
I know it is useful because I've created a huge number of Windows programs with this language as well as console programs.

From what I remember, the official version was going to provide a locking mechanism for files opened in the 'read' mode also, but I guess that never happened before the official work on it came to an end. It is a free download though, and it also has a method for binding the programs into .exe format to create stand-alone programs.

I see, OK.

The drawback of course is that attempting to open a locked file simply fails, theres no option to efficiently wait so with heavily used "mutex" and/or many threads there will be both CPU and performance pains.

 

I know that I am a hardware-only guy, when I read the thread above, and it makes as much sense to me as a discussion in early Ming dynasty calligraphy.

 

I know that I am a hardware-only guy, when I read the thread above, and it makes as much sense to me as a discussion in early Ming dynasty calligraphy.

Well actually the entire discussion ultimately boils down to this:

https://en.wikipedia.org/wiki/Test-and-set

 

I see, OK.

The drawback of course is that attempting to open a locked file simply fails, theres no option to efficiently wait so with heavily used "mutex" and/or many threads there will be both CPU and performance pains.

Hi,

Actually in the language I referred to, there is a way. The CPU does not have to keep sitting there trying and trying and trying and using up cycles just to find out if the file is available. There is a function called "sleep", which puts the thread to sleep for a given amount of time.
It's unfortunate that the params are integers and in seconds not milliseconds like the Windows API "Sleep" function. But there is a workaround, and that is to use "sleep(0)" which puts the thread to sleep to allow other threads to take over. It only awakes again when it is it's turn to work again, which would not be exceptionally long.
Better I think would be "sleep(0.1)" but that's not allowed it has to be an integer, which I think is not too well thought out.
The Official language is meant to be relatively simple and straightforward so it's easy to debug. It does work out that way in practice also.

A better workaround is for me to bind a new copy of the interpreter, which would include a function sleep(0.1) for example. The Windows API has the function "Sleep(n)" which takes 'n' in milliseconds. If I was to bind a new copy, I would probably include that in the modified code. I've already created a copy that does complex math.
I just don't feel like doing that because it takes a lot longer and a lot of thought and I have too many other things going on again.
This means I will probably use "sleep(0)" in multiple calls for now just to get a longer wait time without going to a full second.

You did bring up a very good point though so I can see you fully understand this.

 

I know that I am a hardware-only guy, when I read the thread above, and it makes as much sense to me as a discussion in early Ming dynasty calligraphy.

Hi,

It depends on what language you are using. It may not matter for you at all, ever, but for the next guy it may be useful. That's the way a lot of things are.
I do a LOT of programs in this language and sometimes I just want to through something together for some reason, sometimes that I will only use one time or just to test something about the system or file system or something else.