Do you know if this will work on a cluster if the shared filesystem does not support unix file locking? The "pidfile" approach, which stores the hostname and process ID of the acquirer in the lock file, does not depend on filesystem support for locking. I don't think filelock uses this approach, so I'm a bit worried about robustness.

Filelock advertises itself as platform-independent, and has windows and unix specific functions. I haven't been able to test this on windows, however.

One question I have about this PR is where to write the lock. I have it being written to the source directory of pyjuliapkg, but I suspect it would be better to use some location which is specified by the STATE dict. I just don't know which one makes sense to use here.

Filelock advertises itself as platform-independent, and has windows and unix specific functions.

It can be platform independent and still not work well across multiple hosts on a shared filesystem.

I see what you're asking. The filelock library seems to advise the use of the SoftFileLock type in the event that there are multiple hosts, but this may require sometimes deleting the lock file when someone interrupts the setup. The julia developers seem to have had similar concerns in the design of Pkg, and ultimately I'm not sure of the extent to which this is solved. The solution in this PR should fix bugs on not-shared filesystems, and I'm open to input on how to ensure this also fixes bugs on shared ones. I chose the PID file package off of PyPI with the greatest support that offered a simple reentrant lock, but I'm open to suggestions on others?

Yeah, I understand this is a nontrivial choice. I shopped around for a Python package that ticked all the boxes (basically replicating the Julia solution) but didn't find one that worked well out of the... well, box. Any locking is definitely better than none here and I appreciate your taking this on!

Edit: And I don't mean to sound like I'm a maintainer here. I'm not! Just a user quite invested in making it easier to have python packages that use julia (including for use on clusters).

Looking at package managers in python:

• conda appears to roll its own lock https://docs.conda.io/projects/conda/en/23.7.x/dev-guide/api/conda/lock/index.html https://github.com/csadorf/conda/blob/d14718267ec59cde99079cb8e90b6688f609448c/conda/lock.py#L101
• virtualenv also appears to roll its own lock that depends on filelock https://github.com/pypa/virtualenv/blob/caf03a268e03a3453aa4058ec764b46370f0d543/src/virtualenv/util/lock.py#L78
• poetry used to use filelock, but I can't seem to find any mention of file locking anymore: Use filelock to lock the file cache python-poetry/poetry#6471

In case it's useful, I used a somewhat modified version of pidlock to externally lock juliapkg operations. This worked okay on a cluster, but it again probably doesn't do everything right. For one thing, it's missing a mechanism to delete stale lock files.

Also the github version has a defect, in that it checks for file existence and then, if the file exists, opens the file without exception handling. The delay between existence check and open allows other processes or hosts to delete the file and cause a crash. This can be fixed through handling exceptions on open instead of checking for existence separately.