I don’t really understand this. Unless I am missing something you should always pick the “no GIL” version as that will work with or without a GIL. Thread safe No GIL code would be totally fine to run on python compiled with the GIL with zero modifications.
Because of this I don’t expect there to be multiple versions of any library. Once a library does the (admittedly heavy) lift to no GIL it will just be the main version of that library going forward.
Each library maintainer (probably mostly volunteers) has to decide whether to put effort into making their code thread safe. Clearly it won't be 100% of libraries that "upgrade".
Then on top of that, they know their effort might be for nothing if the decision is made to keep Python GIL-only all along (one of the possible 3 outcomes at the end of the 5 years: ["gil", "nogil", "both supported").
> Clearly it won't be 100% of libraries that "upgrade".
I'm wondering how many libraries with binary extensions are actually in common use. Like, maybe 90% of python projects use a subset of a few hundred such packages?
That's a hassle if you maintain one of those packages, and will be a bit disappointing if in 5 years' time you're still depending on GIL-reliant packages.
But it's nothing like the chaos of the python 2-3 changes, where ~100% of python files in every package and end-user project had to be fixed.
I only learned about this this morning though, it's very possible I'm missing something. A lot of the concerns people are raising look a bit overblown to me.
I take the point that after so many abortive GIL removal attempts, it's harder to be confident this one will happen. But having the go-ahead from the steering council seems like a good indicator this one has traction.
But thread-unsafe code is not the same as incompatible code. That's the point. You can just choose to say "NOT THREAD SAFE" (just as many C libraries aren't thread safe and need to be wrapped in locks to be used by multiple threads) and users will still be able to use it. More importantly, if it's a pure Python extension, you can just not modify the library and the users will still be able to use it whether or not they have gil or no-gil.
That’s true. I was more thinking from the perspective of a library user not library dev. I suspect for some classes of problem going no GIL will be so tantalizing that the work will definitely be done. Either in the incumbent library or an upstart will come out and take over the community with no GIL support.
Current plan says there has to be separate builds per module, as if it is an ABI break. Would be much better if it could be combined into one build. Hopefully necessity triggers some invention here.
There's no way to make it work with the old ABI. Because sizeof(PyObject) is fixed in the old ABI, there's simply no way to attach additional information (e.g. the new cross-thread ref count) to every Python object. The Python ABI (even the "limited" stable ABI) exposes too many implementation details, it's not really possible to make any fundamental changes to the Python interpreter without breaking that ABI.
You could have a single new ABI supporting both no-GIL and with-GIL, but it wouldn't be compatible with the existing stable ABI.
Because of this I don’t expect there to be multiple versions of any library. Once a library does the (admittedly heavy) lift to no GIL it will just be the main version of that library going forward.