But you need to pick your horse. In 5 years time, Python will either be GIL or no GIL, and it is hard to tell which. It might be a setting (which might be more ideal).
If you assume nogil, you need to choose dependencies that support that. You may need to trade off: eschew dependencies that aren't looking like they will be nogil compatible by the deadline. You are stuck on Python 3.18 maintenance branch or whatever, rather than the 3.19 (in reality .. 4.0) version.
Or choose gil then you can use everything. But is there a prisoners dilemma - everyone picks gil, uses whatever dependencies, library maintainers assuming this don't bother to add nogil support, and then the decision becomes to stick to gil, which if you suspect will happen makes you reason even harder not to support nogil.
I don’t really understand this. Unless I am missing something you should always pick the “no GIL” version as that will work with or without a GIL. Thread safe No GIL code would be totally fine to run on python compiled with the GIL with zero modifications.
Because of this I don’t expect there to be multiple versions of any library. Once a library does the (admittedly heavy) lift to no GIL it will just be the main version of that library going forward.
Each library maintainer (probably mostly volunteers) has to decide whether to put effort into making their code thread safe. Clearly it won't be 100% of libraries that "upgrade".
Then on top of that, they know their effort might be for nothing if the decision is made to keep Python GIL-only all along (one of the possible 3 outcomes at the end of the 5 years: ["gil", "nogil", "both supported").
> Clearly it won't be 100% of libraries that "upgrade".
I'm wondering how many libraries with binary extensions are actually in common use. Like, maybe 90% of python projects use a subset of a few hundred such packages?
That's a hassle if you maintain one of those packages, and will be a bit disappointing if in 5 years' time you're still depending on GIL-reliant packages.
But it's nothing like the chaos of the python 2-3 changes, where ~100% of python files in every package and end-user project had to be fixed.
I only learned about this this morning though, it's very possible I'm missing something. A lot of the concerns people are raising look a bit overblown to me.
I take the point that after so many abortive GIL removal attempts, it's harder to be confident this one will happen. But having the go-ahead from the steering council seems like a good indicator this one has traction.
But thread-unsafe code is not the same as incompatible code. That's the point. You can just choose to say "NOT THREAD SAFE" (just as many C libraries aren't thread safe and need to be wrapped in locks to be used by multiple threads) and users will still be able to use it. More importantly, if it's a pure Python extension, you can just not modify the library and the users will still be able to use it whether or not they have gil or no-gil.
That’s true. I was more thinking from the perspective of a library user not library dev. I suspect for some classes of problem going no GIL will be so tantalizing that the work will definitely be done. Either in the incumbent library or an upstart will come out and take over the community with no GIL support.
Current plan says there has to be separate builds per module, as if it is an ABI break. Would be much better if it could be combined into one build. Hopefully necessity triggers some invention here.
There's no way to make it work with the old ABI. Because sizeof(PyObject) is fixed in the old ABI, there's simply no way to attach additional information (e.g. the new cross-thread ref count) to every Python object. The Python ABI (even the "limited" stable ABI) exposes too many implementation details, it's not really possible to make any fundamental changes to the Python interpreter without breaking that ABI.
You could have a single new ABI supporting both no-GIL and with-GIL, but it wouldn't be compatible with the existing stable ABI.
You're missing something, which is that a lot of libraries will be "i-don't-care-about-gil". Only native extensions need to choose GIL or noGIL due to the ABI difference, but pure Python libraries should run with the same code in both variants. And a lot of them will probably be thread safe at some level (function or class) without any changes. For those that aren't thread-safe, I bet that quite a lot can just get away with a "NOT THREAD SAFE" warning and letting the user wrap access to them with locks.
And that's talking about multithreaded code. I bet that even with noGIL, lots of Python code will still continue to be single-threaded, making the gil/no-gil decision irrelevant (save for those native extensions).
If you assume nogil, you need to choose dependencies that support that. You may need to trade off: eschew dependencies that aren't looking like they will be nogil compatible by the deadline. You are stuck on Python 3.18 maintenance branch or whatever, rather than the 3.19 (in reality .. 4.0) version.
Or choose gil then you can use everything. But is there a prisoners dilemma - everyone picks gil, uses whatever dependencies, library maintainers assuming this don't bother to add nogil support, and then the decision becomes to stick to gil, which if you suspect will happen makes you reason even harder not to support nogil.