Back in reality, though, complaining about the GIL as though its a serious barrier to adoption amongst developers that know what they’re doing often says more about the person doing the complaining than it does about CPython.
It was a solid, well-argued piece up to this point. You do yourself and the Python community a disservice by writing off your critics as ignorant. It sounds petulant and childish, and is wrong.
There are valid arguments on both sides of the GIL argument, but neither side's advocates are ignorant or bad programmers.
In my experience with Python, the issue of the GIL comes up far too frequently. I use Python first as a prototyping environment, and once I have a proof of concept I like to optimize for performance (without re-writing the entire program in a more appropriate language). Most of the stuff I do is CPU-bound, but also involves large amounts of data. So, without writing C code as an extension, I can either suck it up and deal with single-core performance, or I can use the multiprocessing module and hope that inter-process communication isn't too expensive or difficult to code (and no, it's not always as simple as just using multiprocessing.Queue). I am under no delusion that the GIL will ever be "fixed", but I am disappointed that I spent years working with Python before I got into more CPU-intensive tasks and eventually hit the brick wall that is the GIL. And of course, comments like you mentioned, which attack my competence as a coder, are not constructive.
I'm interested to hear about your CPU-bound Python programs that also involve large amounts of data. Normally such problems are I/O bound and are better solved in a data parallel way. (Graph problems can sometimes be an exception, but those are still generally I/O bound.)
For writing extensions, have you considered Cython?
1) My current project at work is a GPU-accelerated keyword-matching engine. The project was started before I joined the company, so I had no say in the choice of Python. Keywords change infrequently, while we analyze a continuous stream of incoming text. There are several million keywords, ranging from small to enormous in size. Aho-Corasick (http://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_string_mat...) is a pretty ideal algorithm for this scenario, which we use for the GPU matching kernel.
AC requires some preprocessing of keywords into a deterministic finite automaton (basically a suffix trie). This is very expensive for a large number of keywords with a large number of characters. The DFA grows to something like 10GB while being built.
Meanwhile, the main engine loop has to be running continuously, while updating keywords in the background. The engine is a service available to other systems on our network, so it uses multiple threads for concurrent I/O. The problem is that the GPU performance is so ridiculously high that the CPU can't keep it fed with data. I've profiled it and this is not a memory-bound problem...the CPU simply cannot keep up with the document streams that we send to it.
The concurrent I/O threads cannot reasonably be split across processes because they need a shared memory space for the data structures driving the engine. So clearly, the background keyword updating is a problem if it runs in the same process as the rest of the engine. I spent a lot of time trying to figure out how to get the keyword updating working in its own multiprocessing process. It's a complete hack to work around the failings of Python (I can go into more depth about the implementation issues if you'd like). And this is why I loathe the GIL.
We use Cython for some aspects of the code, but the keyword updating has yielded very little gain. It's difficult to rewrite parts of the keyword updater as more optimized Cython because it uses some language features that do not seem to be supported in Cython.
2) For a personal project, I need to do a lot of timeseries processing. I'm using Python to prototype, with the intention of either optimizing it eventually or possibly rewriting it in a more suitable language. I've found parsing timestamps to be particularly CPU-intensive, while working on gigabytes of data. Most data I send to a multiprocessing process will have to be returned in some form eventually, so communication costs are huge. So huge, in fact, that I only see a 10% speedup from splitting the workloads evenly across six cores. Profiling reveals that the majority of the "processing" time is actually just waiting on data getting sent back to the main process. This would not be a problem with a shared memory space.
Let's be honest: you are upset because you have sometime committed to saying the GIL is a huge deal and you feel insulted by anyone saying it isn't, which is why you are using emotionally loaded words like 'petulant' and 'childish' and 'ignorant' and 'bad'.
The GIL is the single biggest target for language-advocacy FUD against Python by advocates of other languages. I would be a rich person if I got a dollar for every time I saw someone trashing Python as a toy language because of the GIL, without significant knowledge of how to use Python. It's just a much huger issue to someone without real Python experience than it is to people with Python experience.
Setting that aside, there are a vast number of use cases where someone might try to use threads when actually that's not a good solution. It actually isn't incredibly easy to come up with cases where the GIL is this huge fatal flaw. It's really sad how many times I have seen people ranting about the GIL and totally unaware of multiprocessing, unable to give a technical reason why they can't try processes, unaware of greenlets, wholly unaware that the GIL does not exist in Jython/IronPython, etc.
I'm sure that doesn't apply to you, which would put you in the minority. Thus, "often says more about the person doing the complaining".
Threads should be used judiciously, because they dramatically increase the complexity of a program and the difficulty of debugging and reasoning about it. Shared-everything is a great way to blow your foot off if you don't specifically need it. So if you throw threads at every problem indiscriminately, then that really is a weak point in your programming. (Note: I am not saying that using threads is always a bad idea)
If you are a good programmer then you should already know, and not be offended to hear, that threads are a tool of specific applicability, not a panacea to scale up everything.
Let's be honest: you are upset because you have sometime committed to saying the GIL is a huge deal and you feel insulted by anyone saying it isn't, which is why you are using emotionally loaded words like 'petulant' and 'childish' and 'ignorant' and 'bad'.
Thank you for telling me what was inside my head. I was simply unaware of my emotional state and motivations until you helpfully pointed them out to me.
But seriously, "good programmers don't use threads (much)" is your counter-argument?
I have to agree here, despite the two of us rating the importance of the GIL problem quite differently in yesterday's discussion: A better strategy against the GIL FUD is plain old education, not trying to make it a taboo topic. Especially programmers who do know what they're doing won't take too kindly to a community where that becomes prevalent.
I find Nick arguing pretty much to the point and the quote above was ripped from it’s context which is a huge section balancing the arguments against each other.
tldr He argues there are better ways for scaling out than threads and removing the GIL would have enormous consequence _throughout_ the whole code base so it’s removal cannot be warranted.
Reducing 10 paragraphs and 11 bullet points to “only stupid people use threads” is just poor style.
Back in reality, though, complaining about the GIL as though its a serious barrier to adoption amongst developers that know what they’re doing often says more about the person doing the complaining than it does about CPython.
It was a solid, well-argued piece up to this point. You do yourself and the Python community a disservice by writing off your critics as ignorant. It sounds petulant and childish, and is wrong.
There are valid arguments on both sides of the GIL argument, but neither side's advocates are ignorant or bad programmers.