Anything that spawns an unbounded amount of POSIX threads will be bad for performance, so you need an M:N scheduler to execute threads. I feel like this is pretty hard to retrofit onto an existing language (e.g. I don't think you can write a pre-empting scheduler in pure Python as a library either).
AFAIK, only Go does this well (with great results) as a result of it being pretty much baked into the language. I don't think it's an async bandwagon, I think async/await is the best way to get concurrency without requiring the whole language to opt-in (as the Rust devs discovered).
I think it's not necessarily POSIX threads related. In theory there is nothing
wrong with spawning tens/hundreds of thousands OS threads, and Linux does a
decent job at this. But in practise you'll run into issues such as:
* Different OS' handle things differently. macOS limits the number of threads
per process (somewhere around 1500 I believe). Other OS' may be slower, or
impose other limits
* Because of this, sometimes spawning threads is pretty fast (e.g. 10 µsec).
Other times it's super slow (I've seen threads take over 500 ms to spawn)
* Context switches are still pretty slow across OS'. There's finally some work
being done towards lightweight thread support in Linux, but I suspect it
will take a few years to become useful
I _really_ hope that one day we _can_ "just" spawn 100 000 OS threads without
issues, as it would make many concurrency problems easier to solve. Sadly, I
think it will take another 10-15 years.
In theory there is nothing wrong with spawning tens/hundreds of thousands OS threads, and Linux does a decent job at this. But in practise you'll run into issues such as:
The other big problem on Linux is stack sizes - "hundreds" or "thousands" of threads isn't a problem - the C10K problem which proved the dominance of nginx's evented model vs. Apache thread-per-conncetion model was broken 20 years ago. Where it matters is when we reach 100s of thousands or millions of connections on a single node. The first issue you run into is the thread stack size - Linux has a hard default of 8MB; at 100,000 you start to hit OOM issues first. Go (and BEAM) has a ton of work to make this use case possible within the runtime.
What I'm unsure about if Linux even cares about solving this problem - the threading model is pretty general and I'm not sure how important high performance connection handling is. At the end of the day, if you are really I/O bound, an async style, co-operative, event queue will always be more performant than a generic threaded scheduler like the one Linux has; meaning if you care about performance (like Django probably does) they would probably go with an async style scheduler even if Linux could support a million threads.
> The first issue you run into is the thread stack size - Linux has a hard default of 8MB; at 100,000 you start to hit OOM issues first. Go (and BEAM) has a ton of work to make this use case possible within the runtime.
This is not how thread stack sizes work. Thread stack sizes allocate a certain
amount of _virtual_ memory. This means that with a stack size of 8MB you don't
actually immediately use 8MB of physical memory. Only when you start filling up
that memory do you start using physical memory. We can easily demonstrate this
using a simple Rust program:
use std::thread;
use std::time::Duration;
fn main() {
let mut handles = Vec::with_capacity(10_000);
for _ in 0..handles.capacity() {
let handle = thread::spawn(|| thread::sleep(Duration::from_secs(1)));
handles.push(handle);
}
for handle in handles {
handle.join().unwrap();
}
}
We when compile this in release mode, and run it as follows:
/usr/bin/time -f '%M KB RSS' program-name
On my Linux desktop this prints out:
88492 KB RSS
This is roughly 86 MB or RSS memory being used. If the thread stacks would use
physical memory, this would translate to just under 80 GB; and the program would
have been OOM killed because I only have 16 GB of RAM available.
The virtual memory limits in turn are not much of an issue. I think on most 64
bits systems the virtual memory limit is 256 TB. A 256 TB limit would allow up
to 33 554 432 OS threads with a stack size of 8MB. And since you can configure
the limit when spawning threads, you can increase this number by spawning
threads with a smaller limit (e.g. 4MB).
They also read from disk for most requests. So, thread per request and also reading from disk.
I worked at one company where we had many dozens of services with thrift/spring boot and spawned a request per thread and it worked fairly well. We were collecting and aggregating every review on the internet (for our customers), and had dozens of services... There were a couple in a reactive style. So we took the maintenance cost when we needed to.
AFAIK, only Go does this well (with great results) as a result of it being pretty much baked into the language. I don't think it's an async bandwagon, I think async/await is the best way to get concurrency without requiring the whole language to opt-in (as the Rust devs discovered).