Well, that's exactly what I don't care about. Most languages have native threads. I want an actor runtime with transparent green threads, for example. Like Erlang/Elixir.
> I don't think you understand what green threads are. Green threads are simulated threads that run on a single core.
I guess this is just semantics, but it seems to be that this is an overly narrow and stale definition of green threads. The wikipedia article is not even internally consistent in its definition of green threads necessarily being scheduled only onto a single core (it includes goroutines as an example). The only reference to such a definition cites two sources from the very early 2000's.
I think it's safe to say that in 2023 "green threads" simply refers to user space threads rather than kernel threads, and that may include threads scheduled onto multiple cores.
Nothing implies that green threads need to run on a single core. The summary in your own link is pretty good: "In computer programming, a green thread (virtual thread) is a thread that is scheduled by a runtime library or virtual machine (VM) instead of natively by the underlying operating system (OS)."
Which is half of what we're looking for here. A thread that can be managed by the runtime, scheduled onto a native OS thread, and (here's the important part, where Common Lisp falls down on the job), transparently operates like a real thread or in other words, you can make normal blocking operations and the runtime just takes care of it. For example: goroutines
If you want to be a pedant ¯\_(ツ)_/¯ then OK (and I am not sure how "native threads" == "green threads" but I won't argue) -- look up Erlang/Elixir's model, or Golang's, or Rust's async workers.
That's what I am looking for; basically M:N parallel execution model where M can be 50_000 and N (the number of CPU cores/threads) is no more than 32-64.
The rest is a suboptimal mess and, as already mentioned above, I am not looking to invent my own async [task] orchestration runtime for every project I participate in.
> I am not sure how "native threads" == "green threads"
You are still confused. Green threads and native threads are mutually exclusive implementation strategies for threads. Native threads are provided by the O/S. Green threads are implemented at the user level without relying on any O/S capabilities. Green threads, by definition, run in a single O/S process and can therefore only use one core.
> look up Erlang/Elixir's model, or Golang's, or Rust's async workers
I'm familiar with those. Those are not "green threads". But whatever you call them, you can do those things in Lisp too.
FWIW, you are literally the first person I've seen who has insisted that green threads are incompatible with an M:N threading model. Every other definition I've seen for green threads is that they involve userspace scheduling, and there's no requirement that there must be but one thing that something can be scheduled onto.
It's possible that the definition has changed; (human) language is malleable. But the original definition of "green thread", which goes back at least to the 1990s, was in contrast to "native thread" where the latter was implemented by the operating system and the former by the language implementation. The whole point of "green threads" was that they let you do something that looked like parallel processing (but was really just time-slicing under the hood) with no support from the O/S.
If there's a new accepted definition someone should update the Wikipedia page.
Traditional, purist definitions: 1:1 threading = native threads, N:1 threading = green threads, M:N threading = hybrid threads (since it is a hybrid of the native and green approaches)
However, a lot of people nowadays call M:N threading "green" instead of "hybrid". Just Google it you will find lots of people using the term in this way. From a traditional purist viewpoint it is an incorrect usage, but it is also now very common. I guess part of the reason is that green threads in the original sense is less useful today – one of the major historical motivators was it could run on OS platforms which lacked native threads, which was a common problem in the mid-1990s and earlier but nowadays almost never is – so it is unsurprising the term gets stolen for a closely related yet distinct technology with far greater contemporary usefulness.
Wikipedia's article – https://en.wikipedia.org/wiki/Green_thread – essentially contradicts itself. The start of the article gives the traditional definition of "green threads". But then the "Green threads in other languages" section ends up describing lots of things which are much closer to being "hybrid threads" than "green threads". This is the problem with an "encyclopaedia which anyone can edit", it is easy to unintentionally edit an article into contradicting itself, and on highly technical topics it is easy for the contradiction to go unnoticed. I myself am not sure I can fix it, because although I know the true story about the definitions (or at least I think I do), I don't know any reliable source to cite for it.
Their thread article – https://en.wikipedia.org/wiki/Thread_(computing)#Threading_m... – does define M:N as "hybrid threading", whereas it calls N:1 "user-level threading". It also notes "green threads" as a synonym for "user threads"–but "user threads" aka "green threads" exist in both the M:N and N:1 models. It never defines "green threads" as a model as opposed to as a type of thread which exists in two out of the three models.
Thanks a lot for the context. I wasn't aware of the distinction between green vs. hybrid threads and was only using "green threads" I suppose as "suspendable coroutines" or "fibers"? Not sure about that either, gotta admit.
There’s history here. Green threads are as opposed to native, you both agree on that. The original green threads were pretty weak and behaved as described above. People use the green threads term today to mean a much more featured version that is as you describe it. Lisp also has it. So really everyone is right, except for your statement that Lisp does not have green threads. It has native, old style green and new style green threads.
The only implementation I know of that offers these features built-in is Allegro. But the cool thing about Lisp is that it's easy to add features like this at the user level even if they are not a native part of the implementation.
Not convinced. If it's so easy why hasn't anyone done it yet and made it easy to consume for everyone else?
To me this is a red flag. "Just roll it yourself" is not a constructive response in the eyes of a commercial programmer. We come to an ecosystem expecting certain basic building blocks. I am not paid to evolve LISP's ecosystem. Not to mention nobody will give me the time in a world where deadlines and milestones are a fact of life.
If I was doing programming purely as a hobby -- sure! But I don't. And I'd bet most commercial programmers don't as well.
> why hasn't anyone done it yet and made it easy to consume for everyone else?
Dunno. No demand? No standard?
I recently wrote a library like this for my job. It took me about a day. I haven't published it because it was a work for hire and so it's not mine to publish. But it's the basis for my confidence that it's not hard to do.
If you want to write up a spec, I'm available for contract work (and I'm sure you would have people lining up if you actually were willing to pay someone to do this work). But I suspect you would find that writing the spec is the hard part, and if you actually did it you would find that it's easier to implement yourself than to try to hire someone.
I mean, high-quality work should be paid. Obviously.
But now we're making a full U-turn to my initial argument: LISP doesn't have features that I find are (or should be) basic building blocks so I am just taking my business elsewhere.
I think you are being willfully disagreeable here. There is a Lisp with the style of threads you claimed doesn’t exist. So you’re simply wrong with your claim. Also even if that didn’t exist, no one can say that a day’s worth of work is a prohibitive barrier to getting things done.
I suppose part of the context got lost here but no, I haven't gotten out of my way to disagree specifically; more like I got frustrated because the other guy seemed to purposefully miss the point from where I was standing.
I don't want just a best-effort-with-what-we-have-in-our-runtime-that-was-not-ever-designed-for-it implementation however, I want proper preemptive scheduling and a transparent M:N threading model, very akin to what Erlang/Elixir have.
I was made aware of Clojure's core.async but after reviewing it, it's IMO not good enough. Though as others have said it can be used as a building block for something better. That seems to be true as well.
But it does. Allegro CL has it. Other implementation might have it too. It's hard to say because you've been very vague about exactly what it is that you want.
Was I vague? Or were you being overly pedant about Wikipedia-sourced definitions that, as another commenter pointed out, are kind of outdated? And now trying to frame me as not knowing what I want? And pretending that the examples I gave about three other languages don't paint the picture well enough?
You seem to be arguing in bad faith from where I am standing.
...And okay -- I'd like a M:N green threads runtime (or library if you prefer) with preemptive scheduling or at least not a completely manual cooperative scheduling. And yes, we're talking full 100% CPU usage on all cores when needed. Not single-thread.
And to repeat, since you kind of pretended I never said anything about what other languages do, and if you are indeed aware of how Erlang's BEAM VM is doing it -- that is what should be present in more than one language / runtime IMO. Failing that, Golang's goroutines and Rust's async workers+channels are quite fine too.
To me, in 2023, no language has an excuse for not having something like that. Modern example: OCaml team worked on it for years and recently delivered it.
> Where is the stuff like Erlang/Elixir's green threads or Golang's goroutines/channels and Rust's async runtimes workers/channels, and OCaml 5.0's recent multicore runtime (and the emerging libraries around it)?
> I want the higher-level stuff. I want to solve problems and not invent a new async orchestration runtime for every project I work on.
That seems pretty vague to me. You mentioned four different languages (Erland, Go, Rust, OCaml). Do you want the intersection of the features of all those languages? The union? Some subset?
> You seem to be arguing in bad faith
I didn't even realize this was an argument, so I guess the problem must be that I'm just too dim to glean the correct meaning of your words. Sorry about that.
Well it is an argument insofar as you keep insisting that the LISP ecosystem has what I want, and I keep disagreeing with that claim.
But yep, I want Erlang-style concurrency / parallelism and, failing that, something like Golang builtins or Rust's libraries.
So not a subset or intersection, more like a priority-ordered wish-list: I want what I perceive as an ideal model (Erlang) but if that's not available, there is stuff 1-2 floors below that are good enough (Golang, Rust).
But native threads or fully cooperative opt-in parallelism are practically a drag (or were in the projects I used them) and aren't cutting it for the work I do.
I'll grant you that the building blocks for something like what I need are there but I am not willing to put the work to create the runtime / the library, nor pay anyone to do it for me. Hence my original post: I am commenting on the status quo, not on how it could theoretically change at any time. I find the latter of no consequence.
OK, but you must want something besides that because...
> an ideal model (Erlang) but if that's not available
It is available. In Erlang. So if that's what you want, why are you not just using Erlang? Why even bother with the contingency of "if that's available"? Why are we having this discussion at all?
This is the reason that Erlang-style concurrency is not available off-the-shelf in Lisp. There's no market for it. The people who really want Erlang-style concurrency just use Erlang. No language will ever do Erlang-style concurrency better than Erlang. Erlang-style concurrency is Erlang's defining feature. The whole point of Erlang is to do Erlang-style concurrency. You can't do better than Erlang at "Erlang-style concurrency" by definition.
The point of Lisp is not do X-style-anything better than X. The main benefit of Lisp is that it allows you to explore a much larger space of possible solutions in a much shorter time, which is a big win when you don't know what you want, which, I submit, is most of the time.
> So if that's what you want, why are you not just using Erlang?
I do.
> Why are we having this discussion at all?
I saw an article praising LISP, I decided to chime in with realism because certain fandoms (LISP's included) seem very unaware of the realities of the commercial programming outside of their niche hobby language. As a senior dev (who also worked as a CTO a few times) I have learned to evaluate technology and to never wear rose-tinted glasses even for my favorite tech stacks. They became favorites based on merit and nothing else. (In fact I am starting to dislike working with Elixir for certain projects, even though I loved doing them with it in the past.)
LISP is not cutting it for commercial work in general, so I strive to bring nuance to articles (or discussions) that to me seem heavily tilted to the "I am a fan!" direction. And forgive me if "LISP has longevity" and "it's future-proof" seem like hand-waving to me. I don't see factoids, I see people reinforcing their own positive feeling based on actual factoids (like the powerful macro system and a REPL, for example).
And we keep chatting because you seem to insist that either LISP has what I deem good (disagreed, it doesn't) or that it's not important / there's no market for it (disagreed).
> There's no market for it.
You're doing post-hoc rationalization. You don't know that for a fact. I know I would have coded much more Racket and Gerbil Scheme if they had proper async runtime a la Erlang or Golang or Rust (OCaml these days as well though the story there is still unfolding after their recent 5.0 release).
> You can't do better than Erlang at "Erlang-style concurrency" by definition.
Loose definitions then. Rust and OCaml are making very serious strides. I have hope they can surpass Erlang in the next 2-5 years. Both are faster, much stricter with types (lack of those is an endless pit of bugs) and more memory-safe than Erlang (though anything with a GC like OCaml is prone to some of the nastiest problems Erlang has, like cycles between big objects but... topic for another time).
> The point of Lisp is not do X-style-anything better than X. The main benefit of Lisp is that it allows you to explore a much larger space of possible solutions in a much shorter time, which is a big win when you don't know what you want, which, I submit, is most of the time.
We finally got somewhere productive, thank you.
I use Elixir (lives in the Erlang's BEAM VM so has access to everything Erlang) for the same and I agree that being able to explore quickly is very valuable. I've only made the mistake to prototype stuff with Rust once. Never again. Nowadays I use Elixir and Golang for prototyping and the end products either remain that or get rewritten in Rust.
Also REPL story can be better with Clojure but I'll admit it at least exists, unlike that of many other languages. Startup time is not ideal but then again, neither is Erlang's sadly. Editor support I haven't checked in a long time, might be good. Library coverage is very hit and miss depending on which LISP you use. I keep hearing CL has a lot, maybe that's true but I am 50/50 there; judging by your attitude -- "it took me a day to roll it myself" -- I remain unconvinced that the library story is good, it's more like some pie-in-the-sky goodness that's eternally out of reach. Still, if I reach for LISP again in the future I'll evaluate that aspect in detail and will know for a fact.
So yeah, on the "LISP is quick to prototype stuff with" I agree completely. To me it doesn't go all the way however, hence my initial comment.
You can think of me as "dislikes anything that looks like shilling".
There is no place for feelings in our work. When I retire and if I still want to code then, maybe I'll make decisions based on feelings. Before that -- no.
> I saw an article praising LISP, I decided to chime in with realism because certain fandoms (LISP's included) seem very unaware of the realities of the commercial programming outside of their niche hobby language.
Lisp's detractors often seem equally unaware. I've been using Lisp in a commercial setting for the last ten years. And before that I used it in a research setting for 15 years, and even got Lisp sent into space. It worked great.
> LISP is not cutting it for commercial work in general
Lisp is rarely tried for commercial work nowadays, in no small measure because people like you keep taking pot shots at it from the side lines. Have ever actually tried using Lisp in a commercial setting? I have. It works great.
> > There's no market for it.
> You're doing post-hoc rationalization. You don't know that for a fact.
Well, I pitched the idea to you and you didn't bite, so there's a data point.
Yes, it's possible that there's a huge untapped market for Lisp out there if only it had Erlang-style concurrency. But I'll give you long odds against that being the limiting factor. The limiting factor from where I sit is ignorance and prejudice.
> To me it doesn't go all the way however
That's fine. But please don't assume that because it doesn't go all the way for you that it can't go all the way for others. Different goals give rise to different requirements.
No, I go not think this particular feature can be added at the user level. It's something that requires significant support from the runtime to handle all the blocking -> thread parking shenanigans. Attempts to do it at the user level are terribly limited in that respect and just end up like clojure's core.async.
You're not wrong, but most modern CL implementations provide all the primitives you need even if they don't go all the way and provide a fully-fledged Erlang-style parallelism library out of the box.
The first two links, when it comes to green threads, have nothing that fit the bill. They are not transparent. They have all the problems of clojure's core.async. What you need is to be able to call existing blocking code and have the runtime take care of it. A macro based solution cannot do this, because macros are purely source to source transforms. If it can't access the source, it can't modify it. It must be done by the runtime.
i very much doubt that there is anything in lisp that can prevent that from being implemented at a library level (library because lisp is ANSI standardized). if you can prove that lisp is inherently unable to do what you call transparent threads im pretty sure it would be a significant cs journal paper
anyway it seems to me that this discussion evolved from lisp doesnt support multicore/ concurrency/parallelism (false) to "lisp aint erlang" (truism)
if you want erlang just use erlang, or a lisp version of it https://lfe.io/
