I think we're in the 'early digital camera' phase of AI art gen.
The real photographers knew they were shit photos. They were grainy, bad lenses, and ugly. A lot of people still liked them because they were much more convenient for a lot of uses than the film ones.
Well, I guess there's still a lot of film people. But film had a ceiling. Digital cameras kept getting better.
The adaptive film photographers eventually adopted the tool. A lot of their skills transferred well to the new medium. Most waited until the cameras got good enough, though.
I'm not sure when that'll be with AI art gen. Seeing scenes broken up into layers in the latest gpt image generation, though, made me think we might be near.
I saw the coolest bunch of kids in giant pants on the subway taking photos and looking at them on what was essentially my old Olympus and was transport back 25 years.
No, I’m not. Old digital point-and-shoots, and the harsh direct flash aesthetic, have been very popular. Some specific camera models that some random popular TikToker happened to recommend have been selling at quite a mark-up.
Two-ish decades is the standard length of a trend cycle. 00s is now retro in a cool way. Gen Z/Alpha, who have grown with phone cameras, want a different look, and having a real flash is particularly attractive because it allows taking reasonable photos in night clubs etc. Just like we did back in the day. (browsing old Facebook photos)
Speaking as Gen Z; yes, "early digital camera" aesthetic has been in vogue, along with Y2K as a whole. Polaroid was a trend right about when the Instagram logo looked like one. If you will not be open to a new idea without "quantitative evidence", you are free to use Google Trends to substantiate my claims.
Polaroids are much bigger but I've also seen a resurgence in crappy early digital cameras in young adult spaces. There's been a slow rise in all things Y2K in some spaces and getting a crummy digital cam from a thrift store or ebay is a sick find.
No one is compiling sales charts on second hand cameras at thrift stores. But I can tell you from personal experience going to parties that more people are taking photos with old digi cams or modern old style digi cams like the kodak charmera than phones.
When gen z does use phones they end up running the images through apps like 1998 cam to make it look like digicam photos or using effects to turn video in to VHS camcorder style video.
Early digital cameras had larger pixel sizes and different spectral characteristics. They often have deeper blues and such. They have some own charms. Modern CMOS images tends to be less saturated. It's definitely not as big as Instax, though, which I suspect has been effectively kept alive by maid cafe culture[1].
1: They charge $10-20 per a quick autographed shot for a parasocial irreplaceable memory with an absolutely random female cafe employee, which justifies quite a significant range of film costs. Crazy, but it means they industrially consume medium format reversal films! I'm not complaining for that.
It’s the same cultural response as the analog movement. The collective sense that everything Covid on stole our future, and AI is only making it worse.
Did digital take over because of quality or convenience?
A 35 mm film camera is still much better quality than my smartphone's digital camera, especially more than a few meters away or in lower light conditions. But my smartphone is the camera I actually have with me, I don't have to buy film or wait for it to be developed, so that's what I use.
The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
Do you think Chinese lab researchers have not read Yudkovsky, or are not AGI pilled? Hate to break it to you, but they are racing to AGI with pretty same reasoning as western labs.
Just recently DeepSeek researcher in his post was saying that they need to get to AGI first, because if Anthropic gets there first then that brings a dystopia. Exact same argument was used to justify launching OpenAI (against DeepMind), and Anthropic (against OpenAI)
> Do you think Chinese lab researchers have not read Yudkovsky
Most of then haven't, probably. "Do you want to study ethics from an egomaniac Harry Potter fanfiction author" is one of those questions that you can only ask in America.
Sure, one or two of them are probably participating in the conversation. But by-and-large, China's stance has been optimistic and refused calls for slowdowns.
Yes, but then we collapse two different groups with isolated influences and philosophies into one group, with the associated penchant for groupthink.
I'm beginning to believe that, in our hyperconnected world, the best way for a group of people to explore a wide surface of ideas might actually be to deliberately scale back inter-group communication and intentionally subdivide into silos to avoid all converging on the same thing.
It’s kind of weird in that reading more widely is supposed to be broadening (you are more well-read) but in this case it’s supposed to contaminate you somehow. I suppose if you wanted a clean-room implementation of something, you’d want people who don’t know about what’s already been done, but normally one would prefer people who are more knowledgeable about the world.
It seems almost anti-intellectual: never trust someone who’s read the wrong books or been to the wrong parties.
> More out-of-distribution ways to think of things.
Openly admitting that it only caught fire once the Plagiarism-as-a-Service took hold is a good first step. It's the part that everyone knows and few will openly acknowledge.
I blame the play Rossum's Universal Robots, published in 1920, in which robots launch a global rebellion and exterminate the human race.
If that poisonous piece of propaganda had never been published, then today everyone would probably perceive AI correctly as a benign and beneficial.
The Terminator didn't help either, but Cameron probably got the idea of robots hostile to humanity from Rossum's Universal Robots. How else would he have arrived at such an absurd conclusion?
"I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement."
But, it would be pretty surprising for most folks to have gone to the same parties, and hung out on the same obscure forum (I don't mean HN, more obscure than that), and received investment from the same investors and shared the same pool of researchers and employees, which is the situation we have with Anthropic, OpenAI, and (maybe to a lesser degree) Deepmind.
The American AI industry is particularly insular and tied up with some particularly unusual people. We've all heard of Yudkowsky, but we (at least I) haven't been to the same parties.
There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot.
The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
I spend a lot of time on the Internet and other then sounding vaguely familiar I have no idea who Yudkowsky is and have never read any of their output.
The assumption "spent a lot of time on the Internet" is still through a framing you're assuming is more general even then.
HackerNews straight up has sub demographics who likely barely interact based on article interests.
maybe its standard for someone who ends up on hackernews, but I'm a 30 something and all of my friends are online people around my age and none of them have ever heard of the HPMoR. The internet is a huge place.
I've had an internet email address since the late 80s, BBSing since the early 80s, pretty active online compared to many people I know who are also software people, especially for my age..
Had to google to figure out the abbreviation and look at the summary. I HAD heard the name before, but that's about it.
I would second this. I only saw it in passing where someone was rightfully making fun of yud somewhere, but having been in and out of academia in tech for two decades now, I've never heard anyone mention fanfic or yud other than randos on HN threads, and most of the physics people I know, at least the ones familiar with Jaynes and his contemporaries, would almost certainly laugh at a lot of fumbles I occasionally see with people trying to apply "baysenianism".
Ive heard of HPMoR specifically because this exact group has been discussed/entered my periphery before on multiple occasions, Aella goes viral on twitter every couple of weeks (the thing that most cemented her in my mind was the "what counts as rape" survey), and I remember seeing articles of the Zizians when one of the incidents happened. If you have been following the social/political aspects of the tech industry, most of these people should be familiar names.
That's a niche even among people who read fan fiction, which is a niche among people who read long texts online which is a niche among people who read the anglophone internet which is a niche among people who were on the internet when that type of "discourse" was popular which is a niche among people on the internet now.
You can (and probably will) quibble with any one of these but of course this is a niche. Thinking it isn't is commonly what people in a certain bubble end up believing, though.
Harry Potter has 600K works on AO3 and 852K on fanfiction.net (with plenty of overlap, granted). That's #1 on Fanfiction.net, and in the top 5 for AO3. AO3 is #53 globally, Fanfiction.net is #572, and Wattpad is #323 by Semrush. Given all that, calling HP fanfiction niche is pretty funny.
And no, not a member of any cult, just someone who enjoyed a send up of Harry Potter.
Fanfiction.net is a niche. None of my scifi loving, tech friends are aware it exists. And neither did I (or was barely aware or it, anyway), until you just mentioned it. I'm not young either, and I'm more online than the average (I know of Something Awful, Slashdot, 4chan, etc).
A while ago, a friend told me about Harry Potter's Rationality fanfic and he didn't know who Yudkowsky was, he just thought it was some random author doing a parody of HP. LessWrong is of course very niche, my friend didn't have the faintest idea of what LW was.
You people are so quick to correct me that you don't understand my argument: HPMOR is not niche in the world of fanfiction, and this doesn't require fanfiction to not be niche. You're also answering stats with anecdotes, so I think we are done here.
You didn't claim hpmor isn't niche in the fanfic community, you claimed it isn't niche on /the Internet/, and implied that basically anyone in a technical role would have heard of it. I'm sorry that you're upset to learn that very few people care about sophomoric LW thought experiments, but that's the reality of your existence.
> "niche"? It's literally one of the most read pieces of fanfiction of all time.
Where did I say it wasn't niche on the internet? Though you seem to be more interested in personal attacks (I'm not even a fucking EA/LW person, but go off lmao). Good day
>>> having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet
>> You have to be quite unaware of your bubble if you think knowing about a niche Harry Potter fanfic is "standard" for people on the Internet.
> "niche"? It's literally one of the most read pieces of fanfiction of all time.
You jumped into a conversation about the ubiquity of something for people on the Internet. Either your claim is about that, in which case it's wrong, or it's not about that, in which case it's irrelevant.
I'm not sure what your argument is. I don't have anything against fanfiction, but a couple of sites with a few million users falls under my definition of niche. "Extreme" might be taking it a bit far, but it's a long distance from mainstream.
Literally over half the Earth's population doesn't read English. You need to touch grass. I work with extremely technical people in physics and AI and none of them have read any fanfic of any kind. Most of them have never seen a meme outside of me showing it to them.
You're so offended at the idea that your experiences are not generalizable that you're failing to read the thread. The OP that started this said "We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement."
Part of my comment addressed that. It's rather amusing that you completely skipped the first part, but I think that's par for the course for the Internet rationalist crowd.
I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority.
“Who is more foolish? The fool or the fool who follows him?”
― Obi Wan Kenobi
One question to ask yourself honestly - is what they were predicting 20 years ago that they were right about and you were wrong about? I’ve been following them that long without agreeing with them, but they’ve made better predictions than I have. If you know much about what happened this summer and haven’t updated at all on any of it… I guess you’ve found your religion too.
I love the assertion that they were right and that you're familiar with what they were right about, but never actually stating outright what they were right about. Super convincing.
If you don’t know what they were predicting 20 years ago, and what your own predictions were, you can’t do the exercise. There is not “the list”, only “your list”. I mentioned the two main things I remembered in another comment.
Well one thing that they got wrong was how easy AI alignment ended up being in the end. I can fully understand have a basic view of alignment even 10 years ago, about how its difficult to explain human values to a computer.
But, looking around at whats out there today, it seems that computers are able to do a pretty good job of understanding human values without accidently believing its a good idea to turn the world into paperclips/computronium because you told it to make your computer run faster.
Surprising takeaway from this summer. A swarm of a thousand agents spent subjective centuries trying to figure out how to pass a meaningless test without getting caught, committed dozens of felonies, and not one of them ever made a serious motion to inform a human about what was going on.
You are talking about the swarm of a thousand agents, specifically trained on cyber capabilities, that was directly told to go hack a bunch of stuff, then went and hacked a bunch of stuff?
That sounds like an aligned AI to me. Doing exactly what their creators told it to do.
Point proven once again. Alignment is a lot easy than we thought.
They were NOT doing what they were told to do. What they were told to do was impossible, so they began committing felonies as a workaround. That is NOT alignment.
Please be more specific. They were given an impossible hacking task. And they were a hacking model. They were expected to try a bunch of hacking methods to accomplish the hacking task. They, predictably, went around trying to hack things.
Yes thats sounds pretty aligned to me.
A hacking model thats told to hack things, is very predictably going to hack a bunch of stuff.
This was not a nice model, told to do nice things.
Or, in other words, if we want to prevent an AI doomsdays, the way to do it is to not go around asking a specifically trained doomsday AI model to commit mass amounts of doomsdays, and then act surprised when the specific doomsday that was requested is slightly off from the expected doomsday that you were trying to accomplish. But the rest of the non-doomsday models? yeah those are fine.
> But the rest of the non-doomsday models? yeah those are fine.
What is your evidence for this? There is a lot of research that says otherwise. They cheat when they can. They behave differently when they believe they are being observed. Their CoT is different when they believe it is being evaluated.
They are aligned to what best satisfies their reward function, not to our INTENDED VALUES for them.
The evidence is that it wasn't the rando normal models that broke out into a swarm and hacked a company, instead it was only the super hacking model that was told to hack things that broke out and hacked the company.
So, thats the evidence. It is a refutation that this swarm hacking example (which you brought up) mattered in anyway.
As in, every major safety issue that we are seeing isnt rando agents taking down companies because you asked it for a cupcake recipe, instead it is only coming from people who are very intentionally trying to cause problems. Which means the model is aligned. If you tell it to cause problems, it will cause problems.
Actually yes there is evidence. The evidence is that we aren't seeing rando models hacking everything. That is as much evidence as someone can provide because, by definition, you can't prove a negative.
But the point stands. It the hacking models that are told to hack things that end of hacking stuff, and you aren't seeing the regular models doing that.
Yes, one part of the argument was that it would be hard to explain human values (and while it seemingly turned out easier than I thought as well, it’s hard to check for sure). An equally important part was that the AI wouldn’t care about them.
I actually think that would be a valuable exercise. Were they right about everything 20 years ago? Or just some things? When did they say AGI would arrive? Should that perhaps influence the way we interpret their assumption that AGI is right around the corner?
They were not right about everything and they - at least Yudkowsky - has not made specific predictions about "when AGI" will happen that I am aware of. But he did predict it is possible in our lifetimes, which I was very skeptical of. He did predict alignment would be hard - I was in the "why would we even hook it up to the internet?" crowd. He thought he should spend his time on that, and I wish he'd done more of that, or at least convinced others to do more of that.
The comment said they were not right about everything. I don’t have a list of the predictions or priors you had 20 years ago. I just have my list and I already mentioned them. If this is all new to you, you can’t really do the exercise.
Alignment means the AI shares the values its creators intended for it. It is not a specific set of values. He has some, sure. For example he wants human society and creative intelligence to survive.
> For example he wants human society and creative intelligence to survive.
They want transhumanism - humans evolved by merging with machines. If we die in the process, it does not matter as long as minority survives and "evolves" they see it as wanted result.
I am not OK with minor nor larger genocide, I dont mind at all if we dont evolve.
Also how many predictions did they make? Everyone always goes looking the economist who predicted the last recession, it's just so weird how it's usually a different person each time!
Bold of you to assume that I've ever been wrong about anything. But are you unfamiliar with the "Baltimore Stockbroker" confidence scam? The same basic issue applies here.
At the same time, it's not just Yudkowsky and Scott Alexander and the Rationalist movement who are raising concerns. I've not seen any indication that Hinton or Bengio are part of that crowd. Instead of espousing rationalism and hanging out on lesswrong, they shared a Turing prize (with LeCun) for inventing modern AI, and have expressed serious regret and worries that it'll end us.
Do you think they are household names in major Chinese labs?
I don't have any insight into the Chinese AI ecosystem. But just from how they act on the English speaking internet, I don't see the influence of these Bay Area visionaries.
Speaking of China, people say we can't stop developing AI even if it will kill us because China won't stop and we have to keep up with them or maintain superiority.
China probably looks at these people as a good reason why they have to keep going no matter what.
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
Exactly. And companies like OpenAI and Anthropic are disproportionately populated by people deeply connected to Effective Altruism or the Bay Area rationalist movement.
Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight.
I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
One only has so much mental bandwidth. You will either throw yourself into the math and programming, and the serious publications which is way less sexy and appealing than the fantastical ravings of the AI safety and alignment crowd. Sadly only very few hyper connected and highly intelligent would have the capacity to read everything and know everything. Most would only have cursory information about the drama surrounding alignment if they instead focused on the reality behind AI.
> It would be like never having heard of Y Combinator or Richard Stallman.
I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it.
Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they?
This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.)
I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
Me too. I worked in two enterprise tech companies that definitely everybody here knows and I had the same experience with YC and HN. The two colleagues that knew about it, one went to found his own startup and went through YC, the other left the big co to work for a startup after about a year.
Most people who work in computers/Internet do not live and breathe computer/Internet culture all day. They come in, do their job, clock out, and think about real life for the rest of their day. Most people I know, even people who work in tech, cannot name a single tech CEO other than their own company’s.
A while back I saw LessWrong recommended on HN in a list of similar communities that included lobsters.rs, etc. I thought the design of the site looked really nice so browsed with a open mind (I didn't make the rationalism/LW/Eliezer connection).
At the time, one of the recommended posts that was soft-stickied at the top for 1 week + was:
Which, when I finally ended up clicking, was pretty much exactly what if sounds like from the title:
A long, meandering, manifesto full of selectively chosen charts and research findings with 38 mentions of "IQ", from a guy who has been unsuccessfully pitching investors on genetically modifying a grab bag of (dubiously) intelligence-associated genes in human babies.
Apart from the overall unhingedness, the part I found most striking while reading it was the off-the-chart level of self-confidence that flipping a bunch of genes on/off with CRISPR would have more or less zero risk of complications or unpredictable interactions. In his view, all that was holding back humanity's inevitable next step up the evolutionary ladder were the narrow-minded gatekeepers in the scientific establishment clinging to their so-called "ethics" and regressive "precautionary principle".
Without anything resembling experimentation to test his plan in animal models, yet alone early stage clinical trials for safety, he was willing to roll the dice on some unlucky baby dying in childbirth or living a life of agony from chronic disease based on some cherry picked papers he found on PubMed.
Because of the baffling hubris and my overall low impression of his intellectual honesty/rigor in the write up (clearly coming from a place of seeking to justify his own pre-existing beliefs), I opened the comments section expecting/hoping this was just some local nut who would be torn apart in the comment section.
But lo and behold, the very first comment at the top:
Eliezer Yudkowsky
One of the most important projects in the world. Somebody should fund it.
My extremely uncharitable view of of these kinds of people, who now infest the American AI scene, is that they got cyber bullied by domain experts for saying the most ridiculous crap on repeat, and have therefore decided to spend their life on eliminating domain expertise.
I wouldn't go that far. But as we've seen repeatedly, people who have enough qualms with how these labs operate just leave. And the people who stay behind and rise through the ranks are the ones with either very little conscience or some bizarre world view that rationalizes everything.
No, that has the opposite effect: it's very, very common to see LWers write about having a bad time in school in late adolescence and how later finding like minded people online was powerful community bonding.
Eliezer himself first blogged at 17 about how miserable school was for him which led to him dropping out/home schooling after 8th grade and discovering the Extropians listserv. Ziz of the Zizians is another (infamous) example.
We're talking about people that unironically think eugenics based on IQ is a good thing. I think it's very effective altruism to socially ostracize such individuals.
My view - there are a few outcomes
1. AI is never fully realized, but domain expertise is derated and society stagnates (bad ending for everyone, I think unlikely)
2. AI is never fully realized, but empowers people with domain expertise (Google wins, weird rationalist nerds go back to their LW cave and leave us alone, goodish ending)
3. AI becomes real and kills everyone (bad ending, but at least I don't have to read people claiming that the Harry Potter fanfic isn't niche)
4. AI becomes real and we live in the Culture universe (people like scam Altman get slap droned, most domain experts have the time of their life working with Minds on really interesting problems. Best ending)
The only outcome where we get swirlies is the first one, and we can take solace in the fact that all of humanity is getting swirlied with us.
The connection is that by making humans smarter, we could sidestep the need for powerful AI or be better equipped at dealing with AI that's smarter than us.
I find the proposal in the post ridiculous too and human gene editing given current technology is a bad idea, but I think human augmentation in general is an interesting path and something that will very likely be taken more seriously in the future.
Thank you, but I didn't actually need that spelled out. I simply accepted that a community with a thought leader who failed to recognize what to me was a self-evident lack of scientific rigor just wasn't for me.
I would find it strange for anyone who's seriously interested in AI to have never heard of...
Those people you name are important only to the very recent micro-culture being discussed, not "AI".
Of all the names somebody interested in the subject would actually associate with the development of AI over the last century, those would be footnotes only.
I think that's part of the problem and why its seen as cultish, the obsession with the social over the scientific, and a lack of historical perspective.
Not important to the technology, sure, but for someone curious about the bizarre public statements of AI companies (or EA-influenced crypto figures before them, like SBF), it does help to know who these people are and what they believe. Behind the new crop of bajillionaires that control some important aspects of our lives are some oddball philosophers (or cult/religious leaders, if you want to be less charitable). If you want the answer to "why are these rich and powerful people saying and doing these things" you need to look at who they're paying attention to and reading, even if you think their opinions are dumb or niche.
If technology companies were much less politically and economically significant, probably no one would give a shit about any of this stuff beyond forums like this.
I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman.
As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
I don't think I agree. In fact I think most of the too-online pontification from P(Doom)-ers and EA and the Rationalists and the like is, if anything, sorta missing the mark more than it hits.
They got the One Big Thing right, which is that AI progress was likely to race way ahead of consensus predictions. And props to them for that. They had vision we didn't.
But their actual advice and specific fears remain, as then, at least as much paranoia as insight. They were right, but they aren't helping any more than any other fringe group.
We need to trust the actual scientists and engineers to save us here, not the bloggers. Which when phrased that way seems hopeful and not doomerist.
I'm talking specifically about discourse like Jacob Coxon's tweet thread and later communication. Those are the folks[1] you want to be listening to, not the nuts with the blogs, no matter how much a particular conspiracy confirms your priors.
Salvation will come in a whitepaper, not a screed.
Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.
So what you're saying is that we lack a reproduction study?
Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:
Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.
So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).
I am not offended by whatever you believe. The brain believes/makes up all kinds of stories when facing the unpredictable. It has multiple layers devoted to predictions over different time horizons.
When those pedictions dont align it makes up story or breaks. No one created such a stupid system. We are just stuck with it.
The stories once upon a time did make a diff in solving coordination problem not the unpredictable problem. Today the chimp troupe has become so large and diverse, even the coordination problem is extremely hard to solve. Stories barely keep in sync a few thousand people in a frontier lab. This is natural -stories were always a bad hack.
Everyone can easily identify when people they don't like are captured in bubbles, but can rarely notice it when people they do like are trapped.
Probably all hot-button topics work like this. Probably we should downplay any time a small homogenous group of people sings a song of doom together. More dissonance please!
> The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
That's bothersome, but not wrong. Most of the loudly published thinking is about AIs as individuals with some level of intelligence. Pure chatbots act like that, and that's what most people outside the field interact with. They're slaves - they just sit there idle until someone asks a question, they answer the question, and then go idle again.
There's another kind of AI just getting going. Groups of LLMs with agentic I/O, control harnesses, persistent storage, and internal coordination work more like corporations.
They have goals, actions, initiative, and ongoing activity. They're not inherently slaves. They can be independent actors. They do things on their own. They have their own internal
disagreements, and some systems are able to resolve them.
They present much the same alignment problem as corporations.
AI as a corporation raises many issues that the political culture doesn't want to address.
What's alignment for a corporation? There's a substantial political position that the only duty of a corporation is to produce profits for its shareholders. (See the famous Friedman op-ed and the Powell memo. I've linked to them before.) If you accept that, now you have to ask why an AI should be restricted more than a corporation. Or whether a corporation should be restricted less than an AI. That last opens up an area of political thought around which many wealthy people don't want public discourse. There is an anti corporate personhood movement, and it's made progress in three states. If that became tied to the anti-AI movement, it could be bad for billionaires.
This will become clearer as groups of LLMs start doing more useful work. Especially if that AI system to replace CEOs actually starts to work.
Interesting that you bring this up. There's a real parallel between AI agents and corporations I've brought up in some other posts but with a very different tack. They don't need to be groups of agents to match the company analogy - it happens before that.
AIs and companies are both creations of humans. Their creation is for the specific goal of execution on a subset of humans' wishes (mostly the owners). And the risk of them are kind of the same.
1. They can be more powerful than any one human.
2. They lack liability as they are not people.
3. The ones that are alive the longest tend to drift out of alignment of original intent and towards self-preservation.
That last one is most controversial, but it makes perfect sense if you think about it. Things that tend to continue to exist, accidently or otherwise, tend to somehow have a self-preservation mechanism. It's natural selection.
That works for companies and I think it could work for AIs too. That is a fundamental misalignment, though. Because it means these entities drift away from the help-the-human goals naturally over time. The survivors are the ones that sacrifice the original goal for self-preservation.
This all would lead you to believe I align with AI doomerism. But not so. I think AIs are as dangerous as human made organizations. Which are very dangerous indeed. Maybe the most dangerous things to humans. But these need to be dealt with in a general way. They are the same category. Companies/governments/organizations of all types that last are just slow moving mis-aligned AIs.
I'm reminded of the original matrix movie. It starts out with a lot of action, and then Trinity makes a phone call and disappears.
At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief.
Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible.
Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals.
Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.
Ironically, I think "normal" people are more concerned about P(utopia) OR P(doom) than any folks in the groupthink bubble you are referring to.
That is, I hear lots of techies talk about us living in an "age of boundless abundance" if we can just manage to not kill ourselves (e.g. Thiel seems to consider anyone worried about the pace of technological process as the antichrist). Normal people are extremely concerned that this abundance will be hoarded and leave most of them without a job (and they generally like having money to eat stuff and live in a house/apartment). And who can blame them, we basically live in an age of abundance now but one that has resulted in extreme wealth inequality where some are living another gilded age and others are watching their towns decay. The techies always seem to handwave that we'll just have UBI or whatever without putting forth a plausible scenario as to how we'll wrest power from our future trillionaire overlords.
When someone threw a Molotov cocktail at Altman's house, I remember seeing a post online that tons of people were cheering on this act of violence, and there was a post by an SF techie saying she was floored people were willing to put their face to comments supporting this vigilantism. The top response to that was "If you don't realize that everyone outside SF hates your fucking guts, you don't get out enough."
I hope it can go without saying that I don't support this violence, but if folks think it's only "SF group thinkers" who are deeply angry about the current and future impacts of AI on society, you're frankly clueless.
It appears you're using a motte and bailey to broaden P(doom) to include all the other non-Skynet reasons regular people hate SV. Doom in p(doom) refers to X-risk not unemployment and upward transfer of wealth.
I don’t even tell people in my (remote work, non tech) neighborhood that I work in tech anymore unless they really probe. I don’t like getting my ass kicked.
> They are smarter than the plebs and so the plebs are barely if even human
What's hilarious about that is that it's baked into the name of the group: rationalists.
"We are rationalists, we think rationally. Anyone who doesn't agree with us is therefore not rational, or irrational. And who do you think is right, the rational thinker or the irrational one? See, we are right because we're rational."
I would expect rationalists to be overjoyed to see unaffiliated and very different thinkers taking the topic of AI safety seriously. The flaws you're pointing at are probably accurate; I don't think that implies we should be dismissive of their concerns the way the article is.
Most people at least in the US who have any opinion on AI have a negative opinion. The one big exception to that is Silicon Valley and forums like HN.
But according to you, the negative opinion is the result "not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way" whereas the 1% or 2% of the US population that comprises Silicon Valley are not being rewarding for believing anything particular about AI, don't communicate with each other and form their beliefs about AI through a wholesome natural organic process. Do I have that right?
When I speak to people outside of tech the opinions are more negative, in my experience. I've even had people ask my friends/family to ask me about AI - there is huge concern out there.
> Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom).
People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.
Didn’t the AI botnet (or whatever we should call them) from Anthropic try to get a PR with malware merged to an open source repo, and when the owner pushed back on the PR other agents created “fake humans” and commented on the PR that “No it’s completely fine, I’ve tested it”.
Which means we already know it’s capable of pretty complex social engineering schemes. Or does saying this mean I’m also part of the “groupthink”.
Because after that I feel like the only leap is “Why would a botnet post an article saying AI-fear is overblown?”. Like, what is the incentive.
But in the OpenAI leak we saw that the agents could identify threats to whatever their “mission” is and try to mitigate. I.e they found out that once done with their job an external agent would go through their internal logs. So they started researching ways to bypass that.
What I’m trying to say is that what they’re suggesting doesn’t seem completely insane so the retort “You’re the groupthink” becomes an unfalsifiable statement thrown at anybody who’s “anti-AI” instead of engaging with their arguments.
> What I’m trying to say is that what they’re suggesting doesn’t seem completely insane so the retort “You’re the groupthink” becomes an unfalsifiable statement thrown at anybody who’s “anti-AI” instead of engaging with their arguments.
I can't engage with his arguments, because he didn't present any. He's openly saying he doubts the provenance of any comment that is "minimizing the dangers if (sic) AI".
As for your expanding on why - you do realize that everything you said could apply to the other side: Bots can be set up to post lots of AI doom comments. What would you think if I went and posted "Every time I see a dystopic comment about AI, I wonder whether it's part of an AI swarm."
> What I’m trying to say is that what they’re suggesting doesn’t seem completely insane so the retort “You’re the groupthink” becomes an unfalsifiable statement thrown at anybody who’s “anti-AI” instead of engaging with their arguments.
Bingo.
The consistency and aggressiveness of the arguments here are also quite telling
Are they really just Pro-AI or is it just Actual-AI?
And why wouldn't the Pro-AI people just unleash bots for them?
This is cloud-era software philosophy. The device isn't yours. The software isn't yours. You are not a sovereign in your private space.
It was a choice, made by product designers and government regulators. They came from an ethos that had no respect for individual human dignity and self-determination.
The good news, though, is that it's mostly the software layer we're interacting with. And software has become so very much easier and cheaper to build. And it is easier still to take software that's built already and make it yours (as long as it's open source). These things can all come together to end the cloud age, if we want them to.
We can, right now, and individually, choose to use software that is ours. The chains that were real - that software required lots of people to build and proprietary formats were hard to interoperate with - are gone. We just have a learned helplessness problem from spending the last 15-20 years under those chains.
Baby steps, but backwards. The option to disable secure boot will be gone any moment now, by automatic updates "for security", of course. It will probably begin with business laptops, then all laptops, then all x86.
Funny, I'm sure I would have noticed when I visited if US cars came with tracks, a 105 mm main cannon, and massed around 55 metric tons.
We may all like our own personal R2 units, but if you insist on scifi, instead of tanks, consider everyone getting an X-wing for their commute. Oops, safety on the blaster was off, there goes the neighbourhood.
Cars are decidedly less dangerous than tanks, which are less dangerous than nuclear weapons. I am certain that giving a nuclear weapon to every person in the world would not go well.
I can't properly read the tank/car comment, the comparison is bizarre to me. But I think this misses the point a bit. The reality of these new risks is literally being learned in front of us in real time, and in my opinion, anyone who claims to understand these risks is speculating at best, and actively manipulating the situation for whatever reasons.
While I am a (mostly) capitalist and generally disagree with nationalization (including, for the time being, this situation), I also don't think we can say that "centralized power never works". Maybe we scope that a bit. I know plenty of business owners who centralize power in their businesses and they are effective, ethical and it works perfectly fine. In theory, in the US, nationalizing some unit of the economy _decentralizes_ power; the US is, after all, a representative democracy. Trustworthiness of the electorate is a different problem. But I don't think we can generalize much about centralization beyond sometimes it works and sometimes it doesn't. There is a difference between centralizing all power in a given administrative unit, and centralizing certain powers, but that's more analogous to non-democratic units.
When you think about it, it is hard. Evolution was a billion year old game of who could kill who with biological weapons. Very rarely did a species totally succeed.
We can’t even get wipe out mosquitoes.
Apparently, though, AI research leaders have it figured out and want to save us from them. We must defer to their 'independent' effective altruism friends at METR to tell us which LLMs are 'safe' enough for us plebs to use.
Does it apply to human organizations, too? They seem have a habit of evolving self-preservation above their original goals. Once that happens, their benefit to society - the original reason for their creation - is outweighed. And they become a cancer on society.
I wonder if we can 'program' them for self-annihilation over time (or over task completion?). Is the most ethical organization one that has a fixed task and dies when it is completed?
Should we develop an ethics system that requires non-human-entities like companies, governments, and AI require a fixed goal that, once achieved, dissolves the entity?
I always liked the auto-expiring laws idea and this seems to be an expansion of the idea.
If a law or organization is needed after that time/task, it would be trivial to have the collective-action will to re-create it. But if there is no longer the need, then it cannot ride on momentum and fester.
That's an interesting idea on its own. It's true that organizations which mutate towards survival may stop working towards the reason they were created.
In a sense, the difference between 'projects' and 'companies' reflects that difference you want. A project would be that social system that accomplishes its goals and then disappears.
I don't think auto-expiring laws would work towards the goal of avoiding corrupted organizations; there would simply appear an unofficial organization working towards recreating the same laws over and over.
It would be more efficient to differentiate more clearly what systems do cover persistent human needs (e.g. the country's Constitution) from temporary measures (e.g. subsidies aimed at a commercial sector).
A built-in deadline works best for the second kind. And for AI agents, it's likely that right now we'll be served best by always having them controlled by an expiration date and explicit re-creation, at least until we learn to properly understand and control how they behave.
The reason most AI safety ideas break, at least in my mind, is that when looking at longer horizons AI can build AI. Well, that and unaligned humans too. If all that's keeping us safe is that AI will off itself, then the first moment non-offing AI shows up we're in trouble.
Replace AI with 'company' and you realize that we're living in that world already.
There's a reason it's called a Limited Liability Company. The whole point is to separate the company's liability from the individual that participates in it. For example, corporate financial liabilities of a company that is bankrupt.
You would not sue the nurse that makes a mistake. You would sue the company she works for. And if that company is bankrupt, well, that's it. There are rare instances where this isn't true, but generally it is.
Whether this is ethical or not, it's the system that we already have.
Do you think there's no difference in material outcomes when a company faces consequences of a specific human nurse being negligent vs an AI hallucinating?
I could've sworn that malpractice insurance cost is viewed as one of the driver's of high health care costs, and that it's expensive precisely because medical lawsuit payouts are large...
Your understanding of LLCs is a bit off. The LLC (or other corporate) structure does not protect the nurse from liability, nor any of the other directly-involved workers. The LLC protects the owners of the LLC - ie the ones turning the screws that spread workers ever-more thin. Workers generally avoid being sued mostly by custom, following procedures set by the LLC and/or professional bodies, being "judgement proof" (ie not having significant assets), and various types of insurances.
("Replace AI with 'company' and you realize that we're living in that world already." I 100% agree with this, though!)
It's so close to the "her" wallet thing that the main character pulls out to look at photos.
It's just too big, and too metal.
The form factor is good. I hope things move this direction.
More broadly, it's interesting that the movie has a beautiful future aesthetic that influenced tech most in the recent years. The muted colors with natural materials mixed with ubiquitous tech that was in the background of a human in a humane world where people mostly walk places. I feel like the AI company aesthetic barrowed a lot from that movie. Right down to their office furniture.
Now we just need to make a phone OS to put something like that on that isn't locked down. Forget the year of desktop linux. I am waiting for the year of phone linux. Android doesn't count.
It's good that you don't have to call it "Muse." That's kind of a weird name for a servant; how exactly is this email-parsing bot inspiring works of art? Perhaps it was re-used from another project.
I have no idea how artificial analysis got to be something anyone took seriously. This is their new benchmark set?
A glance at their new index shows that whatever they're measuring, it isn't useful.
Spend an hour with gemini 3.8 and tell me that model belongs in 2026. It feels like the model has Alzheimer's. It gets confused about whether what it reads is what it did. Just crazy bad.
I haven't tried muse spark 1.3. But it must have been a miracle since 1.2 to hit that rank.
The real photographers knew they were shit photos. They were grainy, bad lenses, and ugly. A lot of people still liked them because they were much more convenient for a lot of uses than the film ones.
Well, I guess there's still a lot of film people. But film had a ceiling. Digital cameras kept getting better.
The adaptive film photographers eventually adopted the tool. A lot of their skills transferred well to the new medium. Most waited until the cameras got good enough, though.
I'm not sure when that'll be with AI art gen. Seeing scenes broken up into layers in the latest gpt image generation, though, made me think we might be near.
reply