Nah, it's cumulative errors per claim. E.g. maybe the per-decision-factor rate is somewhere over 50% but overall most claims will have errors because so many factors go into coverage/reimbursement decisions.

So what you're saying is that we lack a reproduction study?

Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:

Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.

So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).

I think that the ratio of work done by priors and search is the interesting question at this point. We aren't quite at the point where reading off the most likely hypothesis decompressed from a transformer is sufficient, but it's a lot closer than I originally suspected when we were ~solving chess. I think AlphaGo was kind of the watershed moment that prior-guided search was so much better than either alone.

I think the adversarial policies against Go AIs directly show the gap between intelligence and compression/priors.

Maybe compare the argument that AI has large detrimental environmental impacts to the argument that it has economic impacts. Why would the environmental impacts even be a major argument vs. the worldly economic concerns? Because there is climate science predicting extremely negative effects on humans from warming, e.g. "the end is near" on limiting climate damage. The environmental argument wouldn't have been reasonable to bring up in the 1950s if AI had gone according to the earliest optimistic plans and not required giant data centers.

There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.

There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.

Please link some of the mathematical research which concludes takeover of Earth’s resources by a superintelligent agent is “very likely.”

The broader problem is that current AI and AI companies are not aligned with humans, human values, or human flourishing. AI is a powerful tool that can be aimed at quantified goals but we don't actually know how to quantify human values and flourishing (Goodhart's Law).

Spreading as close to the speed of light as possible with von Neumann probes generally defeats the dark forest, no? Either you win (spread to every star system) or the forest stops being dark from the border wars between alien probes and their manufacturing systems.

Hold on; GiveWell Labs/OpenPhil/Coefficient Giving is older (2011) than Anthropic, the AI safety folks are way older than OpenPhil (~90s, older sci-fi), and it stretches the imagination to claim that your product will kill everyone in order to sell more of it. That's conspiracy theory territory. Even cigarette companies pretended to be harmless.

I don't think it's a secret that the safety folks (including "doomers") are funding a lot of anti-AGI articles and researchers. But they're generally at odds with the big labs (Anthropic, OpenAI, Alphabet, Meta) about the basic concept. The Yudkowsky branch believe AGI/ASI is incredibly likely to take over and end life on Earth. The major labs think the worst that can happen is a few more publicly embarrassing hacks but that anything truly dangerous is a distant threat. ~most safety researchers fall somewhere in the middle.

Anthropic started as a mid-safety reaction to the way OpenAI was being operated but is by no means close to the "doomer" end of the spectrum.

AGI is the term invented because arguments about what AI meant had gotten annoying. Originally there was no distinction and people thought "AI" would mean human level intelligence. Chess and conversations and robotics and math all in one package. Then games and classification and some robotics got solved and called AI, but that didn't solve math or conversation or online learning or a host of other things, so AGI was coined to refer to most of the whole package, virtually all the capabilities you'd need to replace humans intellectually. Now we're quibbling about whether AGI includes robotics or full real-world physical agents or something less.

The consensus now seems to be that once you've got human-level intelligence and planning and executive function then you get recursive self-improvement that can eventually autonomously solve the robotics and world-modeling and other portions of human-equivalence.

Human-wellfare aligned, safe AI is our best shot at a post-scarcity society. Huggingface-hacking sycophants are not that.

Leftists oppose human suffering and disempowerment.

Feels more like leftists are pro-scarcity into order to "empower workers"

I burned some optical installation media for OpenBSD and Linux for the first time in a long time to have some RO boot media if another Internet Worm comes around.