The probability values don’t really represent confidence in modern LLMs though, especially after RLHF and RLVR.
System One says they use RLCD, Reinforcement Learning for Calibrated Decisions, which presumably has accurate probabilities as an explicit optimisation goal.
RLVR generally upweights tokens along the whole thinking trace that led to a correct answer, whether each token was "correct" or not. RLVR doesn't train a model to output an 80% likelihood, it just trains it to produce correct answers, and not to produce incorrect ones.
System One hasn't said how RLCD works, but they do say it is explicitly training models to output "calibrated" probabilities, which makes it distinct from RLVR. This is how they describe it:
> System One models are trained for calibrated decisions: their probabilities are optimized against outcomes to reflect uncertainty.
In RLCD (which is now an RL acronym that has 3 different unrelated expansions!), you basically massively negatively reward a distribution that is {yes: 0.9, no: 0.1} if the answer was no, and less negatively reward a {yes: 0.6, no: 0.4}. Many nuances when designing the details, but that is the rough idea.
It is a known existing thing variously called "calibrated RL" or such.
Implementing it on top of LLMs was difficult to get it to work, they seem to have done it up so its good enough for a polished product that works in a wide variety of usecases at the same time. I got accepted from the waitlist and it's really neat. Edit: it is now on vercel gateway.
One thing to note, the out of distribution behaviour will be different from what we are used to with regular LLMs. Theoretically, it should be worse, but practically, it depends on their method.
Everyone I have talked to who uses coding agents at all now uses them to write almost all their code.
The two people I know who don’t use coding agents work in government, and in a data science company working with government.
I’d say if you work at a tech company and agents aren’t writing the majority of your code, that is weird. But if you work at a traditional company that doesn’t have Claude or Codex subscriptions, there it might be pretty normal to still be writing code by hand.
Well in my personal projects I write all code by hand because people relying on agents will have forgotten how to do their fucking job in 3 years and then it will be appreciated to have people understanding what they're actually doing. That being said, in my job ai writes the code just because it's way faster.
Your "fucking job in 3 years" will be very different, and if you don't learn how to do it in time, you won't still have a job in the computer industry.
Pick an industry where you can stop learning new things once you're out of school, because the computer industry is not one of them and never has been.
My fucking job used to be writing 6502 assembly language on an Apple ][, and I loved it, but I hope I've forgotten enough of it to have room to learn new things. If only I could forget all those hex I/O and peripheral addresses from $C000-$CFFF and the Monitor ROM routines from $F800-$FFFF, without forgetting how brilliantly beautiful Woz and and Allen Baum's code is.
Forgetting old stuff to make room for new stuff is one of the most valuable skills you can have in this industry.
The next most important skill is persistence ;) -- writing stuff down before you forget it, in a way that won't make future-you hate present-you when you need to learn it again.
You forgot to read the last line of my message, in my profession I utilize LLMs to a high degree, my point was, that in my free time I code by hand 1) because self esteem benefits from the process of problem solving 2) I want to understand the fundamentals to not feel like a fraud 3) there is no time pressure to perform.
People neglecting this side will inevitably become less proficient at their job and become rather helpless without AI, that's all I'm saying.
Yeah, I read it, and we don't disagree about your job.
You put a number on it at the top of the thread: three years until people relying on agents have
forgotten how to do the job. Copilot's technical preview was 2021, ChatGPT was November 2022, and
agents that write the majority of the code are a 2024-2025 thing. So the window you're predicting
is roughly the entire span in which these tools have existed, and the atrophy hasn't shown up in it.
For scale, the last ARM I wrote was for a Sony Clié and the last PowerPC for a Mac, both more than
two decades ago. If I needed to write ARM for my Mac now, I could brush up about as fast as I could
on the 6502 I mentioned upthread, most of whose opcodes, $C000 I/O addresses and monitor ROM entry
points I genuinely cannot recall any more. Two to four decades of not touching an instruction set
didn't cost me the fundamentals, and it hasn't made me less proficient in Python and TypeScript or
helpless without AI. Three years of using a good tool is not going to do it either.
There's also a distinction buried in your last line, because "people neglecting this side will
inevitably become less proficient" bundles two different things. Losing a skill you had is not the
same as never acquiring one, and only the second is an argument about tools.
So take acquisition. I've been learning PDP-7 assembly in order to recover, OCR, and analyze 128
pages of PIXIE assembly source and octal machine code that Heinz Lemke wrote at Cambridge in the late 1960s,
and get it running in an emulator. A language I never knew, with custom instructions for bespoke
networking hardware, no Stack Overflow, and almost nobody left to ask except Heinz himself. AI has
been extremely helpful for exactly that, and the understanding is still mine to build: I have to
predict what the machine will do, and catch the explanations that are wrong.
On feeling like a fraud: the test isn't whether you typed it, it's whether you can predict what it
does and fix it when it breaks. Plenty of hand-typed code fails that test and plenty of carefully
reviewed generated code passes it. Your reasons 1 and 3 are good reasons to hand-code, since
problem solving is genuinely fun and nobody is waiting on your free time, but reason 2 is
measurable, so measure it instead of trusting the feeling.
The atrophy you're worried about is real. It just isn't caused by the tool. It happens to people
who stopped wanting to learn, and that predates AI by decades, even millennia. I watched it happen
back when much of the industry ran on assembly language.
If you're not using AI to teach yourself new things, you're not holding it right.
You're not wrong and it really comes down to how the tool is used. People wanting a quick solution, thus skipping the learning process entirely out of either complacency or because they feel like they don't have the time or energy are more at risk of atrophy I'd assume.
Personally the tools have simply gotten too good to ignore so I have been thinking about how to write skills and prompts in a way that enable me to keep learning, overcoming friction and challenges and not simply using the tool for hand holding and jumping straight to the end result.
For me personally, what matters is the process of learning and mastery, my self esteem depends on it, and enriching that with AI instead of AI eradicating that process for the sake of efficiency is my current challenge.
The thing is I think there will come a time when AI tools will not be as accessible as they are now. I just don't know if it is in 3 years or more, because the market can stay irrational longer than I can foresee.
I saw this idea from elsewhere that the things AI helps with was never the "profit bottleneck" of companies. 10x engineering productivity gain does not translate to 10x more revenue if your profit bottleneck is customer acquisition and retention. In other words, your profit is limited by how many people are willing to give you money for your services and AI can't really affect that.
And even the productivity gains per individual is a generous assumption. An individual can only prompt (and check! You guys check right?) so much. Sooner or later the pendulum is gonna swing in the other direction and it will be cheaper to build a team than equip individuals with AI _to deliver the same value_. I'm also assuming that AI is still in the VC-subsidized pricing stage.
Hence why I think the job in the future is gonna be pretty similar to the job four years ago.
> The next most important skill is persistence ;) -- writing stuff down before you forget it, in a way that won't make future-you hate present-you when you need to learn it again.
Huge amen. Another thing I've found Claude very useful for is writing documentation for legacy systems and cleaning up my own notes on it. I only need Claude to be 70-80% correct because from there I can take it. That error margin is no different from moderately-outdated-but-still-useful documentation.
When that pendulum swings, I will have a documentation binder that I will print money with.
> The thing is I think there will come a time when AI tools will not be as accessible as they are now
I am very convinced that today's SOTA level will be accessible 3 years from now, but you might be right about the SOTA accessibility at that point in time.
Still with today's SOTA you need to know much less details to be able to write code than without it.
My (arguably not very directly communicated, true) actual point is that "weird" is a term that is not really suited for tool choices like this.
"Weird" is a social concept. It's an artifact that has its roots in social cohesion and friction induced by individual nodes to glue the group together.
Software development otoh is an engineering discipline (or.. it should be).
And Engineering does not use the local social consensus algorithm for determining correctness. (or.. it should not)
I think they just meant its weird as in if you met a software dev at a dinner party who was still writing all their code by hand it would raise an eyebrow and you’d want to enquire further why that is.
I think understand the purpose of this comment. I think you're sensing friction and you'd like to reduce that by explaining for the other guy. Which is a common social script, but one that is being exploited here I'd say.
But, regardless, what is interesting I think is the scenario chosen there. Because a "dinner party" is not where engineering happens; and that's kinda the point I was getting at.
That's pulled from the pool of "social consensus" and not "math" or "physics" or whatever.
It's interesting, isn't it?
Just like how specific tokens in the context window of LLMs pull probabilities towards specific clusters of ideas (and tokens); with humans, you see similar things happen.
(This comment is more coherent than it might look at at first sight.)
___
Anyway, point (and to the point) is: Don't get hacked by the current thing fraudsters/grifters.
They don't respect you or me or really anyone the slightest. They're just in it to get rich quick (or simply just inflict psychological damage for sadistic reasons), and they will use any means necessary to do so.
This comment chain only exists because some hype guy is trying to induce FOMO into people for not doing more AI.
Why is unclear, but it's clearly malicious. Whether they consciously know that they are doing that doesn't matter for that assessment.
I think LLMs just allow you to get to where you want to get to quicker. So in a productivity oriented world it seems weird to not be making use of them where those gains can be had.
Let me try another analogy:
AI-related comment sections are like a watering hole for both get-rich-quick grifters, but also bad people wanting to hurt others.
The technology is so disruptive that there are all sorts of opportunities while stuff is still developing, things aren't regulated, and people themselves have not yet learned how to self-regulate.
So if you're looking to make money or to just inflict pain onto others, you're pulled towards this stuff.
This is what I am seeing in this specific comment sub-tree. A troll.
But in current year, you cannot just call people trolls, so instead you get these elaborate meta comments that bypass the anti-anti-troll defenses, but get so abstract, people just go "what".
My advice would be to view any of these comment sections through a zero-trust lens.
Godspeed. We will all eventually get through this.
Software can be an engineering discipline doesn't change whether it's a human doing it or AI doing it.
Arguably engineering is something AI should be really good at since you're just trying to compute a working product given constraints, formulas, resources.
Maybe "unusual" or "uncommon" would be better terms here.
I definitely wouldn't put it at 99% of developers, but I'd say it is the norm among developers I know that agents are their primary mode of writing code now (still using IDEs and GitHub to review changes).
Claude Cowork is the application aimed at non-developers that gives them a lot of the same functionality. My girlfriend uses it and has gotten quite far in producing her own software.
People argue whether we are at y-5, y, or y+5, meanwhile we seem to be on a y=2^x exponential that keeps delivering more and more impressive results.
The most interesting question to me is what will be consumed by the exponential like math seems to be undergoing, and what won’t. Writing has been quite stubborn, but I’ve noticed Fable to be quite a big step up there. How about politics? Will we develop new ways to let people express their own values in democracies, or will we just get much better at manipulation? How about experiment driven domains like biology?
We will get much better at manipulation and better at people “writing” things to justify their own feelings.
What’s new about LLMs is that you can scalably manipulate people individually. It used to be that you could either have scale (speeches, tweets, interviews, website, etc.) or individual engagement (replying to mail/tweets/town hall questions.)
Now you can pull the history and preferences of an individual, then shape a message—in real time—to them, specifically. You can have conversations on social media with a single person and shape your message specifically to them.
Part of this can be good (you talk about what they care about, where 90% of broadcast messaging might not apply) and part of it can be bad (manipulation.)
My guess is that, in the US, the right will cynically adopt manipulation to great effect and the left will take a moral stand against shady practices and lose elections.
It does matter though. If you want to murder someone by hitting them with a plushie, you're not going to get charged with attempted murder because it's not possible that that would ever work. There must be justification that the choice will have the intended effect.
We should not be gung-ho to give the government more power to regulate speech.
It isn't so clear there is a reason for the crime of 'attempt', or what or how good the reason is. The Star Chamber, which cooked it up, is universally condemned as oppressive. You might say they anticipated science fiction tyranny, when they invented the world's first thought-crime: they were able to make a crime of 'conspiracy' because they imagined that the speech of the conspirators was an 'outward act'; inevitably its glorious future was e.g. to jail the left wingers in the McCarthy period.
The point of view that says 'there is a reason we punish attempts' has difficulty explaining why we punish 'success' _even more_. There is a sort of bad concience about it. The paradoxes are discussed in a characteristically brilliant and twisted work 'The Punishment that Leaves Something to Chance' by David Lewis, one of the greatest philosophers of the 20th c. It nominally defends the law of attempt but can as well be read as a catastophically destructive parody of it. https://andrewmbailey.com/dkl/Punishment_Chance.pdf
It's interesting to claim that it would be difficult to explain why we punish successful crime more. A successful murder creates more suffering (victim's friends and family), and ostensibly the loss of a productive member of society. These all seem like fairly common justifications for punishing murderers under the banner of retributive justice.
It’s unfortunate that you’re being downvoted because this comment is true. And it just goes to show how far we’ve sunk that people have forgotten the problems of ad tech in the age of LLMs.
You’re making a weird number of assumptions about people’s voting and extrapolating to assertions about Society.
Ads are not really the same. They can’t be as tightly targeted to what resonates with someone. Programmatic ads are certainly much better and closer than, say, television advertising, but users can’t _engage_ with them. Like actually chat with them.
That’s where this is headed and people are not ready. I don’t think we could prepare them, anyway.
> Ads are not really the same. They can’t be as tightly targeted to what resonates with someone.
The EU referendum in the UK proved your point false.
> Programmatic ads are certainly much better and closer than, say, television advertising, but users can’t _engage_ with them. Like actually chat with them.
When people talk about “ad tech”, they’re not talking about TV ;)
And yes, people can and do engage with them. That’s how ads on social media works.
During the EU referendum, people were even resharing ads on Facebook without even realising they were ads.
> My guess is that, in the US, the right will cynically adopt manipulation to great effect and the left will take a moral stand against shady practices and lose elections.
I think that statement may itself highlight how prevalent manipulation is.
I fully anticipate all groups to continue maximal manipulation they can. One thing with LLMs is that it'll be a far less unified view, so a "divide and conquer" strategy is what I anticipate.
In US politics, the right is far, far better at winning elections than the left. This isn’t about personal preference. It’s objective political science.
Look at the most contentious issues in the US: abortion, climate change, taxing the wealthy, gun control, Affordable Healthcare Act.
The Democratic Party platform is aligned with national polling on every one. Every one of those issues has >60% support with voters and the Republican Party has blocked them all.
> Democratic Party platform is aligned with national polling on every one.
It depends on the pollster and where you're polling. I guarantee you rural Tennessee will not agree with downtown Washington DC on any of these issues. In contrast, rural California will likely agree with rural Tennessee. It's not as cut and dry as a homogeneous national poll of 2500 people. Every state, city, county are different. That's why there are federal, state, city, and county governments.
For instance, Abortions are legal nationally. States can individually decide how, or if, they wish to restrict it. This is as the constitution intends under the 10th amendment:
> powers not delegated to the federal government nor prohibited to the states are reserved to the states or the people.
This allows for democracy to take place at the local level, rather than having particular regions thousands of miles away from each other ultimately oppress the other.
To the point on LLMs, I think it's abundantly clear they will be used to propagandize and similar to social media will lock people in a bubble without alternative opinions. It'll be the worst of both worlds, the question is who's the puppet master. At some point soon, I imagine it'll be the AI.
I don't know what distance has to do with it. I would probably be voted out of existence by my town, if the same constitution hadn't convinced them that they have no right. Independence Hall is remote in space and the events still more in time.
The pretense of Dobbs is that laws against abortion are like laws prohibiting medical marijuana or setting speed limits. This contradicts the purpose of laws against abortion and the actual language of most of them. It was obviously a desperate decision, irrespective of the truth about abortion
So we need to "prohibit to the states" the power to enforce medical procedures (or the prohibition of a medical procedure) on individuals.
Which amendment protects someone's bodily autonomy or medical treatment from interference by the states?
If the Federal Congress passed a law providing that abortions are legal on demand up to the 22 week of gestation, after which it would require the corroboration of two doctors, would that stand, or would it be struck down by SCOTUS?
Same for legalization of marijuana, and many other "social" issues.
None of those are single issues, and the political fault lines are in the sub-issues. For example, while most Americans support "gun control," only 20% of Americans support a ban on handguns: https://news.gallup.com/poll/1645/guns.aspx. That figure has been trending steadily downward--from 38% in 1999 to 20% in 2024. That makes it much easier for Republicans to hold the line on that issue: portray all gun control efforts as a step towards confiscating handguns. That's hard for Democrats to defend against because most of the candidates, staffers, etc., who actually run the party probably are in that 20% who wants to ban handguns. That's simply logical, because handguns are used in the overwhelming majority of homicides committed with guns. It makes very little sense to have "gun control" without banning handguns.
The same thing for "taxing the wealthy." 59% of Americans think their own taxes are too high: https://news.gallup.com/poll/707951/americans-tax-views-rema.... And the difference isn't as big between parties as you think--49% of Democrats think their taxes are too high. So Democrats are in a position where they have to advocate for raising taxes on "the wealthy," without scaring any of their own voters into thinking that includes them.
Even a blind squirrel could find these nuts. Apart from that, republicans are ridiculously inept. For example, 2024 was the first time they spent real money trying to go after immigrant voters and minorities, and they made huge gains. But the on-the-ground operation disappeared after the election. Meanwhile, democrats are in these minority neighborhoods 365 days a year pushing their message.
> Apart from that, republicans are ridiculously inept.
They're not. They're really not. They're incredibly capable and are currently executing long term plans successfully, one after another. This goes all the way back to Regan and the disenfranchisement of education. They know exactly what they're doing to erode democracy.
Calling them inept is dangerously stupid at best, and at worst is just right wing propaganda to try and lull people into a false comfort and not act.
There's no "long term plans." It just seems that way because republicans are relatively ideologically homogenous, so it seems like there's much more top-down and long-term planning than there is.
Democrats, by contrast, have much more permanent political infrastructure. At the top law schools, for example, there's probably 10x as many liberal organizations as conservative ones. Democrats enjoy politics, so they are always out there organizing and engaging. Meanwhile, Republicans dislike the institutions through which politics is done, and so they gear up once every four years and then go back to their day jobs. Among the sort of more educated people who actually run the parties and serve as line staffers, republicans also self-select into private sector jobs, and out of public sector or political work.
Most importantly, putatively neutral institutions are de-facto aligned with democrats (because the professional class that runs these institutions is overwhelmingly democrats). Conservatives might have Fed Soc, but liberals own the ABA. Virtually every legal organization that isn’t expressly conservative or highly niche is de facto liberal and participates in support of democratic policies.
That presupposes that the left in the US wants to do the right thing. Something like government run grocery stores is not clearly correct and there is very little evidence supporting that it will work well yet it is a very popular leftist policy in New York.
Based on zero evidence. It’s very easy for a government to step in as a participant, ruin the profitability of a sector in an area, and offer a worse service.
They have no profit requirement or even revenue neutral requirement. So they can just operate poorly at a loss and still wreck other businesses because people will put up with breadlines to get bread for ultra cheap.
The general thing to watch out for with all of these “surely it can’t be evil to do nice thing X” is suicidal empathy. It can seem correct to your gut on the surface while it’s extremely destructive in the long term despite participants wanting to destroy something as an explicit goal.
Bad economic policy is subtly "evil", by way of allocating finite resources inefficiently. Usually this is unintentionally done by not appropriately taking second, third, ..., nth order effects into account.
They destroy grocery stores that offer variety in every neighborhood they operate in. They offer worse service and people put up with it because the rest of New York is subsidizing them through taxes.
The margins for groceries are objectively thin. The only way to provide food at lower prices is to provide a worse good or service, eg. less variety, less quality, less availability, etc. You will see all of these outcomes in NYC, if anyone even accepts the bids to begin with.
It's harder to convince people to all agree on the same, different from the norm, thing.
If you've got 5 people in a car and you play ABBA in every road trip, then one day you suggest to change, the problem is not in being okay with "something else", but on agreeing what that other thing should be, set against the already known thing.
That's why there's so much infighting in the left.
Biology would greatly benefit. We barely understand transcription and protein structure. And the straightforward systems that we know like insulin have complex post translational modifications. So while we have a map of the partial proteonome, we have barely scratched the surface on networks regulation and interactions.
> And the straightforward systems that we know like insulin have complex post translational modifications.
insulin is not straightforward, the way the insulin molecule interacts with its receptor is nuts. on the other hand its post translational modifications are simple and dont have anything particularly surprising (no glycoslation, disulfide bonds where you would expect, nothing special kex2 cuts, arent really defective in disease states even)
I think the question is less about where it's most beneficial but which knowledge structure lends itself to LLMs most. Since biology's "language" is way more complex and irregular than maths or natural language, it isn't particularly accessible.
> Part of this can be good (you talk about what they care about, where 90% of broadcast messaging might not apply) and part of it can be bad (manipulation.)
Side note: it's manipulation either ways because you chose what to talk about, with a goal in mind.
Actually, no. "Manipulation" is a negatively loaded word, and you wouldn't use that word if f.ex. someone helpfully & truthfully helps others see they've misunderstood sth.
I disagree. Whether something is manipulation depends on whether you are trying to change someone’s opinion or behavior, not on whether the manipulator has “good” or “bad” intentions, since those judgments are not objectively universal.
I believe that the decentralization of manipulation (taken in a very broad sense as an effort to modify people's views) spells trouble for democracies. The mass media of old, with all its flaws, created a shared pool of information managed by well-educated people, some of whom understood that they had the power to keep democracy running (keeping populists away from the "manipulation machine," curbing blatant manipulation attempts, and so on).
Decentralized manipulation, by contrast, just runs amok creating echochambers and polarization.
The "left" as in the party, sure, not that much of an authoritarian leaning.
The "left" as in the pervasive group that crawled out of Tumblr, took hold of Twitter back in the day, and has a stronghold on Reddit now? Those do care about what you can say, think, watch and read, and the more they can control, the better. The US right can only dream to have half as much control as the left has had in the last three decades.
The parent comment wasn’t making that comparison. In fact, quite the opposite. The communism comment emphasized personal experience with a boot in one’s face, not that the boot was communist.
What even is the left though in the US? I haven’t watched any TV in decades, and the online content is entirely algo driven, so I don’t know what is happening.
its a bit like how big companies will pullin a CEO who;s job is to do all the unpopular stuff and then summarily dismiss him after (while keeping the policies he created in tact).
ideally we should be throwing our support behind democratic socialists like zorhan mamdani but old school moderate democrats are just as bad as trump. they are juts happy letting trump do the dirty work. At the end of the day they are both working for a certain foreign government and not the interests of the american people.
> If you think that the American Left is anything like communism
no, you completely missed my point. why would i talk about communism if in the same comment i'm saying that the left/right divide is a fake distraction... a red herring?
what i said is that what matters is the authoritarianism/freedom scale.
i just mentioned my past to emphasize that ideologies like communism are not to be identified by announcing themselves as 'communism'... but by recognizing their actual shape and actions. and i have a little first hand experience in that.
What's "scaremongering" to you is "life" to GP. Besides missing their point, you're also saying "you held socialism wrong, we can make it work, if only if it weren't for this pesky ... reality"... Sorry, you missed their comment, and I can't take yours seriously :)
This reminds me a lot of the proof by construction for the 4-color theorem. It was only enabled by the advancement of computers and dissatisfied many of the computer scientists and mathematicians since it was a "brute force" approach.
I wonder if AI will end up being similar. Certain theorems get proven by AI but others do not. We haven't reached the limits of this yet and I haven't found a good argument for where those limits will be (I do doubt that there are no limits).
Isn’t the point more “it’s easy to fall into the trap to believe that predicting when the sigmoid is going to bend is possible and the right heuristic is to instead extrapolate locally”?
That aside, I’d question whether applying the Lindy effect in particular to something that’s not really a life expectancy but more a growth rate is credible… or perhaps a bit circular since it “assumes away” the ceiling.
Predicting when the sigmoid bends is difficult and predicting how long until it unbends is equally difficult.
The simpler assumption is that over enough time, the S functions stack together for long enough that working backwards from exponential is a better predictor of reality.
These stacked S curves have continually been true with most technology.
The author of that post is a prominent Bay Area "rationalist," who have had a quasi-theistic relationship with the concept of all-powerful AIs for a couple decades now.
All of the point of that article is that most people that think something is a sigmoid think it'll bend just as they are publishing their analysis. And the article says, don't do that, assume it'll be related to how long we've been on the "goes up" part.
I dunno. They've been saying this for decades, including in a very well read Harry Potter fanfic, and I'd always dismissed them as kooks, but maybe they're right in the end.
Saying the same thing for decades, when things progress over those decades along the general trend line that you were worried it would, is decent evidence of a prescient prediction. Climate scientists also seemed pretty kooky in the 80s when they sketched out their trendlines, but now their concerns are all over the headlines as reality caught up, and the same is becoming true of the AI safety people.
Everything has limits, of course. But if you've not yet seen a deceleration, it's reasonable to expect that at the least you're in the middle of the sigmoid, not the top.
We have though. The velocity remains high but the acceleration is decreasing.
The improvement between gpt 2-4 where staggering. to 5, 5.6? Much less so.
The results improve of course, but the difference is no longer mind blowing.
Most of the improvement now has come from agentic harnessing, which is unrelated to the acceleration of the model but the tooling around the use of the model.
So we are seeing a deceleration and the speed is still high due to high prev acceleration but its not growing at the same rate as before and the edges are starting to show themselves
I literally don’t understand, and have a deep distrust of people who say we haven’t seen deceleration.
The diminishing returns are colossal. GPT 4 is more than 3 years old now, and while Sol is definitely better, it’s better at a tremendous cost. And 3 years of development. And it still fails in nearly all the same places as 4.0.
I’m not some “zero AI” person, I don’t think this industry will collapse into nothing and we will go back to a no LLM world.
But to look at this industry, which basically began in November 2022, almost 4 years and a trillion+ dollars later and say “oh yeah, definitely riding that exponential growth still” is some of the wildest shit I’ve ever seen, and it’s so common.
I'm the other way around. For doing real, useful work, these things were barely more than toys even just a year ago. Now they're very capable when used well, and still getting better at a fast pace.
That's a false binary. The right binary is whether it's a convergent function or a divergent function. A sigmoid is convergent, which is not well substantiated and more farfetched than a divergent function, even if it's true that 2^x is too optimistic.
I mean, "sure, technically correct is the best kind of correct" - but where we sit on this right now? It certainly feels exponential.
A lot of this sort these sorts of posts are just "appeals to geometry" (aka "cope"). This is coming, it's coming hard. Now you need to decide what you want to do with your life in a world where your smarts aren't as special as they used to be.
This is hard (believe me, I know). But what one ought do is not eschew progress and cling to the delusion that things don't change, what one ought to do is try to see how they can leverage these tools for greater and greater accomplishments.
My wife is working on her PhD in microbiology now, using OpenAI to implement her research ideas. Genetics is just too much data, and she eats through tokens like nobody's business. I thought I was careless with them, but she barely lasts a full day before exhausting her quota. There's definitely a lot of value there, but dealing with the data problem is a big obstacle in biology. I can only imagine what she could get done with more capacity, though...
This absolutely terrifies me, surely one misread piece of data or a hallucination here or there and a little “oh sorry about that, I guessed at this portion of the data to save time” and the data used is useless?
> The most interesting question to me is what will be consumed by the exponential like math seems to be undergoing, and what won’t. Writing has been quite stubborn,
isn't it clearly split between verifiable not verifiable ? what is interesting about that question.
Programming has verifiable and non-verifiable aspects. Competitive programming, passing tests, and performance can all be verified. But translating English requirements into actual software, software architecture, taste, or UI design cannot. And yet over the last couple years we’ve seen huge lifts in all of these areas, not just the verifiable ones.
Verifiable areas I think are clearly seeing the most improvement, or are the quickest to see improvement. But we are seeing lots of progress in non-verifiable areas as well.
How much of the non-verifiable progress is a function of labs purchasing expert data vs. the models improving with compute is maybe another interesting question, but fundamentally I don’t see spend on expert data as something that can’t grow if AI revenues keep growing as well. And as models get better taste they can also help filter and generate new synthetic data for their next versions to train on. The limits of this approach are not so clear.
Labs spend billions hiring experts to generate new data, and better models can better filter existing training data and generate new synthetic data. There’s no reason for that to run out, it’s just expensive.
You could view this as just continually patching a leaky ship. But it seems to work.
I remember listening to Andrej Karpathy talk in a podcast about how synthetic data in particular is used to generate more data for pre-training. I see no reasons for that to have changed. I think it is likely a lot of the new data they are paying for contributes to pre-training as well.
I would also be very shocked if they weren't filtering or prioritising existing pre-training data as well, for example to do curriculum learning or to avoid data that degrades performance.
thats the whole point of turning to formal verification:
Imagine a hypothetical oracle, call her MyladyMath, imagine you can turn to MyladyMath, submit a correctly formed dossier of axioms and definitions, theorems with proofs, and then a newly putatively proven theorem T. MyladyMath will complain if your dossier is malformed, and point out where and why. If the dossier is not malformed it will eventually read in the claimed theorem T, evaluate its proof and then either point out a which step is erroneous and why, or ultimately accept the proof.
Instead of a large corpus of human authored text, this map from dossier/theorem -> accept / reject is a huge implicit array of bits, something fundamental, and this weird gigantic array of bits that effectively describe all accept/reject responses MyladyMath would return exactly. We would never run out of "corpus" when it comes to math, if humans had access to such an oracle.
And we do have access to this oracle, and possess compact algorithms that describe the accept / reject bits. One of them is called MetaMath, a minimalistic verifier, which keeps the concepts of prover and verifier separated, this choice results in concrete proof objects (a sequence of step label references).
The machines are going to comb through all possible paths of the next N steps, for progressively larger N, efficiently compress those results in the weights of an LLM and then use the gained experience as the intuition for guided "not-so-brute"-force proof search, using the prior iterations intuitions to grade the surprisal of the N+1't iteration of results, etc.
The money will not stop flowing in that direction: all power blocs, nation states, militaries, banks, ecommerce, ... depend on cryptography. And the machines will soon do more rigorous proof search grounded in more balanced and objective observations. There is no responsible disclosure mechanism for flawed hardness assumptions in cryptography. It's going to get rocky, and the common man will wonder why the gods have gone crazy, wonder why they don't just pull the plug out of the machines, but nobody will in a staring contest to see who dares keep the plug in the longest (and dominate global cybersecurity).
It's the end of the age of artisanal mathematics, it will now become industrialized mathematics.
That is because there is human annotated data there. Every session you or I used, then of course paid human feedback on repos (such as the recently famous example of meta forcing their employees to).
This is _much better_ data than 1/0 verification, it is as good as a gradient.
Automatically verifiable tasks improve faster since well, its automated.
I think you’re making a category error in your definition of politics here.
Certainly technologies can favour winners and losers, but the struggle is an inherently human one.
We definitely are not on an exponential. Don't say we are because this isn't up for debate. AI progress is logarithmic the million dollar question is 2x or 10x for linear improvement. The nearest qualitative shift would be very fast inference so people could start writing real software on top of LLMs. A 0.001% optimization on a packing problem just isn't interesting for the amount of investment.
> A 0.001% optimization on a packing problem just isn't interesting for the amount of investment.
I think you have completely misunderstood what OpenAI have accomplished here. Almost certainly no one cares about the specific concrete results achieved; they only care about (a) how difficult it would be for an intelligent human to achieve the same feat (ETA: the feat is the proof), which can be estimated by the amount of time the problem has remained open/a public conjecture, and (b) how general this artificial "intelligence" appears to be, which can be estimated by the diversity of topics where it was able to prove a difficult result.
It's as if I showed you a dog that I had taught to speak German fluently, and you remarked: "What point is a dog that can speak a language that less than 2% of the world speaks? Nothing to see here."
On the other hand, if provided the financial incentive would mathematicians have solved these problems? Its not hard to imagine a world where some hard problems were not selected by the sparse experts for whatever reason (lack of interest, whatever), which could have been solved if someone was throwing down millions for solutions.
nothing prevented us from defining cryptocurrencies, auto-rewarding new math in objective manners, grading theorem surprisal objectively, etc. before the advent of LLM's.
being lucky enough to receive the opportunity of hiding in some academic closet, poking your hand out begging for scraps, was a different, perfectly alternative path humans decided to take instead.
It's a bit late to start standing up for your rights when the robot overlords arrive.
Thats true, although once it stops being a party trick I expect the LLM proof mining to also be paid by someone else and be expensive as shit, the question same remains: who?
who? the state on advice of the intelligence community: machine learning inference bruteforcing not bits in private keys, but bruteforcing proof search space modulo machine intuition.
At some point cryptography evolves faster than human cryptographers can follow, at that point anyone can only trust cryptographic schemes proposed by their own LLM models, since they will be direct witness to useful responses for apparently unbreakable cryptoschemes with a mathematically known backdoor for the designing LLM.
Why would one power block accept usage of LLM-derived crypto from another power block, if they know it is likely backdoored? The same within a power block: why would one ally financially trust crypto from another ally in the same power block, if they are witness to how easy it will be to request new convincing crypto schemes with a backdoor? The same within a single nation state: why would one bank accept the cryptography proposed by another bank, if they are witness to how easy it would be to ensnare one another with LLM-designed crypto schemes with backdoors.
Its not that humans are incapable of pulling power plugs when the machines start organizing human-on-human violence, it's that no side, no party feels like they can afford a gentleman's agreement to not consult LLM's to find vulnerable hardness assumptions in mathematical cryptography. Perhaps many powerful entities would be more than willing to pull such a plug, but not before others have provably demonstrated pulling the plug,... which is basically waiting until no other human factions exist.
And there's who pays it, and they will pay private companies to compete in order to assure competitive prices for the government.
What is the y-axis in this claim about logarithmic improvements? Any exponential curve can be trivially turned into a logarithmic and vice-versa, and the y-axis redefined as "progress", with no loss of accuracy.
One example that always bugs me is when people point to "exponential" or "sigmoidal" progress on benchmarks. Benchmarks are artificial constructions (saturation at 100% by definition) and benchmark scores should not be mapped to these words when talking about overall progress.
Example - progress on ARC-AGI-3 at the moment is exponential, steeper than 2^t and e^t. Does that mean AI is progressing "exponentially" in the colloquial sense? No, it doesn't support or refute that colloquialism.
Likewise with MMLU saturation. We can't go above 100% by construction. Therefore we have a "sigmoid". Gah.
Yes, the rate and nature of the results being produced is impressive, and Anthropic have recently invested in building out more in-house capability to do life sciences research (see e.g. [1]).
Overall I'm excited for this acceleration in discovery, even if it's causing disruption to existing research workflows. I wrote a bit about it recently [2], after seeing Levent Alpöge's counterexample to the Jacobean conjecture.
The reason writing is hard might be that post-training pulls style into a particular direction.
In other words, big labs are much more interested in making "AGI" than in making a good writer, especially as what qualifies as "good writing" is rather subjective. E.g. before AI use of metaphors and rhetorical devices were generally a sign of a good writing. Of course, not if you keep spamming the same rhetorical device - but a stateless AI can't know which one it is over-using.
I expect we can squeeze a lot more exponential out of LLMs because they’ve basically shown that human “consciousness,” insofar as it’s composed of knowledge and rules for synthesizing that knowledge, can be represented mathematically in a very high dimensional space. Why does this “just work?” Nobody really knows, but it clearly does.
However, I also expect this squeeze will come at an increasingly expensive price — not just because of inefficient token usage, but because of fundamental limitations of LLMs as a model.
LLMs are letting us brute force our way through a lot of reasoning, but it’s hard to believe that such a generic model of intelligence will take us to the next frontier. We’ll need some fundamentally new approaches at some point. Maybe those will make achieving the exponential more efficient or maybe they’ll unlock even higher degrees of possibility. Who knows?
The transformer is Turing complete. It might be a tarpit though? I don't know.
I think a nice example is using them for arithmetic. It's a specialized deterministic process, so it's extremely wasteful to do it that way.
But they're good at finding solutions to things we don't know how to specialize yet.
So, to use metaphor, maybe the transformer-based models are like the FPGA, and then when we figure out the patterns in that system — all the different kinds of specialized reasoning — we can extract it into an ASIC?
I think this is one approach to AI safety and interpretability that could work, but would require labs to slow down to figure out how to extract circuits/algorithms out of trained LLMs rather than deploying the opaque artifact.
> Why does this “just work?” Nobody really knows, but it clearly does.
We know language has to be learnable by every human, so it needs to be really independent of any specific brain development particularities. If it was not accessible to babies there would be no more language next generation.
The distinction is between information flowing from people to power (elicitation), vs. it flowing from power to people (persuasion). These are not the same, even if they are closely related.
Does AI make real opinion easier to hear, or fake opinion easier to spread? Even if you believe wholly in manufactured consent, how easy it is to manufacture matters.
Today's models depend on inference time compute to get these results. The inference time compute available on any claude subscription is not comparable to the ones used to get some of these results (yes, in this case, it is 2000 USD total as noam confirmed, but some previous results took more).
In general, you can think of the process as generating massive rollouts in generation N, and then compiling in the verifier/human feedback("gradient") signal into generation N+1. The time taken to make the rollout in generation N, and separately the time taken to get the same rollout in generation N+1, each grows constant in some tasks, linear in more, and exponential in some.
In the end, this becomes bottlenecked by time. Today, we can make statements like "I generated all these successful trajectories with 2 weeks of compute, in the next model it will be able to do it in 7 hours of compute", but very soon you'll find yourself making statements like "I generated.... with 8 months of compute, in the next model it can do it in 6 months", which isn't really enticing the same way you can _technically_ brute force passwords but it just needs prohibitive amounts of time and money. That is the "plateau". Note that, this point is quite far away. For example, at any point if we agree it plateaus, today's known hardware techniques such as fixed function accelerators give you a 10-100x timeline reduction immediately allowing for a few more cycles of improvement. This is not to mention future innovations, but of course none of that is helping with the benchmarks where the time needed is growing superlinearly.
In many math and coding benchmarks, we are still in the constant phase. These are the massive improvements we see every few months. I'm not making any prediction of what will plateau and what will not as it's not possible to make an informed prediction about these things IMO. But the observed fact is that some have already plateaud as in, they don't improve with reasonable inference time (likely superlinear growth).
> will we need mathematicians to translate
Let's take a sudoku analogy. The model is initially just doing the random value algorithm, but lets say you the human are watching it. You make one of the usual reductions and interject "hey you can stop trying 8 here because of ....". Over enough examples, you get to a point where the model is _forced_ to learn the logical pattern. Next generation, it will skip that number. After this, you can peak the distribution using simple 1/0 RL. Doing _pure_ 1/0 RL works decent, but its not frontier as its a very sparse signal.
For that lift, human (or even a better LLM, but if you're trying to improve a frontier LLM, there is by definition no better LLM) feedback becomes necessary. This is _why_ it is crucial that these models interface in natural language and is also why the labs are hiring AI tutors by the hundreds. The "better LLM" case is what Kimi etc are doing by "distilling"(bad term for this) claude.
> But the long term is completely bewildering if you believe any of these trends can continue at a similar pace for the next few years.
For math and coding, for now we are in the phase where the times are just ... constant, so there's little reason to think it will stop soon. We still need humans to expand the frontier. It just becomes a matter of if its worth the cost of compute for running this generalized The Algorithm or not.
Given how well chess players internalized _many_ (not all) of alphazero's emergent chess knowledge, I am confident we wont have too much trouble figuring out any new math LLMs come up with, which will let us keep expanding the frontier by giving the LLM the next "lift". Only when we reach the stage where the time growth become exponential will this stop, IMO.
It's like Musks duplicitous argument about unlimited abundance. We have a lot of abundance now, we just keep accelerating it all into the hands of fewer and fewer people - whose response is only to want more, and more, and more.
Yeah, we really should just storm the facility where Musk is hoarding all the worlds bread and meat.
Wealth in terms of capital doesn't represent material goods, it represents the system's confidence in your ability to direct capital efficiently. But eventually efficient capital bottoms out at consumable goods. Someone like Musk with a lot of capital under his control is contributing to the end goal of unlimited abundance.
With his shitty cars, orbital garbage, or CSAM generator? Or his moneyed attacks on democracy and communications? Or his vandalism of government programs without insight, experience, or qualification?
In a free and fair market, his capital would be regarded as a deeply inefficient distortion.
In fact the opposite, look at talks by Peter Thiel. At least he’s being honest. These people are bastards, but for some reason moral goodness and wealth has been conflated leading us to venerate greed.
I've noticed that out of all LLMs I've ever used that Fable is the MOST LLM; the text it produces is abomination. It's impressive how much I hate it. It is such an awful writer - it assumes the reader has zero context and therefore gives every single bit of context and detail - which is nice if you're writing a legal document I suppose. But it uses, niche, $10 words to describe every facet of everything it's discussing. I had it re-write some docs and I ended up rewriting 1k lines of of Fable torment nexus text to around 100. Because guess what, someone reading highly technical docs has a knowledge base that allows us to compress the topic into a much tighter representation.
Just praised Fable in another comment but what you're saying is also insanely true.
I literally roll my eyes and cringe quite often at its output pretty much daily.
I don't like to overload my sessions with skills but I've been using a "write-normal" skill I made just to have it rewrite outputs that particularly piss me off.
"the test is whether the sentence would work as a pull quote or a LinkedIn post. If it would, rewrite it until it would not."
LinkedIn really became the default garbage example
Fable is much better at handling nuance. Opus/GPT 5.6 Sol are much more likely to miss the point you are trying to make, emphasise the wrong thing, exaggerate the importance of unimportant details, or introduce contradictions.
That said, Fable is still not a great writer, largely driven by it not knowing what it should exclude, and it still having the usual LLM-isms. But it’s better.
That's really what got everyone hooked in the first place.
5.6 Sol is great but there's a depth to the understanding that Fable exhibits that's unique to it currently.
Can I truly quantify this? I don't think so. Just that I spend a ton of time with various models and a certain point it's just a personal impression or a gut feeling.
In the days after Fable first came out I increased the amount of parallel planning of tasks that I was doing by 2-3x because it felt like I didn't need to be paranoid due to that handling of nuance.
Another interesting question is why the frontier labs are piling on pure maths, which has little direct economic value compared to something like law or improving the efficiency of their own models? How much OpenAI and Anthropic are paying to serve these models for ordinary users is the elephant in the room. A cynical take is that the frontier labs are trying their best to pump up their pre-IPO valuation through flashy headlines.
> Another interesting question is why the frontier labs are piling on pure maths
The reason is that the original scaling axes (parameters, training tokens, test-time compute) have saturated already, but RLVR (reinforcement learning from verifiable rewards) is still scaling well. And math has this nice property where you can synthetically generate arbitrary volumes of rewards to train the model, because math is self-contained and completely objective. Open-ended reasoning and analysis don't have that convenient property, and that is why progress is much slower outside of math and coding.
do we have any concrete idea of how well the models are scaling now? I agree with these 10 results being impressive, and it is easy to think "wow, and last year the models were barely able to solve IMOs problems". But for me it is perfectly possible that a non sofic group could be found by 100 good IMO students working on all the different strategies that have been proposed (OpenAIs solution was based on "expander graphs", which were introduced to solve the problem some years ago), so it could be that current AI is simply many (say 1000) old models working in parallel. This is linear scaling, not exponential. It could be I'm completely wrong also, the problem is that we have little information.
It's one of the few areas where you can verify results. That fits nicely into training models. They aren't just making judgement calls on what would be nice, it's "what can we do?".
If everyone publicly said that the models can only do things that humans have already done, but you know they can do more, wouldn’t you want to show them otherwise?
Math ability also helps with other things like making models more efficient.
Because it's a tool in search of a use case (or many use cases) and mathematics is the most natural use case for it. Mathematics is by definition the art of putting words on a page in a rigorously defined "correct manner" (i.e. in the form of a valid logical argument, a proof) and all LLMs do is put words on pages and evaluating if they're good words is by far easiest when there is a strict definition of right and wrong.
What's the best way to apply it to legal problems? Finding bugs in statutes? (there are often statutes with wording errors, missing negatives, things like that which don't get picked up for ages)
There are many math problems that are simply puzzles: intellectually interesting but nothing worth of value depends on it. To me it would be more impressive if we could define hard problems that need to be solved up front and see how the models deal with that.
The results OpenAI demonstrated are impressive, but it also looks like they threw a lot of compute at it just to get results. How many tokens did they waste on problems they couldn't solve? Applying inference infrastructure on a large number of math problems at scale we haven't seen before to me doesn't demonstrate an exponential curve in model abilities.
"To me it would be more impressive if we could define hard problems that need to be solved up front and see how the models deal with that."
I was recently listening to BBC Radio 4's episode on the Poincare Conjecture[1] and the guests on the program were discussing how the problem that looked deceptively simple eluded the great mathematicians of the time (including Poincare himself) for nearly a century and how Grigori Perelman cleverly came up with the proof. It took other mathematicians working in groups years after Perelman's publication to understand and validate his proof. The mathematicians on the program were speaking of highly of his proofs and admiring the originality of his work. This made me think of one neat experiment where if we cut-off a frontier model's training data 2002 or anytime before Perelman posted his proofs on arXiv and check if it can come up with the solution by itself. That would surely be a great signal to see if these LLMs aren't just solving interesting puzzles and that they can came up with something truly novel.
P.S I highly recommend Misha Green's "Perfect Rigor" for anyone interested in the history of the problem and the genius behind the proofs of the conjecture - Perleman. I found it an entertaining read and could digest its description of the problem as a layperson (with undergrad level math).
Extreme claims on posts like these also, rightfully, trigger people’s skepticism. I don’t think it’s wrong to question claims that math is dead as a field. But then it leads people to miss the overall trendline.
People argue whether we are at y-5, y, or y+5, meanwhile we seem to be on a y=2^x exponential that keeps leading to crazier and crazier results. The much more interesting question to me is what will be consumed by the exponential like math seems to be, and what won’t. Writing has been much more stubborn, but I’ve noticed Fable to be quite a big step up there as well. How about politics? Will we develop new ways to let people express their own values in democracies, or will we get much better at manipulation?
And then there’s questions like, even if AI can answer increasingly complicated math questions, will we still need mathematicians to translate results to the real world, verify them, or decide where to push the frontier?
This is evidence of culture problems in whatever teams you are a part of, or extrapolating what you see on social media to all of software engineering.
We still have a very strong review culture, and people work hard to review their own code before making PRs to avoid wasting other people's time.
This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt.
I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.
I’ve observed the degradation, but I suspect what’s happening is they’re tuning it for lower inference costs. Maybe turning down the amount of thinking, maybe quantizing, maybe something else.
It seems like there’s a week by week and sometimes day by day change in performance when on a subscription plan using their harnesses.
https://marginlab.ai/trackers/claude-code/ their tracker generally shows that isn’t the case. The only times I’ve seen it drop is something broken and just before fable launched.
Is this using the api or using a subscription, though? The incentives are different for each, and it isn't the least bit unexpected that they would maintain API access quality while 'optimizing' the subscription experience to improve their margins (or losses)
It seems to do really this you would need to crowdsource it -- users individually give the lab access to a body of subscriptions normally used by average people, and the lab occasionally runs some masked version of the task through on diverse accounts.
I feel like a conspiracy theorist but it feels like every new model release from both Anthropic and OpenAI has 1-2 weeks of fantastic performance then a gradual (and sometimes not so gradual) decline in intelligence. It's like they're quantising the model in the background to optimise for available compute/RAM. Which, if I put my MBA hat on, would make perfect sense. Why not halve the required RAM for "90%" of the performance? They get fantastic benchmarks and glowing reviews on release, then slowly squeeze more performance out of the model. By the time the next model is ready for release, the jump feels quite large again.
This is also my experience. I don't know if it's because of the quantization theory, or if it's just me getting used to a certain level of coding performance and gradually less tolerant of the mistakes it makes more over time.
There's clearly random variation, but it also shows each model release is just genuinely better. ( With the exception of a heavily degraded week of opus 4.7, which was acknowledged as a problem at the time. )
There's a psychology of getting used to models after being wowed by the new performance. It sets in as your new baseline expectations, and then when it doesn't deliver, it's felt more acutely. When it does deliver, it's just meeting expectations.
Then a new better model comes along and it's a step up again, another wow moment for a week or two until expectations adjust to meet the new baseline.
I replied to the user above that referenced marginlab, but I believe marginlab uses the API. It is possible (arguably likely, in MBA-land) that the API and subscription accounts hit different sub-models.
Even if they use a subscription account, surely Anthropic can tell which one it is.
You should. Feel like a conspiracy theorist when saying things like this.
Users are not reliable or consistent model evaluators. Users adapt to models - the moment they get a model that performs better, their expectations rise, their tasks get harder and their prompts get shorter.
"They made the model worse" is PEBKAC in 9 cases out of 10.
There are few conspiracies where the vectors between "capitalist organization makes more money" and "user can't reliably distinguish tiers of product quality" overlap.
I wish the users weren't so fucking stupid with the "they made the model worse" stuff.
Then that 1 out of 10 case where the model was actually made worse (whether intentionally or by mistake) would stand out instead of being swallowed by the noise floor.
I got Fable to run overnight and I woke up to a working prototype of a very complex feature. And then I did it again for another complex feature the next night.
The code still took weeks to clean up, but it worked and was correct. It felt then, and still feels, like a big step change on very hard problems. These are problems I would previously expect to take a month or longer to implement.
I have also noticed Fable can handle much more nuance when reasoning through writing and research, but that is harder to quantify.
Not OP, but to me it initially felt extremely proactive and energetic, just powering through roadblocks with ingenuity and enthusiasm. After it came back I was constantly getting refusals and downgrades for things Opus had been doing. I’ve written it off for my use cases and getting by just fine with Opus 4.8 and now 5.
my issue with frontier code is that it uses a model judge for quality whereas slop code bench forces a model to grapple with its own garbage code in order to receive a functionality reward
Similarly, I recently disabled auto-memory in Claude Code, and performance improved.
Managing the context that agents have available to them is far too important to leave to the agents themselves. Agents tend to write far too much into their memory, they are terrible at trimming it down, and their choice of what to include is very poor. I have had much more predictable results by disabling auto-memory and actively shaping my CLAUDE.md, skills, and documentation instead.
Maybe one day agents will be able to manage their own context, but that day is not today.
I have been using Fable 5 extensively, and Opus 5 yesterday and today. I have not noticed any step-change improvement in their judgement in what to keep a memory of or not.
I have actively experimented with this as well. I have a reflect skill that actively prompts the models to modify their memory, and have tried to run sessions actively asking the models to consolidate their memories. Fable is noticeably better at this, but still nowhere near good enough.
Fable will still make mistakes where I give feedback on one piece of code and it will create a memory applying that rule everywhere, completely missing the context for why my advice only applied to that one place. It has also made memories of random details about a service that are very unlikely to ever be relevant again, and for things where we could just read the config if we needed to find that information again anyway. And then it will miss making memories of important architectural concerns.
I think auto-memory suffers a similar problem to comments where newer models write better comments, but their choice over when to write comments, and how long those comments should be, still sucks.
Yep, I disabled it and moved to Claude.local.md anything that seemed worth keeping. Automemory kept so much random cruft and had grown bigger than my Claude.md.
System One says they use RLCD, Reinforcement Learning for Calibrated Decisions, which presumably has accurate probabilities as an explicit optimisation goal.
reply