How do you even measure those? Will that show up under something like VTune?

Yes, VTune is good.

I'm on AMD and uProf isn't that good (it often fails to match PDB to source, it's slow / crashy, stuff like that) so I mostly use Superluminal myself (has godly UX) and only drop down to uProf when I have a suspicion and want to gather hardware stats.

You can do all that with perf on Linux, but everything is manual.

TIL that agile has its own words for everything.

That was a thing in the early 2000's. You had to have a blog and invent words that describe regular things everyone did.

Yeah, I once asked it to review its own writing with a subagent with a pretty strict writing guide, and it didn't see anything wrong, lol.

I like to imagine that there's this one employee that oversees RHLF, who has a particular style of writing, and that got so ingrained that it's just them x1000.

I remember asking claude in claude code why the "seams" and it instantly in very fine detail said how it's from a book on working with legacy code

so they might be RLHFing on these specific approaches and then it becomes the entire model

just an anecdote but I found it interesting how it went full on that it's from that book vs just "it's technical jargon"

AI model does not know special insight into how model itself was trained. All it tells you is it's prediction of expected explanation.

I checked this, and Claude says "it's a real term of art from Michael Feathers' Working Effectively with Legacy Code (2004), where he defines it as a place you can alter a program's behavior without editing in that place."

While I agree the model doesn't have insight into how it was trained I do think the history of the term itself is interesting.

yeah but I didn't prompt it in a way where I tried to get such an answer, it gave an entire boilerplate rundown just from like me asking "what's up with the 'seams'?"

I might be wrong, but usually it'd be a lot less deliberate, and at least in my mind it wouldn't be surprising if they were heaving training these on these specific "best practices" books/methodologies and thus picking up lingo from them

Those books are definitely in the training set if the model can mention them, but why exactly these terms are so commonly used is not necessarily because the book is in the training material. The model does not itself know how its trained.

I've read that book and it's one of the very core concepts that relates to introducing tests in legacy code ("finding a seam").

Tbh those sound more like Mast Cell Activation Syndrome (MCAS) symptoms...

Not really. I have the above and my partner has pots/mcas. They're nothing alike.

I also find them bad at what I call "abstraction compression." They're really bad at noticing when a helper function is needed, when some structure they repeated five times slightly differently can become a struct, when a whole section of code can be encapsulated in a simpler design.

I'm lucky that for my side project (an interpreter) I've written all the code myself, so I've built up its design in my mind over the past year, and so as I mull over what I'm writing I start coming up with simpler designs.

Interestingly using Opus 5 (and LLMs in general) has made me worse at this, since I don't feel the pain of writing something over and over again. On the other hand, I don't really want to implement a whole stdlib, so I have it write more of the auxiliary code. The hardest thing is that I have to manually manage the context, which is painful when I personally remember every helper function and why. I have to remember to keep the list of helper functions updated, which is irritating.

I have that issue too. One thing I suspect they are amazing at is documentation generation, maybe a good solution is to have it generate a concise summary of helper functions that are available for any given context. Then have it use the same doc when you call in some code generation.

You touched on another baked-in limitation of LLMs: their inability to follow Don't Repeat Yourself. To anthropomorphize a little, LLMs love to repeat themselves. I remember in the early days before reasoning algorithms came along, you could ask an LLM a question and it would often explain the same answer two or three times in a slightly different way. You also see this in image generation with multiple people, they will often do essentially the same person several times with minimal variation. Applied to code, in your functions they saw an earlier pattern and they can't help but write it the same way.

This isn't entirely at odds with producing good code, I often force myself to wait for another occurrence of where a helper function is needed before I write it, lest I create a slew of utilities that are easy to forget about. Maybe it just needs an additional pass: "analyze your output and create a set of helper functions where logic is repeated or the abstraction is wanted."

That's the smoking gun

Is it the smoking gun? Or is it the twist? Or maybe, even, the trap?

So is it the LLM or us that's getting the RLHF? /s

I did this too, but it usually thinks its writing is fine in my experience. Even when spawning a subagent, it thinks its effusive comments are fine. It's driving me nuts. Before I commit I end up ripping out 90% of the comments, and rewording the rest, otherwise I'd be drowning in comments. This is my style guide: https://github.com/smj-edison/zicl/blob/main/CLAUDE.md#style...

I just can't get it to stop writing two paragraphs every time it makes a small ownership bugfix in my code. Every time it has to explain in excruciating detail every internal thought it had while fixing it. I find myself going in after and deleting all of its comments, or severely trimming them. Otherwise it ends with the code being unreadable.