Not So Fast: AI Coding Tools Can Actually Reduce Productivity
-
Note that the takeaway isn't "AI sucks" but rather that developers felt it made them faster even though the numbers showed the exact opposite. That may be due to the output quality, but also due to inexperience with using these tools.
@volpeon the takeaway is devs sucks
-
Note that the takeaway isn't "AI sucks" but rather that developers felt it made them faster even though the numbers showed the exact opposite. That may be due to the output quality, but also due to inexperience with using these tools.
@volpeon I can believe it but it's also for a specific case of developing where the developer has high familiarity with the codebase, as I understand it -
@volpeon I can believe it but it's also for a specific case of developing where the developer has high familiarity with the codebase, as I understand it
@sun Yeah, from what I've read in comments AI tools help people with getting started with things you aren't familiar with, but as you gain experience (provided you're willing to learn from what the AI produced) you may be better off writing things yourself. Makes sense to me
-
Note that the takeaway isn't "AI sucks" but rather that developers felt it made them faster even though the numbers showed the exact opposite. That may be due to the output quality, but also due to inexperience with using these tools.
The coding applications built on those models, like Cursor, are going to keep improving to make better use of the models
This part is funny, though. Just as Cursor is forced to enshittify because Anthropic upped their prices for enterprise customers (which is most likely because they're in trouble themselves). -
The coding applications built on those models, like Cursor, are going to keep improving to make better use of the models
This part is funny, though. Just as Cursor is forced to enshittify because Anthropic upped their prices for enterprise customers (which is most likely because they're in trouble themselves).The problem tools like Cursor have is that unlike classic software, AI is horrible to run at scale. With something like a social network, the cost per user goes down as the number of user increases. With AI, you can't have this kind of parallelism that brings the cost down and that means there's linear growth. Computations on the GPU are specific to one model invocation, and a model invocation can't handle multiple requests at once.
-
The problem tools like Cursor have is that unlike classic software, AI is horrible to run at scale. With something like a social network, the cost per user goes down as the number of user increases. With AI, you can't have this kind of parallelism that brings the cost down and that means there's linear growth. Computations on the GPU are specific to one model invocation, and a model invocation can't handle multiple requests at once.
-
The problem tools like Cursor have is that unlike classic software, AI is horrible to run at scale. With something like a social network, the cost per user goes down as the number of user increases. With AI, you can't have this kind of parallelism that brings the cost down and that means there's linear growth. Computations on the GPU are specific to one model invocation, and a model invocation can't handle multiple requests at once.
When you run an LLM, and then another one for a different user, they will use twice the amount of VRAM and twice the number of cores to get the same performance as the original single run.
Let's say you have a database server used by one application, and then you add another application. How much do the resource requirements increase? Not by another 100%, that's for sure. -
Note that the takeaway isn't "AI sucks" but rather that developers felt it made them faster even though the numbers showed the exact opposite. That may be due to the output quality, but also due to inexperience with using these tools.
@volpeon I remember using TabNine way before AI was big and cool. It was just very quick autocomplete, but I only used it during my last job. And I genuinely have to wonder if I stopped using it because I subconsciously knew it didn't improve my coding performance
-
When you run an LLM, and then another one for a different user, they will use twice the amount of VRAM and twice the number of cores to get the same performance as the original single run.
Let's say you have a database server used by one application, and then you add another application. How much do the resource requirements increase? Not by another 100%, that's for sure.@volpeon All true, but it's worth noting that you can queue up requests for the same model to run one after another on a group of GPU's. Not great scaling, but you could serve potentially a LOT of users from one GPU as long as everyone is willing to wait a little bit.
-
The problem tools like Cursor have is that unlike classic software, AI is horrible to run at scale. With something like a social network, the cost per user goes down as the number of user increases. With AI, you can't have this kind of parallelism that brings the cost down and that means there's linear growth. Computations on the GPU are specific to one model invocation, and a model invocation can't handle multiple requests at once.
-
Note that the takeaway isn't "AI sucks" but rather that developers felt it made them faster even though the numbers showed the exact opposite. That may be due to the output quality, but also due to inexperience with using these tools.
-
@volpeon they actually do batching on inference to handle several requests in parallel, that's how the whole thing even kinda works at search engine kinds of scales
@sergaderg Oh yeah, that completely slipped my mind. And yet, it doesn't seem like it helps a lot considering the massive hardware requirements.
edit: I looked into the performance characteristics and it seems there's a threshold of batch size 64 after which performance stops improving. On a scale of millions of requests, that's pretty much negligible.
