Paper on Perceived vs Real Productivity Gains Using LLM

The whole point is that the premise is wrong. It’s like “is it slower to use a internet or a library?” and the methodology section was “search for the thing on Google, flip through and click through all of the links in the Google search results until the answer is found, or try again”. Then the claim is “using a library is faster because we found finding the results could take many pages, or not exist!”.

The reasoning for the issue then ends up being simply due to how mean time is calculated: most of the “internet” results are found in 20 seconds of Googling vs 5 minutes in the library, but in the cases where the result doesn’t exist in the internet, it takes a few hours with this methodology to exhaustively click through all of the results and know it doesn’t exist, while in the library it’s 10 minutes to go :person_shrugging: I guess there isn’t a book on the subject, and the librarian told me so.

Of course, the problem is, that’s not how people search. Or at least, it’s not how you should use Google search and you’re doing it wrong :sweat_smile:. You google a term, look at the first page, if it’s not there google a different term, otherwise head to a chatroom / Discourse and ask for help (or these days, try an LLM).

You have the exact same issue happening in this methodology. There is a very long tail for how long it can take to finish a PR. I wrote in pretty copious detail, don’t let that tail fool you.

If you just accept that some PRs work and some don’t, it takes ~10 seconds per PR. Maybe like 3 minutes if you let it iterate a few times. Any result that says it takes longer than that… what the heck are they doing? For that problem just give up and do it yourself, in which case it just takes a long as doing it without AI.