"The technology has improved so rapidly that this study is now close-to-meaningless."
You could have said that anytime in the last 3 years, but the data has never shown it to be true. Is there data to show that the current gen models are so much better than the last gen models that the existing productivity data should be ignored? I don't think the coding benchmarks show a step change in capabilities, its generally dev vibes rather than a large change to measurements.
[1] https://www.youtube.com/watch?v=1OzxYK2-qsI [2] https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-gen... [3] https://www.youtube.com/watch?v=JvosMkuNxF8 [4] https://www.faros.ai/blog/ai-software-engineering
"The technology has improved so rapidly that this study is now close-to-meaningless."
You could have said that anytime in the last 3 years, but the data has never shown it to be true. Is there data to show that the current gen models are so much better than the last gen models that the existing productivity data should be ignored? I don't think the coding benchmarks show a step change in capabilities, its generally dev vibes rather than a large change to measurements.