Connect with us


New Copilot study finds declining code quality

A new study on the impact of GitHub Copilot and artificial intelligence on software development speaks to some negative results.

A new study on the impact of GitHub Copilot and artificial intelligence on software development speaks to some negative results.

The study, “Coding on Copilot“, produced by GitClear examines the quality and maintainability of code written with AI compared to code that would be written by a human. In other words, “Does AI code look like the careful, finalized input of a Senior developer or more like the chaotic work of a short-term contractor?”.

The answer to this question is summarized in the study abstract already:

“We found discouraging trends in maintainability. Code churn – the percentage of lines that are fixed or updated less than two weeks after being written – is projected to double in 2024 compared to the 2021 baseline prior to the advent of artificial intelligence. In addition, we find that the percentage of “added code” and “copy/paste code” grows in proportion to “updated,” “deleted,” and “moved” code. In this respect, AI-generated code resembles an itinerant contributor prone to violating DRY principles in visited repositories.”

This serves as a strong complement to the findings of several other studies, including a study of GitHub itself from 2022. In particular, it stated that “developers who used GitHub Copilot completed tasks significantly faster – by 55% – than developers who did not use GitHub Copilot.” In addition to productivity, the GitHub study also showed positive effects in terms of developer satisfaction and mental energy savings.

The GitClear study showed the three most significant changes associated with the growth of Copilot, which relate to “fixed,” “moved,” and “copied/pasted” code:

  • Growing churn: “The bottom line is that Copilot usage is highly correlated with “erroneous code” posted to repositories.”
  • Less code moved implies less refactoring, less reuse: “Combined with the growth of code labeled Copy/Paste, there is little room for doubt that the current implementation of AI assistants discourages code reuse. Instead of refactoring and working on DRY (“Don’t Repeat Yourself”) code, these assistants offer to repeat existing code at the click of a button.”
  • More copied/pasted code means future headaches: “There is perhaps no bigger scourge to long-term code maintainability than copied/pasted code. Essentially, when a line of code that doesn’t contain a keyword is repeated, the code author admits: “I didn’t have time to evaluate the previous implementation”. Re-adding code instead of reusing it leaves future maintainers with the responsibility of figuring out how to consolidate parallel paths that implement reusable functionality.”

The article concludes: “How will Copilot change what it means to be a developer? There is no doubt that with the growing popularity of AI, we have entered an era where lines of code are being added faster than ever before. The better question to ask in 2024 is: who will clean up this mess?”.