For context, I’ve been coding on and off for the past 7 years, 5.5 years as a hobby and 1.5 professionally. As a student, I’ve been able to test out Copilot for free, and it also is integrated into VSCode, so I couldn’t see any reason why not to just test it out and see what happens.
1. It’s kind of annoying
Basically, imagine Copilot as the kid in class that answers the question before you’re able to figure it out. Sometimes they get it wrong, sometimes they get it right, but it’s kind of annoying either way. Either they get it wrong and interrupt your thought process, or get it right and prevent you from the fulfillment of figuring it out yourself.
Sometimes I’ve been trying to work through a piece of code and Copilot keeps recommending huge, wrong pieces of code that take me off track and I’ll end up just angrily disabling Copilot.
2. It’s wrong… But not super wrong.
This goes back a bit to 2, but essentially Copilot will predict code that looks almost like it could be right, but if you take a closer look you’ll realize that it’s really not doing the right thing at all:
Like, this piece of code? What’s annoying is that this looks plausibly correct, but it’s wrong. June 2, 2021 is a waning gibbous; this says that it would be a new moon. A security analysis by New York University said that 40% of the code produced by Copilot is vulnerable to security threats, and from working with it I would agree that it makes frequent mistakes when generating large pieces of code.
What’s really frustrating about this is that the amount of time that this takes to check is more than just straight up writing the code yourself.
3. Security much?
If you’ve heard anything about Copilot, you’ve probably heard about the concerns surrounding security. We know that we lose control of our code to a certain extent when we make repositories public, and technically Copilot falls into a bit of a gray zone, but it is interesting how, in some cases, it does copy verbatim code that already exists. Ex. literally writing part of the licensing agreement for a Dart repo word for word:
Should we be using it with private, potentially confidential repositories? Probably not. The wording of the terms and conditions seem intentionally vague, and, although you can apparently disable GitHub using your own code to train its model, I have noticed it repeating certain words, phrases, and organizations that I’ve mentioned in other projects in completely new projects, although it doesn’t seem as if that is a common issue.
Although I’m not claiming to know Microsoft’s intentions, it seems to me a bit of a red flag that they offer such a long free trial (60 days); that seems to suggest that having a lot of users to collect data from in this early stage of Copilot is more of a priority than having paying customers.
If this is something you’re interested in, from October 11-13, 2022, there will be a virtual conference held by the Open Source Initiative (OSI) on the ethical ramifications of projects like this.
4. Unnecessary Stuff
Because it’s trained on large existing projects dependent on large frameworks, sometimes it’ll recommend using huge frameworks that aren’t necessary to solve a problem.
- It’s GREAT for learning a language.
I’m a full-stack kind of coder (NodeJS, HTML/CSS/JS, etc). I’ve done some bits in C++ and Java, but outside of that I haven’t really experimented much in other languages. Then Tauri came along and gave me a reason to start learning Rust. Rust is a great language, but it was made a lot easier by how good Copilot is at writing stuff in Rust.
Don’t know how to write a vector in Rust? No problem, Copilot’s got you. Don’t know how to make a TCP socket? Copilot knows what to do.
Don’t get me wrong, going through documentation is great to learn how to do things the right way, but it’s hard sometimes to even figure out what to look up when you’re a total newbie to a language, and having to look up what to do for every single line of code is less than ideal. Although Copilot makes mistakes, it’s got the basics (creating variables, pushing to a list, etc) pretty much down, which makes it a great little learning helper.
2. It is REALLY good.
Don’t get me wrong. After seeing “The Bad,” it seems like Copilot is a terrible piece of copyright-infringing garbage that takes pieces of public code and slaps them together. My opinion after testing out Copilot regularly for 3 months is that it’s helpful… in moderation. It’s good at figuring out small, general stuff, like sending an HTTP request or creating a UDP client. It’s also really good at commenting code. The problems almost always arise when you start trying to make it do big stuff that realistically just needs to be done by hand. This article by 0xbad1dea shows a some critical shortcomings of Copilot.
From my own experience and other articles I’ve read, I think Copilot is good, but not great. When you first start using it, there’s definitely a “wow!” moment where you’re wondering, how on earth does this work so well? Then you take a closer look and realize that the code it recommended has a lot of mistakes in it. I also found it really distracting how if I stopped typing for a second to think, I would get a suggestion.
What’s also interesting to think about is, as Copilot gets used more frequently, it will get trained on code that it is generating, which could lead to some interesting problems.
Ultimately, of the people who are vehemently opposed to Copilot and those who want to use it all the time, I would say that I’m somewhere in the middle; I’ll probably keep using it as a learning tool and as a way to write small pieces of code that I understand well myself. Beyond that, I’ve found that debugging the problems that Copilot creates takes more time than actually just writing the code yourself.
