GitHub Copilot's AI Tested: Mixed Coding Success Leaves Me Baffled

Exploring the Inconsistencies in AI Coding Tools

It's truly puzzling how AI tools, all built on the same foundational large language model, can yield such varied results. For instance, ChatGPT, Perplexity, and GitHub Copilot all leverage OpenAI's GPT-4 model. Yet, my recent tests showed stark differences in performance: while ChatGPT and Perplexity's pro plans excelled, GitHub Copilot had a 50% success rate.

I conducted these tests using GitHub Copilot integrated within a VS Code environment. I'll share a detailed guide on setting this up in an upcoming article. For now, let's dive into the specifics of the tests I ran.

If you're curious about my testing methodology and the prompts used, you can check out my detailed guide on evaluating an AI chatbot's coding capabilities.

TL;DR: GitHub Copilot managed to pass two out of the four tests I conducted.

Test 1: Writing a WordPress Plugin

This test was a complete disappointment. It was my initial experiment, leaving me unsure if GitHub Copilot struggles with coding or if the interaction constraints within VS Code hinder its capabilities.

Here's the context: I asked the AI to develop a fully functional WordPress plugin that includes an admin interface and operational logic. The plugin's task was to accept a list of names, sort them, and separate any duplicates to avoid adjacency.

This task stemmed from a real-world need from my wife's digital goods e-commerce business, where she manages an active Facebook group.

While five out of the ten AI models tested passed this test entirely, three passed partially, and two, including Microsoft Copilot, failed completely. GitHub Copilot, despite being given the same prompt, only produced PHP code. Although the problem could indeed be solved with PHP alone, GitHub Copilot attempted to reference JavaScript without actually generating it.

When I tried to prompt GitHub Copilot from within a JavaScript file to complete the task, it bizarrely responded with more PHP code, still referencing a non-existent JavaScript file.

Test 2: Rewriting a String Function

This test was relatively straightforward: I provided a function meant to validate dollars and cents but only checking for whole dollars. The challenge was for the AI to correct the function.

GitHub Copilot did modify the code, but the result was problematic. It assumed that any input string was valid, which would cause errors if the string was empty. Additionally, the updated regular expression couldn't handle various edge cases, such as inputs like "3.", ".3", or "00.30". For a function meant to validate currency, such oversights are unacceptable, marking another fail for GitHub Copilot.

Test 3: Finding an Annoying Bug

Here, GitHub Copilot shone. This test was based on a real coding challenge I faced, where the error message didn't directly point to the actual issue. It's a bit like a coding riddle, requiring deep understanding of WordPress API calls to solve.

While Microsoft Copilot, Gemini, and Meta Code Llama stumbled on this test, GitHub Copilot nailed it, showcasing its capability to tackle complex, real-world problems.

Test 4: Writing a Script

GitHub Copilot also succeeded in this test, where Microsoft Copilot fell short. The task involved creating a script that needed to integrate AppleScript, the Chrome object model, and a Mac-specific utility called Keyboard Maestro.

To pass, the AI needed to recognize and address the nuances of all three environments, and GitHub Copilot did just that.

Final Thoughts

It's disheartening to see GitHub Copilot, which uses the advanced GPT-4 model, fail half of the tests. Given GitHub's status as a leading source management platform, one would expect its AI coding support to be more dependable.

However, the world of AI is ever-evolving, and I'm optimistic that GitHub Copilot's performance will improve over time. We'll revisit this in a few months to see how it's progressed.

Do you rely on AI for coding assistance? Which AI tool is your go-to? Have you given GitHub Copilot a try? Share your experiences in the comments below.

Stay updated with my daily project progress on social media. Don't forget to sign up for my weekly newsletter, and follow me on Twitter/X at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, on Bluesky at @DavidGewirtz.com, and on YouTube at YouTube.com/DavidGewirtzTV.