Meta AI Fails to Compete with Llama, Gemini, and ChatGPT in Coding Test

How Well Do AI Tools Write Code?

Over the past year or so, I've put several large language models through their paces to see how effectively they tackle basic programming challenges. The idea behind these tests is straightforward: if they can't handle the basics, it's unlikely they'll be much help with more complex tasks. But if they do well on these foundational challenges, they might just become valuable allies for developers looking to save time.

To establish a baseline, I've been using four distinct tests. These range from straightforward coding assignments to debugging exercises that require deeper insight into frameworks like WordPress. Let’s dive into each test and compare how Meta's new AI tool stacks up against others.

Test 1: Writing a WordPress Plugin

Creating a WordPress plugin involves web development using PHP within the WordPress ecosystem. It also demands some UI design. If an AI chatbot can pull this off, it could serve as a helpful assistant for web developers.

Results:

• Meta AI: Adequate interface but failed functionality.
• Meta Code Llama: Complete failure.
• Google Gemini Advanced: Good interface, failed functionality.
• ChatGPT: Clean interface and functional output.

Here’s a visual comparison: (Note: Replace "/path-to-image/" with the actual path to the image file.)

ChatGPT delivered a neater interface and positioned the "Randomize" button more logically. When it came to actually running the plugin, however, Meta AI crashed, presenting the dreaded "White Screen of Death."

Test 2: Rewriting a String Function

This test assesses an AI's ability to improve utility functions. Success here suggests potential assistance for developers, while failure implies room for improvement.

Results:

• Meta AI: Failed due to incorrect value corrections, poor handling of multi-decimal numbers, and formatting issues.
• Meta Code Llama: Succeeded.
• Google Gemini Advanced: Failed.
• ChatGPT: Succeeded.

While Meta AI stumbled on this seemingly simple task, Meta Code Llama managed to shine, showcasing its capability. ChatGPT also performed admirably.

Test 3: Finding an Annoying Bug

This isn’t about writing code—it’s about diagnosing issues. Success requires deep knowledge of WordPress APIs and the interactions between different parts of the codebase.

Results:

• Meta AI: Passed with flying colors, identifying the issue and suggesting an efficiency-enhancing tweak.
• Meta Code Llama: Failed.
• Google Gemini Advanced: Failed.
• ChatGPT: Passed.

Surprisingly, despite its earlier struggles, Meta AI excelled here, proving its potential but also highlighting inconsistencies in its responses.

Test 4: Writing a Script

This test evaluates knowledge of specialized tools like Keyboard Maestro and AppleScript. Both are relatively niche but represent a broader spectrum of programming skills.

Results:

• Meta AI: Failed to retrieve data from Keyboard Maestro.
• Meta Code Llama: Same failure.
• Google Gemini Advanced: Succeeded.
• ChatGPT: Succeeded.

Gemini and ChatGPT demonstrated proficiency with these tools, whereas Meta’s offerings fell short.

Overall Results

Based on my six-month experience using ChatGPT for coding projects, I remain confident in its reliability. Other models have yet to match its consistency and effectiveness. While Meta AI showed flashes of brilliance, its overall performance leaves much to be desired.

Have you experimented with these tools? Share your thoughts in the comments below!