X's Grok Surpasses Expectations in AI Coding Tests

When X first launched its chatbot, it was tucked behind a paywall. But, as the saying goes, there ain't no such thing as a free lunch (TANSTAAFL), until recently when X decided to open up Grok to everyone. Curious about its capabilities, I decided to put it through my programming tests.

I've always had a soft spot for Grok, thanks to its name, which was coined by Robert Heinlein, one of my all-time favorite sci-fi authors. Heinlein's works played a significant role in shaping my young mind. My parents, who were quite strict about the media I consumed, allowed me to dive into science fiction at our local library, under the assumption that anything labeled as 'science' must be educational.

Heinlein's stories were not just entertaining; they were thought-provoking, challenging societal norms and weaving in scientific themes with social commentary. The term "grok," introduced in *Stranger in a Strange Land*, embodies a deep, fundamental understanding, making it a fitting name for an AI chatbot.

However, there's a catch...

When I inquired about the large language model (LLM) Grok uses, it mentioned being inspired by the wit and rebelliousness of *Hitchhiker's Guide to the Galaxy*. While *Hitchhiker's* certainly has its charm, it doesn't actually use the term "grok." But let's move on to the programming tests.

1. Writing a WordPress Plugin

This test required the AI to demonstrate PHP programming skills and knowledge of WordPress plugin development. It stemmed from a real-life request from my wife, who needed a tool to randomize names for her e-commerce site's monthly involvement device. The twist was that some users could have multiple entries, so the randomizer needed to ensure these names weren't placed side by side.

The code also had to be user-friendly, allowing her to simply paste names, click a button, and get her list. Grok passed this test with flying colors. The interface was clean, functional, and did exactly what it was supposed to do.

2. Rewriting a String Function

The second test involved fixing a user-reported issue with a function meant to validate dollar and cent amounts. My original code only accepted integers, so $5 was valid, but $5.25 wasn't. Grok rewrote the regular expression, coming close to a win. However, it failed to recognize numbers like .5 as valid currency, and it used an inefficient method with double conversions. So, it's a loss on this one.

3. Finding an Annoying Bug

This test required understanding the WordPress framework and API to pinpoint a subtle bug. Many LLMs, including myself initially, struggled with this. But Grok nailed it, providing a correct and useful solution. That's two wins out of three.

4. Writing a Script

The final test was a challenging one, requiring knowledge of Keyboard Maestro, a niche Mac scripting tool, and the ability to write code for multiple environments simultaneously: Keyboard Maestro, Chrome, and AppleScript. Only Google Gemini and ChatGPT with GPT-4 or higher had passed this test previously. Grok, however, aced it, securing three wins out of four.

Final Thoughts

Grok held up well in these tests. If it had just allowed currency values without a leading zero, it would have been perfect. Despite my mixed feelings about the changes at X since it replaced Twitter, Grok has proven to be a robust chatbot, especially in terms of programming skills.

What's your take on Grok? Have you tried it out? And what about *Stranger in a Strange Land* or *Hitchhiker's Guide to the Galaxy*? Share your thoughts in the comments below. So long, and thanks for all the fish!