How I Actually Evaluate an AI Coding Assistant
I’ve tried more AI coding assistants than I’d like to admit. Most of them look great in a five-minute demo and then quietly stop earning their keep once the honeymoon period ends. Over time I’ve settled on a short list of questions I ask before letting a tool anywhere near my actual codebase.
Start with the failure cases, not the wins
Every tool demo shows you the happy path: a clean function, a well-scoped bug, a green test suite. What I actually want to know is what happens when the task is ambiguous, the codebase is large, or the fix touches five files instead of one.
Does it know when to stop?
A tool that confidently rewrites a file it doesn’t fully understand is worse than no tool at all. The assistants I keep using are the ones that ask a clarifying question, or narrow their change to exactly what was requested, instead of “helpfully” restructuring everything nearby.
Does it surface its assumptions?
If an assistant makes a judgment call — picking a library, assuming a data shape, choosing an error-handling strategy — I want that decision visible, not buried in a diff I have to reverse-engineer.
Measure the review overhead, not just the generation speed
Fast code generation is not the same as fast delivery. A tool that writes code in ten seconds but requires ten minutes of careful review has not saved me any time; it has just moved the cost around. The metric that matters is end-to-end: from typing the request to merging a change I trust.
Test it on real, messy code
Toy examples don’t reveal much. I run every new tool against:
- A legacy module with inconsistent naming and no tests
- A task that requires touching multiple files with an existing convention to follow
- A bug report with incomplete reproduction steps
How a tool behaves here tells me far more than any benchmark score.
The verdict I keep coming back to
The assistants that stick around are the ones that make me faster at the parts of the job I don’t enjoy — boilerplate, migrations, first drafts of tests — while leaving the parts that require judgment clearly in my hands. That balance, more than raw capability, is what I’m actually evaluating.
