Almost every engineer you interview today will probably tell you they use AI tools.
A year or two ago, that answer might have been useful. Today, it tells you very little. AI has become such a normal part of software development that asking someone, “Do you use AI?” is starting to feel a little like asking, “Do you use email?”
The more important question is what happens after they open the tool.
Do they know how to use it well?
Do they notice when it gives them the wrong answer?
Do they question what it produces, or do they simply assume that something generated by AI must be correct?
That's where the difference between candidates starts to become much clearer. Two engineers can both say they use AI every day, but one might be using it thoughtfully while the other is simply accepting whatever it gives them. Most interview processes aren't very good at telling those two people apart.
We've found one question that gets much closer to the answer, and interestingly, it isn't really about AI tools at all; it's about what happens when the tool gets something wrong.
Why “Do You Use AI?” Isn't Enough Anymore
AI is no longer an unusual skill for engineers. The Stack Overflow Developer Survey 2025 found that 70% of developers use AI tools daily. When something becomes that common, simply having experience with it stops being a useful way to separate candidates.
The problem is that many companies are still interviewing for AI skills as if they're something new. They ask candidates which tools they've used, how often they use them, or whether they know how to write good prompts. Those questions might tell you whether someone has opened an AI coding assistant before, but they don't tell you whether that person can use it responsibly.
Think about it this way. You wouldn't hire someone to drive just because they told you they've driven a car every day for five years. You'd also want to know whether they know when to slow down, how they react when something goes wrong, and whether they pay attention to what's happening around them.
AI-assisted development isn't that different. The useful skill isn't simply knowing how to use the tool. It's knowing when to trust it, when to question it, and when to stop and do something yourself.
Try This Question Instead
Here's the question we'd recommend asking:
Tell me about a specific time an AI tool gave you something wrong, and how you caught it before it became a real problem.
That's it. You don't need to ask candidates to explain how a particular AI model works. You don't need to test them on AI terminology. You don't even need to ask which tool they prefer.
You're asking for a real story, and that story can tell you a surprising amount about how someone actually works.
Here's what that question reveals:
First, it tells you whether they genuinely use AI as part of their work. Someone who uses these tools regularly will usually have a few examples they can talk about without having to search their memory.
Second, it tells you whether they check the work. AI can produce something that looks perfectly reasonable while still being wrong. If someone has never noticed an AI mistake, either they've been unusually lucky, or they're not looking closely enough.
Third, it shows you where they draw the line between trusting AI and trusting their own judgment. That's becoming one of the most important skills for engineers today.
Finally, it shows whether they take responsibility for the result. When something goes wrong, do they say, “The AI messed up,” or do they explain what they should have checked and what they changed afterward? That difference matters.
Listen for the Story
The strongest answers usually have one thing in common: They're specific.
A candidate might tell you that an AI tool generated the wrong database query, suggested a change that would have broken an existing feature, or misunderstood an important part of the application, and more importantly, they'll explain how they noticed the problem.
They won't just say, "I always double-check AI-generated code." Instead, they'll tell you what they checked, what looked suspicious, what they changed, and what happened afterward.
Good candidates are usually comfortable talking about mistakes. They don't need to pretend the AI tool was perfect or that they caught everything immediately. In fact, being able to say, “I almost missed this, but then I noticed…” can tell you more than a polished success story.
You're looking for judgment, not perfection.
A weaker answer usually sounds much more general. The candidate might say, “AI sometimes makes mistakes, so I always review everything,” and leave it there. There's nothing technically wrong with that answer, but it doesn't give you much evidence about how they actually work.
When you ask a simple follow-up like, “Can you give me an example?”, you'll usually learn much more.
What If They Say AI Has Never Been Wrong for Them?
This is where the question gets particularly useful. If someone tells you they've never had an AI tool give them a wrong answer, that shouldn't necessarily impress you. It may actually be something worth exploring further.
AI tools make mistakes; sometimes they're obvious, sometimes they're subtle, and sometimes the answer looks completely reasonable until you check it against the actual requirements or the existing system.
So ask another question: “Can you think of a time you weren't sure whether the AI's answer was correct?” That gives the candidate another opportunity to explain how they verify their work without turning the conversation into a trick question.
The goal isn't to catch someone out. It's to understand how they think.
Does This Work for Junior Engineers Too?
Yes, although the examples will naturally be different.
A junior engineer might talk about catching a small logic error in generated code or realizing that an AI suggestion didn't match what the task actually required. A more experienced engineer might describe something more complex: spotting a risky database change, an incorrect assumption about an existing system, or a security issue that wasn't obvious at first glance.
The important thing isn't how dramatic the mistake was. It's whether the candidate can explain what happened, how they noticed it, and what they learned from it.
Don't Make This Your Entire Interview
This one question can be useful, but it shouldn't replace the rest of your technical interview. You still want to understand how someone solves problems, communicates, works with other people, and handles the technical responsibilities of the role.
One useful addition, though, is to let candidates use AI during part of the interview. Give them a practical problem and allow them to use the tools they would normally use at work. Then watch how they approach it.
Do they blindly accept the first answer?
Do they ask better questions when the result isn't right?
Do they check what the tool produces?
Do they notice when something doesn't make sense?
That gives you a chance to see their judgment in action rather than simply hearing them describe it.
What You Learn From Two Candidates
Imagine you're interviewing two people for the same engineering role.
The first candidate says, “AI tools sometimes get things wrong, but I always review the code carefully.”
The second candidate tells you about a recent project where an AI tool suggested a database change that would have caused problems with existing data. They explain what looked suspicious, how they checked it, what they changed, and what they now do differently when reviewing similar suggestions.
Both candidates use AI, but you've learned something very different about them.
The second candidate has shown you that they don't just use AI to move faster. They understand that moving faster only helps if they can still recognize when something is wrong.
The Skill That Actually Matters
The best AI-related interview question isn't really about AI. It's about judgment.
Tools will keep changing. The tool a candidate uses today might not be the tool they use next year. New models will appear, existing ones will get better, and the way engineers work with them will continue to change.
What won't change as quickly is the need for someone to look at the result and ask, “Does this actually make sense?”
That's the person you want on your team.
So the next time a candidate tells you they use AI every day, don't stop there. Ask them about the last time it got something wrong. Their answer might tell you much more than the tool they use ever could.
And if redesigning your interview process to find that kind of judgment isn't where you want to spend your time, that's where MyNextDeveloper helps by connecting startups with vetted engineers and AI talent who are evaluated not just on what they know, but on how they actually work.
TL;DR
Nearly every engineer you interview will say they use AI tools every day, so that question doesn't tell you anything useful anymore. The one that does: Ask them about a specific time an AI tool got something wrong, and how they caught it. A strong answer comes with real details, an honest story about what broke, and a clear sense of what they'd do differently, not a vague "I always double-check everything."
If someone claims an AI tool has never let them down, that's actually a warning sign, not a green flag. This one question quietly tests exactly the judgment that separates engineers who use AI to move fast safely from those who just move fast.
Looking to build a high-performing remote tech team?
Check out MyNextDeveloper, a platform where you can find the top 3% of software engineers who are deeply passionate about innovation. Our on-demand, dedicated, and thorough software talent solutions provide a comprehensive solution for all your software requirements.
Visit our website to explore how we can assist you in assembling your perfect team.
