AI engineer interview questions with scoring rubrics
Fifteen open questions for hiring managers, what weak and strong answers sound like, and a scorecard you can copy into your own process.
By James Pentalow, organizer of Building with AI Charlotte · Published November 11, 2026
The best AI engineer interview questions are open questions that make a candidate explain real work. Ask what they built, how they measured it, and how the model behind it works. Score every answer on the same 1 to 4 rubric. This guide gives 15 questions, what strong and weak answers sound like, and a scorecard.
15 questionsGrouped by the three traits to screen for
1 to 4 scaleOne rubric for every question and every candidate
1 scorecardCopy it and score right after each interview
How to use these questions
These questions follow the three-trait framework I use when I talk to people who build with AI: a systems-builder mindset, curiosity, and an understanding of how AI systems are built. The full framework is in our guide to hiring AI talent in Charlotte. This page turns it into an interview you can run.
Run it as a structured interview. Google's re:Work guide describes structured interviewing as asking candidates a consistent set of questions and using clear criteria to judge the answers. The same guide says research shows structured interviews can predict how a candidate will perform, even in jobs that are not structured themselves.
Pick five or six questions per interview. A 45-minute slot has room for that many plus follow-ups. Split the 15 across two or three interviewers.
Ask every candidate the same questions. You can only compare answers if the questions match.
Follow up twice. Ask how they know, then ask what they would change. Rehearsed answers run out after one follow-up.
Score alone, right away. Give each trait a 1 to 4 and write the evidence before you talk to other interviewers.
Ask leading questions, not quiz questions. A quiz question asks for a definition. A leading question opens the topic and lets understanding show up on its own. Ask why the same prompt can give two different answers. Do not ask the candidate to define temperature.
The 1 to 4 scale
Use one scale for every question. Four points forces a choice, because there is no middle score to hide in.
Score
What you heard
1 · Weak
General statements or buzzwords. No example from their own work.
2 · Partial
A real example, but gaps on how it was measured or what went wrong.
3 · Solid
A clear example with numbers, trade-offs and an honest failure.
4 · Strong
Several examples, answers the follow-ups before you ask, and ties the work to a business result.
Systems-builder questions
These five questions test whether the candidate thinks in whole systems. You want to hear what they built, how they built it, how they measured success, and whether they can connect technical detail to business value.
Question 1
Walk me through an AI system you built end to end. What went in, what came out, and where could it break?
Why ask it: It shows whether they see the whole system or only their part.
Weak (1): Lists tools and frameworks. Cannot name a failure point.
Strong (4): Describes inputs, outputs and users, names at least two ways it could fail, and says what they did about each.
Question 2
How did you know it was working? What did you measure before and after?
Why ask it: Measurement separates a shipped system from a demo.
Weak (1): Judged it by looking at a few outputs. No baseline.
Strong (4): Names a metric, a baseline and a target that was set before launch. Anthropic's guidance on evaluations describes good success criteria as specific, measurable, achievable and relevant.
Question 3
What failed after launch, and how did you find out?
Why ask it: Everyone who has shipped has a failure story.
Weak (1): Says nothing failed, or blames the model.
Strong (4): Describes a specific failure, how it was detected, the fix, and what they changed so it would be caught sooner next time.
Question 4
What did it cost to run and how fast did it respond? What did you trade off?
Why ask it: Cost and speed decide whether an AI feature survives.
Weak (1): Does not know the cost or the response time.
Strong (4): Gives rough numbers for both and explains a trade-off they made, such as a smaller model for a simpler step.
Question 5
Who used it, and what changed for the business because of it?
Why ask it: This is the bridge from technical detail to business value.
Weak (1): Cannot say who used it or why it mattered.
Strong (4): Names the users and a result such as time saved, revenue or risk reduced, and can explain what the number means.
Curiosity questions
These four questions test curiosity and self-motivation. Curious people ask next-level questions about your organization, teach themselves, and carry good practices from one job to the next.
Question 6
What did you teach yourself in the last three months, and why did you pick it?
Why ask it: AI tools change often, so recent learning matters more than old credentials.
Weak (1): Nothing recent, or only what the job required.
Strong (4): A specific skill, the reason they chose it, and something they built with it.
Question 7
Which AI tool did you try recently and stop using? Why?
Why ask it: Dropping a tool for a clear reason shows judgment.
Weak (1): Has not tried anything new, or has no reason for the choice.
Strong (4): Compares two tools on a real task and explains what made one lose.
Question 8
What practice from a past job did you bring to a new team?
Why ask it: Good people carry habits with them and spread them.
Weak (1): Cannot name one.
Strong (4): Names a habit such as testing, documentation or review, and describes how they got a new team to adopt it.
Question 9
What do you want to know about our data, our users and how we ship?
Why ask it: The questions a candidate asks are evidence too.
Weak (1): No questions, or only questions about pay and perks.
Strong (4): Asks second- and third-level questions that build on your answers and show they are already thinking about the work.
How AI works questions
These six questions test whether the candidate understands how the sausage gets made. You do not need a researcher. You need someone who knows roughly what a model is doing, so they can tell where AI fits and where it fails.
Question 10
What happens between pressing enter on a prompt and seeing the reply?
Why ask it: It reveals their mental model without naming any terms.
Weak (1): Treats the model as a search engine or as magic.
Strong (4): Explains that text is split into tokens and the model predicts what comes next, one token after another. Google's Machine Learning Crash Course describes large language models as predicting a token or a sequence of tokens.
Question 11
Why might the same prompt give two different answers?
Why ask it: It is the leading version of a quiz question about temperature.
Weak (1): Calls it a bug, or has not noticed it happens.
Strong (4): Explains that the model chooses among likely next tokens, that settings control how much the choice varies, and what that means for testing.
Question 12
A model gives a confident wrong answer. What is going on, and what would you try first?
Why ask it: Every AI product hits this problem.
Weak (1): Says to write a better prompt and stops there.
Strong (4): Explains that the model produces likely text and does not check facts on its own. Proposes giving it sources to answer from, then measuring the error rate on a test set.
Question 13
How would you get a model to answer questions about our internal documents?
Why ask it: It opens retrieval without saying the word.
Weak (1): Suggests training a new model on the documents and nothing else.
Strong (4): Describes retrieval: find the relevant passages, put them in the prompt, and answer from them. The 2020 paper that introduced retrieval-augmented generation combined a model's built-in memory with an outside document index, and noted that the index can be replaced to update what the model knows. Strong candidates also raise access permissions and how the index stays current.
Question 14
What starts to go wrong when a conversation or document gets very long?
Why ask it: It tests whether they understand context limits.
Weak (1): Has not thought about it.
Strong (4): Knows the limit is counted in tokens and has a plan for what to keep, summarize or retrieve. Google Cloud's glossary defines a context window as the number of tokens a model can process in a given prompt. Strong candidates also mention cost and speed.
Question 15
How would you build a test set for this feature before you ship it?
Why ask it: Evaluation is engineering work, and many candidates skip it.
Weak (1): Plans to try a few prompts by hand.
Strong (4): Collects cases from real usage, includes edge cases, defines what passing means, and automates grading where possible. Anthropic's guidance says evaluations should mirror the real task distribution and factor in edge cases.
Scorecard to copy
Fill this in right after the interview. One score per trait, with the evidence that earned it.
Trait
Questions
Score (1 to 4)
Evidence to write down
Systems-builder mindset
1 to 5
The system, the metric and baseline, one failure, the business result
Curiosity
6 to 9
What they learned, what they dropped, the questions they asked you
How AI works
10 to 15
Which ideas they explained in plain words: tokens, context, retrieval, evaluation
Candidate:
Role:
Interviewer:
Date:
Systems-builder mindset (Q1 to Q5) Score: _ / 4
Evidence:
Curiosity (Q6 to Q9) Score: _ / 4
Evidence:
How AI works (Q10 to Q15) Score: _ / 4
Evidence:
Strongest signal:
Biggest concern:
Recommend: yes / no
The rule I use: hire people who score at least 3 on all three traits, and 4 on the trait the role depends on most. For the reasoning behind each trait, see the three things to screen for in the main hiring guide.
Red flags across all 15 questions
Describes tools instead of outcomes.
Cannot say how success was measured.
Credits every result to the model.
Has demos, but nothing anyone used.
Repeats terms without explaining them in plain words.
Asks you no questions.
An interview shows how someone talks about work. To see the work itself, add a short paid work sample test with a small evaluation set. The scorecard section of the hiring guide explains how the two fit together.
Meet AI builders before you hire
You learn a lot by watching someone explain what they built. That happens every month at Building with AI Charlotte, a free, non-profit meetup at Hygge Coworking. Each meetup has two 25-minute builder pitches with a live demo. See the upcoming events, or RSVP for the December 2 meetup. More hiring and builder guides are on our resources page.
What are good interview questions for an AI engineer?
Good questions are open and tied to real work. Ask the candidate to walk through a system they built, how they measured it, what failed after launch, and what happens between a prompt and a reply. Avoid questions that only test definitions.
How do you score an AI engineer interview?
Use the same questions for every candidate and score each trait from 1 to 4 right after the interview. Write the evidence next to each score. Compare notes with other interviewers only after everyone has scored alone.
What is a leading question in an AI interview?
A leading question opens a topic without naming the answer. Asking why the same prompt can give two different answers lets a candidate show they understand sampling. Asking them to define temperature only tests recall.
How many questions should you ask in one interview?
Plan five or six questions for a 45-minute interview. That leaves room for follow-ups, which is where most of the signal comes from. Split the 15 questions across two or three interviewers so each trait gets covered once.
Where can I meet AI engineers in Charlotte?
Building with AI Charlotte is a free monthly meetup at Hygge Coworking, 933 Louise Ave, where local builders pitch and demo what they made. The next meetup is Wednesday, December 2, 2026. See the events page.