A good AI engineer work sample test is a short, paid task that mirrors the job. Give the candidate made-up data, two hours and their normal AI tools. Ask for a small working system, an evaluation set and a one-page memo. Score it on a fixed rubric, then spend 30 minutes asking how they made each choice.
What a work sample test is
The U.S. Office of Personnel Management defines it this way: work sample tests require applicants to perform tasks or work activities that mirror the tasks employees perform on the job. A take-home assignment, a live build session and a paid trial day are all versions of the same idea.
OPM lists three things in their favor. The tasks are very representative of the job. Applicants often see them as very fair. And generally little or no performance difference is found between men and women or applicants of different races, though that depends on the skills being tested. OPM also names the cost: work samples may be costly to develop in time and money, and they need updating when the work changes.
Why it matters for AI roles
An interview shows how someone talks about work. A work sample shows the work. For AI roles the gap between the two can be wide, because the vocabulary is easy to learn and the judgment is not.
Use both. A 2022 study in the Journal of Applied Psychology by Paul Sackett and colleagues revised the long-used estimates of how well hiring methods predict job performance. As summarized by Criteria Corp, the revised figures put structured interviews at 0.42 and work sample tests at 0.33, with job knowledge tests at 0.40 and cognitive ability tests at 0.31. The older 1998 estimate for work samples was 0.54. So a work sample is useful evidence, but it does not replace a structured interview. Our AI engineer interview questions cover that half.
The work sample is where you see the three traits from our guide to hiring AI talent in Charlotte in action: a systems-builder mindset, curiosity, and an understanding of how AI systems are built.
How AI changed take-homes
Any task a coding agent can finish alone no longer tells you much about the person. Anthropic described this problem with its own hiring test in January 2026. Tristan Hume, who leads its performance optimization team, wrote that more than 1,000 candidates had completed the team's take-home since early 2024. The time limit started at four hours and was later cut to two. Then the company's own models caught up. Hume wrote that Claude Opus 4.5 matched the best human result within two hours, so the team could no longer tell top candidates from the model under the test's constraints.
Three points from that write-up apply to any team:
- Keep AI allowed. Anthropic explicitly permits AI assistance on that take-home, because candidates should show their skills alongside the tools they would use on the job.
- Test judgment, not typing. Hume wrote that the original test worked because it resembled real work, and the replacement works because it simulates novel work.
- Hold on to the basics. His principles for a good test: representative of real work, high-signal, free of narrow domain knowledge, and fun.
Most companies are not hiring performance engineers, and you do not need a puzzle. You need a task where the hard part is deciding what good looks like. That is why this test centers on an evaluation set.
Six design rules
- Mirror the job. Pick a small version of something your team does every week. If the role builds support tools, the task is a support tool.
- Cap it at two hours. A short cap respects the candidate's time and keeps people with jobs and families in your pipeline.
- Pay for it. A flat fee tells candidates you value the time. Say the amount in the brief.
- Use made-up data. Never hand out customer data, and never ask for work you plan to ship. The task is a test, not free labor.
- Allow their normal tools. Ask them to list what they used and where they overrode it.
- Fix the rubric before you send it. Write the scoring rules first and give every candidate the same brief.
The brief to copy
This version suits a company that wants an AI engineer who can ship internal tools. The business is made up. Swap in a scenario close to your own work and write the files yourself, or have a teammate do it.
A common take-home in applied AI hiring has the same shape. One 2026 interview guide gives this example: here are 50 markdown docs, build a Q&A system and an eval set of 20 questions, and report the results. The brief below is a smaller version with a memo added.
WORK SAMPLE: SUPPORT ANSWER DRAFTER
Time: 2 hours of work, at your own pace within 5 days.
Pay: $[amount], paid whether or not we move forward.
Tools: Use any AI tools, models and libraries you like.
THE SCENARIO
Lakeside Outfitters is a made-up online store.
You get two folders:
/docs 12 short help articles (returns, shipping, sizing)
/tickets 30 real-looking customer emails
THE TASK
1. Build a small tool that drafts a reply to a ticket
using the help articles.
2. Build an evaluation set of at least 15 cases.
Decide what a good reply is and how you will grade it.
Include cases where the right move is to not answer.
3. Run your tool against your evaluation set
and report the results.
WHAT TO SEND BACK
- Your code and how to run it
- Your evaluation set and results
- A one-page memo:
What you built and why
What it gets wrong today
What you would do with one more week
Which AI tools you used, and one place
you disagreed with them
WHAT WE SCORE
The evaluation set and the memo count for more
than the code. We do not score polish.
NEXT STEP
A 30-minute call where you walk us through it.
Seed the tickets with a few traps: a question the docs do not answer, two docs that disagree, an angry customer, and a request the store should refuse. Strong candidates find them. Their evaluation set shows whether they looked.
Scoring rubric
Score each row from 1 to 4, alone, before you talk to other reviewers. The rows map to the three traits. The rubric and the pass rule are my own framework.
| What you score | Trait | Weak (1) | Strong (4) |
|---|---|---|---|
| The system | Systems builder | Does not run, or one giant prompt with no structure | Runs from the instructions, simple design, handles the case where no doc applies |
| The evaluation set | How AI works | A few happy-path examples checked by eye | Cases drawn from the tickets, edge cases and refusals included, a clear pass rule |
| The grading method | How AI works | Trusts a model's score without checking it | Uses code checks where they fit and spot-checks any model grader against their own judgment |
| The memo | Systems builder | Describes the tools used | Names failures with numbers, ties next steps to what a support team needs |
| Use of AI tools | Curiosity | Cannot say what the tools did or where they were wrong | Explains what was delegated, what was checked and one disagreement |
The grading row follows Anthropic's own guidance on evaluations. It says to design evals that mirror the real task and factor in edge cases. It calls code-based grading the fastest and most reliable. For model-based grading it says to test for reliability first, then scale.
Candidate: Reviewer: Date: The system Score: _ / 4 The evaluation set Score: _ / 4 The grading method Score: _ / 4 The memo Score: _ / 4 Use of AI tools Score: _ / 4 Traps found (of 4): Strongest signal: Biggest concern: Move to debrief: yes / no
The rule I use: move forward if the evaluation set and the memo both score 3 or higher. Working code with a weak evaluation set is a no.
The 30-minute debrief
The debrief is where you learn who did the thinking. Anyone can submit work an agent wrote. Only the person who understood it can explain the choices. Ask leading questions, not quiz questions.
- Walk me through one case your tool gets wrong. Why does it fail?
- How did you pick these test cases? What did you leave out?
- Where did your AI tool give you something you threw away?
- If this went live Monday, what would you watch first?
- What would change if there were 5,000 help articles instead of 12?
- What did you want to ask us before you started?
The last question matters. Curious candidates usually had questions about the customer, the tone or what counts as a good reply. Some will have sent them before starting, which is a good sign.
Adjust it by level
| Level | What to change | What to weigh most |
|---|---|---|
| Junior | Give a working starter tool. Ask only for the evaluation set and memo. | Care in the test cases and honest reporting |
| Mid-level | Use the brief as written. | The evaluation set and the grading method |
| Senior | Add a cost and response-time target, and ask for a rollout plan in the memo. | Trade-offs and the link to a business result |
Hiring for marketing, sales, operations or support? Use the 30-minute work sample in our guide to testing AI fluency in non-engineering hires.
Mistakes to avoid
- Asking for unpaid real work. It costs you good candidates and your reputation.
- Letting it grow past two hours. Long tests filter for free time, not skill.
- Banning AI tools. You cannot enforce it, and it tests a job that no longer exists.
- Scoring without a rubric. Reviewers drift toward whoever writes code the way they do.
- Skipping the debrief. The submission alone cannot tell you who understood it.
- Using it too early. Send it after a first conversation, to people you would hire if it goes well.
See how builders work before you hire
A work sample is a small look at how someone builds. You can get a similar look every month at Building with AI Charlotte, a free, non-profit meetup at Hygge Coworking. Two builders pitch for 25 minutes each, with a live demo and questions from the room. See the upcoming events, or RSVP for the December 2 meetup. More hiring and builder guides are on our resources page.
Frequently asked questions
What is a work sample test for an AI engineer?
It is a short task that mirrors the job. The candidate builds a small AI tool on made-up data, writes an evaluation set for it and explains the results. You score the submission on a fixed rubric and follow up with a call.
How long should an AI engineer take-home be?
Cap it at about two hours of work and let candidates choose when to do it within a few days. Anthropic cut its own performance engineering take-home from four hours to two. Longer tests tend to filter for free time.
Should candidates be allowed to use AI tools on a take-home?
Yes. The job involves AI tools, so the test should too. Ask candidates to list what they used and one place they disagreed with it. Then check their understanding in a 30-minute debrief.
Should you pay candidates for a work sample test?
We recommend it. A flat fee respects the candidate's time and widens your pool. State the amount in the brief and pay it whether or not you move forward. Never ask for work you plan to ship.
What should you score in an AI take-home?
Score the evaluation set and the memo above the code. Look for test cases drawn from real inputs, edge cases, a clear pass rule and honest reporting of what fails. Clean code alone tells you little now that agents can write it.
Is a work sample better than an interview?
Use both. In the revised estimates from Sackett and colleagues, structured interviews scored 0.42 and work sample tests 0.33 for predicting job performance. They show different things: how someone explains work and how they do it.
Keep reading
Sources
- U.S. Office of Personnel Management: Work samples and simulations
- Anthropic Engineering: Designing AI-resistant technical evaluations (Tristan Hume, January 21, 2026)
- Criteria Corp: summary of the revised validity estimates from Sackett et al.
- Sackett, Zhang, Berry and Lievens (2022), Revisiting meta-analytic estimates of validity in personnel selection, Journal of Applied Psychology 107(11). doi.org/10.1037/apl0000994
- Anthropic: Define success criteria and build evaluations
- techinterview.org: What applied-AI engineer interviews test in 2026
Last checked November 11, 2026. The brief, rubric and pass rule are the author's own framework. This is not legal advice; check your local rules on paying candidates. Spot something out of date? Tell us.