Hiring for AI Fluency: 9 Questions That Test Judgement, Not Prompting

Hiring for AI fluency has almost nothing to do with prompt writing. Prompt engineer did not appear in the top 10 roles leaders said they planned to hire in Microsoft's 2025 survey of 31,000 knowledge workers. The scarce skill is judgement, and the evidence on that is blunt. In three preregistered experiments run at Wharton, participants adopted a wrong AI answer 80% of the time. Below are 9 interview questions that test that skill, with what a weak answer sounds like.

The short answer

AI fluency is the ability to decide what to hand to a model, verify what comes back, and own the result. Test it with structured behavioural questions about real past tasks, scored against written anchors. Do not test prompt syntax. Prompting is teachable in a week, and the model increasingly writes the prompt for you.

What "AI fluency" actually means when you are the one hiring

Most job descriptions use the phrase without defining it, which is why the interviews that follow it test nothing. The most usable definition available comes from the AI Fluency framework built by Rick Dakan and Joseph Feller and published as an Anthropic course in 2025. It breaks the capability into four competencies rather than one.

Only one of those four is prompting. Delegation is deciding what work goes to a model and what stays with a person. Description is communicating the task clearly, which is the part everyone calls prompting. Discernment is evaluating what comes back. Diligence is owning the result, including disclosing how it was made.

Three of those four are judgement. One is craft. The industry spent 2023 and 2024 hiring for the craft, and I think that was a category error rather than a reasonable bet that aged badly.

A title survives when the work behind it is a full-time job for one person, and prompting failed that test almost immediately. It became a component of product management, support, marketing, analysis and engineering, in the same way that writing a good search query never became a job.

Microsoft's 2025 Work Trend Index asked leaders which AI-specific roles they were considering. The top answers were AI trainer, AI data specialist, AI security specialist and AI agent specialist, each at roughly 30%. Prompt engineer did not make the list. The clearest signal in that survey sits elsewhere: upskilling existing staff ranked as the leading workforce strategy at 47%, ahead of any hiring plan.

The title faded while the skill spread everywhere

Those two facts get treated as contradictory in most coverage. They are not. A skill becoming universal is exactly what causes its dedicated title to disappear.

Titles multiplied. One title did not.

Indeed Hiring Lab counted 264 distinct AI-touched job titles in the US in 2022, equal to 2.6% of all titles with meaningful posting volume. The count dropped to 159 in 2023, then reached 822 by the first quarter of 2026, or 8.3% of all titles.

The composition matters more than the count. In the US, 63% of those AI-touched titles now sit outside technology occupations, in healthcare, education, marketing, logistics and management. This is not a specialist labour market any more. It is a general one.

The premium attaches to the skill, not to the title. Lightcast analysed more than 1.3 billion job postings and found that postings listing AI skills advertised salaries 28% higher, worth about $18,000 a year. In its data, 51% of AI-skill postings sat outside IT and computer science. That premium is available to a marketing manager who uses the tools well.

Indeed's own tracking shows the same pattern at posting level. AI mentions reached 4.2% of all US job postings by December 2025, and AI-related postings ran 134% above their February 2020 baseline while total postings sat only 6% above it.

Judgement is the scarce half, and the research says so

If prompting is teachable and universal, the interview has to test something else. Three separate lines of evidence point at the same capability, which is telling a good output apart from a plausible one.

People follow wrong answers 80% of the time

Steven Shaw and Gideon Nave at Wharton ran three preregistered experiments with 1,372 participants across 9,593 trials, using an adapted Cognitive Reflection Test. When the AI was correct, accuracy rose 25 percentage points above baseline. When it was wrong, accuracy fell 15 points below the score of participants who had no AI access at all.

Participants followed a wrong answer 80% of the time and reported confidence 11.7% higher than the group working alone. The authors call this cognitive surrender. It is a working paper rather than a peer reviewed result, and it uses reasoning puzzles rather than workplace tasks, so treat the exact figures as directional.

Confidence in the tool predicts less thinking

A Microsoft Research and Carnegie Mellon team surveyed 319 knowledge workers about 936 real generative AI use cases and published the result at CHI 2025. Higher confidence in the tool was associated with less critical thinking. Higher self-confidence in the task itself was associated with more of it.

That is a hiring signal you can act on. You want a candidate whose confidence sits in their own domain knowledge rather than in the model. The same paper found that workers described critical thinking in AI use as goal setting, prompt refinement and verifying outputs against external sources. Those are observable behaviours, which means they are interviewable.

Experts misjudge their own speed

METR ran a randomised controlled trial with 16 experienced open-source developers across 246 real tasks. The developers took 19% longer with AI tools enabled. They had predicted a 24% speedup beforehand, and after finishing they still believed AI had made them 20% faster.

METR is careful about what this does not show, and so am I. It covers 16 developers on mature open-source repositories using early-2025 tooling. It does not prove that AI slows down most work. What it does establish is that self-reported productivity is unreliable evidence, and self-reported productivity is exactly what a candidate offers you in an interview. The same measurement problem shows up in deployments that returned nothing on paper.

The 9 questions, grouped by what they test

These are behavioural questions about work the candidate has already done. Ask for a specific instance with a date and an outcome attached. If the answer stays hypothetical after one follow-up, score it low and move on.

Notice what is missing. There is no question about prompt structure, no request to write a system prompt on a whiteboard, and no trivia about context windows. That omission is deliberate, because those are the parts a competent hire picks up in their first fortnight.

Delegation: does the candidate know when not to use it

Question 1. Tell me about a task in the last month where you deliberately did not use AI. What made it a bad fit? A strong answer names a property of the task, such as an unrecoverable error, a legal exposure, or a case where the candidate needed to hold the reasoning themselves. A weak answer says they use it for everything now.

Question 2. Walk me through the last thing you automated with a model. What was the manual version costing you, in hours or in errors? Strong answers have a before number, even a rough one. Weak answers describe the tool rather than the saving.

Question 3. Where in your current workflow would a model error be most expensive, and what stops it there? The best answers describe a check that already exists, which is the practical version of deciding where a human belongs in the architecture.

Discernment: does the candidate actually verify

Question 4. A model gives you a figure with a confident-looking citation. What are the next three things you do? Listen for opening the source rather than searching until something agrees. Candidates who describe asking the model to check its own work have told you something important.

Question 5. Describe a time the model was wrong and you caught it. How did you catch it? The useful detail is the mechanism, whether that was domain knowledge, a cross-check, a test that failed, or a colleague who queried it.

Question 6. How do you check a long output you are not qualified to review line by line? Strong answers spot-check the parts that would fail loudest, or narrow the task until it becomes checkable. Weak answers accept the output because it looked internally consistent.

Diligence: does the candidate own the result

Question 7. What do you tell a client or a colleague about the model's role in work you hand over? There is no single correct policy here. There is a clear difference between a candidate who has thought about disclosure and one who has never once considered it.

Question 8. You are past deadline and cannot verify the output. What do you ship? The answer I want is that they ship it flagged, with the unverified part named explicitly. Candidates who say they would always verify are either bluffing or have never worked to a real deadline.

Question 9. Something you produced with AI turned out to be wrong in front of a customer. Talk me through the hour afterwards. This tests accountability, and the tell is the pronoun. Candidates who say the model got it wrong have not accepted authorship of their own work.

Score it, do not sense it

The questions are worth little on their own. What makes structured interviewing outperform everything else is the scoring, and scoring is the part most teams skip.

A 2022 re-analysis of the personnel selection literature by Sackett, Zhang, Berry and Lievens corrected a long-standing statistical overcorrection and reordered the field. In the revised estimates, structured interviews came out highest at r = .42, ahead of job knowledge tests at .40, work sample tests at .33 and cognitive ability tests at .31.

Structure means the same questions in the same order, with written score anchors agreed before anyone interviews. Write down what a 1, a 3 and a 5 sound like for each of the three groups above. Interviewers then score independently and compare afterwards, because discussing first collapses three opinions into one.

The 45-minute work sample that beats any question

If you can only run one exercise, run this one. Give the candidate a model output that contains two planted errors, one factual and one logical, alongside the source documents. Give them 30 minutes, full AI access and no instruction beyond "tell us what you would change before this goes to a client".

Score three things. Did they find both errors, did they say which parts they did not check, and did they use the model to challenge the output rather than to confirm it. This is a review task rather than a production task, which is the right shape, because review is where the bottleneck has moved. Teams building internal evaluation suites are solving the same problem one level up.

Where this argument is weakest

Three honest problems, and I do not have a clean answer to the third.

Most of the evidence is lab work and self-report. The Wharton result uses reasoning puzzles, is a preprint, and has not been replicated in a workplace. The Microsoft and Carnegie Mellon study is self-reported, which measures what workers believe about their own thinking rather than the thinking itself. METR covers 16 developers on a narrow class of task.

None of that makes the direction wrong, and all three point the same way. It does mean that anyone quoting the 80% figure as a settled fact about your team is overreaching. The right use of these studies is to justify testing for verification behaviour, not to predict how often your particular hires will fail.

Candidates now have AI in the room. Behavioural questions leak, and within a quarter of a list like this being published, candidates will have rehearsed answers written by a model. That is a real cost of writing it down. The vendor data on interview cheating rates comes from companies selling detection and varies wildly, so I would not quote any of it.

Two defences hold up. Follow-up questions about specifics are hard to fake, because a prepared answer has no second layer underneath it. And the work sample above is close to un-gameable, since using AI during it is encouraged and the thing being scored is what the candidate does with what the model returns.

The third problem has no good answer. Nobody has published a measurement of verification behaviour inside a real workplace, so the link between interview score and on-the-job judgement is assumed rather than demonstrated. Structured interviewing gives you validity for job performance in general. It does not yet give you validated evidence for this specific competency, and anyone claiming otherwise is extrapolating.

Rewriting the job description

The description determines who applies, so fixing the interview without fixing the posting wastes most of the gain. The words you choose have a price attached to them.

One title genuinely does describe a new job, and it is worth separating from the noise. Running agents in production is a real operational discipline with its own on-call surface, which is covered in the piece on what an agent ops engineer is hired to do. Most other AI titles in 2026 are existing jobs with a new adjective in front of them.

Be careful at the entry level too. More than a third of AI-touched titles now sit in non-technical fields, and junior roles are where verification skill is least developed and most needed. The structural version of that problem is set out in the analysis of how the junior pipeline is breaking.

Frequently asked questions

Where to start this week

Pick one open role and rewrite two lines of its description using the replacement column above. That takes 20 minutes and changes the applicant pool before it changes anything else.

Then build the review exercise. Take a real model output your team produced last month, plant two errors in it, and run it past a current employee first to calibrate what a good score looks like. You will learn as much about your existing team as you do about candidates. If your team's answers surprise you, the relevant gap is described in the piece on the distance between AI usage and AI trust.

Related reading