From Shubhi K | Product & Market Analysis
Technical Interviews When Everyone Ships With AI: What Separates Candidates
On this page
Anthropic rebuilt the same take-home test three times in two years, because each new Claude model solved the previous version. That is the clearest published account of what has happened to technical interviews. The artifact a candidate submits is no longer evidence that the candidate can produce it. What still separates people is live reasoning under constraints that change while they work.
Key takeaways
- The take-home has stopped working as a filter. On Anthropic's performance engineering take-home, more than 50% of candidates would have scored better by handing the whole task to Claude 3.7 Sonnet, according to the team that ran it.
- Assessment difficulty is now the majority view among engineering leaders. 71% of 400 engineering leaders surveyed by Karat say AI has made technical skills harder to assess, up from a reported 20% to 30% a few years earlier.
- Candidates want the harder format, which nobody expected. Gartner found 62% of 3,000 candidates were more likely to apply when a role required in-person interviews. Verification reads as seriousness, not as friction.
- The format that survived is the one the research already favoured. Sackett and colleagues put structured interviews at r = 0.42 against work samples at 0.33, so the live structured conversation was the stronger predictor before AI touched any of this.
The short answer. Take-home tests and portfolios no longer separate candidates, because a model produces the same artifact in minutes. Signal has moved to formats where the work happens in front of you: framing an underspecified problem, debugging code the candidate did not write, and defending choices when a requirement changes mid-session. Structure the loop around those three, and let candidates use AI while you watch.
The artifact stopped being evidence
Hiring has always used a substitution. You cannot watch someone do the job for six months, so you look at something they made and infer the person behind it. A portfolio, a take-home, a GitHub profile, a case study.
That substitution held for as long as the artifact was expensive to produce. It is not expensive any more. The output is now cheap for everyone, including people who could not have produced it themselves 18 months ago.
This is not primarily a cheating story. A candidate who uses a model on a take-home is doing exactly what they would do at work. The problem is narrower and worse: the artifact no longer distinguishes between two candidates, so it has stopped being an assessment at all.
The take-home died first
Take-homes were always the most vulnerable format, because they are unsupervised, time-boxed, and usually built from a problem class the model has seen many times. interviewing.io, which runs anonymous mock interviews at volume, describes the algorithmic take-home as being in fast decline, replaced by project-based work at startups.
Karat's survey still found 45% of United States employers using take-home projects. That gap between what is defensible and what is deployed is the real finding. Most hiring loops are running a filter their owners already suspect does not work.
Volume broke the top of the funnel at the same time. The tools that hollowed out the take-home also multiplied the number of applications each role receives, and screening capacity did not multiply with them.
Gartner's candidate research adds a second problem underneath the volume. In a survey of 3,000 candidates, 6% admitted to interview fraud, meaning they posed as someone else or had someone pose as them. Gartner projects that by 2028 one in four candidate profiles worldwide could be fake.
Treat that 2028 figure as a forecast from an analyst firm rather than a measurement, because that is what it is. The 6% is the number to act on, and it is a self-reported floor.
What interviewers actually report
Two surveys give the clearest picture available, and both come from companies that sell into hiring. Read them with that in mind.
interviewing.io surveyed 67 interviewers, mostly at large technology companies. 81% suspected a candidate of using AI to cheat. About a third had actually caught someone. 75% believed AI assistance was letting weaker candidates through.
Karat surveyed 400 engineering leaders across the United States, India and China. 71% said AI has made technical skills harder to assess. 62% of their organisations still prohibit AI use in technical interviews, while those same leaders estimate that more than half of candidates use it anyway.
That last pair of numbers is the most damning thing in either dataset. A rule that most people break, and that you know most people break, is not a policy. It is a fiction that quietly advantages whoever is most comfortable ignoring it.
The split nobody expected
The interesting result in the interviewing.io data is not that things changed. It is who changed. None of the 52 respondents at large technology companies had moved away from algorithmic questions. 58% had changed the kind of algorithmic question they ask, using unusual phrasing and problems requiring two or more data structures.
Startups went the opposite way. 67% of startup respondents said AI had meaningfully changed their process, including dropping algorithmic questions and take-homes entirely in favour of AI-assisted formats.
So the market has not converged. Big companies are making the same test harder. Small companies are changing the test. My view is that the first group is buying time and knows it, because every reformulation has a shelf life set by the next model release.
Three redesigns in two years, published in full
The best available case study comes from Anthropic's performance engineering team, and it is unusual because they published the failure rather than the success.
The team built a take-home in 2024: optimise a parallel tree traversal on a simulated accelerator, with memory management, instruction packing and vectorisation. Roughly 1,000 candidates completed it. Dozens were hired, including engineers who went on to build Trainium clusters. The best human submission came in at 1,363 cycles, better than any model attempt at launch.
Then the models improved. By Claude 3.7 Sonnet, more than half of candidates would have done better delegating the task outright. Team lead Tristan Hume told TechCrunch that each new Claude model has forced a redesign of the test. Claude Opus 4.5 eventually matched the top two-hour human results at roughly 1,790 cycles.
What each revision changed
Version two kept the shape of the problem but added machine features the models had not seen, rewarding what the team called clever optimisation insight rather than known technique.
Version three abandoned resemblance to the job. It borrowed from puzzle games: a tiny, heavily constrained instruction set, scored on minimum instruction count. The point was to find a problem space with thin training coverage.
The design principles the team names are worth copying directly. Prefer problems that sit outside the model's training distribution. Include judgement calls about approach and tooling, not just execution. Use structural constraints that force a workaround. Extend the time horizon, because bounded windows of two to four hours favour the model, while long-horizon work still favours the human.
The tradeoff Anthropic named out loud
Anthropic's own conclusion is the part most hiring teams will skip, so here it is. Version three is less like the job than version one was. The team wrote that realism may be a luxury they no longer have.
That is an honest and uncomfortable finding. Every AI-resistant test drifts away from the work it is meant to predict. You are trading validity for verifiability, and the trade gets worse each release cycle. Anyone selling you an assessment that is both fully AI-proof and fully job-realistic is describing something that does not currently exist.
Prohibition and permission are both real strategies
Employers have split into two camps, and the split is not about sophistication. It is about what each company is trying to verify.
The prohibition camp wants to confirm identity and unassisted fundamentals. Google, Cisco and McKinsey have all reintroduced in-person rounds, and Cisco's talent lead has said publicly that AI should be treated as off-limits unless a candidate is explicitly invited to use it. Gartner reported that 72.4% of recruiting leaders were conducting interviews in person to counter fraud.
The permission camp wants to confirm judgement while using the tools. Canva replaced its computer science fundamentals screen with a competency it calls AI-Assisted Coding in June 2025, on the reasoning that nearly half its frontend and backend engineers already use an AI coding tool daily. Rippling, Red Hat, Meta and Shopify have moved in similar directions.
| Dimension | Prohibition, plus in-person verification | Permission, with AI use observed |
|---|---|---|
| What it verifies | Identity, unassisted fundamentals, composure | Delegation judgement, review of generated code |
| What it costs | Travel, scheduling, smaller applicant pool per week | Rubric rewrite, interviewer retraining, harder scoring |
| Shelf life | Short. Each reformulated question ages with the next model. | Longer. Judgement is the thing being measured. |
| Failure mode | Tests a version of the job nobody does any more | Rewards tool familiarity over engineering ability |
| Where prohibition genuinely wins | Fraud and identity. Nothing else verifies a person is who they say they are as cheaply as being in a room. | No equivalent. Remote permission formats cannot settle identity. |
The last row matters most. Most arguments for in-person interviewing are framed as being about skill assessment, where the case is weak. The case on identity verification is strong, and it is a different argument.
What still carries signal in a technical interview
Strip away the format debate and one question remains. What can you observe in a live session that a model cannot supply on the candidate's behalf?
Three things, consistently. All of them are about the work before and around the code, not the code itself.
Framing an underspecified problem
Give a brief that is deliberately incomplete. No stated scale, no stated latency budget, no stated failure tolerance. Then watch which questions get asked.
Strong candidates ask the two or three questions that change the design, and they say why each one changes it. Weak candidates either start building or ask for a complete specification. A model will happily produce a plausible design for an underspecified brief, which is precisely the behaviour you are trying to detect in a person.
This is the highest-value 30 minutes in any loop I have seen run, and it needs no code at all.
Debugging code the candidate did not write
Hand over a small repository with a real failure in it. Not a puzzle, an ordinary bug: a race condition, an off-by-one in pagination, a cache that returns stale data under a specific sequence.
The signal is the search strategy. Does the candidate form a hypothesis before touching anything? Do they narrow the space, or change three things at once? When the model suggests a fix, do they check whether it addresses the cause or the symptom? This is the daily reality of working alongside generated code, and it is also the review bottleneck that AI-assisted teams keep hitting.
Let them use whatever tools they use at work. Ask them to narrate. The narration is the assessment.
Changing a constraint halfway through
The single cheapest upgrade to any existing interview is a mid-session constraint reversal. Fifteen minutes before the end, change one requirement. The data now arrives out of order. The third-party API has a 200 millisecond budget. The customer needs this to work offline.
Rehearsal does not survive this. Neither does a prepared answer fed through an overlay, because the new constraint has to be reconciled with decisions the candidate made in the previous 45 minutes. You are watching whether the plan was understood or recited.
This is the closest thing to a free lunch in interview design. It costs one sentence and it discriminates more than the hour that preceded it.
A five-stage loop you can run next week
Here is a concrete structure. It assumes AI is available at every stage except identity verification, and it is designed so that permission costs you nothing.
| Stage | Time | What it measures | What defeats it, and the fix |
|---|---|---|---|
| Framing call on an underspecified brief | 30 min | Which questions they ask, and why those questions | Rehearsal. Fix: change one fact about the brief mid-call. |
| Debug an unfamiliar failing repository | 45 min | Hypothesis formation, narrowing, cause versus symptom | A leaked repository. Fix: rotate the bug quarterly. |
| Build with tools allowed and screen shared | 60 min | Delegation judgement, review of generated code | Little. Observation is the control. |
| Constraint reversal inside the build | 15 min | Whether the earlier plan was understood or recited | Little. This is the highest-signal segment. |
| One-page written decision memo, sent after | 20 min | Whether they can defend choices they actually made | Model drafting. Fix: grade only specifics from their session. |
Two scoring notes, because a rubric written for the old loop will quietly sink this one.
First, score process separately from outcome. A candidate who reaches a worse answer through sound reasoning is a better hire than one who reaches a good answer they cannot explain. That inversion feels wrong and it is correct.
Second, define what good AI use looks like before you run a single interview. Canva's published experience is that candidates without AI experience struggled on judgement rather than on coding, failing to spot when a suggestion was poor. If your interviewers cannot agree in advance on what a bad delegation decision looks like, you are not measuring anything. The same discipline applies to the specific questions that test AI fluency and to the performance criteria you apply once the person is hired.
Where this argument is weakest
Four objections, and the two that land.
Where the live format punishes the wrong people
Live, observed assessment favours people who perform well under supervision. That is not the same population as people who do the job well. Take-homes existed partly because they gave quieter candidates, carers, and people in other time zones a fair route through.
Moving to live formats and in-person rounds takes that route away. Gartner's 62% preference figure is an average, and averages conceal the people for whom the answer is the opposite. If you make this shift, you owe those candidates an alternative path, and you should expect to argue about it internally.
The second objection is the vendor incentive running through this data. Both survey datasets in this post come from Karat and interviewing.io. Both sell interview services. Both benefit commercially from the conclusion that interviewing is hard and needs redesign.
Neither sample is large. 67 interviewers and 400 leaders are useful signals, not measurements of a market. The strongest evidence here is Anthropic's, and that is because it reports a measured outcome on roughly 1,000 submissions against a named model rather than an opinion survey. Weight it accordingly, and treat every percentage in this post as directional.
The two objections that do not land: that this is a temporary problem models will stop causing, and that good interviewers could always tell. The first is contradicted by three consecutive redesigns in two years. The second is contradicted by 75% of experienced interviewers saying weaker candidates are now getting through.
If you are the candidate side of this market
The same shift changes what you should prepare, and most preparation advice has not caught up.
Practising problems you can already solve is now close to worthless, because the interviewer has stopped scoring the solution. Practise narrating instead. Talk through a design out loud, including the two questions you would ask before starting and the reason each one matters.
Prepare for the constraint reversal specifically. Take a project you built, change one requirement, and work out what breaks. That is the exercise most interviews will end on, and it is the one nobody rehearses.
On tool use, follow the instruction you are given and ask when it is unclear. Anthropic's published candidate guidance is a reasonable model of where policy is heading: draft your own materials first, then use AI to refine them, and do not use it in assessments unless invited. The market is moving from prohibition toward permission, unevenly, and asking is free.
One thing worth knowing about the market you are entering: entry-level routes are being reshaped at the same time, which is covered separately in the piece on what is happening to the junior developer pipeline.
Frequently asked questions
Are take-home tests still worth using in 2026?
Only as a signal about interest, not ability. Anthropic's performance engineering team found that with Claude 3.7 Sonnet, more than half of candidates would have scored better delegating the whole task. Karat's survey still found 45% of United States employers using take-home projects, which means most loops are running a filter that no longer separates people. If you keep one, keep it short and unscored.
Should candidates be allowed to use AI in technical interviews?
Yes for anything measuring how they work, no for anything measuring identity. Karat found 62% of organisations still prohibit AI while estimating that over half of candidates use it anyway, which makes the ban unenforceable and unfair to the people who follow it. Canva replaced its fundamentals screen with an AI-assisted coding round in June 2025 and now expects tool use.
How do you stop AI cheating in a remote technical interview?
You mostly do not, so change what you are measuring. Overlay tools and second devices defeat screen sharing. The practical controls are structural: change a constraint mid-session, ask why rather than how, require the candidate to explain code they did not write, and verify identity in one in-person or supervised round. Gartner reported 72.4% of recruiting leaders now use in-person interviews against fraud.
What should you ask instead of LeetCode questions?
Ask for problem framing and debugging. Give an underspecified brief and score which clarifying questions get asked. Hand over a small repository with a real bug and watch the search strategy. Then change one requirement fifteen minutes before the end. Large employers have mostly kept algorithmic questions and made them harder, but that approach ages with each model release.
Do in-person interviews put candidates off?
The available evidence says the opposite. Gartner surveyed 3,000 candidates in the second quarter of 2025 and found 62% were more likely to apply when a role required in-person interviews. The likely reason is that half of candidates in the same research doubted the jobs they applied to were even real. Requiring presence signals a genuine role and a genuine process.
Can you interview for AI fluency directly?
Partly. Canva reported that candidates without AI experience struggled on judgement rather than coding, failing to recognise when a suggestion was poor. That is observable: give a task, allow tools, and score whether the candidate catches the flawed output. What you cannot assess this way is depth of unassisted fundamentals, which needs a separate round if it matters for the role.
Where to start this week
Pick one interview you already run and add a constraint reversal to it. One sentence, fifteen minutes before the end, changing a requirement the candidate has already designed around. Run it with the next five candidates and compare the spread of scores against your existing stages.
Then take your current take-home and give it to the best model you have access to, under the same time limit you give candidates. Score the output on your real rubric. If it passes, you have your answer about that stage, and you did not need a survey to get it.
Related on hiring and AI-assisted engineering
The interview is one surface. The others are the collapsing junior pipeline, the gap between developer usage and developer trust, and the three incompatible scoreboards used to judge coding tools.
References
- Anthropic, Designing AI-resistant technical evaluations, 21 January 2026. Used for the three take-home revisions, the 1,000 completed submissions, the 50% delegation finding, cycle counts and the design principles.
- TechCrunch, Anthropic has to keep revising its technical interview test, 22 January 2026. Used for the Tristan Hume quotes.
- Canva Engineering, Yes, you can use AI in our interviews, 11 June 2025. Used for the AI-Assisted Coding competency, the daily-use figure and the judgement finding.
- interviewing.io, How is AI changing interview processes, 8 October 2025. Survey of 67 interviewers. Used for the 58%, 81%, 75% and startup split figures.
- Karat, Engineering interviews in 2026, 7 January 2026. Survey of 400 engineering leaders in the US, India and China. Used for the 71%, 62% prohibition, and format usage figures.
- Gartner, Just 26% of job applicants trust AI will fairly evaluate them, 31 July 2025. Survey of 3,000 candidates, 2Q25. Used for the 62% in-person preference, the 6% fraud admission and the 2028 projection.
- Sackett, Zhang, Berry and Lievens, Revisiting meta-analytic estimates of validity in personnel selection, Journal of Applied Psychology, 2022. Used for the validity coefficients.
- Computerworld, To counter AI cheating, companies bring back in-person job interviews, 26 August 2025. Used for the Google, Cisco and McKinsey moves and the 72.4% Gartner recruiting-leader figure.
The weakest thing about this source base: two of the three survey datasets come from companies that sell interview products, and both samples are small. The Anthropic account is the only source here reporting a measured outcome against a named model rather than stated opinion.
Related reading