Assessment Validity After Sackett 2022: What Changed and What Is Still Disputed

Cognitive ability tests lost 0.20 of their predictive validity in 2022, and not one new hiring study was run to cause it. Paul Sackett and three colleagues reanalysed the same evidence base and found the old corrections had been applied in the wrong places. The revised estimates of pre-employment assessment validity put the structured interview first, at 0.42. That reorders how you should build a hiring process, though less dramatically than the headlines suggested.

Which pre-employment assessments predict job performance best? On the 2022 revised estimates, structured interviews lead at 0.42, followed by job knowledge tests at 0.40, empirically keyed biodata at 0.38, work samples at 0.33, and cognitive ability and integrity tests at 0.31 each. Unstructured interviews and conscientiousness scales sit at 0.19. Combinations of predictors beat any single test.

What the 2022 validity revision actually changed

For a generation, the reference point for hiring science was a single table. Frank Schmidt and John Hunter's 1998 meta-analysis in Psychological Bulletin ranked selection methods by their correlation with job performance. Cognitive ability sat near the top at 0.51, and that number travelled into textbooks, vendor decks and interview training for 24 years.

A validity coefficient is a correlation between a test score and later job performance. A value of 0 means the test predicts nothing. A value of 1 would mean perfect prediction, which no hiring method gets near. Most published estimates are "operational validities", meaning the raw correlation has been statistically corrected for known distortions.

The 2022 paper, published in the Journal of Applied Psychology by Paul Sackett, Charlene Zhang, Christopher Berry and Filip Lievens, did not gather new hiring data. It went back to the meta-analyses behind the old table and asked whether one of those corrections had been applied correctly. Mostly, it had not.

The mistake was in the correction, not the studies

Range restriction is the core issue. If you only hire people who scored well on a test, your employees will show a narrower spread of scores than your applicants did. A narrow spread shrinks any correlation, so researchers correct upward to estimate what validity would be across the full applicant pool.

That correction is legitimate. The authors were explicit that they do not question the formulas. Their argument was that the formulas had systematically been applied in ways which are not appropriate, and that the error ran in one direction.

Why concurrent studies broke the old numbers

There are two ways to validate a test. A predictive study tests applicants, hires some of them, and measures performance later. A concurrent study gives the test to people already in the job and compares their scores with their current ratings.

In a concurrent study, nobody was hired on the test being validated, so restriction on that test is usually small. Yet the correction factors in the big meta-analyses came mostly from predictive studies, where restriction is large. The 2023 follow-up states it plainly: such factors overcorrect substantially if applied to concurrent studies. With roughly 80% of studies concurrent, the inflation reached almost every estimate.

My reading is that this is the most consequential methods correction the field has published in decades, and Sackett himself called it the most important paper of his career. The revisers did not prove that tests stopped working. They proved that the numbers everyone quoted had been inflated by a bookkeeping habit.

Pre-employment assessment validity, old and new, side by side

The table below sets the 1998 estimates against the revised 2022 values for the eleven predictors where the authors had enough data to recalculate. The SD column is the standard deviation of true validity across settings, which matters more than most readers assume.

Values as tabulated by Master International, March 2022, from the Journal of Applied Psychology paper. We checked every change value against the two columns. The headline values match the HumRRO and SIOP summaries and the authors' 2023 article. Years of education, reference checks and several other methods could not be recalculated and are absent, not zero.

The biggest falls hit the most popular shortcuts

Work samples lost 0.21, cognitive ability 0.20 and unstructured interviews 0.19. Two of those three were the methods most often sold as the scientific answer to hiring. The third, the unstructured chat, is what most of your hiring managers default to when no process exists.

The experience figure deserves its own sentence. At 0.07, years in a similar role predicts almost nothing about how someone will perform. Yet it remains one of the most common filters in job descriptions. The gap between what predicts and what gets filtered is covered from the labour market side in the analysis of conflicting entry-level hiring data.

Validity varies by setting more than the averages admit

A mean validity is an average across hundreds of studies, jobs and organisations. The 2023 follow-up in Industrial and Organizational Psychology pushes this point hard, and it is the part of the revision that practitioners most often skip.

The structured interview's mean of 0.42 comes with an 80% credibility interval running from 0.18 to 0.66. The authors suggest reading it as "0.42, plus or minus 0.24". At the bottom of that range, a structured interview predicts no better than an unstructured one does on average.

Read the lower bound before the mean

The authors give a worked example that should change how you compare tools. The structured interview beats biodata on the mean, 0.42 against 0.38. But biodata's lower credibility value is 0.26, against 0.18 for the interview. If you care about the worst plausible case, biodata is the safer bet.

The SD column in the table is how you spot this. Interests carry an SD of 0.25 and integrity tests 0.20, so their real value in your setting could sit far from the average. Biodata, work samples and assessment centres all sit near 0.09, which means fewer surprises. I would rather run a predictor with a slightly lower mean and a tight spread than gamble on a high average I cannot verify locally.

The authors say this directly: their findings should not be viewed as a mandate for preferring one predictor over another regardless of circumstances. A ranking is a starting hypothesis about your hiring process, not a verdict on it.

Hiring assessment batteries matter more than the single best predictor

Almost no serious hiring process uses one method. The more useful question is what a combination of predictors achieves, and how much each one adds.

Across every composite of one to five predictors drawn from cognitive ability, conscientiousness, biodata, structured interviews and integrity tests, the mean validity was 0.51 on the old values and 0.47 on the revised ones. A drop of 0.04 is real but modest. Selection science still works when you combine methods, which is the finding vendors quoting the headline cut tend to leave out.

The validity-diversity trade-off shrank

Cognitive ability tests produce the largest mean score gap between Black and White applicants of any common predictor. Under the old numbers, dropping them carried a heavy validity penalty, so employers faced a hard trade between prediction and diversity.

The revision changes that sum. In a six-predictor composite that adds situational judgement tests to the five above, giving cognitive ability zero weight cut validity from 0.66 to 0.46 on the old estimates, but only from 0.61 to 0.56 on the new ones. Cognitive ability now carries only the fourth-highest weight in that composite.

A 2024 follow-up by Berry, Lievens, Zhang and Sackett in the Journal of Applied Psychology rebuilt the full predictor matrix and reached the same direction of travel. Its abstract states that cognitive tests are not a driving factor in the validity-diversity trade-off, while noting that some validity loss is still needed to reach an adverse impact target. A correction to that paper was published in 2025, so check the corrected version before quoting its specific weights.

Where this argument is weakest: the correction fight is not settled

Everything above rests on one methodological choice. If that choice is wrong, several of the revised numbers are too low.

The case for the old corrections

In 2023, In-Sue Oh, Huy Le and Philip Roth published a critique in the Journal of Applied Psychology. They challenged the recommendation against correcting for range restriction in concurrent studies, arguing that restriction there can be larger than the revisers allowed. Deniz Ones and Chockalingam Viswesvaran responded separately and also questioned the assumptions.

The debate continued into 2024. A focal article in Industrial and Organizational Psychology by Foster and colleagues took a position its title states outright: selection tests work better than we think they do, and have for years. Oh and Le added a commentary arguing that operational validity coefficients remain the right way to judge selection procedures.

What the revisers conceded, and what they did not

Sackett, Berry, Lievens and Zhang replied in the same 2023 volume. They endorsed correction whenever the data needed to do it exists. Their objection is to assuming a degree of restriction with no empirical basis. They argued that restriction is small in many real settings, so not correcting produces only a modest underestimate, and that the high-restriction scenarios in the critique are uncommon.

My position is that the revisers have the better argument on method, and the critics have a fair point on size. Some revised numbers are probably a little low. None of the published rebuttals I found argues for restoring the 1998 values intact, which tells you where the floor of the debate now sits.

There is also a pull in the other direction. A 2022 conference poster by Griebe and colleagues looked at 113 cognitive ability studies from this century and found a corrected validity of only 0.23, which would drop cognitive ability from 5th to 12th among the predictors. That is a poster, not a peer-reviewed paper, so weigh it accordingly.

One limitation neither side resolves: most of these studies measure performance as a supervisor rating. A rating captures what a manager noticed, which is not the same as output. The SOP-level point is simple. A validity coefficient is only as good as the performance measure behind it, a problem examined from the other end in the piece on rewriting performance review criteria for AI-assisted work.

Skills assessment validity when AI can sit the take-home test

Every coefficient in this post was estimated in a world where the candidate produced their own answers. An unsupervised assessment now measures some mix of the candidate and whatever model they had open.

The scale is not hypothetical. In a Gartner survey of 3,290 job candidates in late 2024, 39% said they used AI during the application process, and 29% of those used it to generate answers to assessment questions. That is self-reported, so treat it as a floor rather than a ceiling.

Personality scales look most exposed. Jane Phillips and Chet Robie at Wilfrid Laurier University asked four language models to answer personality measures as an ideal applicant would. GPT-4 faked better than 99.6% of a student comparison group on the easier assessment and 91.8% on the harder one. The comparison group was students, not real applicants, which limits how far the result travels.

A validity coefficient assumes the candidate wrote the answer

Read the revised table through that lens and a sharper ranking appears. The top predictor, the structured interview, happens live. Biodata asks about verifiable history. The work sample, when run as an unsupervised take-home task, is the format most easily completed by a model, and its validity estimate was already cut by 0.21.

Validity values are Sackett et al., 2022. The exposure column is Zan Digital editorial judgement from the studies cited in this section, not a measured result. No published meta-analysis yet estimates validity under AI-assisted completion.

An unproctored take-home test should not carry a validity number in your head at all. The coefficient was estimated on a different test, one the candidate actually sat. How assessment formats break down under AI cheating in more detail, including archival score data, is covered in the analysis of skills-based hiring and AI-exposed assessments. The engineering version of the same problem is in the review of which technical interview signals survive AI.

How to choose hiring assessments using the revised numbers

Put the evidence together and a practical design emerges.

Start with a structured interview as the spine of the process. Structure means the same job-derived questions for every candidate, a written scoring guide with behavioural anchors, and scores recorded before panellists compare notes. Without those three, you are running the unstructured version at 0.19, whatever the calendar invite calls it.

Then add one predictor that measures something the interview does not. A job knowledge test suits experienced hires. Biodata suits volume hiring and has the tightest spread. A cognitive test still adds signal, but the revision says it is no longer the default first choice, and it carries the largest subgroup gap. Do not drop cognitive tests on the strength of the 2022 paper alone if they already work in your local data.

Finally, decide where AI belongs. If the job will be done with AI tools, test the candidate using them, live, and score the judgement rather than the output. The interview questions that separate fluent users from passengers are set out in the guide to AI fluency interview questions. What you should not do is give an unsupervised task, accept polished output, and assume a validity estimate from a pre-2022 table applies.

The last step is the one almost nobody takes: check your own numbers. Twelve months after a hiring round, correlate the scores from each stage with how those hires actually performed. Even a rough local check tells you more about your setting than any meta-analytic mean, because the mean was never about your jobs.

Frequently asked questions

Where to start

Pull up your current hiring loop and label each stage with its revised validity from the table above, using the unstructured value wherever the interview has no written scoring guide. Any stage that is unsupervised and AI-completable gets a question mark instead of a number. That one-page map usually shows where the process is spending interviewer hours on stages that predict very little.

Then pick one role with at least 20 hires in the past two years and pull each hire's stage scores alongside their first-year performance rating. A spreadsheet correlation is crude, and it is still the first evidence about your own setting you will have.

Related reading