NYC's AI Hiring Audit Law: 18 of 391 Employers Posted a Bias Audit

Researchers checked 391 employers in late 2023 and found 18 that had posted the bias audit New York City's Local Law 144 demands. That is under 5%. NYC Local Law 144 was the first US law to make automated hiring tools pass a published audit before use. Two years on, the evidence says it mostly produced silence, and that the silence cannot even be called lawbreaking.

The short answer

NYC Local Law 144 bars employers from using an automated employment decision tool on candidates unless three conditions hold. The tool must have had an independent bias audit within the past year, a summary must be posted online, and candidates who live in the city must get 10 business days' notice. Fines reach $1,500 per violation, and each day of use counts separately.

What NYC Local Law 144 actually requires

Local Law 144 was enacted in December 2021 and took effect on 1 January 2023. The city's Department of Consumer and Worker Protection, known as DCWP, began enforcing it on 5 July 2023 after two delays. The bill text on the City Council's Legistar record is short, and most of what matters sits in three sections.

The obligations fall on employers and employment agencies, not on the software vendor. That single drafting choice explains most of what followed.

The definition turns on one word: substantially

An automated employment decision tool, or AEDT, is any computational process derived from machine learning, statistical modelling, data analytics or AI. It must issue simplified output, such as a score, classification or recommendation. And it must be used to "substantially assist or replace discretionary decision making" for hiring or promotion decisions.

The last clause is the gate. A resume ranker that a recruiter glances at, then ignores, can be argued out of scope. The same ranker used to cut a pile of 2,000 applicants to 50 is inside it. The employer makes that call, and nobody reviews it in advance.

The audit, the posting and the notice

Three duties apply before an in-scope tool touches a candidate. The tool needs a bias audit by an independent auditor, conducted no more than one year before use. A summary of the most recent audit, plus the distribution date of the tool version it covers, must be public on the employer's website. And each candidate or employee who lives in the city must be told at least 10 business days before use, with a route to request an alternative selection process or accommodation.

Penalties are modest by design. Under section 20-872, a first violation costs up to $500, and later violations cost $500 to $1,500 each. Each day a non-compliant tool is used is a separate violation, and each missed candidate notice is its own violation as well.

One thing the law does not say matters as much as what it does. The statute requires an audit and disclosure. It contains no provision requiring an employer to stop using a tool that shows disparate impact. Local Law 144 is a transparency law, not a performance standard, and buyers who read it as a fairness certification are reading something that is not there.

The 18 of 391 finding, and what "null compliance" means

The most cited measurement of the law comes from Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability. It was written by researchers at Cornell University, Data & Society and Consumer Reports, and presented at the ACM FAccT conference in June 2024.

How the study was run

Between 24 October and 9 November 2023, 155 student investigators searched the websites of 391 employers. The sample was drawn at random from 511 employers that recruit Cornell students. Each investigator spent no more than 30 minutes per employer looking for an audit and a transparency notice, without submitting an application.

The result was thin. 18 employers had posted audit reports and 13 had posted transparency notices. Only 11 posted both in a form the authors believed met the law. Among the 267 employers listing jobs in New York City, 14 posted an audit, which is about 5%.

Even the 18 overstated the variety. Only 13 of the reports were unique, because some employers simply republished their vendor's audit. That detail matters later, when the question turns to what vendors supply.

Why a missing audit proves nothing

The authors named their finding "null compliance" for a reason. Because the law lets each employer decide whether its system is in scope, a missing audit cannot be read as a violation. The employer may not use an AEDT. It may use one and have concluded the tool does not "substantially" assist decisions. It may post notices by email after an application, where no outside researcher can see them.

The paper puts it directly: the law gives employers a level of discretion "that makes the denominator impossible to know." In other words, nobody can calculate a compliance rate. Nobody knows how many employers should be complying.

I think this is the most important sentence written about AI hiring regulation so far. A law that lets the regulated party define its own scope will always look like it is being obeyed. That is a design flaw, not an enforcement failure, and it repeats in newer laws discussed in the breakdown of California's AI laws for software vendors.

Why nearly every published AI hiring bias audit passes

The bias audit itself is arithmetic. For each sex category, each race and ethnicity category, and each intersection of the two, the auditor calculates an impact ratio. That ratio is a group's selection rate divided by the selection rate of the most selected group. Where the tool scores rather than selects, the ratio uses average scores instead.

A ratio of 0.8 is the familiar line. It comes from the four-fifths rule of thumb long used in US employment discrimination analysis, and the city's rule does not adopt it as a pass mark. Local Law 144 sets no threshold at all.

The published audits clear it anyway. The FAccT paper reports that 96% of published audit reports showed impact ratios above 0.8, and that only 9 of 386 individual ratio measurements fell below it. Subgroups were omitted from 20% of impact ratios, mostly because they made up less than 2% of the data, which the rule allows auditors to exclude.

The authors suggest employer discretion explains the pattern. An employer that commissions an audit and dislikes the result can decide its tool was out of scope and publish nothing. The researchers are careful to say their data cannot prove this kind of publication bias. I think the incentive is obvious enough that a buyer should assume it exists until shown otherwise.

That has a practical consequence for anyone evaluating a vendor. A clean published audit is the expected output of the system, not evidence that the tool is fair. If your team uses assessment scores to rank candidates, the deeper validity question sits elsewhere, and the Sackett revision of hiring assessment validity is a better guide to it than any impact ratio.

What the Comptroller found about AEDT enforcement

On 2 December 2025, the New York State Comptroller published an audit of DCWP's enforcement of Local Law 144, covering July 2023 to June 2025. The report's own page blocked automated access during research for this post, so the figures below come from DLA Piper's summary of 30 January 2026. Treat them as Tier 2 until checked against the report.

The summary describes the Comptroller's verdict on enforcement as "ineffective." The detail is more specific than the label.

All rows are as summarised by DLA Piper, January 2026. The summary does not state the number of test calls placed or the number of complaints DCWP received during the period.

The two readings tell you something about the audits too. If a state auditor can find at least 17 potential problems in 32 published audits, the published audit market is not producing consistent documents. That is a reason to read your vendor's audit line by line rather than filing it.

What HR tech vendors now supply, and what they do not

Because the law binds employers, vendors moved to supply the paperwork. Several now commission audits of their own models and publish summaries that customers can host. Eightfold, for example, publishes an audit document for its AI Interviewer prepared by BABL AI, titled for Local Law 144 and versioned 29 June 2026.

Vendor audits carry a careful disclaimer

Vendor-commissioned summaries tend to avoid the one question that decides whether the law applies. Eightfold's BABL AI summaries, as indexed, describe the model as audited as though it were an AEDT, while declining to determine whether it is one. That is a reasonable position for an auditor. It also means the hardest legal judgement stays with you.

This is the right reading of every vendor audit: it is a starting document, not a compliance shield. The vendor can show its model's ratios across its customers. It cannot tell you whether your recruiters rely on the score enough to make your use "substantial".

Pooled data and your data

DCWP's guidance allows a bias audit to use historical data pooled from several employers. But, as Ogletree Deakins summarised the city's FAQ, an employer may rely on such an audit only in two cases. It supplied its own historical data to the auditor, or it has never used the tool. Test data is allowed when historical data is insufficient, with the reason explained. Imputed or inferred demographic data is not allowed.

So a vendor's pooled audit covers a new customer in year one. From year two, it only covers you if your data went in. Ask whether it did, in writing, before the renewal, because most procurement teams never think to.

Meanwhile the more serious liability signal is coming from the courts, not the city. In Mobley v. Workday, Judge Rita Lin of the Northern District of California granted preliminary certification of an age discrimination collective on 16 May 2025. It covers applicants aged 40 and over who applied through Workday's system and were allegedly denied employment recommendations. Workday had argued it was not the employer making the decisions. The ruling did not decide the merits, but it shows that vendor exposure is being tested under federal discrimination law, where Local Law 144's employer-only design offers no cover.

Where this argument is weakest

The 18 of 391 number is easy to quote and easy to overread. Here is where the case above is thinnest.

Low numbers may mean low use

Many sampled employers may not use an in-scope tool at all. The study's follow-up survey drew 26 responses from 387 emails, and 23 of those respondents said the law did not apply to them. If that ratio held across the sample, 18 audits would be close to the true number of in-scope users. Nobody can test this, which is the authors' point, but it cuts both ways.

The sample also skews. It covers employers that recruit Cornell students, which tend to hire into credentialed professional roles. High-volume hourly hiring, where resume screening and scoring tools are most common, may look different in either direction.

The enforcement evidence is second-hand here

The Comptroller figures in this post come from a law firm's summary, because the report itself could not be opened during research. Those figures should be checked against the primary document before anyone relies on them in a board paper. A law firm also has a commercial reason to frame an audit as raising employer risk.

There is a fair case for the law, too. Transparency rules often start weak and tighten, and Local Law 144 did create an audit market, a vocabulary and a public record that researchers could measure. The study exists because the law exists. I still think the scope test is the flaw that matters, but calling the law a failure ignores what it made measurable.

An AEDT audit checklist for HR tech buyers

Suppose you run talent acquisition or HR operations and buy screening, matching or interview tools. The useful question is not whether your vendor has an audit. It is whether that audit covers your use. The same discipline applies to any AI purchase, as set out in the AI agent procurement checklist.

The last row is the one most contracts miss. The audit must be no more than one year old at the point of use, so an audit that expires mid-term leaves you exposed for every day after. Put the obligation in the contract, alongside the questions in the NIST AI RMF RFP questionnaire.

New York is also no longer the only jurisdiction. Colorado's law has been delayed and amended repeatedly, as tracked in the status of the Colorado AI Act, and a vendor answer built only for New York will not cover it. If you are rethinking the screening stage itself, the analysis of skills-based hiring assessments covers what the tools measure in the first place.

Frequently asked questions

Where to start

Pull every tool in your hiring stack that scores, ranks or recommends candidates, and write one line on how a recruiter uses its output. If the honest line is "we shortlist from the top of this list," treat the tool as in scope and act accordingly.

Then send each vendor the five questions in the table above before your next renewal. The answers will tell you within a week which vendors have done the work and which are selling a certificate.

Related reading