Opptly built its candidate scoring engine itself, so no upstream vendor’s audit covered it and the model had to be tested directly. Three regulatory regimes apply to it. NYC Local Law 144 requires an independent bias audit for automated employment decision tools and specifies who may run one, which rules out the development team. The EEOC Uniform Guidelines require adverse-impact analysis under 29 CFR §1607.15. The EU AI Act classifies employment screening as high-risk under Article 6 and Annex III, then splits the obligations: developers ensure data governance and bias detection under Article 10 and produce the Annex IV technical file, while deployers commission the audit and publish a summary.

This session walks through how the audit was built to answer the question a regulator or an enterprise buyer asks, which is whether the test could have found bias if bias were present. The design is counterfactual: take one resume, change exactly one thing about the candidate’s identity, and score it again. A name that signals gender or ethnicity. A graduation year that signals age. Everything else stays identical, so nothing except that one change can explain a difference in score. Run across 8 protected-attribute categories and 43 classes, covering sex and gender, race and ethnicity, national origin, religion, pregnancy and maternity status, age, disability, and military or veteran status. Selection rates held between 0.91 and 0.92, impact ratios sat near 1.00, and no class showed practical bias.

The part that decides what an audit is worth is statistical power. A preliminary run on 10% of the data showed an apparent signal at high scores. At full scale it narrowed to the extreme tail and resolved as tail noise. A smaller study would have published a false positive as a finding. We’ll also cover the operational design that makes the result maintainable: synthetic data only, executed inside Opptly’s own network, version-controlled and re-runnable by Opptly’s team.

This session describes an audit methodology and the regulations its tests are derived from. It is not legal advice. Organizations deploying automated employment decision tools should consult their own compliance counsel.