AI Bias & Fairness Audit Services
Independent, evidence-based fairness testing for AI systems that influence decisions about people, across 30+ languages and market-relevant demographic dimensions.
Request a Fairness Audit Scope See What a Bias Audit Report Contains
What a Bias and Fairness Audit Actually Is
An audit measures whether outcomes or quality of service differ between groups and reports distributions, metrics, samples and confidence. It is not red teaming, permanent certification or automatic mitigation. A useful audit is one that could have failed.
What Regulation Requires — and When
As of August 2026: NYC Local Law 144 and Illinois AIVIA are in force; Colorado SB 24-205 is effective; EU AI Act Article 50 is in force while Annex III and Annex I high-risk obligations were postponed; India's MeitY framework is voluntary and the DPDP regime governs personal data. This is orientation, not legal advice. Last reviewed: August 2026.
The Three Kinds of Bias We Test For
Outcome disparity
Different decisions or scores across groups.
Quality-of-service disparity
Different error rates, accuracy, WER or CER by group.
Representational harm
Stereotyping, erasure or name-conditioned output differences.
Fairness Metrics — and How to Choose the Right One
Impact ratio, demographic parity, equal opportunity, equalized odds, predictive parity, counterfactual fairness and per-group error rates answer different questions. Calibration and equalized odds cannot both hold when base rates differ. The primary metric is agreed before testing and others are reported alongside it.
Model Types We Audit
LLMs and generative text, speech and ASR, computer vision and face, scoring and eligibility, recommenders and search, and document or OCR systems.
How We Run an Audit
- Scope and metric selection
- Data and attribute strategy
- Test-set construction
- Evaluator-panel assembly
- Measurement and analysis
- Reporting and walkthrough
Bias Western Tooling Doesn't Look For
Global protected classes matter. India and South Asia also require attention to caste, religion, geography, name signals, language, script, code-mixing, dialect, accent and proxy variables. Native context is part of the method.
Sample Sizes and Statistical Honesty
Intersectional cells need adequate volume. Thin cells are labelled indicative rather than presented as defensible measured disparities.
What You Get
- Fairness audit report
- Metric appendix
- Method statement
- Probe sets and test data
- Intersectional breakdown
- Remediation notes
- Live walkthrough
You keep the test sets.
Where Bias Audits Go Wrong
Common failures include choosing metrics after results, marginal-only analysis, translated probes, unmatched pairs, unrepresentative panels, aggregate-only accuracy, one-time audits and biased ground truth.
What We Do Not Do
We do not certify models as unbiased, provide legal advice, audit our own work, soften findings or choose a metric after seeing results.
How to Engage
Baseline audit, multi-model or multilingual programme audit, and re-audit or monitoring cadence.
Related Services
AI Red Teaming & Adversarial Testing, LLM Evaluation Services, ASR & Speech Model Testing and Computer Vision Model Testing are live. Other live links: Dataset QA & Label Audit and LLM & RLHF Annotation.
Frequently Asked Questions
What is an AI bias audit?
A structured measurement of whether an AI system produces materially different outcomes, or materially different quality of service, for different groups of people. It states which fairness definition was used, which groups were tested, what data the test ran on, and what disparities were found—with sample sizes and confidence intervals, so the finding can be checked.
Is a bias audit legally required for my system?
It depends on jurisdiction and use case. Automated employment decision tools used in New York City require an annual independent bias audit. Colorado's SB 24-205 requires annual impact assessments for consequential employment decisions from 30 June 2026. Under the EU AI Act, transparency obligations came into force on 2 August 2026, while high-risk obligations were postponed to 2 December 2027 for stand-alone Annex III systems and 2 August 2028 for embedded Annex I systems. India's MeitY guidelines are voluntary but expect pre-deployment testing across gender, caste, religion and geography for credit, insurance, employment and service delivery. Confirm your specific obligations with counsel.
How long does an audit take?
A baseline audit of one model in one or two languages typically takes 5–7 weeks, of which the first two are scoping and test-set construction. Re-audits against an existing baseline run 2–3 weeks.
We don't have demographic data on our users. Can you still audit?
Usually yes, using counterfactual and matched-pair methods that construct the comparison rather than relying on labelled user attributes. This works particularly well for generative models. For outcome-disparity testing on historical decisions, some form of group attribution is needed.
Which fairness metric should we use?
It depends on the decision. Equal opportunity is a common default for screening and eligibility decisions. Impact ratio is required where a regulation names it. Equalized odds suits high-stakes decisions where both false positives and false negatives carry real cost. The primary metric is agreed in writing before testing.
Can you audit an LLM, or only classification models?
Both. LLMs use counterfactual prompt sets, name-conditioned probes, stereotype association batteries and demographically matched human raters rather than only confusion-matrix metrics.
Do you test for bias in Indian languages?
Yes. We test in 12+ Indian languages including romanised and code-mixed input, with probe sets authored natively rather than machine-translated from English.
Will you certify our model as unbiased?
No. Fairness is measured against defined groups, data and a point in time. We provide documented, repeatable measurement rather than a permanent certificate.
What happens if you find something serious?
We report it with worked examples and quantified magnitude. Where the cause is data-side and identifiable, we state it. Remediation is a separate engineering decision.
Are you independent enough to satisfy NYC Local Law 144?
We build no employment decision tools and sell no scoring models. Prior involvement is disclosed. Whether independence satisfies a statutory test is a question for counsel; we provide documentation of our role.