Bias in AI means a system gives systematically worse results for some groups of people. It can come from unrepresentative data, from a stand-in measure that hides unfairness, from technical settings, or from how people use the output. NIST says bias can cause harm whatever anyone intended. Researchers and regulators have documented it in face recognition, health risk scoring and hiring software.
01What do researchers mean by bias in AI?
The US standards agency NIST sorts AI bias into three groups in its Special Publication 1270. The word does not only mean prejudice. Statistical bias, for example, can arise when a data sample does not represent the population, with no discriminatory intent at all.
- Statistical and computational bias: the data or the model does not represent the real population, even when nobody meant any harm.
- Human bias: systematic errors in how people think, which shape the data, the design and how the output is used.
- Systemic bias: unfair procedures and habits inside institutions that end up recorded in data and rules.
NIST states that bias is common across technology processes and can lead to harm whether or not anyone intended it. It also says human and systemic factors are significant sources, so fixing the maths alone is not enough.
02How does bias get into a system?
Published studies point to four routes. Each one has a measured example. The routes can combine in one product: a system can use skewed data, rely on a poor stand-in measure and then be used with no one checking outcomes.
- Unrepresentative data. An audit of three commercial gender-classification systems found error rates up to 34.7% for darker-skinned women and 0.8% for lighter-skinned men. Two common test sets were mostly lighter-skinned faces (Buolamwini and Gebru, 2018).
- A stand-in measure. A widely used US health algorithm used past health costs as a stand-in for health needs. Less money had been spent on Black patients with the same needs, so at the same risk score they were sicker. The authors estimated the bias cut by more than half the number of Black patients flagged for extra care, and that changing the measure removed it (Obermeyer and others, 2019).
- Technical settings. NIST tested 189 face recognition algorithms from 99 developers. False positive rates often differed by factors of 10 to more than 100 between demographic groups, but not for every algorithm, and some highly accurate ones showed no detectable difference in one type of search (NISTIR 8280).
- How the tool is used. The UK Information Commissioner's Office audited AI recruitment tools and found some let recruiters filter out candidates by protected characteristics, and some guessed gender and ethnicity from names instead of asking (ICO, November 2024).
03What have regulators done about it?
In the United States, the Equal Employment Opportunity Commission (EEOC) alleged that a tutoring company's application software automatically rejected women aged 55 or older and men aged 60 or older. The company agreed to pay $365,000 to settle, announced on 11 September 2023 (EEOC).
In the UK, the ICO's audits produced almost 300 recommendations that the audited companies accepted or partly accepted. The ICO told providers to collect accurate information from candidates and to run regular checks to find and reduce discrimination.
Our separate page on algorithmic bias examples covers more cases, including some where the outcome is still undecided.
04Why is "fair" hard to define?
A well-known dispute shows the difficulty. ProPublica reported in 2016 that the COMPAS criminal risk tool wrongly labelled Black defendants as high risk more often than white defendants. The tool's maker, Northpointe, argued the software treated both groups equally. The two sides then argued over which statistics to use, including how to treat different base rates (ProPublica's technical response).
The sources do not settle who is right, and they show that reasonable people can pick different fairness measures. Choosing one is a judgment about values, so organisations need to decide and record which measure they use, and why.
05Questions to ask before you rely on an AI tool
These questions draw on the ICO findings above. They are not an official checklist or legal advice. For your legal duties, ask a lawyer or your regulator.
- Does the tool let anyone filter or score people by a protected characteristic such as age, sex, race or disability?
- Does it guess traits such as gender or ethnicity, and how accurate is that guess?
- Has the vendor tested results for different groups, and can you see the results?
- Who checks outcomes regularly after launch, and who decides what to do if a group does worse?
- Does a person review decisions that affect someone's job, credit, health or freedom?
?What is not known
- People disagree on which fairness measure to use, and the COMPAS dispute is not resolved in the sources we read.
- The error rates above describe specific systems tested in 2018 and 2019. Newer versions may differ, and NIST found the results varied by algorithm.
- The sources do not say how many AI tools in hiring, lending or health are tested for bias before use.
QQuestions people ask
Is AI bias the same as human prejudice?
Not exactly. Human bias is one of three categories NIST describes. The data and the model's design can add bias that no individual intended.
Can bias be fixed once it is found?
Sometimes. In the health algorithm study, changing the stand-in measure removed the bias the authors found. Fixing one cause does not guarantee that others are absent, so testing has to continue.
Does a biased result mean someone broke the law?
Not by itself. A regulator or court decides that, and in some cases the findings are allegations. Ask a lawyer or the relevant regulator about your own situation.
Last checked October 24, 2026. Rules and settings change, so open the source to see the current version.
- NIST SP 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence National Institute of Standards and Technology NIST's 2022 publication; we used its three categories of bias and its point that harm can occur regardless of intent.
- Gender Shades Proceedings of Machine Learning Research (Buolamwini and Gebru) A 2018 audit of commercial gender classifiers; we used its error rates and its finding about test-set makeup.
- Dissecting racial bias in an algorithm used to manage the health of populations UC Berkeley Center for Healthcare Marketplace Innovation A summary page for the 2019 Science paper; we used the finding on cost as a stand-in measure.
- NISTIR 8280, Face Recognition Vendor Test Part 3: Demographic Effects National Institute of Standards and Technology NIST's December 2019 test of 189 algorithms; we used its false positive findings by demographic group.
- ICO intervention into AI recruitment tools leads to better data protection for job seekers UK Information Commissioner's Office The ICO's 6 November 2024 announcement of its audit findings; we used its findings on filtering and inferred traits.
- iTutorGroup to pay $365,000 to settle EEOC discriminatory hiring suit US Equal Employment Opportunity Commission The EEOC's 11 September 2023 press release; we used the allegation and the settlement amount.
- Technical Response to Northpointe ProPublica ProPublica's reply to the COMPAS maker; we used it for the outline of the dispute over fairness statistics.
General information, not legal, medical, financial or investment advice. We are not affiliated with any company, agency or standards body named here. See our editorial policy and corrections, or tell us about a mistake at [email protected].