AI governance failure: four cases from regulators and courts

01What counts as a governance failure here?

We use cases where a regulator, agency or court published what it found or alleged. We do not use press speculation, and we do not name private individuals. The wording matters: a regulator's finding, an agency's allegation and a settlement are different things, and the sections below keep them apart. This page is a reading of public records, not legal advice.

02Rite Aid: facial recognition without testing or monitoring

On 19 December 2023 the US Federal Trade Commission announced a proposed settlement with the pharmacy chain Rite Aid. The FTC alleged that between 2012 and 2020 the company's facial recognition system wrongly flagged customers as matches to earlier suspected shoplifters, and that staff then followed, searched or reported innocent shoppers, with a heavier effect on people of colour.

The FTC alleged these governance gaps: no assessment of misidentification risk, no testing of accuracy before use (including no questions to vendors about their testing), no control of low-quality images, no monitoring of false matches afterwards, and weak staff training. The proposed order would ban Rite Aid from using facial recognition for surveillance for five years, require deletion of images and the algorithms built from them, and require safeguards for any biometric system plus a data security programme with outside assessments (FTC). The release says the order needed court approval because the company was in bankruptcy; check the FTC case page for its current status.

03Dutch Tax Administration: unneeded data in a risk system

On 7 December 2021 the Dutch data protection authority fined the Tax Administration EUR 2.75 million. It found that, for years, the agency handled nationality data on childcare benefit applicants in a way the regulator called unlawful and discriminatory. Data on dual nationality of Dutch citizens should have been deleted in January 2014, yet some 1.4 million people were still registered in May 2018. Nationality was also used as an indicator in a system that automatically marked applications as risky, and the regulator found it was not needed for that.

The agency cleaned up its systems after the investigation, deleting the dual nationality data in the summer of 2020 (Autoriteit Persoonsgegevens). The regulator called the fine an important step but said it could not undo the harm.

04iTutorGroup and Air Canada: hiring software and a chatbot

iTutorGroup. The US Equal Employment Opportunity Commission alleged that the company's tutor application software was set to reject women aged 55 or older and men aged 60 or older automatically, turning away more than 200 qualified US applicants. The company agreed to pay USD 365,000 in a September 2023 settlement. The consent decree added staff training, an anti-discrimination policy, a ban on asking for birth dates, and at least five years of EEOC monitoring (EEOC).

Air Canada. On 14 February 2024 a British Columbia small-claims tribunal ruled on a passenger's claim after Air Canada's website chatbot said a bereavement fare could be requested after travel, which the airline's policy did not allow. Air Canada argued it was not liable for what its chatbot said. The tribunal called that a remarkable submission, held the airline responsible for all information on its site, and ordered it to pay $812.02 in total, covering damages, interest and fees (BC Civil Resolution Tribunal). It is one tribunal's decision on one claim, not settled law.

05What do the cases have in common?

The details differ, but the records describe gaps of a similar kind. The NIST AI Risk Management Framework asks organisations to put controls in these areas: testing, monitoring and review.

• No testing or monitoring of how often the system was wrong.
• Data kept or used when it was not needed.
• A rule or setting hard-coded without anyone reviewing its effect on a protected group.
• A tool treated as separate from the organisation that deployed it.