The DWP’s own analysis found its fraud algorithm treats claimants differently by age and nationality

Somewhere in the systems that process Universal Credit, a machine learning model assigns each claim a fraud risk score. A higher score makes it more likely the claim will be selected for review, which in practice means the claimant is asked to produce documents, bank statements and explanations, and payment can be disrupted while that happens.

Claimants are not told their score. They are not told that a score exists, or which features of their claim produced it.

The department’s own fairness analysis

A fairness analysis carried out internally by the DWP, released only after sustained pressure from campaigners and journalists, found a statistically significant referral and outcome disparity for every protected characteristic it examined, covering age, disability, marital status and nationality.

In practical terms, claimants aged 45 to 54 and above, and claimants who are not UK nationals, were being disproportionately referred for review. They were more likely than others to be asked to find and provide additional evidence in order to keep money they were entitled to receive.

Why the secrecy is the substantive issue

The department has argued that publishing detail about the model would help people game the system. That argument has an obvious limit, which is that it also prevents anyone outside the department from establishing whether the system is operating lawfully.

Under the Equality Act 2010, a public body has a duty to have due regard to the need to eliminate discrimination. Nobody can assess whether that duty is being met against a model that cannot be inspected. Legal challenges and freedom of information work by Foxglove and Big Brother Watch have prised out fragments of the picture, but the department has not published the model in a form that would allow independent assessment.

The imbalance in rights

If a bank declined your credit application on the basis of an automated decision, data protection law would give you rights to an explanation and to human review. When the DWP’s model flags a Universal Credit claim, the claimant receives a request for documents and no indication that an algorithm was involved at all.

The people most likely to be scored are also the people least able to absorb a suspended payment while they prove a negative.

What transparency would require

Publishing a model card setting out the features used, the training data, the error rates and the measured disparity by protected characteristic would allow independent scrutiny without telling anybody how to commit fraud. Telling claimants when an automated system has contributed to a decision to review them would bring the department into line with the standard applied to commercial lenders. And an independent audit on a fixed cycle, with findings laid before Parliament, would establish whether any of it is improving.