Back
UK ICO Audit: AI Recruitment Tools Allow Filtering by , Regulator Makes 300 Recommendations
The rapid deployment of
systems in hiring, lending, and welfare assessment has created a new accountability challenge that legal professionals are only beginning to grapple with. In a
audit of AI recruitment tools, the
found that some systems allowed recruiters to filter candidates by
, while others inferred gender and ethnicity from names. The regulator made nearly 300 recommendations, including measures to improve
and monitor
. Yet when an affected person asks why a decision was made, the answer is often procedural rather than substantive:
"the system produced the result."
That response, critics argue, relocates responsibility into a chain of models, datasets, vendors, and reviewers.
This phenomenon has been termed the "" — a description of how technical complexity is used to make the exercise of power appear ownerless. It is not yet a settled legal doctrine, nor does it argue that software has . Rather, it frames a familiar : an organisation that chooses to deploy an opaque system remains visible when it collects the commercial or administrative benefit, yet becomes harder to identify when the decision must be explained. Legal scholars and practitioners are now examining whether principles from corporate law, particularly those concerning the , can offer a useful framework for piercing this algorithmic opacity.
From Corporate Separateness to Technical Opacity
The starting point for any analysis of is , which established the company as a legal person separate from its shareholders. That principle does not license courts to ignore the reality of control whenever a corporate structure produces an unfair result. In , the treated as narrow: it may be available where a person deliberately interposes a company to frustrate an existing legal obligation. The important point, as the recent opinion on the emphasises, is not that courts may freely look past form. It is that legal form cannot be used strategically to defeat accountability.
The presents a different but related problem. A decision may be shaped by a vendor’s model, a deployer’s choice of data and threshold, a consultant’s integration work, and a nominal human reviewer. No single fact automatically determines liability. Nor is opacity always deliberate: some systems are difficult to interpret because of their design, scale, or the way they are used. But complexity cannot settle the legal question. Where an organisation adopts a system for employment, credit, insurance, healthcare, education, or welfare, it has made a choice about how power will be exercised. It should not be permitted to turn that choice into an for the person affected.
Why Familiar Liability Routes Strain
Traditional doctrines are not empty. Contract, consumer protection, data protection, anti- law, , and public-law duties may each apply depending on the decision-maker and the harm. The difficulty is evidentiary and institutional. A claimant may have to establish what inputs were used, which version of a model generated the output, whether a person could intervene, and whether the organisation followed its own safeguards. Those facts are commonly distributed across parties that owe the claimant no direct explanation.
illustrates the mismatch. Product-liability regimes generally ask whether a product was defective and identify defined actors in the supply chain. Under the United Kingdom’s , liability turns on a defective product and specified categories of producer or supplier. That structure is easier to apply to a finished product than to a service assembled from data, software updates, cloud infrastructure, and local deployment choices. It does not follow that every AI system is continuously self-learning, or that ordinary product law is irrelevant. The point is narrower: the relevant defect, causal contribution, and responsible actor may change across the system’s life cycle.
faces a parallel problem. Opacity does not remove the requirements of duty, breach, causation, and damage. It can, however, make each requirement harder to prove. A deployer may say that it relied reasonably on a specialist vendor; the vendor may point to the customer’s data, prompts, or settings; both may invoke . If neither must preserve meaningful records or explain a contested result, the party harmed bears the cost of an information gap created by those who designed the arrangement.
The problem is sharper where an automated output determines access to an essential service or opportunity. A recommendation may be contestable; an unexplained exclusion from a job, benefit, or loan can have immediate consequences. Calling the result a prediction does not make the choice neutral. The relevant legal inquiry is not whether a machine formed an intention. It is whether a person or institution exercised significant power through a system without adequate safeguards.
A Framework That Keeps Responsibility Attached to Power
The answer is not to treat every developer, data provider, and user as equally liable for every outcome. That would obscure rather than clarify responsibility. A better approach begins with the deploying organisation. It is normally the actor that decides to use the system, sets the purpose and operating conditions, and acts on the output. For a high-impact decision, it should be the first institution required to give a comprehensible reason, identify the system used, and provide a real route to human review. It can allocate risk by contract with its vendors, but those private arrangements should not dilute the affected person’s ability to obtain an answer or a remedy.
Second, accountability must be designed before deployment. The organisation should maintain a decision record sufficient to reconstruct the material path to an outcome: the purpose of the tool, its provider and version, the key inputs, any material changes, the authorised users, and the human intervention available. This is not a demand that every model reveal its source code. It is a demand that an institution using automated power retain enough evidence to justify it. A also allows courts and regulators to distinguish a one-off error from a predictable failure in design, procurement, or oversight.
Third, certain decisions require . In contexts affecting livelihood, liberty, health, housing, or welfare, a human reviewer must have both the authority and the information to question, override, and correct an automated output. A who can only approve a score is not oversight. The offers a useful regulatory direction: for , it requires , , and designed to prevent or minimise risks to health, safety, and fundamental rights. Its Article 14 requires effective oversight by natural persons during use. Such obligations do not themselves resolve every , but they demonstrate that oversight can be treated as an operational responsibility rather than an afterthought.
Finally, legal analysis should separate different roles without allowing the chain to become a shield. A vendor may be responsible for misleading claims, negligent design, or the concealment of known limitations. A deployer may be responsible for using an unsuitable system, accepting discriminatory results, or failing to review them. A data provider may bear responsibility where it breached an independent duty. The rule is simple: each actor should answer for the risk it controlled or created, while the institution that made the remains answerable to the person subject to it.
Procedural Corrections and the
This allocation should also shape procedure. Once a claimant establishes that a was materially automated and that a has occurred, the deployer should be required to disclose the records needed to explain the decision. The burden should not remain entirely with a person who cannot inspect the system, the procurement contract, or the relevant logs. Courts can protect genuinely sensitive commercial information through while still requiring an of the decision. Without that , a duty of oversight risks becoming a promise that cannot be tested.
Courts are accustomed to looking beyond appearances when a legal structure conceals the exercise of power. The does not call for rejecting innovation or treating an AI system as a moral agent. It calls for . The use of a complex tool should not permit the user of that tool to disclaim a decision that it has adopted and acted upon.
The question, then, is not whether an algorithm can be blamed, but who chose to rely on it, who could have prevented the harm, who benefited from the arrangement, and who can make the decision intelligible after it is challenged. If the law keeps those questions in view, technical opacity need not become a .
Implications for Legal Practice
For legal professionals, the concept of the signals a shift in how liability for automated decisions must be analysed. Corporate lawyers advising on procurement contracts for AI systems will need to ensure that accountability obligations are contractually allocated without undermining the deployer’s ultimate responsibility. Litigators will need to argue for and that recognise the between claimants and algorithmic deployers. Regulators like the ICO are already moving toward greater requirements, and the sets a high bar for .
The 300 recommendations from the ICO audit are a clear signal that regulators expect organisations to take proactive steps to prevent and ensure . Failure to do so may expose deployers to , , and . The law is catching up with technology, and the is no longer an excuse for absent accountability. As the opinion concludes, the doctrine must see the decision, not just the code.
