Who Controls AI and Should It Be Regulated?

AI development at the frontier is currently controlled by a small number of very large technology companies, predominantly based in the United States and China, with minimal independent oversight and no binding international framework. The computational infrastructure required to train frontier AI models — the hardware, the data centres, the electricity, and the engineering talent — is expensive enough to be accessible only to organisations with enormous capital resources. This creates a structural concentration of power that is not the result of any deliberate policy choice. It is the natural consequence of the economics of the technology. Cathy O'Neil identified the governance problem in its early form in 2016, before the current wave of large language models transformed public awareness of AI. Her analysis of algorithmic decision-making established a pattern that remains central to the governance debate today: the most powerful algorithmic systems share three structural properties that make them resistant to accountability. They are opaque — the reasoning behind their outputs is not visible to those affected by them. They operate at scale — affecting millions or billions of people simultaneously. And they cause damage that feeds back invisibly into their own inputs, making the harm self-reinforcing and difficult to detect.

"Algorithms are not objective. They are opinions embedded in mathematics. And the people whose opinions are embedded in them are rarely the people most affected by them."
— Cathy O'Neil, Weapons of Math Destruction, 2016

The case for regulating AI is not primarily about constraining innovation. It is about ensuring that the people and institutions deploying AI systems are accountable for their consequences — the same basic principle that underlies the regulation of pharmaceuticals, financial products, food safety, and aviation. Every one of those industries argued that regulation would stifle innovation. In each case, effective regulation ultimately improved safety, built public trust, and created more durable markets. O'Neil's framework identifies why AI regulation is structurally harder than regulating most other industries. The opacity of algorithmic systems means that harm can be invisible to regulators, to deployers, and to those harmed. A pharmaceutical that causes harm produces observable symptoms in identifiable patients. An algorithmic system that systematically disadvantages a demographic group may produce no signal that is visible without deliberate investigation — especially if the people harmed have no way of knowing that an algorithm was involved in the decision affecting them. The counter-argument most frequently made by the technology industry is that AI systems are too complex, too fast-moving, and too varied to be effectively regulated by government bodies with limited technical expertise. This argument has some empirical merit. Regulatory frameworks do lag behind deployment. The EU AI Act, adopted in 2024 and the most comprehensive AI regulatory framework currently in force, classifies AI systems by risk level and imposes proportionate requirements — but critics on both sides argue that it is simultaneously too restrictive for low-risk applications and insufficiently robust for the highest-risk ones.

"Regulation is not the enemy of innovation. The absence of regulation is the enemy of trust — and without trust, even the most powerful technology eventually collapses under the weight of its own failures."
— Cathy O'Neil, Weapons of Math Destruction, 2016

Every AI system makes choices — about what to optimise for, whose data to use, whose interests to weight, and what counts as a good outcome. These are not neutral technical decisions. They are value decisions with political consequences. The hiring algorithm that decides whether to shortlist a candidate, the content moderation system that decides what speech to allow, the credit-scoring model that decides who can access financial services — each of these systems encodes a particular view of what fairness, relevance, and risk mean. Those views were chosen by the people who built the system. O'Neil's central argument is that treating mathematical models as objective and neutral is itself a political choice — one that insulates the people who built the model from scrutiny and accountability. A model is not objective because it uses mathematics. Mathematics can express any set of priorities, including deeply unjust ones. The objectivity is a rhetorical claim that performs a specific political function: it removes the value choices from democratic debate and places them beyond challenge. The question of who decides what values AI systems should have is therefore not merely philosophical. It is a question about democratic legitimacy. Decisions that affect how millions of people are hired, lent to, surveilled, educated, and informed are governance decisions. In a democratic society, governance decisions are supposed to be made through processes that are accountable to the people they affect. Currently, most AI value decisions are made through processes that are accountable only to shareholders and, partially, to regulators.

"The people who build models are not neutral. They have assumptions, incentives, and blind spots. The model inherits all of them — and then presents them as mathematics."
— Cathy O'Neil, Weapons of Math Destruction, 2016

The EU AI Act, which entered into force in August 2024, is the world's first comprehensive binding legal framework for artificial intelligence. It takes a risk-based approach: classifying AI systems into four tiers based on the potential harm they can cause, and imposing proportionate requirements on each tier. Unacceptable-risk systems — such as social scoring by governments and real-time biometric surveillance in public spaces — are prohibited outright. High-risk systems — including AI used in hiring, credit, criminal justice, education, and critical infrastructure — face mandatory transparency, human oversight, and conformity assessment requirements. Lower-risk systems face lighter-touch disclosure obligations. General-purpose AI models above a certain capability threshold face additional requirements around transparency and safety testing. O'Neil's framework provides the most useful lens for evaluating the Act's likely effectiveness. Her three criteria for dangerous algorithmic systems — opacity, scale, and self-reinforcing damage — map directly onto the Act's risk classification. The Act is strongest where it addresses opacity, requiring high-risk systems to maintain technical documentation, enable human oversight, and provide affected individuals with meaningful explanations of automated decisions. These requirements, if enforced, would address the core accountability gap she identified. The Act's weaknesses are structural. Enforcement depends on national competent authorities whose technical capacity varies enormously across member states. The conformity assessment process for high-risk systems relies substantially on self-assessment by developers — the same parties with commercial incentives to minimise compliance burden. The definition of "general-purpose AI" is contested, and the most powerful frontier models may fall into regulatory gaps that were not anticipated when the Act was drafted.

"The solution to mathematical models that cause harm is not to ban mathematics. It is to demand transparency, require accountability, and ensure that the people harmed have recourse."
— Cathy O'Neil, Weapons of Math Destruction, 2016

Genuinely accountable AI requires three structural elements that are currently absent or weak across most high-stakes deployments. The first is transparency: the people and institutions deploying AI systems must be required to disclose what the system does, what it optimises for, what data it was trained on, and what its known failure modes are. This is not a demand for public access to proprietary model weights. It is a demand for the kind of disclosure that allows regulators, auditors, and affected individuals to understand what a system is doing and why. The second is independent auditing. O'Neil is explicit that self-assessment by developers is not a substitute for independent review. The pharmaceutical industry does not self-certify drug safety. Financial institutions do not self-certify their own solvency. High-stakes AI systems — those making decisions about people's employment, credit, freedom, and access to services — should not self-certify their own fairness and accuracy. Independent auditors with genuine access to training data, model architecture, and deployment context are a necessary condition for meaningful accountability. The third is redress. People affected by consequential AI decisions must have a practical path to challenge those decisions and obtain meaningful explanation and remedy. The right to explanation enshrined in the EU AI Act and in the General Data Protection Regulation is a step in the right direction. It is insufficient without the institutional infrastructure — accessible complaint mechanisms, adequately resourced regulators, and legal aid for those challenging AI decisions — that makes the right practically exercisable rather than theoretically available.

"Accountability requires three things: that you can see what the system is doing, that someone is responsible for it, and that the people it harms have somewhere to go."
— Cathy O'Neil, Weapons of Math Destruction, 2016

• AI development at the frontier is controlled by a small number of organisations with minimal independent oversight — the economics of frontier AI create structural concentration of power that existing democratic and regulatory institutions are not currently equipped to check. (O'Neil, Weapons of Math Destruction, 2016)
• The most dangerous algorithmic systems are opaque, operate at scale, and cause self-reinforcing damage — these three properties combine to make harm invisible to regulators, deployers, and those harmed, which is why mandatory transparency is a prerequisite for any effective governance framework. (O'Neil, Weapons of Math Destruction, 2016)
• Treating mathematical models as neutral and objective is a political choice, not a technical fact — every AI system encodes value decisions about what to optimise, whose data to use, and whose interests to serve, and presenting those decisions as mathematical neutrality removes them from democratic accountability. (O'Neil, Weapons of Math Destruction, 2016)
• The EU AI Act is the most significant AI regulatory framework currently in force — it classifies AI systems by risk level, prohibits the highest-risk applications, and imposes transparency and human oversight requirements on high-risk systems, but its enforcement depends on national authorities with uneven technical capacity. (O'Neil, Weapons of Math Destruction, 2016)
• Genuinely accountable AI requires mandatory transparency, independent auditing, and practical redress — self-assessment by developers is not accountability, and the right to explanation is meaningless without the institutional infrastructure that makes it practically exercisable for the people most affected. (O'Neil, Weapons of Math Destruction, 2016)

Who controls the development of AI?

AI development at the frontier is currently controlled by a small number of very large technology companies, predominantly based in the United States and China, with minimal independent oversight and no binding international governance framework. The computational infrastructure required to train frontier AI models — the hardware, the data centres, the electricity, and the engineering talent — is expensive enough to be accessible only to organisations with enormous capital resources. This creates a structural concentration of power that is not the result of any deliberate policy choice but of the economics of the technology. The consequence is that the value choices embedded in the most widely deployed AI systems — what to optimise for, whose data to use, whose interests to serve — reflect the priorities of those organisations. Billions of people whose lives are affected by these systems had no voice in those choices. Cathy O'Neil's analysis in Weapons of Math Destruction (2016) established the structural pattern: powerful algorithmic systems tend to be opaque, operate at scale, and cause damage that feeds invisibly back into their own inputs — a combination that makes them resistant to accountability without deliberate structural intervention.

Should AI be regulated?

Yes — the more productive question is what effective regulation looks like and who has the capacity to deliver it. The case for regulation rests on the same basic principle that underlies the regulation of pharmaceuticals, financial products, and aviation: that the people and institutions deploying powerful systems should be accountable for their consequences. The technology industry frequently argues that AI is too complex and fast-moving to be effectively regulated. This argument has empirical weight in specific cases — regulatory frameworks do lag behind deployment — but it does not follow that the absence of regulation is preferable. The EU AI Act, which entered into force in 2024, is the most comprehensive framework currently in force. It classifies AI systems by risk level and imposes proportionate requirements, including mandatory transparency and human oversight for high-risk applications. Its limitations — enforcement that depends on national authorities with uneven technical capacity, and conformity assessment that relies substantially on developer self-assessment — illustrate the genuine difficulty of the problem rather than the case against regulation. Effective AI regulation requires mandatory transparency, independent auditing, and practical redress for those harmed — not just disclosure requirements that produce paperwork without accountability.

Who decides what values AI should have?

Currently, the values embedded in AI systems are decided almost entirely by the organisations building and deploying them — their commercial incentives, cultural assumptions, and risk tolerances. Every AI system makes choices about what to optimise for, whose data to use, and what counts as a good outcome. These are not neutral technical decisions. They are value decisions with political consequences. A hiring algorithm encodes a view of what qualifications and signals predict job performance. A content moderation system encodes a view of what speech is acceptable. A credit-scoring model encodes a view of what behaviours indicate financial risk. Treating these choices as mathematical objectivity — as O'Neil argues in Weapons of Math Destruction (2016) — is itself a political act that removes consequential governance decisions from democratic debate. In a democratic society, decisions that affect how millions of people are hired, lent to, surveilled, and informed should be made through processes accountable to the people they affect — not solely to the shareholders and regulators of the companies that build the systems.

1. O'Neil, Cathy. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. 2016.