The Disappointing Power of Statistics

When our capability outstrips our understanding

It must have been about ten years ago that I started supervising MSc dissertations in data science. The variety of projects was always striking, students were applying machine learning techniques to a huge range of problems, often producing impressive results expressed in a wide variety of of metrics. Unfortunately those metrics frequently outran the student’s understanding of what those metrics meant - yes, the Recall was 0.06 higher, but what did that mean, what was the real world impact? The more projects I supervised the more I realised that the issue went beyond the outcomes, and it wasn’t just the students. The accuracy figures might have been real; but neither of us really understood how they were being achieved.

I described this to a colleague as the disappointing power of statistics. With traditional rule-based systems, you had been forced to build models that emerged from careful understanding of the problem. You had to articulate the structure of the domain before you could write code that operated on it, and the act of articulation was where most of the learning happened. Data-driven methods broke that requirement. You could now process large datasets with techniques that surfaced patterns and produced solutions without any of that prior knowledge work having been done. The result was capability without the comprehension that traditionally came hand-in-hand.

What I see in AI today is the same problem, accelerated and scaled to a level my MSc students could not have imagined ten years ago. The basic questions I started raising in supervisions have become questions about the constitution of knowledge itself.

Lessons for all

Across every area of computing, and increasingly beyond it, the same issue keeps surfacing: what happens when you replace the system built on rules with one built on learning? For decades the answer was reassuring. Rule-based systems were comprehensible; learned systems were narrow and brittle; performance favoured the comprehensible. That truce has now broken. Across language, vision, scientific prediction, code generation and the design of physical artefacts, methods that learn from data have overtaken methods that encode our understanding, and the gap continues to widen.

This is what Rich Sutton crystallised in his essay The Bitter Lesson: general methods leveraging computation tend to outperform approaches that build in human knowledge, and the historical record of artificial intelligence research suggests this pattern is the rule rather than the exception. Sutton wrote the lesson for AI researchers. But its implications have leaked into every domain where data and computation can be brought to bear, and large language models are now its sharpest contemporary expression. They produce behaviour that looks like reasoning without any transparent account of how that reasoning was performed.

A short history of knowledge blindness

It is tempting to feel that this is unprecedented but that is not the case. Capability has outrun understanding before, repeatedly, throughout the history of technology.

Steam engines drove the Industrial Revolution before we worked out the principles of thermodynamics. Vaccination preceded immunology by more than a century. Fermentation predated microbiology by millennia. The aerofoil flew before circulation theory could account for it (and the popular Bernoulli explanation continues to be wrong in most textbooks). Henry Petroski’s argued that engineering, as a discipline, frequently advances ahead of the science that later explains it; and that this is not a defect of the field but a feature of how technical knowledge moves.

The condition these examples share is what I would call explanatory opacity: an artefact functions reliably whilst the mechanism producing that function resists principled account. Aerofoils lifted regardless of whether their builders understood why. The disappointment of statistics is, in this respect, an old disappointment written in new mathematics.

Locating the present moment within this longer history tempers my reflexive discomfort about the black boxes of AI. The state of using something we do not understand is not new, and it is not necessarily a final position. Theory does sometimes catch up. The question is whether this is another such lag — uncomfortable, but eventually closed — or whether the AI case is structurally different in some respect that prevents the historical pattern from running its usual course.

What kind of understanding are we giving up?

We should start by being precise about the kind of understanding we have inherited, and the kind we appear to be setting aside.

Aristotle distinguished four kinds of cause, four ways of answering why. Material cause: what something is made of. Formal cause: the form or structure that makes it the thing it is. Efficient cause: the agent or process that produced it. Final cause: the purpose or end towards which it tends. A complete account, on Aristotle’s view, requires all four. Modern science retained structure as something to describe but stopped treating form and purpose as explanations in their own right, concentrating instead on efficient causes and the material elements on which they act. Machine learning narrows further. It gives us efficient cause with unprecedented power (the network reliably produces the output) yet it withdraws even the structural description that science still routinely provides. We can describe what the system does and inspect every layer of the mechanism by which it does it, yet we cannot map that mechanism onto a principled account of what kind of computation is taking place.

The debate between scientific realism and instrumentalism is one of the older arguments in the Philosophy of Science. Realists hold that successful theories aim at, and approximate, truth about an underlying reality. Instrumentalists hold that theories are useful predictors of phenomena, and need not be true descriptions of unobservable structures. Machine learning operates as the most radical instrumentalist methodology yet devised. To embrace it, even pragmatically, is in a quiet way to set aside the realist project that has driven Western science since Galileo.

Why this becomes a resurgent question

The mechanism by which statistics keeps winning is now apparent. Compute grows exponentially. The rate at which humans can articulate domain expertise grows roughly linearly. Over time the former overtakes the latter in every domain where it can be applied, and human knowledge encoding becomes a ceiling rather than a foundation. This is the working economic reality of contemporary AI.

The interesting question, however, is not how we got here. It is why this becomes an epistemological problem now, in a way that aerofoils and aspirin did not. The asymmetry of scaling alone does not explain it. We have used opaque artefacts before without panicking about them. Something else must be making the disappointment feel sharper this time.

The aerofoil is opaque, but the aerofoil does not think. We bring our own, comparatively transparent, reasoning to bear on how to use it. The opacity sits in the artefact; our cognition handles everything else. Aspirin, similarly, does not decide whether to take aspirin.

What changes with AI is that the opaque artefact’s function is itself the production of reasoning. The opacity is no longer isolated to the thing. It propagates into the cognitive operations we delegate to that thing - including the very faculties by which we would normally close knowledge gaps. We end up using the opaque engine to perform the operation by which we would ordinarily verify it. The result is an Ouroboros: the meta-tool devours its own tail.

This recursion is what makes the AI case structurally different from the older examples. With aerofoils, we used science to eventually understand them; the faculty we used to do that work was held in transparent cognition outside the opaque artefact. With AI, the faculty we would normally deploy to close the gap is precisely what we are increasingly delegating to a system we do not understand. The same kind of recursion does not have a clean analogue in the history of engineering.

What this produces

The downstream consequence of recursive opacity, if it scales, is the accumulation of a new category of artefact in the knowledge system. Call them opaque epistemic objects: proofs, designs, hypotheses, code, explanations and decisions, generated by AI, that function as if rationally derived but cannot be traced back to first principles by their users.

The first examples have already arrived. AlphaFold-predicted protein structures sit in databases and are treated, for working purposes, as facts. Material designs produced by machine learning enter manufacturing pipelines. Code produced by language models is integrated into deployed systems at a scale that no human review can fully match. None of these objects carries the traditional chain of justification that scientific or engineering knowledge has historically required.

The older epistemic architecture under which knowledge accumulated assumed that anything taken to be known could in principle be unpacked into reasons. Opaque epistemic objects break that assumption. As they propagate, the knowledge system fills with items whose grounds, if they exist at all, exist only inside systems whose reasoning we cannot trace.

This is not what most people mean when they say “we now know” something.

It may be what most people are going to have to mean.

What Explainable AI does, and does not, address

The most extensively developed response to all this so far is the field of Explainable AI. XAI has its roots in research on algorithmic scrutability and accountability in the mid-2010s, and was accelerated by regulatory pressure, most prominently GDPR’s invocation of a “right to explanation”. It has produced a substantial body of techniques: local post-hoc explanations, attention visualisations, counterfactual explanations, and saliency maps. On its own terms, the field has made real progress.

But it is worth being clear about what XAI does and does not address. The mainstream of the field operates on a set of accountability demands. Why did the system produce this output for this individual? Such demands are satisfied by local, post-hoc justifications, useful in regulatory, legal and operational settings. The question this essay has been concerned with is different in kind. It is not “why did the system reach this decision” but “what kind of knowledge system do we end up with when capability scales beyond principled understanding?”. XAI takes the opaque substrate as given and works on individual outputs within it; the disappointing power of statistics is concerned with the constitution of the substrate itself. Improvements at the accountability level, however welcome, do not by themselves discharge the upstream epistemic concern. Loan denials may be made legible to applicants whilst molecular biology gains no principled understanding of how AlphaFold predicts folds.

There is a partial exception. The smaller strand in XAI of mechanistic interpretability (the work that asks what concepts and circuits exist inside trained models) is genuinely epistemic in spirit. It is continuous with the historical pattern of theory eventually catching up with engineering, the move I would have most expected the field to make. Whether it can scale to systems of present and future complexity is an open question. It is the part of XAI that addresses our question, but not the part that gets most of the institutional attention.

Cyborg Scientists

There is a deeper tension worth mentioning. The same systems whose internal opacity I have been describing are also the systems extending our capacity for thought. Scientists, academics and philosophers are using AI tools to think through problems in ways that genuinely augment them, in the Engelbart sense of the term. Some of the best contemporary work on AI interfaces aims precisely at this: making AI a partner that amplifies cognition rather than degrading it.

So we are living through two things at once. AI is letting us think more powerfully, creating a generation of cyborg scientists, and AI that is letting us build things without understanding them. The same technology drives both, sometimes in the same workflow. A researcher might use a language model to clarify her thinking about a result whose foundation she cannot fully derive, and then use it to draft the paper in which that result will appear. The augmentation and the disappointment are entangled, and they entangle by design rather than by accident.

I do not think this dissonance will be resolved with better tools, or better regulation, or better interpretability. The two conditions are produced by the same underlying capability, and they are likely to continue coexisting, each becoming sharper as the other does. That is uncomfortable, but it is the situation.

Living with loss

The intellectual habits now under challenge are deep ones. Our assumption is that to know is to be able to explain. Our expectation is that progress in capability must be paid for in coin of understanding. The conviction is that why and what stand or fall together. These commitments are inherited from a particular moment in the history of Western thought, and they have served us well. They may need to be renegotiated.

The MSc students I supervised could report that precision was rising, or that recall had crossed some threshold, but they could not say whether that movement meant anything in the world. But the shallowness ran deeper than the metrics. They could identify patterns in their data but could not explain why those patterns existed. I did not have those answers either. Alongside them, I felt the same gap, not as a supervisor holding back the answer, but as someone who had also been trained in methods that did not produce the kind of understanding that the work seemed to demand. The disappointment was not a defect of the students’ work, and it was not a defect of mine; rather, it was a structural feature of data science as a discipline. What is happening now is the same structural feature, propagating across science, engineering and culture at a scale that no individual supervision conversation could ever address.

That, I think, is what is at stake. Not whether machine learning works - it manifestly does - but what we lose, what we keep, and what we may need to invent when capability is decoupled from comprehension.