Explaining Behavior 4: Testability
There are worse things than being wrong.
Part 4 in a series. Previously:
Suppose we have a theory that explains some thing or other. The first thing any sensible person wants to know is whether the explanation is correct. After all, many explanations turn out to be wrong. Childbed fever isn’t really caused by overcrowding or miasmas, and economic booms don’t really cause spikes in the suicide rate. And having the wrong explanation can lead to ineffective or counterproductive action. Therefore, one of the most important qualities of any explanation is whether it is correct.
But for us to tell if a theory is correct, we have to be able to put it to the test. And so we might say that even more important than being correct is being testable.
What exactly does it mean for a theory to be testable? The short answer is that the theory makes clear predictions that could, in principle, be contradicted by the evidence. That is, if it were wrong, we could find out.
The long answer involves delving into the logic of general theory and the work of philosopher Karl Popper. We begin with something that philosophers of science call the “problem of demarcation” — the question of how to distinguish science from non-science.
What Makes Science Scientific?
Four hundred years ago, as the Scientific Revolution was just getting started, English philosopher Francis Bacon laid out his thoughts on the scientific method. Bacon’s view was a brand of empiricism, a philosophy of knowledge that privileges the role of observation in generating valid beliefs. In the scientific method, said Bacon, one carefully observed the particulars, and from these reasoned to conclusions about the laws of nature. If our observations were careful and our reasoning was sound, we would make accurate inferences.
A few hundred years later and modern science has become quite impressive indeed. Physicists could predict the paths of the planets with mathematical precision and had even used their theories to correctly predict the existence of a new planet, dubbed Neptune. Perhaps more relevant to the affairs of men, one could also use the equations of physical theory to predict the paths of artillery shells, and to aim them accurately even at great distance. Likewise, one could calculate with certainty how many units of one chemical substance would be yielded by the combination of other substances. The men blasted and gassed in The Great War might not have appreciated these fruits of scientific knowledge, but their doom certainly showed its power. Modern scientific theories clearly had a degree of accuracy and precision far beyond either ancient philosophy or traditional folk knowledge, and its applications had remade the world.
In this new world Bacon’s ideas were influential on a philosophical movement known as logical positivism, which had its heyday in the 1920s and 1930s. Its main exponents were a group of philosophers called the Vienna Circle, which included Moritz Schlick, Rudolph Carnap, and Carl Hempel.
Positivism is one of those pesky words with several meanings. In the field of sociology, for example, it might be used to describe any scientific approach to the field -- usually by someone who thinks the study of humans can’t be scientific. And even those who think a science of human behavior is possible and desirable might use the term for work they view as having a narrow or misguided view of science. In its broadest sense positivism is empiricism, and the logical positivism of the Vienna Circle might be classed as a kind of radical empiricism -- a view few people now labeled positivist would actually endorse.
Logical positivism held that all valid knowledge must be verified by direct empirical observation. Actually, it went further: All meaningful statements were verifiable, and anything that couldn’t be verified by observation was meaningless. But let’s focus on the weaker claim: The notion that theories of nature are valid if we can verify them. For the positivists thought that science was great, and, like Bacon, they thought that what made it great was its foundation in empirical observation. The idea was that the laws of physics, chemistry, and so forth, being the broadest statements we could make about the universe, were themselves composed of lesser statements. One could follow the chain of composition down from the most sweeping laws (of gravity, inertia, etc.) to mere empirical regularities (such as the orbits of the planets) to the particular individual instances from which we infer those regularities (lights moving through the night sky). At the bottom one finds a bedrock of statements (protocol statements) that each describe a concrete empirical observation. The truth of these observation-statements is what verifies the truth of the ever-grander ideas built upon them.
In this way the logical positivists thought scientific theories are derived from facts, and the validity of science lay in reasoning from these facts to the laws of the universe. This was how science worked, this what made scientific theories so accurate, and this is what made the difference between science and philosophy or theology or ideology or superstition.
But if science depends on correctly reasoning from facts to laws, one faces the question of how exactly to do this: By what rules of logic does one go from seeing an apple falling to the notion that all objects in the universe exert an unseen force of attraction in proportion to their mass? Indeed, how does one even go from the particular to the general? Logical positivism thus ran up against the problem of induction.
The Problem of Induction
There are two major kinds of reasoning: deductive and inductive. In deductive reasoning, the truth of our premises guarantees the truth of our conclusion. Thus the classical syllogism: All men are mortal; Socrates is a man; therefore, Socrates is mortal. This is, as we have seen, the type of reasoning used to go from a general proposition or principle (like “all men are mortal”) to a specific implication (like “Socrates is mortal”). This is logic in the strict sense, where information in the premises is expressed in the conclusion. It thus has the character of a math problem: It is either right or wrong, and one can demonstrate that a particular conclusion follows from particular premises.
Inductive reasoning is different. In inductive reasoning, the premises are reasons to believe the conclusion – but they don’t guarantee it. This is the type of reasoning we use to go from specific to general. Reverse the syllogism above: Socrates is mortal; Socrates is a man; therefore, all men are mortal. The conclusion is not a logical implication of the premises in the same way it is with deductive logic. Knowing that one guy is mortal does not necessarily mean that all guys are mortal. What if this one guy was an exception? Our generalization goes beyond the facts on which it is based; it adds new information.
Here lies the problem of induction, as formulated long ago by philosopher David Hume in his Enquiry Concerning Human Understanding. That the sun rose in the East today doesn’t tell you that it will rise there again tomorrow. Neither does the fact that it rose every day for all of your life, or for the lives of your father and father’s father -- the past is no guarantee of the future. Suppose that later today a rogue planet zipping through the solar system collides with Earth and shatters it to rubble: There will be no sunrise tomorrow. The history of sunrises so far does not preclude this possibility. Likewise, not only does Socrates being moral not tell us that all men are mortal: No matter how many mortal men we can list, it doesn’t logically follow that all men, present and future, share his condition.
Every inductive inference involves a leap. It might be sensible and practical to make it, but it is not actually logical. And the truth of the premises – the observations, the evidence – can never guarantee the truth of the conclusion.
We don’t need to rely on hypotheticals. There’s a reason the term black swan has become a metaphor for unexpected events. For centuries, a European scholar could have taken “All swans are white” as being as obviously true as “all men are mortal,” for indeed all Europeans swans are white. The Roman poet Juvenal even made a sarcastic remark about a good woman being as rare as a black swan – meaning that one didn’t exist. Then came the European discovery of Australia, home of black swans. The generalization, borne out and “verified” by God-knows-how-many observations over the millennia, no longer held up.
The problem applies to the general laws of science, with the added complication that a scientific theory is a bit more than just a generalization. The laws of physics, for instance, order our observations by invoking new concepts like gravity or electricity, recasting our individual observations as examples of these forces acting on the world. Perhaps we can be right about the observations, but wrong about these forces, or about the details of their working. The same exact facts have been the basis for competing theories, each explaining them in different ways. Was one of them just doing induction incorrectly? In any case, all the competing theories would contain information that goes beyond just the facts they explain, and none would be a mere logical result of adding up those facts.
In short: The facts a theory explains are not enough to prove the theory.
Perhaps the best historical example of this is Newton’s theories of gravity and motion. The Newtonian system had been so accurate and precise that by the end of the 19th century there were those who would argue it was, indeed, proven. Some physicists of the time even advised against going into the field, which they thought was nearing completion. And then came the revolutionary work of Einstein, and a new theoretical system that largely displaced the old one. Despite all the “confirmations” of his ideas over the centuries, it turned out that space, time, and gravity were not as Newton thought.
Testability and Falsifiability
Philosopher Karl Popper was supremely concerned with the distinction between science and pseudoscience, and also firmly opposed to the position of the logical positivists. He first laid out his argument in the book The Logic of Scientific Discovery (perhaps better translated as The Logic of Scientific Research).
He begins by noting that his job as a philosopher is to examine the logic of science, not the psychology of science. Clearly people do constantly reason from particular to general and draw inferences based on limited information. Human life and culture depend on this. Quite often the results are good enough for us to get by, and sometimes they’re even brilliant scientific theories. If one wants to call this psychological process induction, by all means go ahead, but don’t confuse it with any kind of logical method. And absent any logical method, any clear way of doing it such that the evidence leads to one and only one correct conclusion, induction doesn’t distinguish science from pseudoscience any more than it separates obsolete science from current science.
Popper agrees with the logical positivists that the empirical evidence plays a crucial role in science. Scientific ideas must be tested against the evidence. The mistake is in thinking that the test involves verifying a theory. General theories can never be verified – their testability lay in the fact that they can be falsified.
Consider a logical deduction: All swans are white; this is a swan; therefore, it is white. As we’ve seen, this only works on one direction: Knowing the premises are right tells me the conclusion is right, while knowing the conclusion is right does not tell me the premises are right. But what if the conclusion is wrong? What if this swan is actually black? If the logic is valid and the conclusion is false, it can only be because one or more of the premises were false. The black swan can teach us that we were wrong about all swans being white.
Note the asymmetry: A million white swans couldn’t tell us that this premise was true, but one black swan could tell us it is not.
Hence Popper argues that while a scientific theory cannot, logically, be proven true, it can, logically, be proven false. And for Popper this is the key to distinguishing scientific theory from ideology or pseudoscience is whether it is falsifiable.
What Makes a Theory Falsifiable?
To be falsifiable, a theory has to deal with empirical matters: Things that can, at least in principle, be observed through senses or instruments. Metaphysical ideas and value judgments are not falsifiable because they don’t deal with matters of empirical fact and can be neither verified nor falsified by the facts. (Note that this doesn’t make them unimportant, and that for his part Popper scoffs at the idea only empirical statements are “meaningful.”)
Not all empirical ideas are falsifiable, though. For example, the idea must not be a tautology. The writer Moliere once lampooned physicians by depicting them explaining why smoking opium makes people sleepy: It makes one sleepy because of its dormitive virtue. Dormitive virtue just means ability to make one sleepy, so the whole thing is circular: It makes you sleepy because it makes you sleepy. It’s as unfalsifiable as it is uninformative.
To be falsifiable a theory needs to say something definite about what should or should not be the case. The content of the idea must be such that if it were wrong, we would know how to find out. Any truly falsifiable theory can be recast as a statement that forbids something: If this theory is true, X should not occur.
Furthermore, falsifiability, and thus testability, is a matter of degree. The more definite and precise a theory is in its predictions, the more possible outcomes it forbids, the more testable it is. In Popper’s words, a good theory must “stick its neck out.”
An idea that is sufficiently vague or open-ended can be difficult or impossible to falsify. Consider astrology. The weekly newspaper horoscope works by making predictions that are sufficiently broad that it would be difficult for them not to be true of whoever was reading, regardless of what star they were born under. “During this week you will find yourself questioning aspects of a relationship to someone you know.” This is about as likely to be true if astrology is wrong as if it is right.
Einstein’s theory of general relatively, on the other hand, did not just predict that we could observe starlight being bent by the gravity of the sun: It predicted the precise amount of the displacement, down to the second decimal place. If it were displaced by even a slightly different amount, it would mean the theory wasn’t quite correct and thus in need of revision. Indeed, this was actually the case for an earlier version of theory:
Einstein's 1911 paper predicted deflection of star light on the limb of the Sun to be 0.83 seconds of arc and encouraged astronomers to test this prediction by observing stars near the Sun during a solar eclipse. It is fortunate for Einstein that the weather precluded results by Perrine in 1912 and Perrine, Freundlich, and Campbell in 1914. If results had been obtained they may have disproved the 1911 prediction setting back Einstein's reputation. In any case, Einstein corrected his prediction in his 1915 paper on General Relativity to 1.75 seconds of arc for a star on the limb.
The theories of sociology, psychology, and economics lack such precision, and so are generally less testable. But some are more falsifiable than others.
Conceptual theories are not testable – though since they’re not explanations in the first place, one should not expect them to be. Only explanations stand a chance of being falsifiable, for only they state propositions that might turn out to be right or wrong.
Sociologist Donald Black cites one of his own ideas as an example:
In The Behavior of Law, I thus propose that law varies directly with relational distance. One implication is that cases (such as criminal homicides) involving strangers attract more law (such as more punishment) than cases involving intimates (such as spouses, lovers, or friends). And they do. We can readily observe, for example, that the probability of capital punishment for homicide in modern America is far greater in cases between strangers than in cases between intimates. Because another pattern is, in principle, also possible (such as greater severity in cases involving intimates than strangers), the formulation qualifies as testable. It could conceivably be proven wrong.
Other social scientific claims are less falsifiable. Recall sociologist George Homans’s distinction between an explanatory proposition and an orienting statement. While a good proposition tells us at least the direction of the relationship between one quantity and another, the orienting statement only tell us that one thing will have some unspecified influence on another. He cites the Marxian idea that “the economic base of a society influences other aspects of that society” as something that tell us very little about how a given change in the economy will affect anything else. In Popper’s terms it is also nigh unfalsifiable, as it forbids very little: It is consistent with any effect all and forbids only that there is absolutely zero influence of economy on the rest of society.
Not Even Wrong
The human sciences have produced a number of purported explanations that are difficult or impossible to falsify. They often amount to a system of vague concepts and orienting statements that allow much room for interpretation. Popper himself had some of these in mind when thinking about the distinction between science and pseudoscience. He mentions the example of Freudian psychology, which was quite popular in the Vienna of his youth.
Freudians, he notes, seemed supremely confident that every psychological observation confirmed their theory. Look at anything people say (or don’t say) and anything they do (or don’t do) and you will see a Freudian explanation. And this, to Popper, was the problem. Freudians could, without batting an eyelash, easily explain completely opposite outcomes. A man pushed a child from a bridge to drown in a river – ah, he did it because he was suffering from a repressed Oedipal complex! But suppose the man instead saw a child drowning and dived in to save him: Ah, he did it because he achieved sublimation of his repressed complex! Freudian theory seemed to serve less as a tool for making testable predictions than for making after-the-fact interpretations, and its lack of specificity meant that any evidence at all could be interpreted as consistent with the theory.
Freudian theory was unfalsifiable: No conceivable evidence would ever contradict it, and so force a revision. Which means evidence was irrelevant to it. Which means in turn that Freudianism was not really a scientific theory.
Even of one does not follow Popper in defining all unfalsifiable ideas out of science – I wouldn’t necessarily call a useful system of classification “unscientific” just because classification isn’t testable – it is hard to argue that a falsifiable explanation isn’t far superior to an unfalsifiable one. Indeed, from a scientific point of view it is better for a theory to be wrong than to be untestable. As Donald Black puts it:
To be wrong is better because at least it demonstrates what reality is not -- it eliminates something -- which is better than demonstrating nothing at all. A wrong theory might also inspire a fruitful reformulation. Understandably, therefore, one of the harshest criticisms eminent physicist Wolfgang Pauli might direct at a colleague's theory was that it was "not even wrong.”
True Believers
Some ideas are unfalsifiable due to their content: The idea is not empirical, or it is a definition or tautology, or it is too vague. Others start their lives falsifiable in principle but are then rendered unfalsifiable in practice because the true believers refuse to countenance any falsifying evidence.
Popper sees Marxism as an example of this. The intellectual circles of early twentieth-century Vienna were as full of Marxists as they were of Freudians, and they had the same unshakable faith that every bit of news confirmed their beliefs. But Popper is willing to grant that some of Marx’s ideas were, in principle, falsifiable, and had indeed been falsified.
The living standards of the working class had not continually declined in a steady process of immiseration – standards of living had risen, and the common man was better fed than ever. The class system had not steadily simplified into a mass of destitute, unskilled laborers working on behalf of a small minority of owners – if anything it had complexified, as the middle class grew and the development of joint-stock corporations allowed large numbers of people to own shares of businesses. The global socialist revolution did not come – instead, we had a national revolution in Russia that resulted in anything but a utopia of equality. And the law did not only take the side of capital in labor disputes – it increasingly sided with workers, and mandated limits on working hours, required health and safety conditions, and set minimum wages.
Such facts can provoke two different sorts of reaction. Some might acknowledge that these developments clash with the theory as previously interpreted – that they contradicted expectations. They might reject the framework as a whole, or they might try to salvage some parts of it by revising others. For example, they may decide capital ownership isn’t everything, and that while class interests matter, states and their rulers are autonomous actors with their own interests apart from those of any economic class.
Others, though, will explain away all the apparent contradictions. As with Freudianism, the problem comes from that fact that a bright devotee can explain completely opposite outcomes with equal facility. The government refuses to enact a law to limit working hours – ah, just as Marx’s theory would predict, the state sides with capital every time! Then a decade later, the government does enact a law to limit working hours – ah, the devious cleverness of the state, protecting capital by throwing the workers a few crumbs to put off the revolution for another year. The state sides with capital every time!
Whether a statement is falsifiable in principle is a property of the idea itself; something that can be critiqued on logical grounds. Whether an idea is falsifiable in practice depends on whether people are willing to allow it to be falsified, and this is a matter for methodology – conventions about what sort of conduct is acceptable in any given debate or field. There are even some good grounds for not throwing out a theory at the first sign of trouble – perhaps the empirical observations are mistaken, after all. Was your one black swan really black? Was it really a swan? There are no privileged protocol statements in Popper’s view -- it’s testability all the way down -- and no simple rule of logic lets scientists off the hook from weighing options and making judgments.
But if you want to play the game of scientific theory, you must be able to describe at least some possible observations that would contradict your ideas. And we shouldn’t count evidence as supporting a theory unless it is from a test that the theory could conceivably have failed.
Next installment: Part 5: Handling Anomalies
For a classroom exercise on testability, see: Part 4.5: Teaching Testability (Paid only)
References
Black, Donald. 1995. “The Epistemology of Pure Sociology.” Law and Social Inquiry 20: 829-70.
Homans, George Casper. 1967. The Nature of Social Science. New York: Harcourt, Brace, and World.
Hume, David. [1748] 1955. An Enquiry Concerning Human Understanding, edited by Charles W. Hendel. Indianapolis: Bobbs-Merril Company, Inc.
Kolakowski, Leszek. 1966. The Alienation of Reason: A History of Positivist Thought. Garden City: Doubleday.
Popper, Karl R. [1934] 2002. The Logic of Scientific Discovery. London: Routledge Classics.
Popper, Karl R. 1957. The Poverty of Historicism. New York: Harper and Row.
Popper, Karl R. 1963. Conjectures and Refutations: The Growth of Scientific Knowledge. New York: Routledge.
Popper, Karl R. 1976. Unended Quest: An Intellectual Biography. La Salle: Open Court Publishing.
Popper, Karl R. 1994. The Myth of the Framework: In Defense of Science and Rationality, edited by M. A. Notturno. London: Routledge.