Defining Concepts

The concepts we find in our theoretical sciences do not of course spring into life fully grown. They have a prehistory: an often long period during which they are developed, contested and eventually, as the field of science studies puts it, ‘stabilised’. I’ll talk about this process of stabilising concepts later. For the moment let’s just think about mature concepts in what we take to be reasonably well-established theories. How are we to understand these theoretical concepts, how do we give meaning to them? I’m going to spend some time discussing this both because these are interesting questions in their own right and also because the long road we travel in thinking about these issues leads us to the conclusion I want to underline in this chapter – how important all the other pieces beyond theory and experiment are in the Meccano set of science, so much so that the very notion of theory does not make sense without them.

Some theoretical concepts have esoteric names and are unfamiliar to those who are not specialists in the field. In particle physics we learn about leptons, in low-temperature superconductivity about Cooper pairs. So, what is a lepton, what is a Cooper pair? Cell biologists study the cellular cytoskeleton, including actin filaments, microtubules and intermediate filaments. They examine the endoplasmic reticulum and the Golgi apparatus. Cancer biologists study proto-oncogenes and tumour suppressor genes. What are actin, microtubules and intermediate filaments? What are the endoplasmic reticulum and the Golgi apparatus? What are proto-oncogenes and tumour suppressor genes? Social psychologists study positioning theory. What is a position? Other theoretical concepts have familiar labels, especially in the social sciences, like democracy, socio-economic status and aggressive behaviour. But as these come – our everyday versions of them – these concepts are too vague and open-ended to play a role in proper science theory. The claims of science are meant to be clear and unambiguous. So, whether the concepts they employ have familiar or unfamiliar monikers, the concepts themselves need to be made clear and unambiguous.

The easy way to give meaning to a concept is simply to define it. For instance, ‘leptons are considered to be fundamental particles. They have a spin 1/2 and do not partake in strong interactions [interacting only via electromagnetic and weak forces]. As fundamental particles, some leptons are negatively charged.’⁷ That’s fine – sort of. The problem is that this definition is clear only if what it is to be a ‘fundamental particle’ is already clearly defined, along with ‘spin 1/2’, ‘strong interaction’, ‘weak force’, ‘electromagnetic force’ and ‘negatively charged’. Or consider ‘proto-oncogenes’. These are defined by the US National Cancer Institute as: ‘Gene[s] involved in normal cell growth. Mutations (changes) in a proto-oncogene may cause it to become an oncogene, which can cause the growth of cancer cells.’⁸ The clarity of this definition also depends on the clarity of other terms such as ‘normal cell growth’, ‘mutations’, ‘oncogenes’ and ‘cancer cells’.

What we see here is typical: one theoretical concept defined in terms of other theoretical concepts that are themselves in need of clear and precise definitions. When it comes to social science, this problem is often compounded by the fact that there may be multiple different definitions on offer, definitions that classify items in the world differently. Consider ‘democracy’. Definitions range from:

to

to definitions that provide more formal criteria. For instance, for a state to count as a democracy, its citizens must have:

1. 1. Effective participation

2. 2. Voting equality at the decisive stage

3. 3. Enlightened understanding

4. 4. Control of the agenda

5. 5. Inclusiveness.¹¹

Note that, just like the examples from the natural sciences, these too end up defining one theoretical concept in terms of other theoretical concepts. What, after all, is a ‘competitive political system’, or ‘control of the agenda’, or ‘public participation in political decisions’ – what even is a ‘political decision’?

Can we ever break out of this theory circle, to define theoretical concepts in less esoteric terms? That was the hope in the heyday of operationalism, which was championed by the American physicist P. W. Bridgman. Bridgman’s method for breaking out of the theoretical circle was to define each theoretical concept by an operation by which it is measured: ‘We mean by any concept nothing more than a set of operations; the concept is synonymous with the corresponding set of operations.’¹² Readers may be familiar with this kind of doctrine from behaviourism – sometimes called ‘rat psychology’ – which was dominant in psychology from the 1920s through the 1950s. B. F. Skinner is its most well-known advocate. Skinner maintained that psychology is the study of behaviour, not of some invisible ‘inner mind’, and he urged that all psychological concepts, like anger, revulsion or reflection, must be defined in behavioural terms. Note defined. Behaviour is not a reflection of psychological states or a clue to what someone else is experiencing. Behaving in particular ways is what it is to be angry, revulsed or reflective.

Behaviourism was widely abandoned in psychology because it just didn’t seem to work. We couldn’t articulate distinct kinds of behaviour that fitted with each different nuanced psychological concept that we regularly employ and would surely not want to abandon on the grounds that it must be a chimera since we can’t find a behavioural definition of it. Behaviourism also completely misses out on the ‘what-it-feels-like’ to be in these different states. So, gradually the claim that all psychological phenomena comprise behaviour lost dominance to the more everyday view that they are causally related – though, as I discuss in Chapter 2, in the section ‘Physicalism and Materialism’, objections to the genuine reality of the inner mind and its states remain alive in the programmes to reduce mental states to states of the brain.

Operationalism more widely suffers from two general problems. First, every different measurement procedure introduces a new concept even if, on the more usual way of thinking about it, these procedures measure the same concept but in different ways. Then it seems nature must be littered with huge numbers of principles to bind together the different concepts, all of which take the same value in the same circumstances and that we normally think of as all the same concept, just measured differently; principles like this: ‘Concept 1 and Concept 2 and Concept 3 and … Concept N always have the same value.’ Second, we generally make big efforts to defend the idea that our procedures are good for measuring what they are supposed to. As I noted, when I turn to discussing experiments I use the famous early twentieth-century oil drop experiments for which Robert Andrews Millikan won the Nobel Prize for measuring the charge of the electron. As you will see in the final section in this chapter, ‘You Can’t Build an Experiment without a Gigantic Meccano Set’, Millikan did not just assert that what he calculated from the reading on the voltmeter in his experiment was the charge of the electrons in his experiment. He argued for this claim, both with a theoretical model – you’ll see the core of this in Figure 1.8 – and also in a detailed description of the actual materials which his experiment employed to show that they could play the part required of them in securing an accurate and precise measure of the charge.

Figure 1.8 Millikan’s actual apparatus

Drawn by Adrian Harris especially for this book. Thanks Adrian!

Consider another example, this time from genetics. In his 1929 article on Heredity in the Encyclopaedia Britannica, British biologist J. B. S. Haldane provided what can be viewed as an operational definition of phenotype and genotype. He wrote that ‘[a] class of organisms whose members cannot be distinguished from one another by observation is called a phenotype; a class which can be distinguished from another by breeding tests is called a genotype’.¹³ Such definitions were given because genes or the genetic material had not yet been discovered. At the time, genes were accepted as some unobservable ‘units of heredity’ without any direct experimental handle on them. Hence, a quasi-operational definition was beneficial. Yet, today, with the advent of molecular genetics, which characterises genotype as the ‘variant forms of a gene that are carried by an organism’ and phenotype as the ‘observable physical properties of an organism’ – both of which accept a molecular characterisation of the genetic material in the form of deoxyribonucleic acid (DNA) and the theoretical models used to explain gene expression – we would consider such operational ‘definitions’ superficial, serving at best as indicators of genotype and phenotype but falling short of proper definition. This shows that establishing the proper definition of genotype and phenotype required a lot of material experimental work within some broad theoretical framework that operational definitions fail to appreciate.

Operationalism makes nonsense of these elaborate efforts to defend that our measurement procedures are up to the job of measuring the concept they are supposed to, since after all operationalism holds that the concept just is what is measured by those procedures.

Just looking to the procedures by which a concept is measured may, though, be too narrow a focus in attempting to break out of the spiral of definition of one theoretical concept by another by another by another. In the 1930s, 1940s and 1950s, philosophers – especially logical empiricists – tried valiantly to define theoretical concepts using purely observational terms. They were called logical empiricists because they argued that scientific claims should be made entirely explicit, for instance theories should be formulated formally as systems of axioms from which further claims could be deduced as theorems. The empiricist label was because they wanted science to be able to confirm each claim by empirical observation. What exactly is meant by observation was up for grabs – did it for instance mean observable with the naked eye or was it to allow the use of sophisticated instruments? Whichever way that is decided, what matters for breaking out of the circle of theoretical definition is that for each claim to be confirmable entirely by observation, each theoretical concept in that claim needs to have an observational correlate – some observational states that obtain if and only if the theoretical concept obtains. So, hurray, these observational correlates can serve as definitions for these concepts and the circle is broken for each concept.

The difficulty is that this programme failed miserably. There just seems to be no way to carry it off. As Carl Hempel, who is widely acknowledged as the leading philosopher of science of the time, concluded in a famous paper written originally in 1958: ‘[I]t is clear that theoretical formulations cannot be replaced by expressions in terms of observables only.’¹⁴ This continues to be true even if we become for more liberal, not looking just for correlates among features that can be observed but allowing that theoretical concepts be defined by any terms that we already grasp the meaning of, as Hempel suggested: ‘[W]e might qualify a theoretical expression as intelligible or significant if it has been adequately explained in terms which we consider as antecedently understood.’¹⁵ The result of these repeated failures is that philosophers settled for implicit definitions of theoretical concepts rather than explicit ones that defined them using concepts outside the theoretical circle. All that means is that we give in and take the meaning of a theoretical concept to be given by the axioms of the theory. As the Stanford Encyclopedia of Philosophy notes: ‘This idea has become almost constitutive of the very notion of a theoretical term in the philosophy of science.’¹⁶

The problem with this is that implicit definition leaves it open what our theories are talking about. That’s because no matter how detailed a theory is – how many axioms we add to its formalisation – there can never be enough detail to pick out uniquely what the theory is about. There will always be unintended or ‘non-standard’ interpretations for it. This follows from theorems in model theory in logic, but it is easy to get the gist of why by looking at a caricature example.

Consider the simple theory that has one axiom, an axiom familiar from school physics: F = ma. We think of this as telling us that the force on a material object is equal to its mass times its acceleration. But all we really know from the formula is that there are three quantities and the first is equal to the product of the other two. That is just as true of the area of a rectangle with respect to the length of its two sides. So F could mean area of a rectangle, m the length of the rectangle and a its width. Now let us add more detail. In a world where only gravitation acts we can add the law of gravity to our axiom set: F_G = GMm/r². This tells us what force a system with mass m will experience in the presence of another of mass M a distance r away, where G is the constant of gravity. From this we infer that in this gravity-only world, ma = F = GMm/r². But this works just as well for rectangles if we suppose that the rectangles come overlaid with kites, as in Figure 1.1.

Figure 1.1 Kites and rectangles instead of forces, masses and accelerations

Drawn by Lucy Charlton especially for this book. Thanks Lucy!

Just suppose G = 2, M = the area of the kite and r = _√d.

We can go on thickening the theory of course. As we do so, more and more alternative interpretations will get ruled out. Maybe it will no longer be possible to see the equations of classical mechanics as all about rectangles. It naturally helps in narrowing down the interpretation if some of the concepts that figure in the theory are among that nice collection of concepts that are antecedently understood so that what they refer to is nailed down. For instance, we may suppose that a in the formula F = ma refers to acceleration, which is a concept in our ordinary language vocabulary and we are clear just what it means. (But beware of even that. We take acceleration in the physics formula to be the rate of change of velocity with time (dv/dt) but in some medieval physics the acceleration of a falling body was understood as the rate of change of velocity with distance traversed (dv/dx).) Given this assumption, it gets harder – though not impossible – to interpret F as the area of a rectangle. Until each theoretical term is identified with something we antecedently understand, we cannot be sure of a unique interpretation. Without this, no set of axioms no matter how long can guarantee a unique interpretation. As the philosopher Hilary Putnam argued, you can’t fix what you are talking about just by talking more and more.

So, implicit definition of theoretical concepts by laying out the theoretical principles that are supposed to be true of the concepts doesn’t work to fix what those concepts refer to. Nor can we define them explicitly in terms of other concepts we already understand outside the confines of the theory.

Philosophers worried about this problem a lot. But I don’t know of any scientists who do. I think there’s a good reason for this. Theory isn’t just a set of claims that stands there to inform us of what the world is like – it’s not there just to describe the world. Theories are tools that we use to do things in the world. We use theories to build models and we use the models to make predictions about what will happen in the world and to design experiments and technologies and policies and measurement procedures that we then implement, jostling the world, picking up bits and changing them around, to learn more about the world and to try to make it more to our liking. These are what break us out of the spiral of theory defining theory defining theory. When we position our specially designed radar sensors at the ends of the court at Wimbledon, train these on the ball that Serena Williams serves and read out 122 miles per hour, we know we are dealing with velocity and if we calculate its time rate of change (so a = dv/dt) and associate that with the formula F = ma, then we can be sure that F = ma is not about the areas, lengths and widths of rectangles.

We must be careful, though, about how much all these successful interjections of theory into the world can buy for us. They serve to rule out unintended interpretations of our theoretical concepts, but they do not ensure that these theoretical concepts refer to things that are really there in the world as we conceive them in our theories. Just think about all those theories that we used in the past to make successful predictions about what will happen in the world and to change things, like phlogiston theory that was used successfully to produce breathable air, inflammable air, shiny metal and ‘calx’ (metallic ash).¹⁷

Stabilising Concepts

Scientific concepts do not, of course, appear full blown and fully formed out of nowhere. Rather, they are constructed within and by science and its surrounds, contested and over time stabilised. There are various ideas of how this happens. To the early twentieth-century bacteriologist and philosopher Ludwick Fleck, concepts were developed within a community of experts and spread outward to the general public. At the same time, the public reinforces the thinking of experts. As Fleck urged, scientists are ‘more or less dependent, whether consciously or subconsciously, upon “public opinion”’.¹⁸

By contrast, in the late 1970s the philosopher Bruno Latour and the sociologist Steve Woolgar argued that a fact is produced within the scientific community through persuasion – scientists must constantly convince one another that certain statements using those concepts should be treated as facts. The point at which a concept becomes stable, Latour and Woolgar write, is when a statement using the concept ‘rids itself of all determinants of place and time and of all reference to its producers and the production process’.¹⁹ The meaning of a concept becomes fixed when it does not seem to have been constructed in the first place. And of course, as time goes on and new things are learned and new ideas and influences arise, concepts can also destabilise and even eventually disappear.

Aside from the obvious social processes it takes to get a concept entrenched, the stabilisation of any one concept in science depends on many products of science, including not just other concepts. Concepts become stabilised in part because of how they are gradually interrelated with other pieces of knowledge and practice, especially those from fields of research other than the one that first developed the concept.

Take, for example, the concept of a ‘neuron type’. According to neuroscientists, there are not just different types of cells in the nervous system but different types of nerve cells in terms of their shape, electrical properties, gene expression and, perhaps most importantly, function. Although the concept of a neuron type is not new, since the early 2000s neuroscientists have developed materials and methods that support and constrain what the concept of a neuron type can amount to. They have used genetically modified animals and/or genetically modified viruses to express light-sensitive proteins in specific neuron types within the brains of mice. Upon delivering flashes of light, they could manipulate the activity of these neuron types, such as those that produce a particular neurotransmitter. Consequently, they could also explore which behaviours these types of neurons control.²⁰ In this case, products that were not concepts or theories, and specifically coming from genetics and virology, helped to stabilise the current concept of a neuron type. It is now well accepted that the genetic variability of neurons is important in understanding the structure and function of the brain.

Misusing Concepts

Since at least the Scientific Revolution on, it has been commonly – though definitely not universally – supposed that proper scientific theory is required to lay out exact relationships between cleanly delineated features of the world, as in physics and economics. When I come to talking about powers, tools and laws in Chapter 3, in the section ‘Where Is Physics Successful?’, you will see that this is not something I entirely subscribe to. But where it is required I have wanted to underline how important it is to characterise the concepts that theory employs to refer to those features carefully and unambiguously and to provide ways to measure them precisely and accurately.

These demands that concepts be well specified and cleanly measurable creates problems however. As one of the founding members of the Vienna Circle, Otto Neurath, noted, many concepts that we find useful and that science helps us learn more about are loose congestions of different ideas and criteria with vague boundaries, like socio-economic status, learning readiness or implicit bias. Much of the world that science studies does not lend itself to description by precisely, rigidly delimited theoretical concepts.

That means that theoretical concepts that are tightly constrained by a web of mathematical laws and by highly precise criteria for application may not be universally applicable but at best constrained to pockets of reality – and that attempting to apply them more widely can often lead us into trouble. To underline that this is no idle philosophical worry, I will illustrate it in detail with a well-known concept at the heart of social science – probability. The lesson to be drawn from the case of probability about concepts and what I shall term ‘small worlds’ hold widely across both the social and the natural sciences.

You will already be familiar with the fact that broad swathes of social science research are given over to establishing, analysing, generalising, theorising about and using statistical associations that are manipulated with the assumptions of probability theory.

This makes sense if probabilities can be attached to broad swathes of the phenomena that social science is meant to deal with. But can they? Here we face the same issue that you will meet when I discuss the assumption of universal determinism: is the social world really that orderly? Perhaps it is my failure to see the forest for the trees, but when I look at various studies across the social sciences, from psychology, sociology and political science to economics and public health, I often cannot see grounds for this assumption, I sometimes see good evidence against it and I also see places where it seems to be leading us astray, with respect both to the accumulation and the use of knowledge.

The popular view that the reliability of real science requires precision is reflected in the account I have been giving of the importance of clear concepts and measurement in producing scientific theory. If the social sciences are to aspire to the same reliability, surely they too must aspire to precision! For example, cost–benefit analysis purports to combine inputs and outputs by quantification which allows the computation of an unambiguously best outcome: the optimum. This appears to require very precise knowledge of causes and effects, for example that if you increase demand by this (precise) amount prices will go up by this (precise) amount. But everybody knows that you don’t know exactly what will happen. You are uncertain. Maybe 2 per cent, maybe 3.5 per cent. Not precise. Precision is to be re-established by the notion of the probability distribution. If you assign probabilities, as you do with horse-racing odds, to various outcomes, then you can maximise by calculating the expected outcome of this or that input, and you are back with what is called mathematical precision. This is more than assessing carefully in qualitative terms which outcome is more likely than that. It requires quantification to enable commensurability and allow calculation of a precise optimum.

This account says that you assign probabilities. For that to be legitimate, there have to be probabilities to assign. And the paradigms of tossing dice and drawing cards do a lot of work to reassure us that this is a fair assumption. Of course, in real life, where the social sciences operate, it may be hard to find probabilities. And you may not be very sure that you have got them right. But surely it is reasonable to ask of any input, ‘what are the odds that it will produce this result?’ and thereby produce usable probability distributions. And you should use them, it would be irrational not to use them, not just because they have the properties you need to carry out the manipulations you want but because, even if shaky, they are better than nothing, they are better than the alternatives. So this familiar trick is used in the social sciences to tidy up the messiness of uncertainty and turn it into a well-behaved set of numbers.

But what if there are some problems which do not allow this? What if there are some uncertainties which are not like ‘I am uncertain whether I will throw a 6’, uncertainties which are fundamental? This is a source of troubles for the social sciences that I have long worried about. But if this is an obstacle to successful social science and successful social decision-making, as I think it is, it is an obstacle we bring on ourselves. For it involves the misuse of the otherwise valuable tool, reasoning with probabilities. Probabilities are a genuine aspect of the social world – under some circumstances. But problems arise when we assume there are probabilities where there are none and then base our theories and our expectations on them.

Where there are none? Really? Isn’t there always some probability or other for an event? I think not. And that’s part of what makes trouble for social science. Probability is a superb tool, like a thirty-piece set of first-class chromium screwdrivers. But it may be that not all of our problems are loose screws.

What do social scientists say about these worries about the assumption that there are probabilities here, there and everywhere? One person who has defended them to me is the Oxford econometrician David Hendry. Hendry assumes that econometrics would ideally like to discover ‘the complete structure of the process that generates economic data and the values of all its parameters’.²¹ He seems to presuppose that this process is always appropriately modelled with a probability measure. In conversation he has supported this assumption with the remark that the sampling procedure can ensure that a probability measure is appropriate. I can make sense of this from the point of view of the statistician looking at samples from a particular population in order to infer characteristics of the population as a whole. But even in this case it makes a lot of difference how one thinks the sample has been drawn.

Consider a simple example. We have a pack of ten playing cards: four hearts, two diamonds, four spades. You draw one card at random. What is the conditional probability that the card drawn is a diamond, given it is red? Here there is an answer: 1/3. Next consider a more complicated procedure. First, we do two totally separate and independent flips of a fair coin. If the results are HH, you draw one card at random from the ‘pointed’ cards in my population (i.e. diamonds and spades), otherwise from the non-pointed cards. Again, it is entirely correct to assign a probability measure to the outcomes, and under that measure the conditional probability that a drawn card is a diamond given it is red is 1/4. But this depends on more than the sampling procedure – drawing one card at random. It depends on what is ‘sampled’.

To see how it does so, imagine I give the cards to the person sitting closest to me to order on their aesthetic preferences – put the card they find most attractive first, then the second most attractive next and so on; then I pick the top card. Now we have to argue the case, and I do not know any good reasons to suggest that this situation should be represented by a probability measure. What argument is there to show that there’ll always be some probability or other for the outcomes from a data-generating method like this, characterised by features that we have no idea how to assign a probability to – like the various possible preferences of the person who happens to be sitting next to me? To insist that every data-generating process, regardless of its features, is correctly described by some probability measure or other is just to assert the point at issue, not to defend it.

Not all economists take the view that Hendry seems to have been defending. The economists John Kay and Mervyn King express the same worries I have in their recent book, Radical Uncertainty: Decision-Making for an Unknowable Future, which takes issue with the attempts to use what they call ‘small-world’ methods for making decisions in real-life large-world settings. Kay and King never explain what they mean by this but they make clear that a small world contrasts with much of ‘the world as it really is’,²² full of uncertainty, ambiguity and vagueness. I have more to say about small worlds in Chapter 3, in the section ‘Where Is Physics Successful?’, but I think that you’ll be able to see the point here by thinking about their illustration, which involves a version of a television gameshow:

Philosophers have been worrying about this for a good while. The philosophical basis for it was nicely laid out by the philosopher Ian Hacking in his 1965 book, The Logic of Statistical Inference. Hacking taught that probabilities are characterised relative to what he called ‘chance set-ups’ – like drawing balls randomly from an urn with six white balls and four black balls or picking cards randomly from a fair deck – and they do not make sense otherwise.

Not every situation in the world can be characterised as a chance set-up. In fact, I don’t suppose that many situations that come naturally can be. Looking at my account and that of Kay and King, we can extract two key necessary features a situation should have in order to be classed as a chance set-up. One is that the outcomes and the process are fully specified. The other is that there are enough probabilities built in at the start to ensure that the probabilities you aim to calculate fall out logically. The slogan for this last is: no probabilities in, no probabilities out.

We can easily see how the two requirements are satisfied in the two cases I noted as exemplary chance set-ups. Consider the first case:

1. 1. The outcomes are: {the one card drawn is a heart, the one card drawn is a diamond, the one card drawn card is a spade}, where it is clearly meant to be supposed that hearts and diamonds are red and spades are black.

2. 2. The probability ‘in’ comes from the description that one card exactly is drawn ‘at random’ from a pack of ten cards with the frequencies of each suit as specified. ‘Random’ is a technical term. It does not mean just ‘drawn without thought’ or ‘arbitrarily’, It means: each card has an equal probability of being drawn. The fact that all hearts and diamonds are red and all spades are black guarantees certain conditional probabilities about the colour of a card given its suit: Prob (a card is red/the card is a diamond) = Prob (a card is red/the card is a heart) = Prob (a card is black/the card is a spade) = 1. These together are enough to fix the conditional probability we want: Prob (the card is a diamond/the card is red).

We can do the same with my second example.

Why is the game described by Kay and King not a chance set-up? Well, start with the very first step. You are invited to choose envelope one or choose envelope two. If you do either you are guaranteed some money – or you are so long as it is part of the defining characteristics of the game that what you are told is true. Let us suppose that is certain. Then it is a no-brainer to join the game. Well, yes – unless it is run by some dreadful organisation you don’t want to be associated with, you haven’t time, there’ll be publicity you don’t want, etc. The possibilities are endless and they fall in no orderly scheme. Perhaps we can use the ordinary English idiom of likely/unlikely for many, but there is no way to put a proper probability measure on them.

Suppose you join the game. Now you have to choose envelopes. How do you decide? You might decide to choose the envelope with the number of the age of your youngest child or the odd number because you associate ‘odd’ with ‘eccentric’ and you like eccentrics, etc. Still no genuine probabilities yet.

Now suppose you have indeed chosen an envelope – envelope one – and it contains $100. How do you decide to keep it versus switching? Knowing a little about rational choice theory, you might think in terms of expected gain and loss and try to maximise the ‘expected value’. And you might be happy to take the face values of the dollar amounts as appropriate measures of how much value those amounts provide for you, which you might not always do. (Suppose for instance you drew envelope one and the $100 seems a godsend – it will just allow you to pay your mortgage but half that definitely will not.) Let’s suppose that the game show claim that the contents of the second envelope are either half or twice that of the first can be trusted. In that case the possible outcomes of switching and of staying put are fixed too. But to calculate expected gains and losses you need probabilities.

So where do the probabilities come in? Kay and King say, ‘[t]he hidden assumption is that there is a 50–50 chance of gaining or losing irrespective of the amount in the envelope’. That’s not given in the information defining the situation. That comes about from the assumption that ‘you apply the Indifference Principle, and judge each of these outcomes equally likely’.²⁴ The Cambridge Press collection, The Theory of Decision under Uncertainty, tells us:

One conventional reason for this is that you are told, ‘when you don’t know anything that favours one outcome over another, it is rational to treat each possible outcome as equally likely’. But you are not told how to divide up the outcomes. Real-life problems seldom come with the outcomes pre-set. The Theory of Decision under Uncertainty gives this example:

This is not the problem in the envelope-switching game though. There the outcomes are clear. Keep the envelope you drew and take home the amount of money found in it. That we have supposed to be certain. If you switch envelopes, we may suppose you have not been lied to and you take home either twice or half that amount. The problem is with the principle of indifference. This instructs you to treat outcomes as equally probable when ‘there is no reason to view one as more likely than another’.

What King and Kay point out is that given the information supplied there is plenty of reason to think they are not equally probable. For instance, as they suggest, you may well have views about ‘[w]ho is putting the money in the envelope’ and ‘what … their financial resources [are]’. You may also have views about whether the game show likes to produce big winners or not, or when across their series of shows they like to do this and how far along in that process you are, or whether their audience share has dropped recently and they want a big win to entice back viewers or much else relevant.

Of course, you may not know which of these pieces of information you have are really reliable and you may not be sure which option they favour, if either, or what it amounts to all put together. I suppose in this case you might throw up your hands and say, ‘oh well, might as well flip a coin’. But we should hardly count this as a way to proceed that rationality dictates. We would certainly be shocked to learn that a jury did this in deciding on a verdict of guilty or not.

This, though, is all about invoking the principle of indifference and assigning a 50–50 probability to the two outcomes that you’d get from switching. Behind Kay and King’s worries, and mine, is a more serious problem: why think there are any probabilities applicable in this case? If there aren’t any, then surely it cannot be rational just to act as if there are, then assign them in some way or another and act accordingly.

Consider where these probabilities could come from. How for both envelopes would we get 50–50 chance of there being half or double that amount in the other? The most obvious starting idea I suppose is that the game show organiser chooses some amount of money they are prepared to start with – say $100 – and flips a coin to settle which envelope to put it in. Imagine that’s envelope one. Then they again flip a fair coin to decide whether to put $200 or $50 in envelope two. In this chance set-up, Prob ($50 in envelope 2) = Prob ($200 in envelope 2). But the reverse is not true. Whether there is $50 or $200 in envelope two, Prob ($100 in envelop 1) = 1. So, the odds of doubling versus halving are not equal for both envelopes, as was proposed for calculating expected values.

‘Ah’, one might object, ‘still we don’t know that’s the chance set-up we are in’. That’s part of the point of calling this a ‘decision under uncertainty’. Here we are not just uncertain what is in the other envelope, but we haven’t a clue how that was decided, nor how it was decided to put $100 in envelope one to begin with. Maybe they decided to put $50 in envelope two and then flipped a coin for envelope one’s contents, or maybe they were short of money so only had $25 to put there. In these circumstances, whether we act as if we don’t know anything else and apply the principle of indifference or whether we look at all our information and come up with some different numbers, these just represent our subjective probability. And here I mean purely subjective, not anything based in facts we know about the world. It is just an arbitrary number we decide to use so we can bring the apparatus of expected loss and gain to bear to guide our choice.

Now, there’s a very great deal written back and forth about this: what is the rational way to make decisions when we know very little? There’s a whole field devoted to it, echoed in the title of Kay and King’s book: decision-making under uncertainty. But that’s not the issue I am discussing. Rather, here I am exploring reasons why the social sciences may not be so good at prediction as we might hope, nor at establishing detailed theories on which there is widespread agreement, nor at devising stable explanations of social phenomena – ones that stick around. So here the issue is whether part of the reason for this could be that social science predictions, explanations and theories are often based on models that presuppose that there are probabilities where in fact there are none.

I’ll illustrate this with a case I’ve worked on recently – the idea of an ‘effect size’ of a treatment or a social intervention. Effect size is big business in social science right now, as witnessed in these remarks:

• Nearly every discussion about educational improvement today refers to ‘effect sizes’. Education organizations compare effect sizes in planning professional learning programs. District and school leaders consider effect sizes when selecting the strategies to include in school improvement initiatives. Even classroom teachers evaluate effect sizes in deciding what practices will be most effective in helping their students learn.²⁷

• Effect sizes are the currency of psychological research.²⁸

• Increasing emphasis has been placed on the use of effect size reporting in the analysis of social science data.²⁹

The effect size of a cause on a given outcome is an average in the numerator plus a denominator that is supposed to make these averages comparable across different interventions, like smaller class sizes versus more teaching assistants. The denominator introduces very serious problems of its own, but here I want to focus on the fact that the effect size numerator involves an average. Thus enters probability. It is an average across the individual effects of the cause across a given population, so each of these individual effect sizes is supposed to occur with some probability in that population in order that the average makes sense.

Here I should pause for a moment’s clarification. Take any given population, like the students in my UCSD 2018 Philosophy of Social Science class, and any given quantity that we can assign a number for to each individual, like their numerical marks in the first exam. I can always calculate an average of those in the usual way – multiply each grade outcome by the number of students who got that grade, add those numbers together and divide by the total number of students who received grades. Consider the grade 66. We can call the relative frequency of students receiving a 66 the ‘probability’ of getting a 66 in the population of students who received a mark. And it is true that relative frequencies are probabilities in the sense that they satisfy the fundamental axioms of probability. But to then use that probability to calculate an effect size for that kind of exam or my kind of teaching that educators should sit up and take notice of requires much more than the fact that it is a true relative frequency in my class. It must have some broader, more stable significance. There are many ways of thinking of probability that do not involve frequencies at all – probabilities are disposition-like objective chances or, as ‘Bayesians’ do, degrees of belief. But those who do think in terms of frequencies are not content with just finite frequencies in a single case, like the twenty-two students in my class. Rather they define probabilities as limiting relative frequencies in an infinite run of repeat trials on similar populations. The infinite repetition is supposed to supply the generality or stability required for attaching the label ‘probability’.

It is just this kind of stability that I worry about with respect to these many effect sizes calculated and promulgated in social science nowadays. To explain why, I have to say a few things about the kinds of causes or interventions whose effect sizes are under consideration. The cause or intervention under discussion is very seldom if ever enough on its own to produce a contribution to the named effect. It needs supporting or helping factors to be enabled to do so. The same is generally true of causal claims we make in both science and daily life. The stock philosophers’ example is that striking a match causes it to light. Well yes, supposing there is oxygen in the room, there are no heat-sensitive sprinklers that will douse the match as soon as the match head begins to heat up, there’s no eager firefighter ready to blow on it at the merest sign of a flame, etc. etc. etc.

In statistics these supporting factors are called ‘interactive’ or ‘moderator’ variables because they interact with the cause to moderate how much, if anything, it will contribute to the effect. They are often graphed in causal pies or cakes, like those pictured in Figures 1.2, 1.3 and 1.4. In formal treatments, as in econometric models or proofs about what can be accomplished with randomised controlled trials, the relation between causes and effects are represented in equations like this

$Y_i=b_i{X}_i+W_i$

where X_i is the cause in question of the outcome Y_i for individual i, b_i is the interactive variable representing the net effect of all X’s supporting factors that obtain for i and W_i is the net contribution of all the factors that contribute to Y for i independently of X. Note that the value of the supporting factors b moderates how much a given value of X contributes to Y as it should.

To see my worries, consider a simple two-valued case: either the cause is present for i (e.g. student i is in the small class) – X_i = 1 or it is not (that student is in the usual large class) – X_i = 0. The difference in Y’s value between the two, holding fixed all the independently operating causes for i (represented in W) is just what we mean by the individual effect size of X on Y for i. By inspection you can see that the average of the individual effect sizes across all the different individuals in a population is just the expectation of b in that population.

But b represents the supporting factors. And there’s the rub. Why think that the causes and interventions for which we are purporting to calculate effect sizes are operating in small worlds where these probabilities exist rather than in the larger worlds of the kind that Kay and King worry about? Indeed, before even we get to probabilities over a set of supporting factors, we should worry about supposing that there are these fixed sets of supporting facts in the first place to assign probabilities to – namely, that my first requirement that outcomes be fixed is satisfied. Why suppose that the ones necessary for one individual are the same as for others?

Consider, for example, the following discussion from Eileen Munro, who authored the UK’s 2011 Munro Review of Child Protection:

Figure 1.2 A cake diagram for Person A (abused as a child).

From Munro et al. Reference Munro, Cartwright, Hardie and Montuschi2016

Figure 1.3 A cake diagram for Person B (not abused as a child).

From Munro et al. Reference Munro, Cartwright, Hardie and Montuschi2016

Figure 1.4 A cake diagram for Person C (abused as a child: conditions not combined).

From Munro et al. Reference Munro, Cartwright, Hardie and Montuschi2016

It’s because of these kinds of problems with probabilities that I worry that talk of effect sizes is very often misplaced. The concept of probability on which it depends is too often stretched beyond where it will reach. We end up calculating something that has no real significance in the large worlds that effect sizes are supposed to help us navigate in. As a result, we adopt policies that fail and neglect others that could succeed in the setting at hand.

I turn now from concepts to look briefly at some other ingredients that are also necessary to constitute meaningful, useful theory.

Exploring

What philosophers call ‘exploratory experimentation’ is conducted without any regard to theory; it is not intended to test theory. Nor is it intended to develop theory or estimate values of theoretical parameters (like the gravitational constant) or to fill in missing details. For the most part, it does not take direction from theory at all, even low-level theory. It is there to probe, to explore, to discover. It is blithely living a rich life of its own, far from the beady eye of theory.

Philosopher and historian of science Richard Vagnino provides us with a nice example, illustrating the care and detail that go into this kind of exploratory experimentation:

Creating Phenomena

Hacking tells us that experiments create new phenomena that never before existed, like the Zeeman and Stark effects, which are the shifting and splitting of spectral lines in light emitted by atoms and molecules, both of which gave rise to Nobel Prizes. The second, which was important in the early development of quantum mechanics, is splitting due to an electric field. The first is splitting due to a magnetic field. It was first produced in Leiden in 1896 by the Dutch physicist Pieter Zeeman. It plays a significant role in our knowledge of the electron. As historian Theodore Arabatzis reports, the Zeeman effect ‘not only provided evidence for the existence of the electron but also led to a specification of two of its properties, its charge to mass ratio and the sign of its charge’.⁴⁴

Hacking explains that we speak of physicists discovering these effects, but, he claims, this is misleading. It downplays the battery of experimental practices, including the construction of ingenious apparatuses that allow for their stabilisation and reliable reproduction. Only with this network of background activities and products of science do such effects turn into actual phenomena: noteworthy events that occur regularly in very specific – small-world – circumstances. Hacking’s point is that we have been so caught up with the use of experiment to test theory and with the theoretician’s attempts to explain phenomena that we have ignored how experiments often create the very phenomena to be explained.

Another of Hacking’s examples is the Hall effect produced in 1879 by the American physicist Edwin Hall as part of the research for his PhD at Johns Hopkins University. The Hall effect occurs when a current passes through a conductor in a magnetic field. The magnetic field exerts a force on the moving charges perpendicular to their line of motion. This pushes them to one side of the conductor. The charge build-up on the conductor creates a voltage difference at right angles to the field and to the current.

Did Edwin Hall’s experiments really create the Hall effect? Can it not take place in nature without the need for experiment? Hacking’s answer is ‘yes and no’. He explains:

Reconstituting Phenomena

Besides testing theory and creating phenomena, there’s lots else we do with experiments. We use them to help develop new concepts, to fill in missing gaps in narratives and to understand better how various measurement techniques work. They not only produce new phenomena never before seen but we use them to find out more about phenomena that we are already familiar with. Often this involves getting a better measure of some of properties of the phenomena: like using a superior radar gun to find that a car’s top speed is 220.335 kpm rather than 220 kpm. Sometimes, though, in the effort to find out more about an already familiar phenomenon experiments tell us that we weren’t that familiar with it in the first place. These kinds of cases call for a more drastic, qualitative redescription of the phenomenon. When this happens, philosophers say that the phenomenon gets ‘reconstituted’. Let’s look at this in more detail as another illustration of what more we can do with experiments than just test theory. I use an interesting case in molecular biology, as described by philosopher Andrew Bollhagen:

As the story illustrates, experimental efforts to further describe a phenomenon can lead to drastic revisions of ‘familiar’ descriptions. As philosophers like Bollhagen⁴⁷ put it, experimentation can lead researchers to ‘reconstitute phenomena’.

Figure 1.5 Kinesin walking cargo down a microtubule trail

Drawn by Nicki Shaw/@nickisdoodles, especially produced in black and white for this volume. Both Bollhagen and I are grateful for Shaw’s permission to use it. Thanks Nicki!

Figure 1.6 Kinesin pushing microtubule along

Drawn by Faith Bollhagen especially for this book. Thanks to Faith from both Andrew and me!