2.1 Crucial Experiments and Underdetermination

In his seminal book Representing and Intervening (Reference Hacking1983), Ian Hacking introduces Francis Bacon (1561–1626) as “the first philosopher of experimental science” (246). Hacking is not alone in this appraisal. Centering Bacon specifically in our historical narrative can be challenged (cf. Reference KheirandishKheirandish 2009). Nevertheless, it is instructive for our purposes to take a look at what sort of methods Bacon advocated, and to what end. With the explicit aim of overhauling the approach to natural philosophy descendent from Aristotle, Bacon wrote the Novum Organum to promote and illustrate induction as the appropriate method for learning about nature and for putting that knowledge to practical use. For Bacon, understanding of the natural world was not to be purely “invented” or “imagined” by “premature reflection” but discovered by applying reason to thorough natural and experimental histories – collections of documented diverse instances of the matter of interest (Book II, Aphorisms IX and X). Bacon famously uses the example of the nature of heat to illustrate his approach. In investigating the nature of heat, a natural philosopher should first list many and diverse instances of heat – from lightning, to warm springs of water, to fresh horse shit – and then analyze these in comparison with nearby instances in which heat is absent (Aphorisms XI and XII). These instances could be observations of a general sort, such as that heat attends all flames, boiling liquids, and sparks struck from flint, and could also include observations made in the context of “experiments” purposefully aimed at investigating instances of heat, such as attempting to focus moonlight with a burning glass (Aphorism XII). Indeed, in his illustrative tables pertaining to heat, Bacon calls for several experiments that would need to be performed in order to fill in the details that he deems relevant to the inquiry (Aphorism XIII). After examining instances of presences and absences in these special tables, Bacon’s method recommends investigating the extent to which the nature of interest varies as a matter of degree, by making such observations as: “Animals increase in heat from movement and exercise, from wine and eating, from sex, from burning fevers and from pain” (Aphorism XIII, point 9). With all of this information set out in advance, Bacon instructs that the human mind can then be set to work in discerning the nature in question.

Induction, then, is the necessary process by which mere mortals can hope to acquire knowledge of the true nature of things.

Bacon’s process involves setting reason to the task of uncovering what the features of the elements in the collection reveal about the nature of things and the laws that govern them. Moreover, induction is an iterative process that needs rounds of refinement. Working with the tables of presence, absence, and degrees, considering how to characterize the nature of something like heat that would capture and exclude the appropriate instances, the faculty of understanding may be permitted to hazard a “first harvest” characterization. The aim is to expose the nature of heat such that it admits no contradictory instances and accounts especially for the revealing instances, instantiae ostensivae (Aphorism XX). Bacon’s summary of his first harvest regarding the form of heat is: “Heat is an expansive motion which is checked and struggling through the particles” (Aphorism XX). From this initial attempt, the natural philosopher proceeds in induction by way of various other “aids to the intellect” (Aphorism XXI) such as considering instances of specific types in a systematic way, such as the revealing instances mentioned earlier, “which reveal the nature under investigation naked and independent” and, correspondingly, concealed instances, instantiae clandestinae, which “exhibit the nature under investigation at its lowest strength” (Aphorisms XXIV and XXV). The most historically notorious of these types of instances are those Bacon calls the crucial instances, instantiae crucis. Bacon draws the name of this type of instance from the signposts that label different ways to go at an intersection. He describes these instances as follows:

In other words, when seeking an account of the nature of something and faced with two reasonable possibilities, one can attempt to find an instance that clearly displays the nature in question and reveals either of the possibilities as separable, unnecessary for that nature, such that the true cause can be ascribed to the other possibility. Bacon uses the example of the ebb and flow of the sea, which “must of necessity either be caused by” rocking as in a shallow bowl or by the rising and falling of the waters from the depths, as when boiling water rises and sinks again when the boiling subsides (Reference DumitruDumitru 2013, 49). Bacon suggests that a crucial instance in this context would be supplied by the observation that the high tide in Spain and Florida is simultaneous with the high tide in Peru and eastern China. Such an instance would rule out the sloshing possibility. Sometimes, Bacon says, these instances are gifted to us by nature, but more often they follow from investigation designed for that purpose.

A simplified distortion of this reasoning, inspired by but not faithful to Bacon’s original, has come to be known as a “crucial experiment” (see Reference HackingHacking 1983, 249–251). The caricature instructs us to sufficiently narrow the research question such that carefully constructed circumstances should yield a decisive answer. The problem, of course, with that is this: How can one be assured that the possibilities considered are exhaustive? If the possibilities considered are not exhaustive, then the purported crucial instance or the result of the purported crucial experiment will not be decisive after all. The case of the ebb and flow of the tide, for one, does not seem watertight. Are there any true crucial instances or true crucial experiments at all?

As Pierre Duhem lucidly explained, when an experiment fails to produce a predicted phenomenon, “The only thing the experiment teaches us is that among the propositions used to predict the phenomenon and to establish whether it would be produced, there is at least one error; but where this error lies is just what it does not tell us” (Reference Duhem1991/1954, 185) and “the physicist can never subject an isolated hypothesis to experimental test, but only a whole group of hypotheses; when the experiment is in disagreement with his predictions, what he learns is that at least one of the hypotheses constituting this group is unacceptable and ought to be modified; but the experiment does not designate which one should be changed” (187). Indeed, Duhem goes so far as to embrace holism regarding physical theory:

For Duhem, there can be no “crucial experiment” in physics because having one would require the physicist to “enumerate completely the various hypotheses which may cover a determinate group of phenomena; but the physicist is never sure he has exhausted all the imaginable assumptions” (190). Logic alone does not preclude different physicists from responding to mismatch between theoretical predictions and the results of experiments in diverse ways – one could choose to cut out the heart of the theory, while another could choose to pursue more superficial modifications. However, Duhem stresses that the loose directive logic furnishes in such circumstances is not the only stricture the physicist obeys. In addition there are “motives which do not proceed from logic and yet direct our choices, these ‘reasons which reason does not know’ and which speak to the ample ‘mind of finesse’ but not to the ‘geometric mind,’ constitute what is appropriately called good sense” (217). Of this “good sense” Duhem writes:

Yet, for those interested in the epistemology of experimental physics, such an appeal to the significance of “vague and uncertain” inclinations and the need for moral fortitude may be unsatisfying. It is well and good, perhaps, to entreat experimental physicists to render in themselves a lucid and vigilant sense of how to proceed in the face of underdetermination, to cultivate their characters so as to become impartial and faithful judges. But how exactly? Once cultivated, what does such good sense recommend? The epistemology of experiment would be more transparent if we could fill in the details of Duhem’s “good sense” somewhat more. Transparency is especially desirable here because without it, in the murky realm of unarticulated sensing, it is more difficult to ward off the suspicion that non-epistemic interests are playing the decisive role.

Claudia Dumitru offers Robert Boyle’s (1627–1691) inaugural use of the phrase “experimentum crucis” in his 1662 A Defence of the Doctrine Touching the Spring and the Weight of the Air as “an almost perfect example of a Baconian crucial instance” (Reference Dumitru2013, 55). The example in question is worth knowing, and will be of further use to us in what follows. Take a tall glass vial and fill it with liquid mercury. Upend the tube and, keeping the open end momentarily well sealed, place it oriented vertically in a dish of more mercury. Remove the seal and observe the level of the mercury in the upside-down tube fall somewhat, although not entirely out of the tube. This is a Torricellian tube. In Boyle’s day, this phenomenon, the space made at the top of the inverted tube under such circumstances, attracted competing explanations. Boyle’s preferred hypothesis was that it was the “spring of the air” surrounding the tube keeping the mercury up. In sketch, the competing explanation was that the would-be vacuum in the Torricellian space was so unnatural that some other subtle fluid must be present in that space, keeping the mercury up. The experiment that Boyle refers to as an “experimentum crucis” is the one Blaise Pascal arranged to be carried out by his brother-in-law, who brought a tube prepared in this manner up the Puy-de-Dôme (a lava dome nearly 1,500 meters high in central France) in stages, the mercury falling as he went. Remarking on this experiment, Boyle wrote: “since this noble phenomenon seems to follow from ours and not upon our author’s hypothesis, it seems to determine the controversy” (quoted in Reference DumitruDumitru 2013, 55). In the next section we will see that the matter is more subtle.

2.2 Calibration and Regress

The final paragraph of Shapin and Schaffer’s well-known book Leviathan and the Air-Pump foreshadows much of the debate between philosophers interested in the epistemology of science and sociologists of science at pains to dismantle any preferential treatment for science qua knowledge-producing activity over and above other human pursuits. Shapin and Schaffer examine the conflicting perspectives of Boyle and Thomas Hobbes with respect to the nature of philosophy and the role of experimentation. To simplify, they portray Hobbes as an antagonist of experimental methods, highlighting his efforts to undermine the trustworthiness of Boyle’s experiments with the air-pump by calling attention to its propensity to leak. Shapin and Schaffer ultimately conclude that revealing the labor involved in arriving at experimental results undermines their authority: “As we come to recognize the conventional and artifactual status of our forms of knowing, we put ourselves in a position to realize that it is ourselves and not reality that is responsible for what we know. Knowledge, as much as the state, is the product of human actions. Hobbes was right” (Reference Shapin and Shaffer2011/1985, 344).

These final words of Shapin and Schaffer’s book neatly express a position diametrically opposed to my overall message in the present work. Shapin and Schaffer’s view is that seeing the scientific sausage being made, so to speak, undermines the credibility of scientific claims. I think the opposite is true: having a detailed and nuanced understanding of scientific practice helps us appreciate the justification for science’s special role as our best route to learning about nature. There is an important caveat. Someone under the mistaken impression that science proceeds according to the algorithm presented in introductory textbooks could understandably be thrown off course by learning about actual scientific methods. But the correct lesson to draw in that case is not that scientific results are merely the products of human actions, and thus not to be taken especially seriously, but that the surprised person had an overly idealized view of what science is all about.

It seems to me that the biggest threat to the epistemology of experiment is encapsulated in what Harry Collins has called the “experimenters’ regress.” Any empiricist epistemology of science (as opposed to a rationalist one) bestows a key role on empirical results. In order to argue that the results of one’s experiment are to be taken seriously, one has to argue that the experimental apparatus is working properly, but to argue that the apparatus is working properly – so says the regress – one demonstrates that the apparatus has produced the expected “correct” results (Reference CollinsCollins 1992/1985, 84). Collins argues that it is in virtue of this looping in experimental reasoning that social factors like power, prestige, and personality get traction in decisions about how the results of experiments are interpreted and received by researchers and the scientific community at large. If Collins is right about the existence of this sort of regress, then the epistemology of experiment faces a serious problem. The experimenters’ regress cannot be abolished by good research ethics alone. All fraud could cease, but social factors would still be required to truncate the regress. If social factors were to play the decisive role in the deliverances of science, then there is not much to recommend a special epistemic status for the enterprise. Shapin and Schaffer explicitly invoke Collins’s notion of the experimenters’ regress in their discussion of attempts to replicate Boyle’s experiments with the air-pump (Reference Shapin and Shaffer2011/1985, 226). In order to replicate Boyle’s experiments, other experimenters had to be able to judge when such replication had been accomplished. The only way to do this was to use Boyle’s phenomena as calibrations of their own machines. To be able to produce such phenomena would mean that a new machine could be counted as a good one. Thus, before any experimenter could judge whether his machine was working well, he would have to accept Boyle’s phenomena as matters of fact. And before he could accept those phenomena as matters of fact, he would have to know that his machine would work well (226).

In their narrative and analysis of these replication attempts, Shapin and Schaffer emphasize the role of social conventions for decision-making in these experimental contexts. In particular, they argue that social conventions played a role in when an apparatus under construction was accepted as an air-pump, when the results of trials made with that apparatus were taken seriously, and when the results of trials made by anyone besides Boyle himself would be believed over Boyle’s reports. If Shapin and Schaffer are correct that Boyle’s experiments with the air-pump suffer from the experimenters’ regress, then the epistemology of Boyle’s experimental methods would be dubious. Are they correct? To answer this question, it will be helpful to examine Boyle’s approach in a little more detail.

Robert Boyle is widely regarded as one of the most significant founders of experimental methods, if not the primary founder. For instance, writing of Boyle’s experiments with the air-pump, Michael Nauenberg states: “The construction of the vacuum pump and the discovery of the physical properties of air were, undoubtedly, the most important development in experimental science at the beginning of the scientific revolution” (Reference Nauenberg2015, 330–331). Taking up the project outlined by Bacon, Boyle set about collecting observations and performing experiments in order to amass empirical reserves that could serve to inform theoretical understanding of nature and useful practical applications. Boyle was evidently humble enough to suppose that great theoretical systems take significant time to emerge from natural and experimental histories, a process that might outlast his own lifetime. He saw himself as mining the marble from the quarry from which others might sculpt masterpieces later on (Reference Sargent and HunterSargent 1994, 64; see also Reference SargentSargent 1995). Reference AnsteyPeter R. Anstey’s (2014) account, which groups the approaches of Bacon-Boyle-Hooke (BBH) into a single philosophy of experiment, emphasizes that the experimental philosophers were keen to maintain healthy intellectual distance between their observations and experiments on one hand, and speculation on the other, while nevertheless recognizing the eventual aim of systematizing the results of accumulated investigations:

Indeed, Anstey notes that a “significant portion of Boyle’s experimental work had no immediate bearing on speculative theory whatsoever” (122). Something of Boyle’s methods and personality can be gleaned from reading his detailed and entertaining reports of his investigations. Take for instance “Some Observations about Shining Flesh,” a letter that Boyle presented to the Royal Society. Boyle recounts how he had been getting ready to go to bed when he was informed that one of his servants had found a shining neck of veal in the larder. Despite having a cold (from experimenting with a new telescope earlier that day in high winds), he stayed up to make several observations and experiments that very night, although not “long enough to make all the tryals that I thought of and judg’d the occafion worthy of” (1672, 5108). Boyle explains the origin of the veal, the number of shining spots on it, the size and shape of the spots, where they were located on the piece of meat, the strength of the light as compared to that of glow worms, the color of the light, the lack of accompanying heat, the state of the meat (not stinky), the condition of the larder in which the veal neck had been kept, including the orientation of the window thereof, the prevailing meteorological conditions such as the direction and character of the wind, temperature, quarter of the moon, and the reading of the barometer. He investigates the effect of removing a shiny part from the larger neck piece. He tries rubbing it on his hand, compressing it, placing it in a crystal vial of Spirit of Wine, in a teacup of cold water.

Then he calls for the Pneumatical Engine (the air-pump) to be set up, puts one of the largest shining pieces in the device’s receiver, and has the pump worked in the dark so as to observe how the shining meat behaves as the air is evacuated from the chamber.

When the light does not appear to diminish to any considerable degree, Boyle wonders if the air-pump, “having been managed in the dark, had leaked all the while” (5112). The candles are brought back in, a Mercurial Gage is procured, the seal between the glass of the receiver and the engine is reestablished securely, and the trial is performed again. This time, Boyle witnesses the diminution of the shining light with the evacuation of air from the receiver (which evacuation is corroborated by the appearance of the gauge). When air is let back in the receiver, the increased luminosity of the piece of veal returns to its former strength. The experiment is repeated and Boyle wonders if the shining light would vanish entirely if the pumping continued long enough, but eventually he feels it is unreasonable to stay up any longer that night and that he really should go to bed.

However, while he is undressing, it occurs to him to see if other parts of the veal are “innobled” with the strange shine and, sure enough, a leg is brought to his chamber! Boyle then explains that the next day he was distracted from the shining veal by the illness of his niece, and by the time he was disposed to perform further observations and experiments, the veal of interest had been disposed of, leaving him only the little piece preserved in the crystal vial, which he observed for several more days until the glowing diminished to imperceptibility, along with a piece of fowl with some shining parts (although not as many as the veal) that was also retrieved from the larder later.

Clearly, Boyle’s methodological approach reached far beyond recording careful observations, although he certainly did that too. Taking cues from craftsmen and tradesmen, he contrived many artificial circumstances, manipulating natural conditions in order to perform particular experiments of interest. As Sargent explains, Boyle clearly recognized the difficulties that come with preparing experiments in this way:

Subtle differences in experimental conditions can affect the success of the trial as whole, as Boyle observed from attending to the practices of smiths and glassworkers, noting, for instance, that “none but an artist expert in tempering of iron would suspect that so small a difference of time of its stay in the flame could produce so great a difference in its tempers” (quoted in Reference Sargent and HunterSargent 1994, 70). Boyle’s awareness of the subtleties and fickleness of experiments can be easily discerned from his two fascinating essays “Of the Unsuccessfulness of Experiments” and “Of Un-succeeding Experiments.” In the first, Boyle recounts that he has learned that experiments that succeed once, but that the diligent experimenter cannot manage to replicate, may fail on account of the materials employed. He complains at length of the difficulties involved in procuring Spirit of Salt unadulterated by Spirit of Nitre, even from supposedly reputable Chymists. Indeed, Boyle goes on to explain that even when one has managed get one’s hands on what really ought to be some pure material, one may still encounter variation that can affect the outcome of experiments. Urine, for instance, which was a commonly used ingredient, differs according to its origin such as between “that of healthy and young men abounding much more with volatile Salt than that of sickly or aged persons; and that of such as drink Wine freely being much fuller of spiritous and active parts than that of those whose drink is but Beer or Water” and how long it has been kept before use in experiments (Boyle 1999/1661, 49). He reports “great odds there may be betwixt some Experiments made with recent and putrifi’d Urine,” as evidenced by the properties of a stock of urine that he had buried in a dunghill and, having been distracted by other things, left for four or five months instead of the intended five or six weeks (50). Assuming consistency among mineral samples can be particularly troublesome. He remarks that “there is as well a difference in Minerals of the same kind, as there is in Vegetables and Animals of the same species,” owing to the conditions of the minerals’ origins, which means that what appears to be a pure sample “may have lurking in them Minerals of quite other nature, which may manifest themselves in some particular Experiments” (46). Of course, prepared materials can exhibit diverse effects on experiments as well, depending on the shape of the glassware, for instance, and can even be over-purified for the purpose at hand:

In the latter essay on un-succeeding experiments, Boyle discusses other contingencies, circumstances, and abstruse causes, besides the variety of materials that can spoil experiments. The purpose of the essay is twofold. First, Boyle aims to impress upon other experimenters examples that should inspire them to “try those Experiments very carefully, and more than once, upon which you mean to build considerable Superstructures either theoretical or practical, and to think it unsafe to rely too much upon single Experiments, especially when you have to deal in Minerals” (77). Sometimes an experiment that works on a small quantity of matter will fail on a greater quantity. An experiment performed at one time (such as certain dissections of animals) may not produce the same results performed in another time of the season. Whether, as Bacon apparently himself reported, roses can be made to bloom again in autumn may depend both on the kind of rose and on the prolific nature of the individual bush (59). Moreover, since even the most skilled craftsmen, like dyers and glassworkers, can sometimes fail to reproduce some technique or material that is exceedingly familiar to them, “it need be no such wonder, if Philosophers and Chymists do sometimes miss of the expected Event of an Experiment but once, or at least but seldom try’d, since we see Tradesmen themselves cannot do always, what, if they were not able to do ordinarily, they could not earn their bread” (64–65). In general, the particular circumstances of an experiment that “are very difficult to be observed, or seem to be of no concernment to an Experiment, may yet have a great influence on the Event of it” (65). There are issues with the testimony of experimenters too. Boyle suggests that the reported results of some experiments depend very much on the pride of the experimenters who perform them, such as reports of certain plants frozen in ice: “’tis strange to observe what things some men will fancy, rather than be thought to discern less than other men pretend to see” (62). Boyle cautions experimenters not to be lazy, of course, in attempting to discern the causes of varied experimental outcomes (as when differences in the buds of grafted cherry trees foretell whether they will fruit the first or second year after grafting). Nevertheless, the essay is full of examples where subtle differences in materials and circumstances evade and frustrate even the careful experimenter for some time.

Second, the essay is meant to serve as a kind of “Apology for Sober and Experimental Writers” in the event that one attempts to perform an experiment described by another experimenter and the trial fails to produce the anticipated result (77). When these failures occur, the reputation of the writer should not be immediately and utterly undermined, since perhaps subtle differences in the circumstances of the experiment are at fault. Along the same vein, Boyle intimates that when reading the reports of an experimenter he does trust, even if the reported results contradict other observations or widely held hypotheses, he will not immediately discount the results since “sometimes there happen irregularities contrary to the usual course of things” and sometimes the contradiction is only apparent (80). In short, Boyle cautions his fellow experimenters to be watchful and wary, but not to despair so much as to forsake the endeavor because even in the mistakes there can be interesting discoveries.

According to Sargent, Boyle regarded the results of experiments as “merely signs that provide hints about how nature works” (Reference Sargent and Hunter1994, 71), and he accepted theories that enjoyed the support of many complementary experiments:

In particular, due to the multifarious subtle factors involved in even the most careful experiments, a single experiment should not be taken as proof nor as refutation of a theoretical claim (71).

Other natural philosophers wanted to build their own air-pumps and to perform experiments and explore phenomena to which Boyle had drawn attention. Partly because the construction of an air-pump was an expensive endeavor, there were few initial successes in producing rival apparatuses. The pumps themselves were temperamental, and as Shapin and Schaffer stress, prone to leaking, which we have already seen from Boyle’s own observations on the shining veal. Much care and tinkering was therefore required in order to construct an operational pump suitable for investigation of phenomena. Shapin and Schaffer describe in detail the attempts of a particular challenger of Boyle’s who created his own pump based on details of Boyle’s pump as conveyed by Robert Moray. According to Shapin and Schaffer, “Christiaan Huygens was the only natural philosopher in the 1660s who built an air-pump outside of the direct management of Boyle and Hooke” (2011/1985, 235). In the course of attempting to build his own serviceable pump, Huygens introduced several modifications to Boyle’s design. For instance, Huygens’s pump had a copper valve where Boyle’s had a wood one, had a closed top whereas Boyle’s had a port, and used different sealant materials (237). Like Boyle, Huygens made subsequent incremental alterations to his apparatus in an attempt to improve its functioning. Huygens told others that his pump was better than Boyle’s and offered as evidence an inflated bladder that remained inflated in the pump all night, whereas Boyle had reported that bladders deflated (slowly) in his apparatus (237). Once he had his pump running to his own satisfaction, Huygens claimed to have made an interesting discovery regarding the Torricelli-type experiments that Boyle described in New Experiments. Placing a barometer containing “water he had purged of air by leaving it many hours in the receiver of the air-pump,” he discovered that, unlike ordinary water, the level of this water did not fall with the evacuation of the pump (241). According to Shapin and Schaffer, Huygens came to use this “anomalous suspension” as a calibration phenomenon (243). “By February 1662 Huygens took the anomalous suspension of water well purged of air, and the fall of that water when a bubble was introduced, to be marks of a good machine” (243).

The anomalous suspension was a result unforeseen by Boyle, and not immediately replicated by him either. When news came from Holland of this strange discovery, the pumps in England were not in a position to verify the finding – one was under development and the other was temporarily out of commission (244). Without appropriate trials of his own, Boyle had to respond to Huygens. According to Shapin and Schaffer, “Boyle claimed that his machine was better than that of Huygens, so that anomalous suspension was a mark of Huygens’ incapacity to make a good pump,” “denied that anomalous suspension could be used as a calibration of the pump,” and even suggested to Huygens (via Moray) that the latter’s apparatus was probably not sufficiently evacuated and that Huygens should employ a gauge to check (245). Huygens persisted in defending the phenomenon and providing further results to support it, and Boyle continued to blame the anomalous suspension on the integrity of his challenger’s pump (246).

Despite further technical improvements, trials, and arguments from Huygens, by January 1663 Boyle was still not “assured of the truth of the experience” (248). According to Shapin and Schaffer, “in March and April 1663 it became clear that unless the phenomenon could be produced in England with one of the two pumps available, then no one in England would accept the claims Huygens had made, or his competence in working the pump” (249). Failing to sufficiently purge water of air for use in the experiment, Moray even ventured that the difference in experimental outcomes might be due to the water in London being different than the water in Holland (249). Shapin and Schaffer argue that unless the anomalous suspension could be reproduced on Boyle’s turf and to his satisfaction, they were at a stalemate. However, continued technical difficulties in England prevented them from performing satisfying trials. Shapin and Schaffer describe the status of the pump that Boyle had left to the Royal Society, which was put under Robert Hooke’s charge in late 1662, as “an almost permanent trouble because of its obvious leakage” (249). Eventually, in mid-1663, Huygens made the trip to London. Boyle was out of town at the time, residing at his sister’s house in Essex. Within about a week of Huygens’s arrival, Hooke had successfully produced the anomalous suspension (251). Boyle was informed straightaway and witnessed the phenomenon himself in experiments performed after he returned to London in August (252). Nevertheless, Boyle maintained that “in regard they had noe Gage to try how farre they had exhausted ye Aire in the Receiver it seem’d not absurd to coniecture that there might remaine in ye Receiver enough [air] to keep up in ye Tube 3 or 4 foot of Water” (quoted in 252). In other words, even after seeing the anomalous suspension himself, Boyle was not totally satisfied that it was not due to a problem with the instrument. According to Shapin and Schaffer, in experiments made during the fall of 1663, Boyle attempted to dissociate the troublesome phenomenon from the appraisal of a pump’s integrity by producing the anomalous suspension of mercury outside of the air pump entirely (254).

Shapin and Schaffer argue that Boyle never published an account of anomalous suspension because this phenomenon “resisted the recognized explanatory competence of the spring of the air” and “might challenge … the worth of the air-pump” (230, 255). Shapin and Schaffer take this episode to support the main interpretive thrust of their book – the intimate connection between the “problem of knowledge” and the “problem of social order”:

Are Shapin and Schaffer right to interpret Boyle’s judgments on the anomalous suspension as primarily a matter of protecting his preferred hypothesis and his pride in the worthiness of his instrument? Does the anomalous suspension display the decisive role of social convention in settling when air-pumps were working properly and thus when the results of trials made with them ought to be taken seriously?

Interestingly, the issue of anomalous suspension remained mysterious long after the height of the controversy between Boyle and Huygens. In a 2015 article, Nauenberg states that the origin of the phenomenon “was not clarified, and it has remained an unsolved puzzle to the present day” (329). Nauenberg actually undertook to perform and compare the Boyle-Hooke and Huygens versions of the experiment to try to understand the physics from a contemporary perspective. He found that the explanation Shapin and Schaffer offered was largely wrong (340). Using ordinary tap water, Nauenberg saw the water column in his Torricellian tube fall upon the commencement of pumping, which he attributes to “the pressure of the gas formed by the bubbles surfacing inside the tube” (337). When the experiment was performed with water purged of some of its dissolved air, the column of water did not start to fall until the pressure in the receiver was much lower than it had been with the ordinary, unpurged water (337). According to Nauenberg, the anomalous suspension is due to the limitations of the seventeenth-century pumps to evacuate their receivers – he estimates that there would have been a residual air pressure greater than 0.15 atmospheric pressure (336–337). Indeed, he argues that Boyle himself was aware of the air bubble issue (335–336). That is, purged water could have remained anomalously suspended in these early pumps because the pumping mechanism simply was not strong enough to sufficiently remove residual air pressure in the receivers to draw down a column of water without the help of the pressure of a gas bubble released inside the tube itself. Since Nauenberg used a modern vacuum pump, it would be interesting to try these experiments with more faithful replicas of the seventeenth-century instruments, wax and resin seals and all.

As we have seen from Boyle’s two essays on “unsuccessful” and “un-succeeding” experiments, which were originally published prior to the Huygens controversy, Boyle was very aware of the difficulties involved in reproducing an experiment and the many subtle circumstantial differences that could disrupt any particular trial. Boyle need not have been dogmatic to be suspicious of Huygens’s results. Huygens was using an instrument of his own construction with a newly modified design. Boyle knew very well how temperamental his own air-pumps could be. In fact, the reason he could not immediately investigate the anomalous phenomenon was that his pumps were out of commission. Even in low-stakes trials, such as the late-night experiments with the shining veal neck, the seal on the air-pump required constant vigilance. It was not unreasonable for Boyle to suspect that Huygens’s early experiments with designing, building, and operating an air-pump would meet with difficulties, and that Huygens’s initial reports of unusual results should be interpreted cautiously. It was reasonable for Boyle to worry that Huygens’s pump leaked. Even after seeing the experiment performed himself, it was not necessarily dogmatic for Boyle to worry about air being left in the apparatus. This was not special treatment for Huygens – this was Boyle’s cautious attitude about the functioning of these temperamental instruments born of hard-won experience, and a wariness consistent with the general attitude toward experiments expressed in the two essays on the unsuccessfulness of experiments. A multitude of contingencies can affect the outcome of any one experiment.

From the perspective of the experimenters’ regress, however, it is perhaps little comfort to appeal to the reasonableness of caution in light of subtle contingencies. In a given experiment, one wants to know whether those contingencies have spoiled the results for the purpose at hand. One wants to know if the apparatus is working properly. If one can appeal to “subtle contingencies” ad nauseum, then it seems that convention or other social influences must step in to decide the verdict. Thus we see again the connections between underdetermination and the experimenters’ regress in the epistemology of experiment – the connections between the question of auxiliaries and whether an apparatus is working as the experimenter desires. In the following section, we extend this discussion to our third, related issue: excuses. Ordinarily, it is not epistemically permissible to ignore empirical data because they defy one’s expectations. But data are routinely omitted on the grounds that the experimental apparatus was not working properly at the time the data were generated. If these judgments are decided by social factors too, then much of experiment would be just as good as fraud.

3.1 Millikan’s Orphaned Drops

In 1911 and 1913, Robert Millikan published results from his oil drop experiments on the value of the fundamental electric charge, e. Among his aims were to demonstrate that all electric charges were exact multiples of a single elementary charge and to measure the value of this fundamental charge. At the time the experiments were performed, there was still debate in the physics community about whether there might be fractional charges. Thus Millikan sought to produce decisive results on that issue and to provide a more precise measurement of e than was then available.

The basic method of the experiment was to record the time it took a small drop of oil carrying an unknown number of ions to fall between two plates under the influence of gravity, and then to record the time it took the same drop to travel the same distance under the influence of a known electric field. To calculate the total electric charge on a drop, Millikan used the formula:

$\begin{array}{c}{e}_n=\frac{4}{3}\pi {\left(\frac{9\mu }{2}\right)}^{\frac{3}{2}}\left(\frac{1}{g\left(\sigma -\rho \right)}\right)\left(\frac{v_1+v_2}{F}\right)v_1{}^{\frac{1}{2}}\end{array}$

where $v_1$ and $v_2$ are the velocities of fall and rise respectively, calculated from the measured values of the times and distances of drop travel. The other values are known: μ is the viscosity of air, g is gravitational acceleration, σ is the density of the oil, ρ is the density of air, and F is the electric field (Reference FranklinFranklin 2016, 116).

Millikan (1911) describes the method as follows:

As Reference FranklinFranklin (2016) puts it, “Millikan’s technique in manipulating the oil drops was nothing short of spectacular” (116). Millikan would stare at a single oil drop for sometimes more than four hours and took measurements for forty-seven days (116, 119).

Franklin’s scholarship on the Millikan oil drop experiments is particularly interesting. Millikan was evidently “selective” regarding his data. He sometime omitted measurements that he had made on particular drops. As Franklin recounts, in his 1911 publication on the oil drop experiments Millikan was “quite explicit about the exclusion of data” (119). In particular, he left the first four drops (smallest/slowest) and the last four drops (largest/fastest) he recorded out of the final analysis, explaining:

However, in his subsequent paper reporting an even more precise and accurate measurement of the charge of the electron, Millikan was less transparent. Reference FranklinFranklin (1986) tells the story as follows:

Insofar as Millikan ignored data he had no reason to expect were bad data except the fact that they disagreed with his preferred result, he engaged in epistemically detrimental QRPs.

Is the famed experimentalist a fraud? Franklin’s conclusion is that while not an outright fraud (he gives other examples of fraud in science), Millikan can be characterized as a “trimmer” – one who leaves ostensibly relevant data out of analysis because they differ most from the mean, in order to report narrower uncertainties than if those data were included (230). Trimming data gives the appearance of a more precise measurement, a measurement with tighter error bars, than untrimmed data. Following Charles Babbage, Franklin contrasts this sort of trimming from the tails of a distribution with “cooking,” meaning “selection of data to achieve agreement,” for instance, “selecting only those data that will agree with a particular hypothesis or theory” (Reference Franklin1986, 227).

Is trimming really epistemically detrimental? Does it harm the epistemology of Millikan’s experiment? Trimming certainly has the potential to be epistemically detrimental. Suppose two different experimentalists seeking to measure the same physical parameter both discard outlying data in order to achieve a more precise estimate. If their reported uncertainties are artificially tight, then the two results may appear to be discordant, when they would not be had all of the relevant data been included in the analysis. This could lead to confusion and possibly even to misguided inferences about the epistemic status of the hypotheses involved in the experiments. Was Millikan’s trimming in particular epistemically problematic?

Of the forty-nine drops that Millikan observed after Franklin believes that Millikan was confident in his apparatus, for twenty-two of them, Franklin’s research shows that Millikan did not bother to calculate the fundamental charge. Franklin simply states: “The most plausible explanation for their exclusion is that Millikan did not need them for his determination of e” because he “had far more data than he needed to improve the measurement of e by an order of magnitude” (230). Although Franklin is not particularly worried about these twenty-two unused drops, I find it at least somewhat puzzling that Millikan would have omitted them from his analysis. As I indicated earlier in this section, taking these measurements is painstaking work requiring peering for hours through an eyepiece at pinpricks of light slowly falling and rising. Once the patience and energy had already been expended to record the data, why not use them, especially if doing so would not spoil the aim of one’s experiment, as Franklin indicates was the case for these drops?

Franklin does worry, however, about twenty-seven drops for which Millikan did calculate a value of e and then subsequently excluded from his analysis. Of twenty-one of these, Franklin says: “Twelve were excluded because they required a second-order correction to Stokes’s Law, two because of equipment malfunctions, five because they had few reliable observations, and two for no apparent reason, presumably because they were not needed” (230).

The factors motivating selectivity with regard to these twenty-one drops are not all equally compelling. It seems perfectly admissible, for instance, to reject drops because of an independently obvious equipment malfunction. Lack of sufficient reliable observations also seems like a reasonable – although conventional and thus debatable – ground for omission. The issue of corrections to Stokes’s law warrants further investigation. Stokes’s Law gives the drag force on a sphere (the oil drop) moving through a viscous fluid (the air) under certain conditions. In his 1911 paper, Millikan actually explores the limitations of assuming that Stokes’s law strictly holds in the context of his experiments. He argues that the law “breaks down as the diameter of the sphere becomes comparable with the mean free path of the molecules of the medium” – that is, for very small drops in Millikan’s experiment (quoted in Reference FranklinFranklin 2016, 113). He introduced a modification of Stokes’s law to represent the mean free path of the molecules composing the viscous fluid, the size of the drop, and an empirically determined constant (117–121). Given that Millikan introduced this other modification to Stokes’s law, why did he omit the twelve drops on account of those requiring an(other) modification? Franklin offers the following explanation: such a modification “made calculations based on their data unreliable.” He explains in an endnote that he, Franklin, “made an unsuccessful attempt to make a second-order correction to Stokes’s law along the lines of Millikan’s first-order correction” (122, note 24). This explanation strikes me as worthy of further investigation. My comments on the twenty-two unused drops also apply to the two drops omitted “for no apparent reason.”

The remaining six exclusions are even more suspect. Franklin writes that, for five of these,

Franklin suspects that Millikan trimmed his data in this way to support his reputation as a highly skilled experimenter (230). If this was Millikan’s motivation, then the omission of these five drops is epistemically disingenuous in the sense that the decision was not justified on epistemic grounds. If Millikan suspected that these data had been spoiled by unfavorable experimental conditions, that would be one matter. But the fact that omitting these data would protect his reputation does not by itself furnish appropriate epistemic reasons to omit them. Such decisions should not be condoned in the epistemology of experimental physics.

We come to the final drop. Millikan had two methods for calculating e; one involved the total charge on a drop and the other involved changes in charge. As Franklin describes, Millikan used both methods to calculate a value for e from the final drop:

Franklin reports that Millikan stated he exclusively used the method involving the total charge on the drop; however, in fact he used some combination of the two methods or the second method alone for nineteen out of the fifty-eight published drops, which tended to make his results seem more consistent (230). Millikan’s treatment of this last drop in particular, though, seems egregious. He clearly felt his apparatus was working properly, noted “Publish,” calculated the result from the data, and then abandoned the result because it failed to meet his expectations. This is just the sort of practice that “blind” or “double-blind” analyses serve to prevent – without another reason to throw out this result, this decision should be regarded as epistemically detrimental.

Franklin is not too flustered about Millikan’s “selective” analyses. Franklin explains that the “effect of Millikan’s trimming was quite small” and the effect of the cooking “was also small” (232). Indeed, several decades after The Neglect of Experiment, Franklin’s appraisal is roughly the same. “The effects of both the exclusion of data and of the selective analysis of the data are negligible,” and he suspects “that Millikan knew” that the effects of these questionable practices were negligible too (2016, 123). Millikan’s value of e was 4.778 x 10¹⁰ esu with a statistical error of ± 0.002, while Franklin’s reanalysis, including the published fifty-eight drops and twenty-five unpublished drops recorded after February 13, 1912, is 4.780 x 10¹⁰ esu ± 0.003 – that is, in agreement (1986, Table 8.1).

I think Franklin is too generous with Millikan. What matters is not that Millikan’s trimming and cooking happened not to throw off the course of physics when viewed in retrospect. Absent clairvoyance, Millikan could not have known that in the long run his selectivity would not have derailed further research. Condoning Millikan’s decision-making would equally condone QRPs today for which we do not yet have the benefit of hindsight. Surely that would be counterproductive. We can perhaps feel relief that Millikan’s unjustified decisions did not happen to have majorly bad impacts on the state of our knowledge, but that in itself does not justify those decisions.

In the following section we consider another famous case in which physicists left data out of analysis and have been charged with doing so for no good reason. This further case gives us an opportunity to inquire in more detail about the sort of reasons for omitting data that should be considered permissible.

3.2 The Sobral Astrographic Plates

Daniel Kennefick has stated that Arthur Eddington’s eclipse expedition of 1919 “may well have been the most important scientific experiment of the entire twentieth century” (2019, 4). The expedition continues to be touted as the experiment that resoundingly confirmed Einstein’s general theory of relativity despite decades of nuanced scholarship on the epistemology and social/historical context of the results. The near-mythological narrative is very clean: Einstein predicted that the image of a star near the limb of the sun will be displaced from its relative position when away from the sun. The apparent deflection compared to the Newtonian standard should be detectable with careful observation. In 1919, Eddington took advantage of a total solar eclipse visible from Príncipe off the west coast of Africa to make the measurement. Voila! Einstein was vindicated and an instant celebrity. General relativity took its rightful place in the canon, radically changing the landscape of physical theory forever.

There has been a truly enormous amount of scholarship on the 1919 eclipse results, their interpretation, and the epistemic fallout thereof. Analysis of this experiment covers much diverse ground, from arguing that the eclipse results were the pivotal difference-maker in the acceptance of general relativity (see also Reference KennefickKennefick 2009; Reference Kennefick, Lehner, Renn and Schemmel2012), that the public presentation of the results was significantly tilted by political motivation in the immediate aftermath of World War I (Reference Earman and GlymourEarman and Glymour 1980), and that the results were inconsequential for the wide acceptance of general relativity among members of the physics community in comparison to the theory’s success in retrodicting the advance of the perihelion of mercury, which had long vexed astronomers working in the Newtonian framework (Reference BrushBrush 1989; Reference Brush1999). I cannot hope to offer a comprehensive discussion of this scholarship here. Instead, I will endeavor to highlight a few points of particular interest in the work on the epistemology of this experiment over the past several decades. In particular, I will focus on the question of whether Eddington was epistemically justified in omitting certain data collected during this eclipse expedition.

The persistent pedagogical narrative of this experiment papers over the significant logistical difficulties involved in a way that ultimately makes the results seem more decisive than they were at the time. The experiment caricature (just compare two photographic plates and measure the distance between dots!) obscures the expertise and finesse needed to get any useful measurements in this context. John Earman and Clark Glymour characterized some of the fussy yet crucial details of the measurement as follows:

Of course, this sort of constellation of auxiliary concerns is not unique to the 1919 eclipse measurements. Nearly any interesting experiment has them. However, keeping these details in mind is crucial for understanding the epistemic significance of the results of the experiment. The eclipse expedition had two parties: with the authority of his title as Astronomer Royal, Sir Frank Watson sent Eddington and Edwin Cottingham to Príncipe and Andrew Crommelin and Charles Davidson to Sobral in Brazil. The Brazil contingent took plates with two instruments, an astrographic telescope from Greenwich that yielded nineteen plates on the day of the eclipse and another telescope with a four-inch aperture borrowed from the Royal Irish Academy (RIA) that yielded eight (73). Met with cloud cover in Príncipe, Eddington and Cottingham took sixteen plates, only two of which were usable (73). Einstein’s predicted deflection was 1.74” of arc while the Newtonian value is 0.87”. The original analysis of the results yielded values of 1.98” with a probable error of 0.12” (RIA telescope at Sobral), 0.86” with no reported probable error (Greenwich astrographic instrument at Sobral), and 1.61” with a probable error of 0.30” (Príncipe) (74–75). Earman and Glymour calculated the associated standard deviations in seconds of arc, yielding 0.178” (RIA telescope at Sobral), 0.48” (Greenwich astrographic instrument at Sobral), and 0.444” (Príncipe) (75). Earman and Glymour argue:

Note that the results from the Greenwich instrument at Sobral might arguably be consistent with the Newtonian value. Yet the interpretation Frank Watson Dyson delivered to the Royal Society was univocal: “After careful study of the plates I am prepared to say that there can be no doubt that they confirm Einstein’s prediction. A very definite result has been obtained that light is deflected in accordance with Einstein’s law of gravitation” (77). William Wallace Campbell of the Lick Observatory was still unconvinced by 1923, noting that insofar as the British chose to take the Príncipe results seriously but not the results from the Greenwich instrument at Sobral, “the logic of the situation does not seem entirely clear” (78). By 1920, Eddington reported the results as 1.98” from Sobral and 1.61” from Príncipe without mentioning the results from the Greenwich instrument at all (79). The analysis offered by Reference Earman and GlymourEarman and Glymour (1980) concluded: “The British results, taken at face value, were conflicting, and could be held to confirm Einstein’s theory only if many of the measurements were ignored. Even then, the value of the deflection obtained was significantly higher than the value Einstein predicted” (50–51). Indeed, Earman and Glymour put the point quite bluntly: “Dyson and Eddington, who presented the results to the scientific world, threw out a good part of the data and ignored the discrepancies” (85). What reason is there to trust the cloud-contaminated astrographic results from Príncipe whilst ignoring the cloud-free astrographic results from Sobral except that the latter were on Newton’s side? Did Dyson and Eddington engage in QRPs too?

The experiment came hot on the heels of World War I, during a time when there was a lot of skepticism, vitriol even, from scientists of nations in the Allied Powers toward those of the Central Powers, and considerable defensiveness on the part of the latter. Manifestos, public letters, condemnation, name-calling, and suggestions for boycotting members of the international scientific community circulated. Thus, an alternative explanation of Dyson and Eddington’s data analysis decisions and their interpretation of the data was available. Eddington wanted peace and unity among the scientific community after the war and championed Einstein as means to that end.

At the end of the day, Earman and Glymour argued that the personalities and extensive public advocacy of Dyson and Eddington were essential to the reception of the 1919 eclipse results as confirming general relativity (52). They describe Dyson as “a man of solid but not brilliant scientific accomplishments” who “was one of those people, familiar in every discipline, who exercise enormous personal authority well beyond the influence of their published work” (71). They conclude: “For Eddington, one of the chief benefits to be derived from the eclipse results was a rapprochement between German and British scientists and an end to talk of boycotting German science” (83). Was Eddington epistemically justified in leaving out the Sobral astrographic plates from his interpretation of the expeditions’ results? Or did social considerations disrupt the epistemology of this experiment?

Reference Collins and PinchHarry Collins and Trevor Pinch (1993) discuss the 1919 eclipse results in their well-known book The Golem: What Everyone Needs to Know about Science. The overall premise of this book is that science is a golem:

Collins and Pinch stress that all important science is controversial and empirical results are not decisive. Science, as a golem, is “not an evil creature but it is a little daft.” It is “not to be blamed for its mistakes; they are our mistakes,” and “powerful though it is, it is the creature of our art and craft” (2). This view puts science on a level with other human activities, which Collins and Pinch entreat us to appraise with the same ordinary logic we use in everyday contexts. They conclude:

I am sympathetic to this outlook in some respects. Certainly, science is a human endeavor and humans are at times magnificent craftspeople and at times bumbling fools. For Collins and Pinch, however, it is not the weight of empirical evidence that induces epistemic progress in science, it is the management of the scientific community that takes discordant empirical results and “brings order to this chaos, transmuting the clumsy antics of the collective Golem Science into a neat and tidy scientific myth” (151). This gives too much credit to social factors. We can often identify the expert steps from the misguided stumbles. If Eddington threw out serviceable data for no good epistemic reason, we can call foul and ask better of other scientists. Collins and Pinch imply that he did. They describe Eddington’s treatment of the results in the following way:

Collins and Pinch argue that Eddington did not have good empirical grounds to justify omitting the Sobral astrographic plates. They note that Eddington dismissed the plates as suffering from sources of systematic error, and state that if this were indeed true of the plates in question, “then Eddington would have been quite justified in treating the results as he did,” but “that at the time he was unable to educe any convincing evidence to show that this was the case” and instead made “after-the-fact determinations of what the observations were taken to be” (51). Instead of a decisive test of general relativity, Collins and Pinch explain this historical episode as a profound cultural shift toward the broad acceptance of relativity. After Eddington’s eclipse expedition, the interpretation of other previously uncertain phenomena, such as Einstein’s gravitational redshift, began to fall out on Einstein’s side. They use the analogy of crystal formation: “Once the seed crystal has been offered up, the crystallisation of the new scientific culture happens at breathtaking speed” (53). This spread, they suggest, was propagated by selectively omitting data: “Eddington and the Astronomer Royal did their own throwing out and ignoring of discrepancies, which in turn licensed another set of ignoring and throwing out of discrepancies, which led to conclusions about the red-shift that justified the first set of throwing out still further” (53).

Was there good reason to ignore the results derived from the plates taken at Sobral with the Greenwich astrographic instrument? Were there relevant sources of systematic error that would have justified throwing out those data for the purpose at hand? Daniel Kennefick has argued that there was. Kennefick’s interpretation, whether or not it ultimately withstands the scrutiny of further scholarship, illustrates the sort of reasons that would have been good grounds for omitting the Sobral astrographic plates from the data analysis.

The eclipse occurred on May 29, 1919. In Davidson’s diary entry dated May 30, 3:00 a.m., he wrote:

The response of the mirror to temperature posed a significant problem because changes in the scale of the plate image are difficult to distinguish from the phenomenon of interest (see Reference KennefickKennefick 2019, 190–192). A change in scale, just like the predicted phenomenon, would radically shift the apparent position of the stars. There is a difference: a change in scale would induce shifts greatest at the edge of a plate centered on the Sun whereas the phenomenon of interest would be greatest nearest the Sun. However, with few displaced stars to measure, these effects could unfortunately be indistinguishable in practice (191). The loss of focus of the Sobral astrographic instrument distorted the images of the stars to be measured, so that they were not circular. This was problematic because the team was trying to measure a sub-arc second effect using smushed star images that were three or four arc seconds across (192–193). Without being able to pinpoint the center of the star images, any displacement measurements taken would have been of dubious accuracy. The team attempted to determine the change in scale, but Kennefick suggests that their estimate may have been hampered by using only the right ascension coordinates of the images (193–194). Kennefick suggests that Dyson may have suspected that the team had not accurately determined the scale of these plates, and if Dyson indeed had that worry, “he was therefore justified in ignoring any result derived from that data” (195). The researchers were worried about the quality of the Sobral astrographic plates before data reduction had even begun, and thus before they would have noticed the lack of agreement of the deflection calculated from those plates with Einstein’s prediction (201). Collins and Pinch are therefore wrong to think that there was a tight circle of mutual confirmation between theory and evidence here. If Kennefick is right, the reasons that Dyson and company had for omitting the Sobral astrographic plates were independent of the results generated from them.

Reference KennefickKennefick (2009) offers the following possible reconstruction of the experimenters’ reasoning:

Insofar as the scientists involved in the 1919 eclipse expedition had reason to suspect that the usefulness of Sobral astrographic plates had been significantly degraded by the change in focus of the instrument that they noted before beginning data reduction, they had good reason to omit the deflection values calculated from those plates from the final analysis.

3.3 Lessons for Epistemology of Experiment

In this section I have discussed two well-known cases from the history and philosophy of science: Millikan’s oil drop experiments and the Eddington eclipse expedition of 1919. In both cases, the researchers omitted data from analysis. In the case of Millikan, Allan Franklin has argued that while the experimenter omitted some data for no good reason, and thus may be rightfully accused of “trimming” and even “cooking” the data, ultimately these questionable practices did not harm the long-term epistemic impact of the results of the experiments. I have argued that Franklin is too generous with Millikan and that leaving out data for no good reason ought to generally be regarded as epistemically detrimental even if it happens not to derail science in a particular instance. We cannot count on the benefit of hindsight to judge whether such practices are permissible in science. Omitting data without just cause, and in particular, simply in virtue of the fact that results generated from those data disagree with the experimenter’s preferred outcome, is bad practice.

In the case of the eclipse expedition, Eddington has been accused of omitting data that failed to agree with his preferred outcome, in particular omitting the estimates of the general relativistic effect of the sun on starlight near its limb as calculated from one of the three instruments used in the eclipse expeditions of 1919 in which he participated, because that calculation did not confirm the value Einstein predicted. However, Daniel Kennefick has argued that the focus on Eddington is misguided, and when one examines the reasoning of Davidson and Dyson, one can see that they had good reason to omit the data in question from the final analysis. Far from throwing the data out based on their disagreement with Davidson and Dyson’s anticipated or preferred outcome, Davidson and Dyson recognized a problem with the functioning of the instrument during the process of taking data that gave them reasonable grounds to doubt the utility of their results for the scientific purpose at hand.

The lesson displayed in both of these cases generalizes. To throw out data just because they are inconvenient is bad epistemic practice, but it can be reasonable to throw out data due to instrument malfunction. Instrument malfunction is an appropriate excuse. How can one tell that the equipment is not working properly? This question brings us back to the heart of Collins’ experimenters’ regress argument. If there is no way to tell whether the experimental equipment is functioning properly other than by obtaining the expected result, then there is no substantive distinction between throwing out data just because they fail to produce the anticipated result and throwing out data because the “equipment is not functioning properly.” The following section explores this issue still further, by investigating the ways in which experimental physicists set about determining whether their apparatus is working appropriately for the purposes of the experiment they are trying to perform.

4.1 Calibration

As Reference PerovićPerović (2017) explains, Franklin has responded to Collins’s emphasis on the experimenters’ regress and its relation to what Collins characterizes as the social construction of knowledge by drawing attention to the important role of calibration procedures in arguing for empirical results. As a rough-and-ready example of calibration, which we will shortly complicate several times over, consider the process of calibrating a leak detector. A leak detector is a device used to identify and measure the magnitude of leaks in a vacuum system. Such an instrument could be, for instance, a vacuum pump equipped with a small device for detecting the presence of helium. An operator of the leak detector can then join the part or system to be tested to the inlet of the detector, evacuate it using the leak detector pump, and then carefully introduce small amounts of helium to areas on the outside of the part or system being tested to try to localize leaks. It is useful to be able to measure the severity of leaks in the system so as to ensure that the vacuum standards of the experiment in which the parts are to be deployed is met. For such measurements to be accurate, the leak detector must be calibrated. This can be done, for instance, by attaching an external leak standard, a canister filled with helium of known leak rate, probably obtained from the detector’s manufacturer, to the leak detector and setting the detector’s scale appropriately or running whatever more complicated calibration procedure is required. Although a leak detector is usually an auxiliary instrument to the main experimental apparatus, the same sort of procedure could apply to an instrument used for science data collection as well.

Franklin has argued that calibration disrupts vicious circularity in the experimenters’ regress. The vicious regress relies on the idea that a properly functioning experimental apparatus is known to be properly functioning only insofar as the “correct” or expected result is obtained. A vicious regress makes room for non-epistemic social factors to be decisive in the interpretation of the results and the judgment of their success. However, Franklin has stressed that many experiments do not display vicious regresses of this sort. Rather, the apparatus is established to be working properly by calibrating it using something besides the success condition of the experiment before data deemed scientifically interesting are even taken. Franklin characterizes calibration as “the use of a surrogate signal to standardize an instrument” and stresses the invocation of such standardization procedures for arguing that one’s instrument is working correctly or for worrying that it is not (Reference Franklin1997, 31).

Unfortunately, it does not seem that calibration modeled on paradigmatic cases alone can do the work of breaking vicious regresses in more exotic contexts. After describing how the scale of a new voltmeter might be calibrated by way of known voltages, Collins remarks:

In the context of efforts to detect gravitational waves, a context that Collins returns to again and again, the novelty of the phenomenon makes “direct” calibration of the instrument impossible. Gravitational waves, “standard” or otherwise, cannot be procured from the supply cabinet like 5V batteries. In such circumstances, the surrogate nature of calibration standards seems to leave an uncomfortable inferential gap between the success of the calibration stage and the judgment that the instrument is operating properly for the purpose of measuring the previously unmeasured. In a recent conciliatory publication jointly authored by Collins and Franklin, Collins emphasizes the significance of social factors in tricky experimental contexts: “The so-called ‘epistemological criteria’ are necessary for establishing the existence of a new phenomenon (as Allan says) but they are not a sufficient criterion where dispute runs deep” (Reference Franklin2016, 100).

Franklin has responded extensively to Collins’s discussion of Joseph Weber’s attempts to detect gravitational waves, arguing that Weber’s apparatus failed the relevant calibration tests (Reference Franklin1997, 46). That is, before even worrying about the inferential gap involved in interpreting purported detections of novel phenomena, Weber’s instrument had an opportunity to show that it was suitable for the task ahead, and failed. Without handy gravitational wave standards, the detectors in these experiments were subjected to surrogate signals: “Scientists injected pulses of acoustic energy into the antenna and determined whether their apparatus could detect such pulses. Weber’s apparatus failed to detect the pulses, whereas each of the six experiments performed by his critics detected them with high efficiency” (44). According to Franklin, the crucial difference between Weber’s experiment and his competitors’ was the algorithm he used to search for signals in his data. Weber’s algorithm was nonlinear while the competitors’ was linear. Weber was evidently concerned that a linear algorithm would miss the gravitational wave signals. Addressing this worry, the competitors also checked to see if using the nonlinear algorithm on their own data would yield the detection that Weber claimed – it did not (45). Franklin concludes that in light of Weber’s failure to properly calibrate his instrument using plausible surrogate signals considered together with several other problems with Weber’s analysis, the physics community rightfully rejected his detection claims based on epistemological criteria (49).

Eran Tal’s sophisticated account of calibration complicates the interpretations presented thus far. Tal argues that we are mistaken to think of calibration primarily as adjusting the readout of one instrument to reflect that of a standard, concluding that “comparison with a standard is neither necessary nor sufficient for successful calibration” (2017a, 246). If Tal’s argument is correct, then it seems that we will have to revisit the responses to the threat of the experimenters’ regress in order see if compelling arguments against it can nevertheless be made even with an understanding of calibration modified by Tal’s insights. In order to do this, we had better understand Tal’s critique of other accounts of calibration and the nature and implications of his own view.

4.2 Commissioning

Empirical research often relies upon apparatuses that take some serious time and attention to get up and running for their proper use in science. Speaking broadly, we can break the preparation for an experiment into a sequence of phases: design, construction, and commissioning (which may include “engineering runs” and calibration). Depending on the nature of the experiment, there may be other preparatory phases too such as prototyping and simulation. The commissioning phase in particular serves as a basis for the epistemic significance that the researchers accord to the results of the experiment. In this section I offer a preliminary discussion of commissioning and its epistemic significance in the epistemology of experiment, using the KATRIN experiment as an illustrative case.

The term “commissioning” is also used in primarily engineering contexts, such as the operation of nuclear power plants. As one article explains: “The results of commissioning have to demonstrate that the requirements and intentions of the design and the intentions of the designers, as stated in the safety analysis report, have been met and that the unit is ready for a long-lasting and successful operational phase” (Reference Grauf and AlonsoGrauf 2012). The role of a commissioning phase in research contexts is not dissimilar. This is no wonder, since preparing an apparatus for science involves engineering work. For example, Richard Hills, the project scientist for the massive Chilean radio array ALMA, stated in a presentation to the ALMA Science Advisory Committee that, quite simply, the point of their commissioning efforts would be to “Make the system into a telescope – one capable of making the specified astronomical observations” (Reference Hills2009, slide 6). For ALMA, the commissioning phase involved design tests and debugging the electronics, antennas, infrastructure (like power), and software (Reference HillsHills 2009).

To take another example, the publication reporting the results of the KArlsruhe TRItium Neutrino (KATRIN) experiment collaboration’s commissioning of the vacuum system of their main apparatus frames the success of that phase of work as follows: “The vacuum system has to maintain a pressure in the 10^–11 mbar range. It is demonstrated that the performance of the system is already close to these stringent functional requirements for the KATRIN experiment” (Arenz, Babutzka, Bahr, et al. 2016, abstract). For KATRIN, commissioning the vacuum system of the main spectrometer included, for instance, baking everything that could be baked to promote outgassing from the metals used in the vacuum systems and finding and fixing leaks in the system that would prevent achieving the vacuum needed for the desired experiment.

Roughly, then, in scientific contexts, commissioning refers to the preparation of an apparatus for routine performance according to the aims of the research context. These days, at least for larger collaborative experiments, these research aims and the performance requirements they imply are often explicitly stated and published in design articles prior to beginning the construction of the experiment. Indeed, significant research is needed to specify these research aims and requirements in the first place. While the successes of the commissioning phase of an experiment might seem primarily pragmatic – for example, an ultra-high vacuum was achieved – these pragmatic wins have epistemic import, as I aim to make clear in the following discussion.

Generally speaking, there are two particularly important benchmarks in a commissioning phase. The first is when the apparatus is sufficiently assembled and functioning for basic operation. In the context of a telescope this is often referred to as “first light,” when the telescope is sufficiently a telescope to make some astronomical image. That image may be quite far from being of any serious scientific interest, due to much required further work. Similarly, in accelerator physics like that conducted at the LHC, the “first beam” of an accelerator is celebrated when a beam can be successfully produced and steered through the instrument. For KATRIN, it was important to demonstrate the successful transport of electrons from their tritium source (after practicing with non-tritium-sourced electrons and ions) through to the detector end of the apparatus.

A second important phase marks the transition from taking “engineering data” to “science data.” After the primary function of the instrument has been demonstrated (“we can see something!”) operations still need to be developed and checked to have a hope of satisfying the aims of the experiment. Like other aspects of experimental methods, the precise nature of the procedures used to transition an instrument from basically functional to the realization of its full science capabilities will vary with context and are worth investigating in detailed case studies.

As a brief illustration, consider the efforts of the KATRIN collaboration in the period between when they first injected the system with tritium and when they were confident enough to publish their first major science result improving the upper limit on the neutrino mass. The purpose of the KATRIN experiment is to estimate the mass of the electron antineutrino by measuring the shape of the end point of the energy spectrum of beta decay electrons from tritium. Solar neutrino experiments demonstrating that neutrinos oscillate between their three flavors imply that neutrinos are not massless as stipulated by the Standard Model of particle physics. Tritium beta decay yields helium-3, an electron, and an electron antineutrino. By studying the energy spectrum of the electrons produced by these decays, the KATRIN collaboration looks for a small distortion in the tail of the spectrum where the nonzero mass of the election antineutrino should nibble away at energy that otherwise would have been imparted to the electron. The KATRIN experiments aims to measure this neutrino mass with a sensitivity of 0.2 eV. Several significant technical challenges to meeting that goal were identified in the design stage of the experiment:

Any one of these elements is ambitious enough to make a seasoned experimenter nervous. While working as an engineer at one of the laboratories involved in KATRIN, I was told that one collaboration member, overwhelmed by the variety of catastrophic failure modes of the experiment, referred to it as something like “the flying purple unicorn.” A single superconducting magnet can be temperamental and dangerous. Add to that “challenging” high voltage, high vacuum, and cryogenic requirements, plus a windowless gaseous (i.e. scary) highly pure tritium source, and you have a precarious situation. Careful commissioning to test out these diverse and difficult specs of the experimental apparatus were thus essential to ensuring that the desired experiment could run.

The 70-meter-long KATRIN apparatus includes the windowless gaseous tritium source from which the tritium decays, a transport and pumping section through which the beta decay electrons are guided by superconducting magnets to the main spectrometer. The main spectrometer vessel is a stainless steel tank of 1,400 m³ in volume, weighing approximately 200 tons, which is supposed to maintain an ultra-high vacuum. This section of the instrument was constructed in Deggendorf and is so large that it could not be transported directly overland to Karlsruhe. The collaboration explained. “There is a slight problem of transportability from Deggendorf to Karlsruhe: The tank is too big for motorways, and the canal between the rivers Rhine and Danube has to be ruled out too. Thus instead of a journey of about 400 km, the spectrometer has to travel nearly 9000 km” down the Danube to the Black Sea, through the Aegean and Mediterranean Seas through the Strait of Gibraltar, up through the Atlantic to the Rhine from the other end (www.katrin.kit.edu/213.php). There is an outstanding picture from this expedition of the hulking vessel of the main spectrometer escorted by police cars through Leopoldshafen to the delight of a packed crowd of onlookers, at the moment where it is barely scraping between the roofs of two houses, looking like an embarrassed grounded blimp. An enlarged and framed version of this photograph greeted me every day as I walked through the front door to the lab when I worked at CENPA, the feat proudly displayed on the wall next to our mailboxes. Just think: Boyle thought he had problems with leaks!

The mass spectrometer transports only electrons whose kinetic energy is above a certain threshold, which can be altered by the experimenters. Finally, at the business end of the mass spectrometer, there is a focal plane detector that counts the electrons. With the beamline components in place, in 2016, the collaboration tested the alignment of the magnets, demonstrating that the instrument could transport electrons and ions and also block positive ions (Aker, Altenmüller, Arenz, et al. 2020, 3). The following year, they tested spectroscopic performance with a ^83mKr source and checked the calibration of their high voltage system to the parts-per-million level (ibid.). Satisfied with these preliminary commissioning procedures, the collaboration was ready to introduce tritium to the system. In the “First Tritium campaign” (FT campaign), the source was limited to 0.5% of operational activity by mixing the tritium with pure deuterium as a safety precaution. For the collaboration, “[a] key aspect of the FT campaign was to demonstrate a source stability at the 0.1% level on the time scale of hours” (ibid.) They were able to demonstrate time stability of key parameters affecting the stability of the source within design specifications over 12 days — success (ibid.). Other indicators of source stability were also checked. For instance, the rate of electrons that make it through to the detector also indicates source stability, and this rate “was demonstrated to be stable on the 0.1% level over a duration of 5 h” (ibid., 4). The tritium commissioning campaign had other objectives as well:

They tested 30 different thresholds for the electron kinetic energies, in a larger range that would be employed in the actual measurement procedure in order to obtain significant statistics despite the reduced source activity, “gain confidence in our calculation of the spectrum over a wider interval” and “perform a search for sterile neutrinos in the 200 – 1000 eV mass range, which is the subject of a separate publication” (ibid. 5). Based on prior research efforts to measure the neutrino mass, the collaboration was justifiably concerned about the sensitivity of the spectral shape to unknown systematics: “The analysis heavily relies on a precise description of the spectral shape including all relevant systematic effects and a robust treatment of systematic uncertainties. Any unaccounted-for effect and uncertainty can lead to systematic shifts of the deduced neutrino mass” (ibid.) For reassurance, the collaboration assigned two teams to carry out the analysis independently using different calculations and software. When the teams returned results that agreed within 4%, the collaboration reported that gave them “high confidence in our analysis tools” (ibid.) This analysis used only 82 of the 116 “scans” (a full sequence of all of the energy thresholds) recorded during the FT campaign (ibid., 8, although curiously page 5 gives the total scan number as 122 — I have asked a KATRIN collaboration member about this who said she would inquire about it internally, and have as yet to hear the explanation). The collaboration refers to this subset as the “golden” data set. They account for the omitted scans as follows:

The collaboration investigated various sources of systematic error including the column density, tritium concentration, the probability of endpoint electrons losing energy due to inelastic scattering, magnetic fields of the approximately 60 magnets included in the instrument, electric potentials in the source and spectrometer, rotational and vibrational states of the molecular tritium used in the source, and the efficiency of the detector, explaining how these were monitored and the extent to which the variations recorded would be tolerable in experimental conditions. The collaboration observed a 350 mcps background rate, which they hoped to reduce to under 100 mcps with further measurements and refinements (ibid., 12). They were able to demonstrate that the final results of their analyses were independent of the source column density and of the scanning mode (increasing voltage vs. decreasing voltage vs. random mode), the energy range of data used in the fit, and the spatial distribution of the pixels in the detector (ibid., 15). The collaboration concluded: “All these properties are essential prerequisites for the neutrino mass measurements” (ibid., 16). With this work accomplished, the collaboration looked forward to increasing the source activity to the nominal operating value.

Of particular interest in relation to the experimenters’ regress, is the fact that the FT campaign measurements, performed during the commissioning phase, did not seek to produce the phenomenon of ultimate interest. In particular, the experimenters did not attempt to set a limit on the neutrino mass in the FT campaign. Because the FT campaign used a source of significantly reduced activity compared to nominal operating specifications, measurements made during the campaign would only have been sensitive to neutrino masses at around 6 eV, which was significantly larger than the best mass measurements that had already made by other experiments at the level of 2 eV (ibid., 4). In fact, for the purposes of the FT campaign, the experimenters set the neutrino mass parameter to zero in their analysis and instead used measurements of value of the spectrum endpoint “as a proxy to evaluate the analysis results” (ibid.) Rather that aiming to demonstrate the instrument’s capacity to successfully measure the very phenomenon of interest in the experiment to come, the function of these measurements during the commissioning phase was to try out and to compare various possible analysis approaches, test the relevant software and run other validation checks that would ultimately be useful in arguing for their experimental results down the line. One could object to that, like the experiment ultimately desired, the campaign recorded the end point of the beta decay electron energy spectrum thus suggesting that “success” of the commissioning campaign depended on the detection of the very phenomenon of interest in the experiment proper thereby instantiating Collins’ regress. However, this interpretation would miss the fact that success of the experiment would ultimately be achieved at 0.2 eV sensitivity.

In their publication reporting the KATRIN collaboration’s first measurement of the upper limit on the neutrino mass, they present results from four weeks of data-taking (Aker, Altenmüller, Arenz, et al. 2019). By having “commissioned the entire setup by a series of dedicated measurements” over the course of several years, which “demonstrated that all specifications are met, or even surpassed by up to 1 order of magnitude, except for the background rate”, the collaboration was willing to take science data (ibid., 4). The over-spec background rate had been studied in detail and attributed to decay products of ²¹⁰Pb implanted on the inner surface of the spectrometer during construction from exposure to ambient air (ibid.). Another source of higher-than-anticipated background events is ²¹⁹Rn atoms from certain vacuum pumps attached to the main spectrometer. The collaboration attempted to remedy this problem by installing special baffles at the inlets of the pumps, but a layer of H₂O covering the inner surface of the main spectrometer “originating from an imperfect bake out of the prespectrometer” introduced a layer of H₂O on the baffles themselves, disrupting their ability to trap the offending ²¹⁹Rn (ibid., 5). Due to problematic drifts in the source column density, attributed to “radiochemical reactions of T₂ with the previously unexposed inner metal surface of the injection capillary”, the experiment ran at a column density a factor of 5 below the nominal operating value (ibid., 5).

Recall Collins’ experimenters’ regress. Introducing it in his book Changing Order in the context of his extended discussion of early efforts at constructing gravitational wave detectors, Collins characterizes the regress as follows:

Explicating the threat of regress in this way portrays the experimenters’ efforts to check that the instrument is working as needed to perform the experiment of interest and the experiment itself as identical. This is deeply misleading because experimenters do a lot of work to assure themselves that their experimental apparatus is ready for science applications before they are willing to record data that they will take seriously with respect to the primary aims of the experiment. Calibration, often lots of it, is generally a part of what happens during the commissioning phase of an experiment, when this preparatory work is accomplished. But if Perović is correct in his analysis of in-situ calibration at the LHC, calibration of a certain nature may extend far beyond the initial preparatory stages of an experiment, perhaps even becoming intimately intertwined with the measurement process. Even in a much more routine sense, calibration procedures may feature in the experiment throughout its operational phase. In KATRIN, for instance, the rear end of the tritium source section of the beamline is capped by an electron gun for use as a calibration source as needed (Aker, Altenmüller, Arenz, et al. 2020). Calibration is part of what happens during commissioning, but neither exhausts the activities of commissioning nor is limited to the confines of that phase of the experiment.

As can be seen from the example of KATRIN’s FT campaign, which was only one dramatic part of the longer commissioning phase, the tests, checks, and troubleshooting that occur during commissioning are important aspects of the epistemic support the experimenters offer for their ultimate results. In this phase, the experimenters, engineers, and technicians attempt to demonstrate that the instrument “as built” can in fact perform to design specifications demanded by the science goals. They try out different experimental strategies and finalize decisions about how to run the intended experiment. They uncover unanticipated difficulties (like higher than desired background rates) and do what they can to understand and remedy them so that the experiment can go on. The critical work accomplished in this phase furnishes some of the important arguments that the researchers need in order to justify their ultimate interpretations of the results of the experiment. Philosophers interested in the epistemology of experiment would evidently do well to explore the methods and arguments employed in commissioning phases of experiments, in detailed case studies.

Explicitly naming a phase of an experiment “commissioning” is a widespread practice today. For large, complicated, expensive, and technically difficult experiments, such a phase is particularly prudent. Experimenters are often wise to try out part of their instrument (e.g. a small fraction of the antennae that will ultimately form a large array), their instrument at reduced power (as when accelerators initially run at diminished energies compared to that of which they are capable at full operation), or with surrogates that are less dangerous or finicky (such as relevantly similar radioactive isotopes of reduced activity). However, it also seems plausible to look for something functionally serving as a “commissioning phase” in many experimental contexts from the past or at smaller scales. Franklin forgives Millikan for not including the oil drop measurements he made in his trials between October 28, 1911, and February 13, 1912, in his final calculations of the fundamental charge published in 1913, because Millikan was not yet confident in his apparatus (Reference Franklin1986, 230). This seems appropriate. It is reasonable to begin to use and troubleshoot one’s instrument before recording science-worthy data. Indeed, the successes or failures of this preparatory phase of the experimental work is an important part of what eventually makes the results of the experiment compelling or not.

Although these issues deserve further attention, my aim has been to provide some preliminary reasons in support of the idea that grounds for some of the arguments that scientists must make in order to argue that their experimental apparatus works to their satisfaction – arguments that their instruments are properly calibrated and operating, that the functioning of those instruments is adequately understood for the purpose at hand, that plausible sources of significant error have been eliminated or accounted for, and that the data processing to be used is appropriate for the desired application – are often found in calibration and commissioning activities. The latter in particular deserve further philosophical attention.

4.3 Further

My aim in this Element has been to introduce you to some of the main themes of the scholarship thus far on the epistemology of experimental physics and to offer some reflective commentary. I have chosen to recount some of the cases and arguments that stood out to me in particular as I was introduced to the epistemology of experiment in physics, and to focus on auxiliaries, regress, and excuses. Admittedly, this is a conservative approach. Many worthy topics and perspectives have been left out. While Bacon’s observations of hot horse shit and the odd contents of Boyle’s larder are granted several paragraphs, I do not attempt a discussion of Ibn al-Haytham’s much earlier sophisticated investigation of optics or epistemology. I do not discuss the extent to which there are interesting differences between the epistemology of experiment in physics and the epistemology of experiment generally, or in other specific scientific disciplines such as biology. The rise of Bayesian approaches, machine learning, and statistical methods broadly speaking in experimental physics deserve further attention. So do issues surrounding noise, artifacts, systematics, uncertainties, and error. Recent work on the social epistemology of experimental physics, for instance on collaborations and incentive structures, is also absent. The literature on science and values is mostly passed over. Conflict between epistemic priorities and moral imperatives – I think particularly of the reckoning necessary in the astronomy community with regard to relationships among Indigenous peoples and mountains with favorable seeing – deserve thorough and lucid treatment elsewhere.

For those who will continue to explore these topics, the cases and arguments of this Element offer the following advice. Key decisions made in empirical research — the judgment that the instrument is working properly, for instance, or that certain data should be omitted from analysis — require epistemic justification. The fact that such justification is required as a general feature of empirical research can therefore be represented in the epistemology of experiment, perhaps along with some mid-level strategies of the sort that occur in commissioning such as using well-characterized test signals and demonstrating that required design benchmarks have been achieved concretely. Yet the reasons that can appropriately serve as justification for key decisions in empirical research are often furnished by particular contextual details. Philosophical investigation of the epistemology of experimental physics thus needs to attend to such details if it aims to deliver normative arguments regarding science in practice.