I. Cerberus and the Fluid, Relational Patterns of Suspicion

As noted above, in contrast to Helios—a watchlisting system for “known” threats—Cerberus is oriented towards the detection and pre-emption of security risks that are not registered in intelligence and information systems. As one Home Office interviewee explained, “[t]argeting is to identify things you do not otherwise know . . . [and] the whole point of Cerberus is to identify the unknowns.”Footnote ¹¹⁵ The identification of these “unknowns,” as explained above, is based on the distillation of “characteristics” from historical data—“what were the characteristics of the data [that] we were interested in before”—enhanced by a graph of “associations between different entities,” which crystallizes in forms of “concern by association.”Footnote ¹¹⁶ The relation between watchlisting and risk assessment thus follows a “two pass approach”: While Helios provides a picture of “[what] we already know about them,” Cerberus asks: “[D]o they exhibit a pattern we are interested in? . . . [T]hat’s the first branch of your decision tree. And then there’s . . . a second layer to that, which is: Do we know what we want to do with them, and what action do we want to take?”Footnote ¹¹⁷ To situate the importance of these questions, we need to place Cerberus in a longer institutional chain of decision-making at the border:

These leads are designed to take the specific socio-technical form of a color-based risk code, separating “reds,” the “needles in haystacks” who pose a risk or threat, from “greens,” “people [who] don’t do anything wrong” and who “get through,” and “ambers,” who are not quite red because “they haven’t done anything illegal . . . but they’re [also] not a green because they’ve done something abnormal.”Footnote ¹¹⁹ While the metaphor of the “needle in the haystack” has a long legacy in post-9/11 counterterrorism, data mining, and surveillance, Cerberus provides a new perspective on this “comforting pastoral imagery of data agriculture.”Footnote ¹²⁰ “Historically,” a senior data analyst behind Cerberus explains:

Resonating with the translation of standards of “reasonable suspicion” into a construction of “concern by association,” the identification of these “ambers,” an interviewee clarifies, is “fundamentally” about distilling and “applying patterns to data, to find the people who [fit] those patterns of interest.”Footnote ¹²² This determination of risk is not only tied to general “patterns” and “characteristics” detected in data on a population level but also to an individual’s own past behaviors or routines. As one interviewee noted: “[A]bnormality is what we’re looking for . . . [or] anomalies in individuals.”Footnote ¹²³ “Risk” is therefore not tied to particular legal identifiers or individual data points—nationality, country of departure, age, et cetera—but emerges from the relation between these data points and a person’s own history, pointing to possible behavioral “anomalies or abnormalities,” as well as historical patterns with other data points that signal a propensity to “risk”—a mode of “concern by association.”Footnote ¹²⁴ There was an insistence among interviewees that Cerberus nonetheless works as a “rules-based targeting” system,Footnote ¹²⁵ and does not yet employ ML systems that would autonomously alter the assessment criteria—or “risk indicators”—employed at the border.Footnote ¹²⁶ Yet, it is crucial to underline that such “rules” are nothing but a negotiated, evaluated, and temporary formalization of detected “patterns” and “characteristics” composed as chains of associations and clusters of attributes inferred from data, whatever the source and composition of that might be.Footnote ¹²⁷

This displays the first characteristic of the dispositif of speculative suspicion: Its relational, fluid character. Cerberus enacts difference based on shifting combinations of attributes that are continuously open to revision.Footnote ¹²⁸ Many of the key critiques of algorithmic governance—on the fragmentation of publics,Footnote ¹²⁹ the foreclosure of human natality and indeterminacy,Footnote ¹³⁰ the erosion of non-discrimination standards,Footnote ¹³¹ or the problem of transparency—could to some extent be traced to this operationalization of pre-emption based on “patterns” of anomaly and association.Footnote ¹³²

II. Cerberus and the Fragile, Heterogenous Composition of Suspicion

Yet, based on our study of Cerberus, we caution against overstating the agency of the algorithm in this process of detecting and targeting “unknown” threats. Many critical scholars have traced how this process is reshaped by tools of unsupervised ML for pattern recognition,Footnote ¹³³ anomaly detection,Footnote ¹³⁴ or clustering.Footnote ¹³⁵ As a result, shifts in decision-making practice are often diagnosed as a function of the specific AI use cases that are deployed. Such analyses risk overlooking the mundane, analogue institutional practices that mediate—and often overrule—algorithmic systems as well as the composite, heterogenous circuits that these systems hinge upon to become institutionally meaningful and actionable. In short: We caution against analyzing the governance effects of Cerberus as merely resulting from the internal logic of a particular computational code or AI system. This is to some extent a matter of sequence and adoption. While the use of AI and ML is indeed envisaged in the development of Cerberus, the current project is far from the promises of seamless automated algorithmic prediction—“it’s all a bit steam-powered behind the scenes,” one of its designers notes.Footnote ¹³⁶ Yet, if in this section we take you on a tour “behind the scenes,” this is not only to trace the analogue techniques integral to the initial iterations of Cerberus but, more importantly, to show why it is crucial to remain attentive to the inevitable “steam-powered” processes implicated in systems usually described as automated or algorithmic. If we want to grasp and meaningfully problematize the normative operations of infrastructures like Cerberus, in short, we should be attentive to these more-than-human compositions, and cautious of reproducing forms of technological determinism.

To map the infrastructural interrelation between the analogue and the digital, the human and the algorithmic, a useful entry point is to study how the targeting rules that are engendered and deployed through Cerberus are crafted, vetted, and deployed. How does Cerberus arrive at actionable color codes of suspicion? How and by whom are “patterns” and “characteristics” in data observed or distilled? How are the rules emerging from those optimized and revised? How are tentative “hypotheses” of meaningful patterns generated and evaluated? These are crucial questions in our infra-legal study: The regulatory effects emerging from the infrastructural formation of Cerberus can only be grasped by tracing the fragile, distributed and more-than-human conduits by which the essential targeting rules are authored, authorized, and enacted. This analysis allows us to evaluate both the institutional formation of the system—which entails a specific distribution of agency—and the normative nature of the rules on which it hinges. To find our way through the complex, co-authored composition of these rules, it is useful to break down the process into three consecutive stages.

The first stage in this process—the current development stage of Cerberus—was described to us as an attempt to:

In other words, to create a “risking layer” that “get[s] rid of the hay,” one interviewee noted, “we’re effectively encoding professional expertise” by “put[ting] into the system what our people tell us is risky.”Footnote ¹³⁸ There is a technical reason for this approach: “It’s a . . . good way to start, because we know that these are proven patterns.”Footnote ¹³⁹ “[V]ersion one of the [targeting] rule,” in other words, hinges on the codification of “historical, corporate knowledge” reflecting “[patterns] which our teams already know generate positive outcomes.”Footnote ¹⁴⁰ Yet, in addition to the technical reasons for “encod[ing] corporate knowledge,” the central driving concern is of a rather different nature: “[T]here is a cultural aspect to this, which is to say, the size of the transformational step we’re providing to you is small . . . . It’s not scary, and you can use it, and let’s get on with it.”Footnote ¹⁴¹ As one interviewee explained:

This “cultural aspect,” considered crucial in the design of Cerberus and the labelling of data, is further elaborated in Section D. Our key point for now is that the first iteration of the targeting rule is a codification of existing knowledge that could hardly be more analogue:

Importantly, this coding of “historical, corporate knowledge” the “proven patterns” produces a continuous feedback loop: Once you have a “captive audience,” a data analyst explained, “they’re using the system [both] to consume [and to] generate data.”Footnote ¹⁴⁴ This feedback process is infrastructurally mediated by the format in which information is integrated: “[A]s part of our system development, [we are] introducing . . . auto-generated fields, pre-population [and] codified fields. Which means we’re trying to . . . reduce the amount of free text that people are writing in, trying to standardize the data capture.”Footnote ¹⁴⁵ Through this socio-technical mediation in the design of the user interface, data is “structured … to a higher level of quality.”Footnote ¹⁴⁶ This allows the Cerberus team “to label up the data with … features of interest” for “future ML.”Footnote ¹⁴⁷

If the creation of “patterns” and “characteristics” for “rules-based targeting” initially hinges on mundane techniques for “sucking up professional knowledge,” the next stages of Cerberus hinge on the promise of AI and ML. In the second stage, shaped by supervised models, a data engineer explained that the “risk analysis will inevitably be about machine learning based optimization.”Footnote ¹⁴⁸ This optimization process entails a dynamic of testing and tinkering: “[O]ur risk framework may not change, but the configuration, and the weights, and so on, of those risk rules may change.”Footnote ¹⁴⁹ Identified targeting “rules” are thereby “run through an algorithm that says . . . risk rule one is always brilliant, it always leads to a positive outcome, so appoint that. Risk rule three is rubbish, it always leads to a negative one, switch it off.”Footnote ¹⁵⁰ Initial errors are both anticipated and perceived as generative in this process: “[W]e know that our weightings are probably going to be a bit off for a little while, and so we need to . . . run it through the testing process [and] kind of tweak and tune them.”Footnote ¹⁵¹ In this stage, ML tools are deployed to continuously evaluate the performance of existing targeting rules. These rules, in other words, only ever attain a tentative, temporary salience—their weights within the calculus of risk are continuously readjusted as the threshold values of performant patterns evolve with every border passage, or, at least, this is the aspiration of this algorithmic optimization process.

In the third stage, reflecting the ultimate promise of ML that is still far from the capabilities of Cerberus “[u]nsupervised models [could be deployed to] propose new hypotheses” of what constitutes a “pattern” or “characteristic” of “risk.”Footnote ¹⁵² Fed by the standardized and labelled data in the system, unsupervised learning tools can “run over the data, and find patterns, and then present them back to people to evaluate what those patterns are trying to tell us.”Footnote ¹⁵³ As Pottage notes, this approach entails a form of “abductive reasoning” which “elicits and presupposes randomness, incompleteness and contingency.”Footnote ¹⁵⁴ What it produces, Pottage argues, “is not so much prediction as prehension … Whereas prediction still connotes induction—unknowns are anticipatedly brought under a pregiven concept—prehension is a novel faculty, which generates speculative hypotheses in the process of deploying them.”Footnote ¹⁵⁵ Since such “cluster analysis” can, however, perfectly come up with a “mad pattern”—“people on a Wednesday travel with cheese”—this requires further feedback loops where targeting teams “assess the pattern, assess the cluster, and [potentially] disregard it as a risk indicator.”Footnote ¹⁵⁶ The direct incorporation of algorithmically distilled “patterns” as actionable risk indicators is therefore not something currently envisaged in Cerberus: “It wouldn’t be something that we’d just say . . . this is an interesting thing to look at, therefore we’re going to apply it. There would always be that human due diligence.”Footnote ¹⁵⁷ Yet, in producing hypotheses of emergent and potential risks, “it has found you something that no person would have found you. [T]hat’s the … next evolution of this [algorithmic bordering infrastructure].”Footnote ¹⁵⁸

Additionally, frontline operational officers who are primarily actioning existing targeting rules—which are composed by Cerberus and reviewed by targeting teams—can themselves “spot behavioral trends and traits, that no system will ever see, if it’s just all about the data.”Footnote ¹⁵⁹ Rather than passive recipients of actionable data, they are therefore also enrolled as “a source of new hypotheses” for Cerberus.Footnote ¹⁶⁰ They might “find something which we can … explore, and codify when it’s appropriate.”Footnote ¹⁶¹ Existing “professional expertise and professional practice” is thereby not only codified in initial “patterns” and “characteristics” for “rules-based targeting” but also serves to produce “new hypotheses” for patterns of suspicion that can crystallize in actionable rules. In other words, algorithmic and analogue techniques of “hypotheses generation” are “considered as being in concert”: “we spot emerging trends that are different to the ones that they spot, because we’ll be looking at all the data, and they’ll be looking at . . . the one person in front of them.”Footnote ¹⁶²

Describing the incremental process of inferential analysis, one interviewee therefore noted that “the data is effectively like a snowball . . . it goes through different parts of the process, and it accretes additional contributions from . . . the system, or from the person who’s looking at it.”Footnote ¹⁶³ This is perceived to produce a “virtuous circle” where, “over time, that will then lead to a place where we . . . have this flow of data from initial acquisition all the way through to getting the operational response, which we can then feed back in.”Footnote ¹⁶⁴ The “virtual circle” envisaged here ties together frontline officers, targeting teams, and algorithmic tools of pattern recognition in an ongoing dynamic of hypothesis generation, evaluation, optimization, and operationalization, which is itself the source of new hypotheses, et cetera. What is enacted through this socio-technical architecture is a circular, more-than-human cybernetic chain of feedback loops in which we find no centralized location of normative agency from which stable standards of suspicion emerge, and in which targeting rules emerge only as speculative signals tied to patterns temporarily meeting a threshold value.

This three-stage analysis of Cerberus’s targeting processes underscores the fragility and heterogeneity of the speculative suspicion it composes and puts into effect. Data analysts, targeting teams and frontline officers collectively compose actionable risk indicators in a process of “tweaking and tuning” mediated by socio-technical devices ranging from internal interviews to standardized user interfaces. Standards of “reasonable suspicion”—as imposed by the TCA—are thereby shaped both by the affordances of the infrastructural assemblage that Cerberus ties together as well as the cultural resistance and institutional inertia that plagues the introduction of new large-scale information systems. What emerges is a long and jurisgenerative chain of human and infrastructural actants through which targeting rules are enacted—from the “encoding” of “professional expertise” in temporary “characteristics” and the “pre-labelling” of data via the “user interface,” to the “tweaking and tuning” of risk indicators and the “unsupervised” sourcing of new “hypothesis” by both algorithmic “cluster analysis” and the human observation of “behavioral traits” at the border.Footnote ¹⁶⁵ The targeting process that Cerberus puts in motion, in other words, should not be analyzed as a displacement of analogue by algorithmic logics but—in the words of Nick Seaver—as a “steady accumulation of feedback loops, little circuits of interpretation and decision knit together into a vast textile.”Footnote ¹⁶⁶ In this textile, Seaver observes, “[e]very stitch is held together by a moment of human response, a potential rejection shaped by something outside the code, whether it is the arbitrariness of personal preference, the torque of structural bias, or the social force of a formal evaluation framework.”Footnote ¹⁶⁷ We suggest that the heterogenous and fragile fabric of the Cerberus infrastructure is no different: As the targeting rules are tenuously and continuously stitched and restitched together, agency appears not as attribute of particular Home Office decision-makers or technological tools, but as the relational effect of specific sociotechnical compositions for knowing and governing “risk” at the border enacted through complex human-machinic processes.

III. Cerberus and the Speculative Temporalities of Suspicion

A third characteristic of this dispositif is its speculative and retroactive character. This is evident in issues around data deletion criteria shaping Cerberus: “What is it about the data that historically [we] have found [ourselves] viewing … [D]oes that give us any indication of the data we’re going to be interested in, in the future?”Footnote ¹⁶⁸ This transposition from the past to future tense permeates the logic of pre-emption: It is the extension of past patterns forward in time that enables present interventions to modulate or avoid anticipated futures. This speculative movement across temporal scales is enacted through Cerberus both in the transposition of general “historical patterns” in data to future forms of targeting—based on the production of “concern by association”—and in the distillation of “anomalies” and “abnormalities” at the level of the individual, based on deviations from observed paths of past behavior. Cerberus thereby operates through what Antoinette Rouvroy and Thomas Berns have described as a “memory of the future”—a troubling temporal bind where the movement of people is conditioned by a logic of algorithmic anticipation in which the unexpected or spontaneous registers as risky.Footnote ¹⁶⁹ As Ramon Amaro observes, related to practices of ML-based predication in general, this leads to the “freezing of dynamic life into a homogeneous milieu” where the anomalous is rendered suspicious.Footnote ¹⁷⁰

This mediated movement of “patterns” across temporal scales also works in the inverse direction, producing indications of risk that are rendered retroactively actionable. One senior official involved in developing Cerberus gave the following example: “One day, there was a dog, at an airport, sniffing bags of passengers going out. The dog went, bark, bark, bark, bark. So, the officers asked the passenger to open their bag. Inside the bag was a lot of cash.”Footnote ¹⁷¹ From this initial intervention and identification of illicit behavior, the inferential and speculative logic that drives Cerberus unfolds:

This movement back in time generates significant institutional traces tied to the coercive force of the watchlist:

This example indicates not only how “patterns” and “characteristics” are rendered actionable in both temporal directions but also how the associative logic of targeting focused on “unknown” risks feeds into information systems for the identification and prevention of “known” threats—and how practices of risk screening through PNR data analysis are also a key source of input data for the expansion of watchlisting at the border. In this institutional process, as this example illustrates, every border passage is a potentially jurisgenerative moment from which inferences can be drawn—a moment where new “rules” might crystalize that can subsequently be transposed across temporal scales. This speculative and retroactive characteristic is a crucial effect of Cerberus’s dispositif of speculative suspicion.

IV. Cerberus and the Performative Enactment of Suspicion

A fourth characteristic of this dispositif of suspicion is its performative and ontological character. It is now a common observation that “raw data is an oxymoron.”Footnote ¹⁷⁴ Data, from this perspective, is not a representation of an external reality but a performative set of practices—inputting, codifying, labelling, aggregating, tweaking—through which particular realities are enacted.Footnote ¹⁷⁵ One of the developers of Cerberus, in this sense, underlined that “data engineering is manipulation of data.”Footnote ¹⁷⁶ First of all, this “manipulation” is manifested on the level of data capture: When ingesting heterogenous data sources where not “everyone’s speaking the same language,” they noted, this requires a “transformer capability in your architecture”—"[i]f you don’t have [standardized data], and more often than not you don’t, you need to . . . stick a layer in . . . and do something to the data … You’re taking the raw data and you’re transforming it into something that’s more useful.”Footnote ¹⁷⁷ All data is thereby “manipulated” into the “common ontology” of Cerberus, “applying specific fixes to it to get it to your common model . . . [y]ou’re flattening down all the kinds of data you get.”Footnote ¹⁷⁸ This infrastructurally constituted “common ontology” thereby mediates the material realities upon which Cerberus acts. Importantly, these manipulations and mediations should not be seen as a—potentially flawed—representation of reality, but a performative enactment of what matters and what is excluded from mattering.Footnote ¹⁷⁹ The process of “flattening down” data is an enactment of this ontological cut.

Secondly, this “manipulation” is manifested in the infrastructure of interfaces by which targeting rules are presented to targeting teams and by which insights travel back from the frontline. From the six products constituting the so-called “Cerberverse,” three have “user-facing interfaces”—the most important of which is the Central Operations Platforms (COP) where targeting teams … get the direct outputs from Cerberus and review them and say yes, this is a goer, no, this isn’t.”Footnote ¹⁸⁰ From there, “it’s interfaced with the frontline teams [who] have fifteen seconds to make a decision.”Footnote ¹⁸¹ What is presented through this interface is not a “big tabular format under [a] network graph” but an actionable indicator enabling rapid decision-making—“[t]hey don’t have to know the chemistry of the machine, they can just trust it.”Footnote ¹⁸² As Fleur Johns argues, “[i]nterfaces layer interactions and operations in ways that do not presuppose knowledge or even awareness of other interface layers”—they offer “simulated visibility” while “obfuscating the machine and its buried commands.”Footnote ¹⁸³ Buried in the actionable indicator of the “user-facing interface” is the long chain of feedback loops—the “tweaking and tuning” from “hypothesis generation” to “optimization”—by which Cerberus algorithmically generates its targets. Interfaces thereby “establish parameters for and structure practices of use,” “elicit[ing] inputs and assemb[ing] outputs in ways that create impressions of usable coherence and directive capacity.”Footnote ¹⁸⁴

The infrastructural mediations of the interface also impact how professional knowledge is “sucked up” into the system of Cerberus and how “people at the front line” become “a source of new hypotheses” for the distillation of “patterns” and new targeting rules.Footnote ¹⁸⁵ This codification of dispersed operational knowledge follows specific socio-technical pathways—auto-generated fields, pre-population, codified fields—which are designed as part of Cerberus to “reduce the amount of free text that people are writing in [and] standardize the data capture.”Footnote ¹⁸⁶ As one interviewee explained:

The COP interface thereby not only structures how data is rendered actionable at the frontline but also shapes how “frontline teams … record what they did” and thereby foster the feedback loops of optimization and hypothesis generation that drive Cerberus. While scholars have argued on this basis that interfaces “define” or “condition human agency,” we want to push this claim further: At the interfaces of Cerberus, agency is not an attribute of particular human or non-human actants but an emergent effect of relational compositions in which the human and non-human are inherently entangled,Footnote ¹⁸⁸ and through which performative boundaries are drawn of what comes to “matter” at the digital border.Footnote ¹⁸⁹ This opens paths for critical engagement with how the exercise of authority is routed through border interfaces.Footnote ¹⁹⁰

I. Cerberus and the (Re)Production of Race by Proxy

Recent CJEU decisions and related academic literature has extensively highlighted a range of potential harms associated with the automated processing of PNR data—including the risk of discrimination based on protected characteristics, entrenchment of historic bias, and creating new forms of algorithmic inequality and unfairness—and outlined safeguards authorities should take to mitigate those harms and protect fundamental rights.Footnote ¹⁹³ On one level, the policy leads and data engineers designing Cerberus are aware of these legal risks and recognize the systems’ potential to cause algorithmic harm. As one interviewee explained:

The risks of algorithmic bias and discriminatory targeting through reliance on racially biased data—or, the “garbage in, garbage out” problem, as it called in debates on algorithmic fairness—was also acknowledged by one of the Cerberus designers: “We know that historic data sets … contain biases … [that] are either proactively introduced or … a reflection of society at the time … Stop and Search data disproportionately contains men of a certain … ethnic demographic.”Footnote ¹⁹⁵ “So, if you train your dataset on a Stop and Search database, it will look for young black men. It doesn’t matter where the risk is, that’s what it’s going to tell you, because that’s what you told it to look for.”Footnote ¹⁹⁶ “Being able to assess [and] … counter those biases—or to at least, account for them and negate their influence in some way—is going to be something that’s important as well.”Footnote ¹⁹⁷

But Cerberus is also being shaped by other elements and trade-offs that risk embedding racial bias and discriminatory targeting into its emergent infrastructure. These risks are exacerbated by the current lack of data quality assurance and oversight processes to address algorithmic harms, which are either too weak or non-existent in Cerberus, as elaborated below. And they are discounted and deferred through the shared assumption of Home Office data engineers and policy leads that they are only potential future risks for once Cerberus’s ML capabilities are more fully developed, rather than problems that are being infrastructurally configured and designed into the present.

The failure of earlier Home Office IT infrastructure projects like E-Borders continues to loom large over Cerberus, shaping its design and implementation in important ways.Footnote ¹⁹⁸ This failure can be compounded by resistance from frontline border officers and their “historical distrust of giant projects that go wrong.”Footnote ¹⁹⁹ As one interviewee involved in designing Cerberus put it:

However, with Cerberus this potential resistance to a machine that apparently says, “[t]his is where the risk is” is being proactively neutralized through a strategy of enrolment—“So, we have to win them [that is, frontline officers] over”—that is closely enmeshed with the “cultural aspect” of the Cerberus infrastructure development process discussed above and the DSA Unit’s embrace of user-centered design methodologies and processes.Footnote ²⁰¹

One key technique used by Home Office data engineers to defuse resistance and “win over” frontline staff is to make Cerberus’s algorithmic targeting “look like what they do” so “they can trust … [and] begin to accept its output.”Footnote ²⁰² To foster this identification and sense of human-machinic equivalence, Cerberus system designers are engaging with frontline officers to “suck out the[ir] professional knowledge” on key questions like “[w]hat does risk look like [to you]?”, so they can “codify it, standardize it, [and] get the system [that is, Cerberus] to do that” as well.Footnote ²⁰³ Our analysis of this process in the previous section aimed to underscore the relational and fluid characteristics of Cerberus’s dispositif of speculative suspicion and the important role of mundane knowledge practices in digital bordering. But this knowledge extraction and encoding process also has important implications for how algorithmic harms are infrastructurally enabled and mediated. In short: To smooth over anxieties and opposition by frontline staff about data-driven job replacement and work them into this data-analytics enabled bordering infrastructure, the training data, data labelling practices, and first iterations of Cerberus’s targeting rules are all drawn from frontline officers’ intuitions about “risk” at the border. And this extraction and inclusion of “corporate expertise” is taking place with knowledge that frontline intuition, or “copper’s nose,” about “what risky looks like” is often prejudice-based—“I know people from that route, from that country, who look like that … they are a bit dodgy.”Footnote ²⁰⁴ With the Home Office’s troubled history of racial profiling in immigration enforcement,Footnote ²⁰⁵ and acknowledgement that racially-biased training data—“garbage in”—results in racially-biased AI outputs—“garbage out,”Footnote ²⁰⁶ this design choice and trade-off clearly risks embedding racial hierarchies into the infrastructure of Cerberus at an incipient stage of its development to facilitate its wider acceptance and implementation by frontline border staff. It also highlights, as described in the exchange below, the significant ways frontline border staff are being transformed into Cerberus data annotators doing the infrastructural “data work” needed for training Cerberus’s ML models:

The risks of algorithmic discrimination through PNR governance are also well known, and feature in CJEU case law in this area. The EU PNR Directive states that “any assessment of passengers prior to their scheduled arrival … or departure … against pre-determined criteria” must be done in “a non-discriminatory manner” without considering sensitive data about passenger’s race, ethnicity, or religion.Footnote ²⁰⁸ The Home Office operationalize this non-discrimination norm by deleting sensitive data from the PNR they receive from airlines. As one interviewee explained:

But this strategy of trying to delete the risk of discrimination by stripping out protected characteristics information from PNR datasets is particularly fraught when using algorithmic targeting systems like Cerberus.

The deletion of sensitive “race” data from PNR datasets, for example, doesn’t mean that Cerberus’s targeting won’t result in racially discriminatory outcomes. ML techniques can readily infer race from statistical proxy data, like postcodes or travel history, and generate racist results even with the sensitive data about race stripped out.Footnote ²¹⁰ According to Thao Phan and Scott Wark, with algorithmic ML systems like Cerberus, “attempts to remove race as an actionable data point [often] results only in its entrenchment in a novel form.”Footnote ²¹¹ These novel racialization processes—which they call “proxy discrimination” or “race, by proxy”—are characterized by the use of “colorblind” algorithmic techniques that enact race via “latent associations between features of data,” grouping people into emergent clusters from attributes inferred across heterogeneous data, rather than through racial profiling techniques based on shared identities, common characteristics or specified criteria.Footnote ²¹² One way this enactment of race can take place in algorithmic governance is by inferring “risk” through ML pattern recognition and anomaly detection techniques. Consider the following exchange with our Home Office interviewees, reproduced at length and elaborated below, using the hypothetical example of profiling a traveler who has travelled to Saudi Arabia for pilgrimage to understand how the risks of discrimination and racial profiling are handled in Cerberus:

We underscore four key elements from this exchange that help us get at how racialization is enacted through the emergent bordering infrastructure of Cerberus and the wider stakes of the current approach to bias and discrimination. First, this exchange shows just how significantly Cerberus is shifting UK border governance away from rules-based targeting—“If destination equals Jeddah, plus this time of year, then inference equals . . . ”—towards targeting based on the algorithmic detection of patterns and anomalies composed from inferences, “characteristics” of interest and associations drawn across diverse data, as analyzed above. Second, it demonstrates how this kind of algorithmic border governance is understood by the Home Office as an ostensibly colorblind and race neutral technique that offers a “protection against the bias angle,” because it is not based on profiling using prespecified criteria or “a dedicated subset of countries to target” but detects “risk” from emergent patterns and anomalies in data.Footnote ²¹⁴ In the previous section we analyzed this relational patterning as a feature of Cerberus’s dispositif of speculative suspicion. But it is also has important discursive and policy functions—including in supporting government claims that “[Cerberus’s] use of data is . . . even handed and not prejudice to any protected groups” and that “the system and its use fall within the existing powers and remit of the Home Office,” so no new legislation or algorithmic accountability interventions are needed.Footnote ²¹⁵

Third, as analyzed above, characteristics and patterns indicating “risk” and anomaly do not just emerge from “raw data.”Footnote ²¹⁶ They are distilled from heterogenous data sources—including data on past travel, nationality and immigration and border data—including “status check” data on individual’s “proof of entitlement to a range of public and private services, such as work, rented accommodation, healthcare and benefits”Footnote ²¹⁷ and previous encounter data and risk indicators from frontline border officers—policing and security data—both UK and international—and a wide range of private commercial data providers. And because this ingested data is proxy data already “giving weight to race—without even having a defined category for race”Footnote ²¹⁸ explicitly featured, new racialized effects and divisions can readily emerge when Cerberus asks: “Are the things you’re doing risky?” Fourth, Cerberus aims not only to detect anomalies, but to expand anomaly governance as a field of pre-emptive security intervention. As one interviewee noted, recalling the red-amber-green classification system operationalized in Cerberus to sift and govern global mobility flows: “A really good outcome for Cerberus is to massively increase the number . . . [and] proportion of ambers,” with “ambers” understood as anomalies that exhibit characteristics of interest or otherwise indicate “risk”: “They’re not a red, they haven’t done anything illegal . . . but they’re not a green [either], because they’ve done something abnormal. So, they’re in the middle.”Footnote ²¹⁹ If our argument about Cerberus generating new enactmentsFootnote ²²⁰ of race and reconfiguring novel racialized groupings through risk governance processes operating under the threshold of legal perception and out with prevailing non-discrimination norms is correct, then this “expansion of the ambers” and anomaly governance will pose profound legal and political challenges, engaging crucial questions about algorithmic accountability, inequality and racial justice, as Cerberus’s ML capabilities are more fully developed and put into action both at the border and within the UK.

Current oversight and accountability processes in Cerberus are either too weak, non-existent, or functionally orientated towards project delivery to identify, meaningfully grapple with and address these kinds of reconfigurations of power. Whilst data quality assurance and oversight processes for quantifying or mitigating racial bias in Cerberus are being considered within parts of the Home Office, they aren’t yet in place even though Cerberus is currently in operationFootnote ²²¹ and border data is already being classified by frontline staff, generating training data for Cerberus’s ML models. Data ingested into Cerberus is audited and subjected to the usual data quality checks, but only to ensure compliance, prevent “improper access and [system] use”Footnote ²²² and to check that the data is as “complete and correct as can be.”Footnote ²²³ As one interviewee explained: Auditing datasets for bias “will be a longer-term thing because at the moment we’re still kind of building the foundations of our big house. You know, we’ve got to get everything up and running.”Footnote ²²⁴ The main priority for now, in other words, is project implementation. Addressing potential problems of racial bias and other harmful effects of Cerberus can happen later. During interviews we were informed that data ethics guidelines for Cerberus were being developed with input from Cerberus “business leads, who rely on these rules to generate targets” and the Home Office Biometrics and Forensics Ethics Group (“BFEG”).Footnote ²²⁵ Subsequent conversations with BFEG members, however, confirmed that AI ethical principles were drafted by the BFEG for potential use with AI-driven systems like Cerberus, but that these had been indefinitely “put on the backburner” by the Home Office due to other priorities.Footnote ²²⁶

II. The More-Than-Human Loops of Cerberus

The Home Office and BAE staff we interviewed all emphasized the importance of human oversight and decision-making and underscored that Cerberus was not yet capable of supporting unsupervised ML or deep learning techniques that autonomously self-generated their own targeting rules or criteria, even though “the aspiration is certainly to use more and more sophisticated techniques and algorithms” as the infrastructure develops.Footnote ²²⁷ The so-called “human in the loop” was recurrent device brought in used to temper Cerberus’s targeting potential and reaffirm human control over this emergent infrastructure. As one Home Office interviewee put it: “We’re never going to have a situation where something happens—[for example], where an individual is examined [or] treated in a certain way—‘Why did you do that?’ ‘Because the machine said so.’ That’s just not going to happen. That cannot happen.”Footnote ²²⁸ This also resonates with the CJEU’s position on automated PNR governance, which states that any “positive match” in PNR targeting must be individually reviewed “by non-automated means” to identify false positives and minimize “the number of innocent people wrongly identified.”Footnote ²²⁹ The prevailing assumption amongst those designing—and marketing—Cerberus is that it is ultimately an algorithmic assistance tool for human decision-makers—Border Force officers—to “give them the most complete picture of the movement that’s coming across the border” and to allow those frontline officers “to make an informed decision about the appropriate next course of action to take.”Footnote ²³⁰ As one BAE interviewee put it, describing a future scenario where ML techniques were fully integrated into this bordering infrastructure to automate the detection of “behaviors of interest” and risky passengers for Home Office decision-makers to consider alongside their own independent analysis of risk: “There would always be that human due diligence to go, ‘OK. System says this is an interesting thing. Does it actually align up with our analysis: yes or no? OK. It seems like a sensible thing to apply.’”Footnote ²³¹

According to Jake Goldenfein, “the human in the loop” is not a self-evident rule of law principle, but a malleable and generative legal device that “strategically elides questions around the different ways humans are threaded into decision systems and attendant attributions of responsibility,” whilst obscuring the need to understand how specific algorithmic governance systems work in practice.Footnote ²³² Building on this insight, we close by highlighting key elements of Cerberus’s emergent infrastructure below that are obscured or reconfigured through this focus on autonomous human control.

Consider, for example, problems of scale and how “the assembly of techniques, tools and representational conventions … used to know and manage” them enact and give shape to the kinds of decision-making used in border governance.Footnote ²³³ One of the critical elements in human oversight of automated decision-making processes is time—or, more specifically, the amount of time that human decision-makers have to “decide.” As Ben Wagner argues, “the lower the amount of time assigned to the human operator, the more likely it is to be quasi-automated.”Footnote ²³⁴ According to our interviewees, frontline border officers using the targeting leads from Cerberus have ten to fifteen seconds to review its outputs and decide “Yes, this is goer; [or] No this isn’t.”Footnote ²³⁵ And with the exponential growth in global aviation traffic—with UK airport passengers expected to rise from 284 million in 2017 to 435 million by 2050—effectively managing this growth and scale, according to the Home Office, requires digital infrastructure and data governance practices that “ease the flow for legitimate passengers at the border whilst keeping threats away” by making “target[ing] interventions better.”Footnote ²³⁶

These concerns about scale and flow are shaping the design of Cerberus, and the human-machinic relations it is putting into effect, in critical ways: “[T]he whole point of Cerberus,” as one interviewee explained, “is [to] take one hundred per cent of the data and whittle it down to the one per cent that’s actually interesting that we want to look at.”Footnote ²³⁷ To do this at speed, governing potential security risks whilst facilitating circulation, Home Office targeting teams rely on the COP user interface discussed in Section C above. In analyzing governance through dashboards, Kitchin, Laurialt, and McArdie argue it is crucial to grasp their performative dimensions—in other words, that they are not neutral means of displaying data, but generative devices that actively produce and shape what they purport to represent. And to highlight their performative effects, they propose an empirical research strategy of focusing on what user interfaces do.Footnote ²³⁸ What the COP user interface does, in its current configuration, is to “abstract away all the complexity” of Cerberus—revealing the outcome of the Cerberus data analytics to frontline targeting teams on a screen in simplified form, without providing details about how those analytics—or inferences, anomalies, “concern by association,” or “behaviors of interest”—were arrived at or composed, to “not overload them with data.”Footnote ²³⁹ The key rationale offered for this abstraction of complexity is management of scale. As one interviewee put it, frontline staff need to trust Cerberus’s analytics and “do” not “think”:

So, the Cerberus infrastructure is enabling the management of complex scale in UK border governance but in ways that are significantly reconfiguring border decision-making and human-machinic agency in the process. The very notion of Cerberus as an algorithmic assistance tool for border officers to make “informed decisions,” or advanced technology under effective human control, is drawn into question through data interfaces and practices that defer the “thinking” of border governance into Cerberus’s data analytics process and that have border officers effectively executing Cerberus’s targeting decisions and interdicting passengers as potentially risky with little meaningful ability to understand why or how. The human-machinic configuration enacted through this interface reallocates responsibility and authority in distinctive ways. The algorithmic bordering “decision” itself is fragmented and distributed in ways dissimilar from the decision envisaged in public law that the “human in the loop” device cleaves to. A key challenge, as Louise Amoore argues, is that such decisions aren’t authored by a “clearly identifiable human” but rather are “derived from a composite of algorithm designers, frontline security officers, experimental models of the mathematical and physical sciences, a training dataset, and the generative capacities of machine learning classifiers working on entities and events.”Footnote ²⁴¹ And we haven’t yet developed an appropriate political language or legal method for engaging this redistribution of agency.Footnote ²⁴²

The reference to stress testing and clinical trials in the quote above also points towards a very different kind of “human in the loop” being inscribed into Cerberus’s risk governance calculus. That version of the “human” is also different from the public law “human” because they figure either before or after the border decision-making process to interdict—for example, to test the targeting model reliability or performance or to do some post-hoc Cerberus impact assessment.

Critically engaging with these kinds of reconfigurations and power and accountability require a different kind of infra-legalities approach to human-machinic relations in digital borders. It demands analyses of algorithmic infrastructures as emergent phenomena and movement beyond naïve conceptions of agency “that presuppose a field of discrete self-standing entities” towards more fluid and sociotechnical understandings that can map the relational effects generated via arrangements of human and non-human actants in particular settings.Footnote ²⁴³ Only by resituating analyses of digital infrastructures like Cerberus on this kind of ontological footing, using method assemblages fit for the task, can we begin to grasp the broader stakes and challenges they are opening up, reshaping and foreclosing. And this critical task will become even more urgent as the ML targeting capabilities of Cerberus are more fully developed and operationalized.