The term “content moderation,” a holdover from the days of small bulletin-board discussion groups, is quite a bland way to describe an immensely powerful and consequential aspect of social governance. Today’s largest platforms make judgments on millions of pieces of content a day, with world-shaping consequences. And in the United States, they do so mostly unconstrained by legal requirements. One senses that “content moderation” – the preferred term in industry and in the policy community – is something of a euphemism for content regulation, a way to cope with the unease that attends the knowledge (1) that so much unchecked power has been vested in so few hands and (2) that the alternatives to this arrangement are so hard to glimpse.

Some kind of content moderation, after all, is necessary for a speech platform to function at all. Gus Hurwitz’s “Noisy Speech Externalities” (Chapter 12) makes this high-level point from the mathematical perspective of information theory. For Professor Hurwitz, content moderation is not merely about cleaning up harmful content. Instead, content moderation becomes most important as communications channels approach saturation with so much content that users cannot pick out the signal from the noise. In making this particular case for content moderation, Professor Hurwitz offers a striking inversion of the traditional First Amendment wisdom that the cure for bad speech is more speech. When speech is cheap and bandwidth is scarce, any incremental speech may create negative externalities. As such, he writes, “the only solution to bad speech may be less speech – encouraging more speech may actually be detrimental to our speech values.” Professor Hurwitz therefore suggests that policymakers might best advance the marketplace of ideas by encouraging platforms to “use best available content moderation technologies as suitable for their scale.”

Laura Edelson’s “Content Moderation in Practice” (Chapter 13) provides some detail on what these technologies might look like. Through a survey of the mechanics of content moderation at today’s largest platforms – Facebook, YouTube, TikTok, Reddit, and Zoom – Dr. Edelson demonstrates that the range of existing techniques for moderating content is remarkably diverse and complex. “Profound differences in content moderation policy, rules for enforcement, and enforcement practices” produce similarly deep differences in the user experience from platform to platform. Yet all these platforms, through their own mechanisms, take a hardline approach toward content that is “simply illegal” or that otherwise contravenes some strong social expectation.

In Chapter 14, “The Reverse Spider-Man Principle: With Great Responsibility Comes Great Power,” Eugene Volokh examines the hazards that arise when private go-betweens assume the responsibility of meeting public expectations for content regulation. As companies develop technical capabilities that insinuate them more deeply into human decision-making and interaction, there is a natural temptation to require them to use their powers for harm prevention. But as seen in the case of online platforms, these interventions can create discomfiting governance dynamics where entities micromanage private life without clear guardrails or a public mandate. Volokh argues that courts do grasp this Reverse Spider-Man Principle at some level, and that they have worked to avoid its dangers in diverse settings. Tort law, for example, does not generally hold landlords responsible for screening out allegedly criminal tenants, even if such screening might help protect other tenants from violent crime. If it were otherwise, then the law would appoint landlords as narcotics officers, with likely disastrous consequences for individual liberty.

Alan Z. Rozenshtein’s “Moderating the Fediverse: Content Moderation on Distributed Social Media” (Chapter 15) points toward an alternative social media architecture that would address the Reverse Spider-Man problem by dialing down the reach, responsibility, and power of any one community of moderators. This “Fediverse” does not rotate around any single intermediary in the way that today’s mainstream social media architecture does. Instead, the Fediverse is held together by a common protocol, ActivityPub, that allows any user to found and operate their own “instance.” In the case of Mastodon, the Fediverse’s most popular social media platform, each instance works a bit like a miniature X platform with its own content policies and membership criteria. Groups of instances, in turn, can enter into federative agreements with each other: Instance A may allow its users to see content posted in instance B, but not content posted in instance C.

This architecture ensures that no one group of moderators has the scale – or the responsibility, or the power – to set content rules that control the shape of public discourse. But achieving this result would require great effort in the form of a distributed, almost Jeffersonian moderation culture in which a much larger group of users participates intimately in content decisions. Moreover, it is unclear that the Fediverse lends itself to ad-based monetization in the same way that platformed social media does. The seemingly natural behavioral and economic inclination toward market concentration and walled gardens indicates that public policy will have to play some role in encouraging the Fediverse to flourish. Professor Rozenshtein’s chapter offers some suggestions.

12.1 Introduction

A central tenet of contemporary First Amendment law is the metaphor of the marketplace of ideas – that the solution to bad speech is more, better, speech.Footnote ¹ This basic idea is well established in both judicial and scholarly writing – but it is not without its critics. My contribution to this volume adds a new criticism of the marketplace-of-ideas metaphor. I argue that there are circumstances where ostensibly “good” speech may be indistinguishable by listeners from bad speech – indeed, that there are cases in which any incremental speech can actually make other good speech indistinguishable from bad speech. In such cases, seemingly “good” speech has the effect of “bad” speech. I call this process by which ostensibly good speech turns the effects of other speech bad “a noisy speech externality.”

This thesis has important implications. First, it offers a poignant critique of the marketplace-of-ideas aphorism introduced by Justice Holmes in his Abrams dissent.Footnote ² If the marketplace of ideas is subject to significant market failure, correctives may be justified. Market failures, after all, are a standard justification for regulatory intervention. But, second, my contribution goes a step farther, suggesting not only that there are circumstances in which good speech may fail as a corrective to bad speech but also that there are circumstances in which the addition of seemingly good speech may only yield more bad speech. In such cases, the only solution to bad speech may be less speech – encouraging more speech may actually be detrimental to our speech values. If that is the case, then correctives may be not only justified but needed to satisfy an important societal interest. And, third, this chapter presents solutions for content-neutral ways in which to implement such correctives.

The insight underlying this thesis builds on my prior work applying the insights of Claude Shannon’s information theory to social media.Footnote ³ That piece applied Shannon’s work to social media to argue that, at least at a metaphorical level and potentially at a cognitive level, our capacity to communicate is governed by Shannon’s channel-capacity theorem.Footnote ⁴ This theorem tells us that the capacity of a communications channel is limited by that channel’s signal-to-noise ratio. Critically, once that capacity is exceeded, any additional signal is indistinguishable from noise – and this has the effect of worsening the signal-to-noise ratio, further reducing the communications capacity. In other words, after a certain threshold, additional speech is not merely ineffective: It creates a negative externality that interferes with other speech.

Other scholars have made similar arguments, which can casually be framed as exploring the effects of “too much information” or “information overload.”Footnote ⁵ But the negative-externality element of this argument goes a step further. A “too much information” argument suggests that listeners are overwhelmed by the quantity of speech to which they may be subject. This argument suggests that speakers can – deliberately or otherwise – exercise a veto over other speakers by saturating listeners’ information sources. For the listener, it is not merely a question of filtering out the good information from the bad (the signal from the noise): At the point of saturation, signal cannot be differentiated from noise and any filtering necessarily must occur upstream from the listener.

Filtering – reducing the overall amount of speech – has always been a key tool in fighting bad speech. All platforms must filter. Indeed, this is nothing new: Editorial processes have always been valuable to listeners. The question is how they do it, with a related question of the law surrounding that filtering. Under the current approach (facilitated through Section 230 of the Communications Decency Act and built upon First Amendment principles), platforms have substantial discretion over what speech they host. This chapter’s normative contribution is to argue that liability shield should be contingent upon platforms using “reasonable best-available technology” to filter speech – a standard that most platforms, this chapter also argues, likely already meet.

The discussion in this chapter proceeds in four sections. It begins in Section 12.2 by introducing technical concepts from the field of information theory – most notably the ideas of channel capacity and the role of the signal-to-noise ratio in defining a channel’s capacity. Section 12.3 then introduces the traditional “marketplace-of-ideas” understanding of the First Amendment and builds on lessons from information theory to argue that this “marketplace” may in some cases be subject to negative externalities – noisy speech externalities – and that such externalities may justify some forms of corrective regulation. It also considers other arguments that have a similar feeling (“too much information,” “information overload,” and “listeners’ rights”), and explains how the negative-externalities consequence of exceeding a channel’s carrying capacity presents an even greater concern than is advanced by those ideas. Sections 12.4 and 12.5 then explore the First Amendment and regulatory responses to these concerns, arguing that the negative-externalities concern might justify limited regulatory response. In particular, Section 12.6 argues that platforms can reasonably be expected to implement “reasonable best-available technologies” to address noisy speech externalities.

12.2 Information Theory, Channel Capacity, and the Signal-to-Noise Ratio

Initially developed by Claude Shannon at AT&T Bell Labs in the 1940s to study how, and how much, information could be transmitted over the communications channels making up the telephone network, information theory studies how we encode and transmit usable information over communications channels.Footnote ⁶ While mathematical and abstract in its characterization of information and communication, it is quite literally at the foundation of all modern communications networks.Footnote ⁷

To understand the questions that information theory answers, we can start with a counterfactual. Imagine a perfect, noiseless, communications medium being used by two people to share meaningful information between them – say a professor wants to transmit a 90,000-word “article” to a journal editor. Assuming no limits on the part of the two communicating individuals, how quickly can one transmit that information to the other? We can ask the same question in a slightly different way: Because we assume the speakers do not impose any constraints (i.e., we assume that each can speak or listen at any speed), we want to know how much information the communications medium can carry per unit of time – its “channel capacity.”Footnote ⁸

To answer this to a first approximation, one could imagine that a professor could read the article aloud in three hours. So, the channel capacity is at least 30,000 words per hour. But we have assumed that the communicating individuals aren’t the constraint. So, in principle, the professor could read faster, and the editor could transcribe faster – say, 90,000 words per hour, or even 180,000 words per hour, or even 180,000,000 words per hour. In the limit case, because we have assumed that the endpoints (the speaker and listener) do not impose any constraints and that the communications channel is a perfect, noiseless medium, the professor and the editor could communicate instantaneously.

This of course is not the case. But it illustrates two distinct limits we need to be aware of: the ability of the endpoints (speaker and listener) to encode and decode information at a given speed, and the ability of the communications channel to transmit, or carry, that information at a given speed.

Shannon studied both the encoding and carrying questions. We are focused on the carrying question, which for Shannon boiled down to two factors: the strength of the information-carrying signal and the amount of background noise. Taken together, these define the signal-to-noise ratio of the communications channel (mathematically, signal divided by noise, or signal/noise). Increasing this ratio increases the channel capacity. This means that you can increase the channel capacity either by increasing the signal strength or by decreasing the noisiness of the channel.

This should make intuitive sense to anyone who has ever had conversation in a noisy room. It is hard to have a conversation at a loud party – you generally need to speak more slowly and loudly to be heard clearly. You speak more slowly because the carrying capacity of the room (qua communications channel) is reduced by the noise; you speak more loudly to increase the strength of your communications signal.

The example of the noisy room also demonstrates three key takeaways from information theory. What is the source of the “noise” in the room? Mostly other people having their own conversations, or perhaps music is playing in the background. This is not meaningless “noise.” Noise is not merely static or unintelligible sound (though static would be noise). Rather, noise is any signal that is not carrying meaningful information for the recipient. The recipient needs to expend mental energy trying to differentiate signal from noise (if there is enough signal available to reconstruct the intended communication), which slows down her ability to receive information.

Second, all noise is, therefore, reciprocal. Your conversation is noise to everyone else in the room! This means that when you speak more loudly so that your interlocutor can make out what you are saying, you are increasing the amount of noise that everyone else in the room must deal with – you are worsening their signal-to-noise ratio.

Something similar happens when you exceed the carrying capacity of a given communications channel, even if there is no one using that channel other than the speaker and listener. If you think about having a conversation on a phone line with a lot of noise in the background, there is a maximum speed at which you can talk and be understood. What happens when you speak faster than this? The sounds you make become unintelligible – they become noise. This is another lesson quantified by Shannon: When a channel’s carrying capacity is exceeded, any additional information put onto that channel is interpreted not as signal but as noise.Footnote ⁹

This last observation illustrates a third key takeaway: Signal and noise are interpreted by, and at, the receiver. There could be a thousand conversations going on in the room. Only those that reach the given individual’s ears contribute to the signal and noise she must decipher. Similarly, in the online context, the signal-to-noise ratio is a function only of the message that an individual receives, not of the universe of messages that a platform carries. If a platform that carries a billion messages each day only delivers the relevant, meaningful ones to its users, it will have a very high signal-to-noise ratio; if another platform carries tens or hundreds of messages each day but delivers them all to a user regardless of their relevance, requiring her to sift through all of the messages in order to find those of relevance to her, that platform will have a relatively low signal-to-noise ratio.

12.3 Externalities and Speech Regulation

The discussion above tells us something additional about shared communications channels: At a certain point, information added to a communications channel creates a negative externality, reducing the capacity of that channel for everyone using it. But additional foundation is needed before we can look at why this carries an important lesson for how we think about speech regulation. The discussion below revisits the metaphor of the marketplace of ideas, looks at other scholars who have considered the challenges of limited communications capacity, and then introduces the idea that the negative externality created when a channel’s carrying capacity is exceeded can justify regulation. It concludes by discussing some examples to illustrate these concerns.

12.4 Preliminaries of Addressing Noisy Speech Externalities

Whether there is need and legally justifiable reason to support regulating speech in the digital era has prompted substantial debate in recent years. Arguments against such regulations most often sound, on both fronts, in the metaphor of the marketplace of ideas: The ostensible problem we face today is that the cost of speech is too low and the barriers to expression are too few, both of which would typically support the functioning of a robust marketplace.Footnote ³⁰ The discussion below argues that concern about negative externalities derived from an information-theory-based understanding of channel capacity is sufficient to justify and overcome fundamental legal obstacles (i.e., First Amendment concerns) to regulation. It then considers what such regulation could look like, drawing from other settings where the law addresses externalities.

12.5 How We Should Regulate Noisy Speech Externalities

This brings us to this chapter’s ultimate question: How should we regulate online speech in response to noisy speech externalities? My answer is that we should adopt a model similar to that used by the EPA for pollution control – a best-available control technology – but rely on customary industry practices to determine whether such a standard is being met. Unlike the EPA model, the baseline requirement in this setting ought to be a “reasonable best-available technology” requirement, recognizing the vast variation between the capabilities of various platforms and needs of their users. One option for implementing this requirement is to make Section 230’s liability shield contingent upon the use of such technologies.Footnote ⁴⁵

Section 12.4.3 introduced standard legal approaches to addressing externalities, including public law approaches such as environmental regulation and private law approaches such as nuisance and trespass claims. So far, neither of these approaches have been put into practice in the online environment. Rather, Section 230 of the Communications Decency Act creates a permissive self-regulatory environment.Footnote ⁴⁶ Section 230 frees up platforms to moderate users’ speech while making clear that they are under no obligation to do so.Footnote ⁴⁷ Under this approach, platforms are shielded from liability for any harms caused by speech generated by their users, including to the speech interests of their other users.

Section 12.4.1, however, suggested that the government may have a sufficient interest in regulating this speech with narrowly tailored regulations intended to lessen the impacts of noisy speech externalities. We might look to aspects of both public and private law approaches to regulating externalities for a model of how the government could respond to noisy speech externalities. In the final analysis, this approach would combine the self-regulatory approach embraced by Section 230 with an affirmative requirement that platforms use best-available content-moderation technologies as suitable for their scale.Footnote ⁴⁸

Pollution and pollution control are the most traditional legal analogies for thinking about noisy speech externalities and their regulation – and the control-technologies analogy maps onto the concept of content-moderation technologies. A requirement that a platform uses “best-available content-moderation technology” to ensure as best as possible that its users have meaningful access to content on the platform is a content-neutral policy. More-prescriptive policies would run the risk of making content-based distinctions, especially if they were based in concerns that some types of speech on platforms were more or less in need of protection.Footnote ⁴⁹ And while policies such as a common-carrier obligation would likely qualify as facially neutral, their effect on the signal-to-noise ratio of platforms would be to render the platforms useless for the vast majority of communications.Footnote ⁵⁰

Content-moderation techniques and technologies are akin to technological filters or amplifiers that reduce noise or increase the strength of desirable signal to improve the signal-to-noise ratio. These techniques or technologies may come in many forms (indeed, they need not be technological or algorithmic, but could result, for instance, from cultivating community norms or market mechanisms). Their defining characteristic is that they improve a platform’s signal-to-noise ratio, making it easier for users to engage with desired information (signal) or less likely that they will encounter undesired information (noise). As with filters and amplifiers, these technologies can be misconfigured – content moderation can have the effect of amplifying harmful speech or filtering desirable speech. But this is a specific question of how a technology is implemented (including whether it is a “best-available” technology), not a question of the viability or desirability of the underlying technology.

The harder question is who decides what content-moderation technologies are reasonably considered “best available” and how regulation based upon those technologies may be implemented. In the environmental-regulation context, this is done through a regulatory process in which the regulator gathers information about industry practices and dictates what technologies to use. This is not a desirable approach in the speech-moderation setting. As an initial matter, different content-moderation technologies may have different effects on different speech or speakers. Unlike in the pollution context, this potentially creates substantial issues, including embedding content-based distinctions into regulations. That could bring us into the domain of strict scrutiny and concerns about government interference in private speech – a central concern against which the First Amendment is meant to protect.

An additional challenge is the range of speech and the range of platforms hosting that speech. This is a more dynamic environment than the environmental-pollution setting. The EPA regulates a small number of pollutants produced by a small number of chemical processes, which can only be addressed by a small number of control technologies. This makes assessing the best available among those control technologies a tractable task. Courts and regulators are unlikely to be able to keep up to speed with changing needs and capabilities of content moderation – indeed, they are likely to lack the sophistication needed to understand how the technologies even work. And different technologies may be better or worse suited to different types of speech or different types of communication platforms.

It is not unusual for courts to look to industry custom in the face of changing technologies or scientifically complex settings.Footnote ⁵¹ This is a setting where deference to customary industry practices, as opposed to prescriptive command-and-control regulation, makes good sense. And it is an important margin along which online speech platforms compete today. Indeed, platforms invest substantially in their content-moderation operations and are continually innovating new techniques to filter undesired speech, amplify desired speech, and generally to give users greater control of the information that they receive. To be sure, not all of these technologies succeed, and platforms often need to balance the effectiveness of these technologies with the business needs of the platform. To take one example of the former effect, the initial theory behind adding the ability to “like” and “retweet” content on X was to amplify desirable content – but its greater effect was to substantially worsen the platform’s signal-to-noise ratio by increasing the velocity of lower-value content and superficial engagement with that content.Footnote ⁵² On the other hand, tools like verified accounts and the ability for high-reputation users to help moderate posts through systems like X’s Birdwatch or Reddit’s moderator system provide useful amplification and filters that help manage the platforms’ signal-to-noise ratios.

Of course, any regulation of online platforms’ speech practices needs to confront Section 230’s liability shield. Today, Section 230 permits but does not require platforms to adopt content-moderation policies. As suggested above, if a platform is exceeding its channel-carrying capacity without losing users, this suggests there is a market failure keeping those users beholden to the platform. At that point, justification for Section 230’s permissive no-moderation provisions are at their nadir and it becomes reasonable to expect the platform to adopt improved moderation practices.

This is no small recommendation, and I do not make it lightly. Weakening the protections of Section 230’s liability shield significantly increases the cost of litigation, especially for smaller platforms. In its current form, it is difficult for a plaintiff in a Section 230 case to survive a motion to dismiss. Making that shield contingent upon a nebulous “reasonable best-available technology” requirement invites suits that would survive a motion to dismiss. Critically, this raises significant concerns that litigation could deprive platforms’ users of their chosen venue for the exercise of their constitutionally protected speech rights. Any alteration to Section 230’s liability shield should thus be accompanied by specific requirements to counterbalance these concerns, such as sanctions for sham or strategic litigation, fee-shifting requirements, specific pleading or discovery requirements, and safe harbors for smaller platforms. These should be accompanied by a presumption that any industry-standard content-moderation techniques satisfy a “reasonable best-available technology” benchmark. Policymakers might also consider a two-pronged requirement (a) that concerns about a platform’s content-moderation practices must be reported to a state attorney general prior to the commencement of a suit and (b) that a private suit can only be brought after that attorney general non-prejudicially declines to conduct their own investigation.

12.6 Conclusion

A central tenet of contemporary First Amendment law is the metaphor of the marketplace of ideas – that the solution to bad speech is more, better speech. But this is built upon an assumption that more, better speech is possible. Information theory tells us that there are circumstances where any additional speech is necessarily bad speech. This is analytically equivalent to an externality, a common form of market failure and traditional regulatory intervention; and it is analogically equivalent to a market failure in the marketplace of ideas. Indeed, examples of regulation in the face of such failures are common in cases such as pollution and nuisance law – as well as in the First Amendment setting.

This chapter has argued that regulation may be justified, and may survive First Amendment challenges, in cases where noisy speech externalities are likely to occur. It also argues that such regulation should draw from the examples of pollution control’s use of best-available control technologies to mitigate these externalities – but that unlike the pollution setting, courts should look to customary industry practices to evaluate whether a particular platform is using such technologies. This chapter suggests that Section 230’s liability shield, which currently allows but does not require platforms to implement any content-moderation technologies, should require platforms to adopt such technologies. However, this is likely a less radical suggestion than it may seem, as most platforms already actively use and develop content-moderation technologies in the standard course of business; such efforts should be sufficient to satisfy a “best-available content-moderation technology” requirement. Rather, only platforms that actively eschew content-moderation practices, or that otherwise neglect these technologies, would risk the loss of Section 230’s liability shield.

13.1 Introduction

Almost all platforms for user-generated content have written policies around what content they are and are not willing to host, even if these policies are not always public. Even platforms explicitly designed to host adult content, such as OnlyFans,Footnote ¹ have community guidelines. Of course, different platforms’ content policies can differ widely in multiple regards. Platforms differ on everything from what content they do and do not allow, to how vigorously they enforce their rules, to the mechanisms for enforcement itself. Nevertheless, nearly all platforms have two sets of content criteria: one set of rules setting a minimum floor for what content the platform is willing to host at all, and a more rigorous set of rules defining standards for advertising content. Many social-media platforms also have additional criteria for what content they will actively recommend to users that differ from their more general standards of what content they are willing to host at all.

These differences, which exist in both policy and enforcement, create vastly different user experiences of content moderation in practice. This chapter will review the content-moderation policies and enforcement practices of Meta’s Facebook platform, YouTube (owned by Google), TikTok, Reddit, and Zoom, focusing on four key areas of platforms’ content-moderation policies and practices: the content policies as they are written, the context in which platforms say those rules will be enforced, the mechanisms they use for enforcement, and how platforms communicate enforcement decisions to users in different scenarios.

Platforms usually outline their content-moderation policies in their community guidelines or standards. These guideline documents are broad and usually have rules about what kinds of actions users can take on their platform and what content can be posted. These guideline documents often also describe the context in which rules will be enforced. Many platforms also provide information about the enforcement actions they may take against content that violates the rules. However, details about the consequences for users who post such content are typically sparse.

More detail is typically available about different platforms’ mechanisms for enforcement. Platforms can enforce policies manually by having human reviewers check content for compliance directly, or they can employ automated methods to identify violating content. In practice, many platforms employ a hybrid approach, employing automated means to identify content that may need additional human review. Whether they employ a primarily manual or primarily automated approach, platforms have an additional choice to make regarding what will trigger enforcement of their rules. Platforms can enforce their content-moderation policies either proactively by looking for content that violates policies or reactively by responding to user complaints about violating content.

Platforms also have a range of actions they can take regarding content found to be policy violating. The bluntest tool they can employ is simply to take the content down. A subtler option involves changing how the content is displayed by showing the content with a disclaimer or by requiring a user to make an additional click to see the content. Platforms can also restrict who can see the content, limiting it to users over an age minimum or in a particular geographic region. Lastly, platforms can make content ineligible for recommendation, an administrative decision that might be entirely hidden from users.

Once a moderation decision is made, either by an automated system or by a human reviewer, platforms have choices about how (and whether) to inform the content creator about the decision. Sometimes platforms withhold notice in order to avoid negative reactions from users, though certain enforcement actions are hard or impossible to hide. In other instances, platforms may wish to keep users informed about actions they take either to create a sense of transparency or to nudge the user not to post violating content in the future.

13.2 Facebook

Facebook (owned by Meta) has made more information about its content-moderation policies and practices available compared to other social-media companies discussed here. However, it is also the only major platform at the time of this writing that gives an outside body, its external Oversight Board, discretion over the enforcement of its policies.

13.3 YouTube

Rather than a standalone section of its website, YouTube outlines its content policies (“Community Guidelines”) in a section of its Help pages.Footnote ¹³ YouTube prohibits nearly all the same categories of content as Facebook, although the companies’ policies use different nomenclature in some cases and demonstrate different areas of focus. For example, both platforms prohibit sexual content, but Facebook groups this category under the umbrella of “offensive content” while YouTube groups it with “sensitive content.” Similarly, both platforms broadly prohibit fraudulent content, but YouTube focuses more on preventing spam, while Facebook focuses on financial scams.

In contrast to its relatively well-developed documentation around its content policies, YouTube’s documentationFootnote ¹⁴ of its policy-enforcement mechanisms is sparse. The company thoroughly describes how users can flag content that violates policy and how content is reactively reviewed when that happens (always by human reviewers). The policies state that YouTube does, however, “use technology to identify and remove spam automatically, as well as re-uploads of content we have already reviewed and determined violates our policies.”Footnote ¹⁵ Google (YouTube’s owner) also publishes data about content moderation on YouTube in quarterly Transparency Reports.Footnote ¹⁶ In these reports, Google breaks down the share of removals originating from automated systems versus users, with greater than 90 percent of removals originating from automated systems. Google also provides statistics on when in a post’s lifecycle removals happen, breaking down the share that happens before a post receives any views at all, one to ten views, or greater than ten views.

Like Facebook, YouTube employs a “strike” system to nudge users into better behavior.Footnote ¹⁷ YouTube’s strike system is significantly more aggressive, however. Users get a warning with no other penalty attached the first time YouTube finds that they have posted content that violates its policies. After that, users who receive three additional strikes in a ninety-day period will have their YouTube channel permanently removed. YouTube further says that “[i]f your channel or account is terminated, you may be unable to use, own, or create any other YouTube channels/accounts.”Footnote ¹⁸ This implies that channel removal is indeed a complete ban of the user in some cases, but it’s unclear how often this penalty is imposed in full.

13.4 TikTok

TikTok, similar to Facebook, maintains a separate “Community Guidelines” section of its website.Footnote ¹⁹ Content prohibitions are grouped slightly differently, but they generally resemble those of other platforms insofar as they focus on sexually explicit content, fraudulent content, and content deemed to pose a safety risk.

TikTok has released very little information about its mechanisms for enforcement, which violations will result in permanent bans, and how many “strikes” users might receive before getting a permanent ban. In 2021, TikTok published a blog postFootnote ²⁰ announcing that the platform would begin automated proactive content removals for some categories of content. The platform also publishes quarterly Community Guidelines Enforcement reportsFootnote ²¹ with details around content removal and restoration after appeal.

Unlike Meta and Google, TikTok does not give removal statistics by method of initial flagging. Rather, it breaks down final removals by “automated” versus “manual” means. The word “automated” is undefined, but one can reasonably infer it refers to removals without any human review. In TikTok’s case, this appears to be about one-quarter of overall removals, but note that this metric is not equivalent to the ones given by other platforms around initial flagging type, so these numbers are not directly comparable. This is because this metric likely refers to human involvement at any point in the moderation process, instead of solely at the point of initial flagging.

At the same time as its automated proactive-content-removal announcement, TikTok also confirmed that it employs a strike system to ban users who repeatedly post violating content. TikTok does not currently disclose how many times (or at what frequency) users would have to violate policy to receive a ban. Its Community Guidelines make clear that they have a zero-tolerance policy for the most serious categories of violations, such as Child Sexual Abuse Material (CSAM) or violent content. In its transparency reports, the company provides data about the number of accounts removed on a monthly basis. Still, there is no way to connect the number of removed posts to the number of removed accounts without more intermediate data.

13.5 Reddit

Like other platforms reviewed in this chapter, Reddit publishes Community Guidelines that apply across the entire platform.Footnote ²² However, these Community Guidelines are best thought of as a content-moderation “floor” that describes a substantially lower threshold than is actually enforced across the vast majority of the platform. This is because all Reddit content is posted to “subreddits” (also known as channels), each having its own set of policies and practices that users create and enforce themselves.Footnote ²³ Reddit does require that channel moderators post their policies clearly and maintain an appeals process, but communities are otherwise free to self-moderate as they see fit.

This overarching policy of relatively few limitations on what content is permitted on the platform has naturally led to the existence of many groups with a great deal of content that many users would find objectionable for one reason or another. To manage this issue, Reddit has a policy of “quarantining” subreddits that most users might find highly offensive or upsetting.Footnote ²⁴ Reddit will not run ads on quarantined channels, which means they generate no revenue for Reddit. Content posted in these channels also does not appear in feeds of users not subscribed to the quarantined subreddits and will not be discoverable in user searches.

Similar to other platforms we have discussed, Reddit publishes a transparency report with details about its content-policy enforcement. However, it only publishes this report annually.Footnote ²⁵ Reddit has some site-wide enforcement of its content-moderation policies, but subreddit moderators do the majority of content removal, according to its transparency report. To support the enforcement of both site- and community-specific content guidelines by moderators, Reddit makes an extensive set of moderator documentationFootnote ²⁶ and toolsFootnote ²⁷ available to its army of volunteer channel moderators. One community moderation tool unique to Reddit among the platforms we have discussed is that of flair.Footnote ²⁸ Flair are short text tags with single words, phrases, or emoticons. While flair can be used for a variety of purposes, when it is associated with user accounts, it typically conveys a user’s reputation.

Due to the fragmented nature of both content policy and enforcement on Reddit, there is little that can be said about how enforcement decisions are communicated to users when they happen on the channel level. However, while subreddit moderators have broad autonomy to police their channels (and to ban users from them) as they see fit, only Reddit can ban user accounts from the site entirely. Reddit publishes data about both content and user-account removal in its transparency report, but the platform does not outline any explicit thresholds of policy violations (either what kind or how many) that would prompt a user’s account to be suspended.

13.6 Zoom

While Zoom is not generally considered a social-media company, it is still a platform for users to share content. Readers may be most familiar with Zoom as a tool for one-on-one video calling, but Zoom can also be used to host multi-party calls with up to 1,000 participants and webinars with up to 10,000, depending on the host’s account type.Footnote ²⁹ Zoom users can also record videos and save them to Zoom’s cloud so that others can watch those videos at a later time. Therefore, the company has published standards for what content it is and is not willing to host.Footnote ³⁰ In their community standards, Zoom prohibits many of the same content categories as other platforms we have reviewed. These prohibited categories include hate speech, promotion of violence, and sexual or suggestive content, though some other commonly prohibited categories, such as misinformation, are allowed. However, unlike the other platforms we have discussed, Zoom only enforces its policies in reaction to user reports.Footnote ³¹

Zoom appears to have no proactive enforcement of its content policies. Zoom also states that all moderation in response to user reports is done manually, rather than by automated means.Footnote ³² Notably, the company does not currently publish data about its content-policy enforcement. Instead, Zoom’s annual transparency report only includes statistics about the company’s responses to government requests of different types. The company has not made data available about how many pieces of content it has removed or how many users have been banned due to its content-policy enforcement.

Zoom does not have external oversight of its content-moderation decisions – only Meta does this – but interestingly, the platform does have several progressive tiers of internal content-moderation review to which users can appeal decisions. At the highest tier of review, an “appeals panel” makes decisions by majority vote. Panel members are chosen from a pool of Zoom employees and serve for no longer than two years. Panel decisions are documented so they can guide future internal decision-making. In many respects, Zoom’s “appeals panel” is described quite similarly to Meta’s Oversight Board.

13.7 Differences in Content-Moderation Policy

Of the platforms we have reviewed, it is likely no coincidence that the three largest – Facebook, YouTube, and TikTok – have similar written policies on content moderation, as they are all attempting to serve very broad user bases and therefore face similar challenges. They all have platform-wide policies against many of the same types of content. They all take tiered approaches to enforcement, involving banning some kinds of content and limiting access or distribution of other kinds of content. They all describe (in greater or lesser detail) a policy of warning users who post violative content and banning those users who do so repeatedly.

Reddit’s channel-specific approach is different in almost every respect from the approach taken at Facebook, YouTube, and TikTok. While there is a minimum standard for allowable content on Reddit, most policy rules are set by users themselves to facilitate the types of discussions they want to engage in within specific groups. As they are written, Zoom’s content policies fall somewhere between the permissiveness of Reddit and the broad prohibitions against offensive content that the largest platforms have. Zoom prohibits sexual and fraudulent content, as well as explicit calls for violence. However, the platform makes no explicit rules against many other categories of content, including misinformation, that are harder to define. In this respect, Zoom’s content policies are significantly less aggressive than those of Facebook, TikTok, and YouTube.

13.8 Differences in Content-Moderation Enforcement Rules

The starkest differences between the platforms we have studied exist not in their policies as they are written, but in their rules for enforcing these policies. For example, Zoom’s clear statement that it only enforces its policies in response to user reports creates manifestly different conditions for what content is allowed than exists on platforms that engage in proactive enforcement.

There are also meaningful differences between what consequences platforms impose on users who violate platform rules. Most platforms we have discussed employ “strike” systems of some kind, but not all are clear about what penalties will be enforced after which strike, or how long strikes will be counted. YouTube’s clarity on these points is a notable exception. This ambiguity is likely strategic, giving platforms the freedom to adjust their policies in reaction to events without having to communicate every change publicly. It is interesting to note that one of Reddit’s rules for its channel moderators is not to create “Secret Guidelines”Footnote ³³ that aren’t clearly communicated to users, even though Reddit itself is largely opaque about how it enforces its own guidelines.

Reddit and Zoom take a much more reactive approach to content moderation than Facebook, YouTube, and TikTok. Reddit, as discussed above, leaves most aspects of content moderation – including enforcement – to its user community. Zoom’s content policies look much more like those of Facebook, YouTube, or TikTok on paper, but unlike those platforms, Zoom intervenes only in response to user complaints. In effect, then, any given group of users on a Zoom call can effectively agree on and enforce a local content-moderation policy – much as if they were on a subreddit. Unlike Reddit, however, there is no “floor” of allowable content for consenting users, because Zoom only enforces its content policies if it receives a complaint.

However, there do appear to be some areas where the effects of policy enforcement are relatively consistent across platforms, even if the mechanisms for achieving this effect differ. This is particularly true around content that is simply illegal, such as violent terrorist imagery or CSAM (Child Sexual Abuse Material). Every platform we have discussed here makes clear that not only is this type of content prohibited, but that posting this type of content will result in users losing their accounts immediately, without strikes or warnings.

13.9 Differences in Content-Moderation Enforcement Implementation and Transparency

Differences around policy enforcement extend beyond rules for what policy enforcement looks like and what triggers it. There are also serious differences in platforms’ implementation of enforcement systems. Zoom’s all-manual, tiered enforcement system has very different accuracy characteristics than systems that use machine learning to evaluate content proactively. TikTok appears to rely more heavily on fully automated content moderation with an expectation that users will dispute some decisions and some content will be restored after those disputes. These details of implementation create very different user experiences than exist on other platforms.

Some of these differences are the result of platforms’ differing structures. Reddit’s uniquely manually intensive moderation system results from its channel-focused design. Reviewing the resources needed to build accurate machine-learning systems is beyond the scope of this chapter. However, the largest platforms that employ machine-learning techniques to identify violative content in an automated manner can do so, at least in part, because of the enormous training sets of data they can build because of the large volumes of user content they host.

All of the platforms we have reviewed publish transparency-report documents that provide some information about how their policies are implemented in practice. Each of these “transparency reports” have developed independently and, even when theoretically reporting data about the same category, often use different metrics to measure slightly different things. This means that while they can be individually informative, they are rarely directly comparable.

13.10 Conclusion

The platforms reviewed here have profound differences in content-moderation policy, rules for enforcement, and enforcement practices. How, then, can we compare them when they differ on so many dimensions? Ultimately, platforms (and their policies) exist to shape their user experience. This chapter, therefore, proposes that users’ ultimate experience of platforms’ content policies provides the most meaningful basis for comparison. This outcome-focused framework leads us to a series of questions that can be asked about different categories of content on each platform:

• What content are users able to post?

• What content will be taken down after users post it and how quickly will it be removed?

• What content will be visible to users other than the poster?

• What content will be recommended to other users?

• What will the consequences be for users who post violating content?

An example of how to apply this framework to a category of content, in this case sexual content, is shown in the table below.

Platforms and policymakers often discuss aspects of content moderation in isolation. Our exploration of moderation policy and implantation demonstrates the degree to which these dynamic systems are the result of multiple interlocking parts, where aspects of one part of the system impact the efficacy of another. The reality of how policies are experienced by users is heavily impacted by how those policies are implemented. In closing, we encourage the reader, when attempting to make comparisons between platforms or even attempting to understand the impacts of changes to a single system, to consider the whole, rather than the parts.

14.1 Introduction

An entity – a landlord, a manufacturer, a phone company, a credit card company, an internet platform, a self-driving-car manufacturer – is making money off its customers’ activities. Some of those customers are using the entity’s services in ways that are criminal, tortious, or otherwise reprehensible. Should the entity be held responsible, legally or morally, for its role (however unintentional) in facilitating its customers’ activities? This question has famously been at the center of the debates about platform content moderation,Footnote ¹ but it can come up in other contexts as well.Footnote ²

It is a broad question, and there might be no general answer. (Perhaps it is two broad questions – one about legal responsibility and one about moral responsibility – but I think the two are connected enough to be worth discussing together.) In this chapter, though, I’d like to focus on one downside of answering it “yes”: what I call the Reverse Spider-Man Principle – with great responsibility comes great power.Footnote ³ Whenever we are contemplating holding entities responsible for their customers’ behavior, we should think about whether we want to empower such entities to surveil, investigate, and police their customers, both as to that particular behavior and as to other behavior.Footnote ⁴ And that is especially so when the behavior consists of speech, and the exercise of power can thus affect public debate.

Of course, some of the entities with whom we have relationships do have power over us. Employers are a classic example: In part precisely because they are responsible for our actions (through principles such as respondeat superior or negligent hiring/supervision liability), they have great power to control what we do, both on the job and in some measure off the job.Footnote ⁵ Doctors have the power to decide what prescription drugs we can buy, and psychiatrists have the responsibility (and the power) to report when their patients make credible threats against third parties.Footnote ⁶ And of course we are all subject to the power of police officers, who have the professional though not the legal responsibility to prevent and investigate crime.

On the other hand, we generally do not expect to be in such subordinate relationships to phone companies, or to manufacturers selling us products. We generally do not expect them to monitor how we use their products or services (except in rare situations where our use of a service interferes with the operation of the service itself), or to monitor our politics to see if we are the sorts of people who might use the products or services badly. At most, we expect some establishments to perform some narrow checks at the time of a sale, often defined specifically and clearly by statute, for instance by laws that require bars not to serve people who are drunk or that require gun dealers to perform background checks on buyers.Footnote ⁷

Many of us value the fact that, in service-oriented economies, companies try hard to do what it takes to keep customers (consider the mentality that “the customer is always right”), rather than expecting customers to comply with the companies’ demands. But if we insist on more “responsibility” from such providers, we will effectively push them to exercise more power over us, and thus fundamentally change the nature of their relationships with us. If companies are required to police the use or users of their products and services (what some call “third-party policing”Footnote ⁸) then people’s relationship with them may become more and more like people’s relationship with the police.

To be sure, none of this is a dispositive argument against demanding such responsibility. Perhaps sometimes such responsibility is called for. My point, though, is that this responsibility also carries costs. We should take those costs into account when we engage in “balancing,” “proportionality tests,” Learned Hand cost–benefit analysis, or something similar – whether as a matter of adjudication, policymaking, or even just moral judgment – in deciding whether to demand such responsibility.

14.2 The Virtues of Irresponsibility

Let me begin by offering three examples of where some courts have balked at imposing legal liability, precisely because they did not want to require or encourage businesses to exercise power over their customers.

14.3 Practical Limits on Private Companies’ Power, in the Absence of Responsibility

Of course, many such companies (setting aside the common carriers or similarly regulated monopolies) already have great power over whom to deal with and what to allow on their property, even when they aren’t held responsible – by law or by public attitudes – for what happens on their property. In theory, for instance, Prodigy’s owners could have decided that they wanted to kick off users who were using Prodigy email for purposes that they found objectionable: libel, racist speech, Communist advocacy, or whatever else. Likewise, some companies may decide not to deal with people who they view as belonging to hate groups or anti-American organizations, just because their shareholders or managers think that’s the right thing to do, entirely apart from any social or legal norms of responsibility.

But in practice, in the absence of responsibility (whether imposed by law or social norms), many companies will eschew such power, for several related reasons – even setting aside the presumably minor loss of business from the particular customers who are ejected:

1. 1. Policing customers takes time, effort, and money.

2. 2. Policing customers risks error and bad publicity associated with such error, which could alienate many more customers than the few who are actually denied service.

3. 3. Policing customers risks allegations of discriminatory policing, which may itself be illegal and at least is especially likely to yield bad publicity.

4. 4. Policing some customers will often lead to public demands for broader policing: “You kicked group X, which we sort of like, off your platform; why aren’t you also kicking off group Y, which we loathe and which we view as similar to X?”Footnote ³⁴

5. 5. Conversely, a policy of “we do not police our customers” – buttressed by social norms that do not require (or even affirmatively condemn) such policing – offers the company a simple response to all such demands.

6. 6. Policing customers creates tension even with customers who aren’t violating the company’s rules – people often do not like even the prospect that some business is judging what they say, how they dress, or whom they associate with.

7. 7. Policing customers gives an edge to competitors who publicly refuse to engage in such policing and who sell their services as “our only job is to serve you, not to judge you or eject you.”

Imposing legal responsibility on such companies can thus pressure them to exercise power even when they otherwise would not have. And that is so in some measure even if responsibility is accepted just as a broad moral norm, created and enforced by public pressure (likely stemming from influential sectors of society, such as the media or activists or professional organizations), and not a legal norm.Footnote ³⁵ That moral norm would increase the countervailing costs of non-policing. It would decrease the costs of policing: For instance, the norm and the corresponding pressure would likely act on all major competitors, so the normal competitive pressures encouraging a “the customer is always right” attitude would be sharply reduced. And at some point, the norm might become the standard against which the reasonableness of behavior is measured as a legal matter.

Likewise, when people fault a company for errors or perceived discrimination, the company can use the norm as cover, for instance arguing that “regrettably, errors will happen, especially when one has to do policing at scale.” “After all, you have told us you want us to police, have not you?”

Accepting such norms of responsibility could also change the culture and organization of the companies. It would habituate the companies to exercising such power. It would create internal bureaucracies staffed with people whose jobs rely on exercising the power – and who might be looking for more reasons to exercise that power.

And by making policing part of the companies’ official mission, the acceptance of responsibility norms would subtly encourage employees to make sure that the policing is done effectively and comprehensively, and not just at the minimum that laws or existing social norms command. Modest initial policing missions, based on claims of responsibility for a narrow range of misuse, can thus creep into much more comprehensive use of such powers.Footnote ³⁶

Indeed, it appears that something like this happened with social-media platforms. Title 47 U.S.C. § 230 freed online companies of legal responsibility for the content of users’ speech, and many such companies therefore did not exercise their legal power to restrict what users posted, or did so only lightly.Footnote ³⁷ But the mid-2010s saw a combination of social and congressional pressure that held platforms responsible for supposed misinformation and other bad speech on their platforms, which caused the leading platforms to exercise such power more and more.Footnote ³⁸ Platforms have now begun making decisions about which political candidates and officials to deplatform and which important political stories to block (including in the heat of an election campaign).Footnote ³⁹ One might approve or disapprove of such power exercised by large business corporations over public discourse;Footnote ⁴⁰ but my point here is simply that calls for great responsibility have indeed increased the exercise of such power.

14.4 The Internet of Things, Constant Customer/Seller Interaction for Tangible Products, and the Future of Responsibility

So far, there has been something of a constraint on calls for business “responsibility” for the actions of their customers: Such calls have generally involved ongoing business–customer relationships, for instance when Facebook can monitor what its users are posting (or at least respond to other users’ complaints).

Occasionally, some have called on businesses to simply not deal with certain people at the outset – consider Castaneda v. Olsher, where the plaintiffs argued that the defendants just should not have rented the mobile homes to likely gang members. But such exclusionary calls have been rare.

I expect, for instance, that few people would think of arguing that car dealers should refuse to sell cars to suspected gang members who might use the cars for drive-by shootings or for crime getaways.Footnote ⁴¹ Presumably, most people would agree that even gang members are entitled to buy and use cars in the many lawful ways that cars can be used, and that car dealers should not see their job as judging the likely law-abidingness of their customers.Footnote ⁴² If the legislature wants to impose such responsibilities, for instance by banning the sale of guns to felons or of spray paint to minors, then presumably the legislature should create such narrow and clearly defined rules, which would rely on objective criteria that do not require seller judgment about which customers merely seem likely to be dangerous.

But now more and more products involve constant interaction between the customer and the seller.Footnote ⁴³ Say, for instance, that I’m driving a partly self-driving Tesla that is in constant contact with the company. Recall how Airbnb refused to rent to people who it suspected were going to a “Unite the Right” rally.Footnote ⁴⁴ If that is seen as proper – and indeed is seen as mandated by corporate social responsibility principles – then one can imagine similar pressure on Tesla to stop Teslas from driving to the rally (or at least to stop such trips by Teslas owned by those people suspected of planning to participate in the rally).Footnote ⁴⁵

To be sure, this might arouse some hostility, because it’s my car, not Tesla’s. But Airbnb was likewise refusing to arrange bookings for other people’s properties, not its own. Airbnb’s rationale was that it had a responsibility to stop its service from being used to promote a racist, violent event.Footnote ⁴⁶ Why would not Tesla then have a similar responsibility to stop its intellectual property and its central computers (assuming they are in constant communication with my car) from being used the same way?

True, the connection between the Tesla and its user’s driving to the rally is somewhat indirect – but not more so than Airbnb’s. Indeed, Tesla’s connection is a bit more direct: Its product and the accompanying services would get the driver the last mile to the rally itself, rather than just providing a place to stay the night before. Indeed, there’s just one eminently foreseeable step (a short walk from the parking space) between the use of the Tesla and the driver’s attendance at the rally. And conversely, if we think Tesla should not be viewed as responsible for its cars being used to get to rallies that express certain views, what should that tell us about whether Facebook should be responsible for use of its service to convey those views?

Now, Tesla’s sales contract might be seen as implicitly assuring that its software will always try to get me to my destination. But that is just a matter of the contract. If companies are seen as responsible for the misuse of their services, why wouldn’t they have an obligation to draft contracts that let them fulfill that responsibility?

Of course, maybe some line might be drawn here: Perhaps, for instance, we might have a special rule for services that are ancillary to the sale of goods (Tesla, yes; Airbnb, no), under which the transfer of the goods carries with it the legal or moral obligation for the seller to keep providing the services even when one thinks the goods are likely to be used in illegal or immoral ways. (Though what if I lease my Tesla rather than buying it outright, or rent it for the day just as I might rent an Airbnb apartment for the night?) Or at least we might say there’s nothing irresponsible about a product seller refusing to police customers’ continuing use of the services that make those products work.

But that would just be a special case of the broader approach that I’m suggesting here: For at least some kinds of commercial relationships, a business should not be held responsible for what its customers do – because we do not want it exercising power over its customers’ actions. We might then ask whether we should apply the same principle to other commercial relationships.

14.5 Big Data and the Future of Responsibility

There has historically also been another constraint on such calls for business “responsibility”: It’s often very hard for a business to determine what a customer’s plans are. Even if there is social pressure to get businesses to boycott people who associate with supposed “hate groups”Footnote ⁴⁷ – or even if the owners of a business (say, Airbnb) just want to engage in such a boycott – how is a business to know what groups a person associates with, at least unless the person is famous, or unless someone expressly complains about the person to the business?Footnote ⁴⁸

But these days we can get a lot more data about people, just by searching the Internet and some other databases (some of which may cost money, but all of which are well within the means of most big businesses). To be sure, this might yield too much data about each prospective customer for a typical business to process at scale. But AI technology will likely reduce the cost of such processing by enabling computers to quickly and cheaply sift through all that data, and to produce some fairly reliable estimate: Joe Schmoe is 93 percent likely to be closely associated with one of the groups that a business is being pressured to boycott. At that point, the rhetoric of responsibility may suggest that what now can be done (identifying supposedly evil potential clients) should be done.

Consider one area in which technological change has sharply increased the scope of employer responsibility – and constrained the freedom of many prospective employees. American tort law has long held employers responsible for negligent hiring, negligent supervision, or negligent retention when they unreasonably hire employees who are incompetent at their jobs in a way that injures third parties,Footnote ⁴⁹ or who have a tendency to commit crimes that are facilitated by the job.Footnote ⁵⁰ But until at least the late 1960s, this had not required employers to do nationwide background checks, because such checks were seen as too expensive, and thus any such requirement “would place an unfair burden on the business community.”Footnote ⁵¹ Even someone who had been convicted of a crime could thus often start over and get a job, at least in a different locale, without being dogged by his criminal record.

Now, though, as nationwide employee background checks have gotten cheaper, they have in effect become mandatory for many employers: “Lower costs and easier access provide [an] incentive to perform [background] checks, potentially leaving employers who choose not to conduct such checks in a difficult position when trying to prove they were not negligent in hiring.”Footnote ⁵² As a result, people with criminal records now often find it especially hard to get jobs.

Perhaps that’s good, given the need to protect customers from criminal attack. Or perhaps it’s bad, given the social value of giving people a way to get back to productive, law-abiding life. Or perhaps it’s a mix of both. But my key point here is that, while the employer’s responsibility for screening his employees has formally remained the same – the test is reasonable care – technological change has required employers to exercise that responsibility in a way that limits the job opportunities of prospective employees much more than it did before.

Similarly, commercial property owners have long been held responsible for taking reasonable – which is to say, cost-effective – measures to protect their business visitors from criminal attack. Thus, as video surveillance cameras became cheap enough to be cost-effective, courts began to hold that defendants may be negligent for failing to install surveillance cameras,Footnote ⁵³ even though such surveillance would not have been required when cameras were much more expensive.

We can expect to see something similar as technological change renders cost-effective other forms of investigation and surveillance – not just of employees or of outside intruders, but of customers. If it is a company’s responsibility to make sure that bad people do not use the company’s products or services for bad purposes, then as technology allows companies to investigate their clients’ affiliations and beliefs more cost-effectively, companies will feel pressure to engage in such investigation.

14.6 Conclusion

“Responsibility” is often viewed as an unalloyed good. Who, after all, wants to be known as “irresponsible”?Footnote ⁵⁴ Sometimes we should indeed hold people and organizations legally or morally responsible for providing tools that others misuse. People and organizations are also of course entitled to choose to accept such responsibility, even if they are not pressured to do so.Footnote ⁵⁵ And sometimes even if they do not feel responsible for doing something, they might still choose to do it, whether because they think it’s good for their users and thus good for business, or because they think it’s good for society. In particular, I’m not trying to take a position here on what sort of moderation social-media platforms should engage in.Footnote ⁵⁶

My point here is simply that such responsibility has an important cost, and refusal to take responsibility has a corresponding benefit. Those who are held responsible for what we do will need to assert their power over us, surveilling, second-guessing, and blocking our decisions. A phone company or an email provider or a landlord that’s responsible for what we do with its property will need to control whether we are allowed to use its property, and control what we do with that property; likewise for a social-media platform or a driverless-car manufacturer. If we want freedom from such control, we should try to keep those companies from being held responsible for their users’ behavior.

There is value in businesses being encouraged to “stay in their lane,” with their lane being defined as providing a particular product or service. They should be free to say that they “are not the censors of public or private morals,” and that they should not “regulate the public and private conduct of those who ask service at their hands.”Footnote ⁵⁷ Even if, unlike with telephone and telegraph cases, they have the legal right to reject some customers, they should be free to refrain from exercising that right. Sometimes the responsibility for stopping misuse of the product should be placed solely on the users and on law enforcement – not on businesses that are enlisted as largely legally unsupervised private police forces, doing what the police are unable to do or (as with speech restrictions) are constitutionally forbidden from doing.

15.1 Introduction

Current approaches to content moderation generally assume the continued dominance of “walled gardens”: social-media platforms that control who can use their services and how. Whether the discussion is about self-regulation, quasi-public regulation (e.g., Facebook’s Oversight Board), government regulation, tort law (including changes to Section 230), or antitrust enforcement, the assumption is that the future of social media will remain a matter of incrementally reforming a small group of giant, closed platforms. But, viewed from the perspective of the broader history of the internet, the dominance of closed platforms is an aberration. The internet initially grew around a set of open, decentralized applications, many of which remain central to its functioning today.

Email is an instructive example. Although email is hardly without its content-moderation issues – spam, in particular, has been an ongoing problem – there is far less discussion about email’s content-moderation issues than about social media’s. Part of this is because email lacks some of the social features that can make social media particularly toxic. But it is also because email’s architecture simply does not permit the degree of centralized, top-down moderation that social-media platforms can perform. If “ought” implies “can,” then “cannot” implies “need not.” There is a limit to how heated the debates around email-content moderation can be, because there’s an architectural limit to how much email moderation is possible. This raises the intriguing possibility of what social media, and its accompanying content-moderation issues, would look like if it too operated as a decentralized protocol.

Fortunately, we do not have to speculate, because decentralized social media already exists in the form of the “Fediverse” – a portmanteau of “federation” and “universe.” Much like the decentralized infrastructure of the internet, in which the HTTP communication protocol facilitates the retrieval and interaction of webpages that are stored on servers around the world, Fediverse protocols power “instances,” which are comparable to social-media applications and services. The most important Fediverse protocol is ActivityPub, which powers the most popular Fediverse apps, notably the X-like microblogging service Mastodon, which has over a million active users and continues to grow, especially in the wake of Elon Musk’s purchase of X.Footnote ¹

The importance of decentralization and open protocols is increasingly recognized within Silicon Valley. X co-founder Jack Dorsey has launched Bluesky, an X competitor built on the decentralized ATProtocol. Meta’s Mark Zuckerberg has described his plans for an “open, interoperable metaverse” (though how far this commitment to openness will go remains to be seen).Footnote ² And established social media platforms are building in interoperability with ActivityPub applications.Footnote ³

Building on an emerging literature around decentralized social media,Footnote ⁴ this brief essay seeks to give an overview of the Fediverse, its benefits and drawbacks, and how government action can influence and encourage its development. Section 15.2 describes the Fediverse and how it works, first distinguishing open from closed protocols and then describing the current Fediverse ecosystem. Section 15.3 looks at the specific issue of content moderation on the Fediverse, using Mastodon as a case study to draw out the advantages and disadvantages of the federated content-moderation approach as compared to the currently dominant closed-platform model. Section 15.4 considers how policymakers can encourage the Fediverse through participation, regulation, antitrust enforcement, and liability shields.

15.2 Closed Platforms and Decentralized Alternatives

15.3 Content Moderation on the Fediverse

15.4 Encouraging the Fediverse

The Fediverse is still a very small part of the broader social-media ecosystem. Mastodon’s several million users pale in comparison with Facebook’s billion or X’s hundreds of millions of users. Whether the Fediverse ever grows large enough to challenge the current dominance of closed platforms is very much an open question, one that will ultimately depend on whether it provides a product that ordinary users find superior to what is currently available on the dominant platforms.

Such an outcome is hardly preordained. It would require millions of people to overcome the steeper learning curves of Fediverse applications, commit to platforms that are often intentionally less viral than the engagement-at-all-costs alternatives, and navigate the culture shock of integrating into an existing community.Footnote ⁵⁰ After experiencing a mass influx of X users that defected after Elon Musk purchased the platform, Mastodon has seen its active users drop from its late-2022 high of 2.5 million, suggesting that, for many users, Mastodon does not work as an X replacement.Footnote ⁵¹

But Mastodon has demonstrated that, for millions of people, decentralized social media is a viable option. And even if Mastodon’s market share remains modest, other decentralized applications, whether operating on ActivityPub or other protocols (as with the ATProtcol-powered Bluesky) will continue to grow, especially if they combine Mastodon’s emphasis on decentralization with Silicon Valley’s engagement-at-all-costs priorities. In the end, the current dominance of the incumbent platforms may prove illusory. They are, after all, themselves subject to shakeups, as is demonstrated by the meteoric rise of apps like TikTok.

Although decentralized social media will have to stand on its own merits, public-policy interventions could nevertheless encourage its growth. Here I briefly consider four such interventions, ranging from most- to least-direct government involvement.

First, governments could support the Fediverse by participating in it as users or, better yet, as instances. This would both directly contribute to the Fediverse’s growth but, more importantly, would help legitimate it as the preferred social-media architecture for democratic societies. For example, shortly after Musk announced plans to purchase X, the European Commission, the executive branch of the European Union, launched EU Voice, a Mastodon instance that “provides EU institutions, bodies and agencies with privacy-friendly microblogging accounts that they typically use for the purposes of press and public relations activities.”Footnote ⁵² Other governments and international organizations could follow suit.

Second, governments could mandate that large social-media platforms interoperate with the Fediverse. For example, under such a regime, Facebook would be allowed to choose what users or content appear on its servers, but it would have to allow other Fediverse instances to communicate with it. This would allow users to access content that Facebook removes and also still be able to interact with the broader Facebook community.Footnote ⁵³ Such regulation would have to specify to what extent Facebook could block other instances entirely, since otherwise Facebook could effectively defederate. But even a limited interoperability mandate would enable a balance between what are the currently envisioned options: totally unfettered control by closed platforms or common-carrier-type regulations that make any sort of moderation impossible.Footnote ⁵⁴

Such regulation is already being pursued in Europe, where the Digital Services Act would require large platforms to interoperate, a requirement that could easily be modified to include the Fediverse.Footnote ⁵⁵ In the United States, interoperability legislation, which has already been introduced in Congress,Footnote ⁵⁶ would be a welcome alternative to recent overbroad state laws from Texas, Florida, and other Republican-governed states that purport to limit the ability of major social-media platforms to moderate content. These laws, in addition to being poorly thought out and overtly political, may also violate the First Amendment, at least in their more extreme versions.Footnote ⁵⁷

Third, antitrust regulators like the Department of Justice and the Federal Trade Commission could use an incumbent platform’s willingness to interoperate as a consideration in antitrust cases.Footnote ⁵⁸ Interoperability could then be an alternative to calls to “break up” social-media giants, a tactic that is both controversial and legally risky.Footnote ⁵⁹

Finally, policymakers should consider how the background legal regime can be tweaked to improve the incentives for the Fediverse. In the United States, the most important factor is Section 230 of the Communications Decency Act of 1996, which shields platforms from liability as publishers of content created by users.Footnote ⁶⁰ Although Section 230 has come under increasing controversy, especially as it applies to giant platforms, it’s hard to imagine how the Fediverse could function without it. The open nature of the Fediverse – with users being able to travel between and communicate across instances – limits the scope of monetization, since users can choose instances that limit advertisements and algorithmic ranking. But this also means that Fediverse instances will lack the resources necessary to perform the sort of aggressive content moderation that would be necessary were they to be held liable for their users’ content. The rationale for Section 230 immunity when it was enacted in the mid-1990s – to help support a nascent internet – no longer applies to the technology giants. But it does apply to the current generation of internet innovators: the federated social-media platforms.