There is a form of naturalism which runs through Kripke’s account of rule-following. Given the overarching structure of his account, in which Wittgenstein’s naturalism is made to serve the ends of a skeptical solution to the paradox of the regress of interpretations, it inevitably has the effect of giving a reductive interpretation of it. The reductionist aspect was made clear by those, such as Crispin Wright, who made Kripke’s naturalistic element explicit, using non-normative notions in a constructive account of what going by a rule consists in. I look at Wittgenstein’s pivot toward naturalism in the early 1930s and trace its development to the discussion of rule-following in the Philosophical Investigations. I argue for a different understanding of Wittgenstein’s naturalism and its relation to the paradox of PI §201, one which allows it to escape the charges of both reductionism and an unsatisfactory form of quietism.