In Confederates in the Attic, Tony Horwitz offers a disturbing observation: “When Spike Lee's movie on Malcolm X launched a wave of ‘X’ clothing, a counter symbol quickly sprouted on T-shirts and bumper stickers across the white South. It showed the diagonal cross of the rebel flag beside the words ‘You Wear Your X, I'll Wear Mine.’” The distance between the two X's suggests an enduring racial divide between opposing relationships to the past. Malcolm X spoke to this correspondence in 1964 when he declared: “We didn't land on Plymouth Rock. Plymouth Rock landed on us.” In this chapter I make the case for James Baldwin as a thinker who disabuses readers of willful innocence about the complex legacy of segregated memory in America. I explore how present publics inherit the challenges of desegregating memory that Baldwin identified. I then assess the extent to which his contributions will help those seeking to address the threat segregated memory poses to trans-racial democratization.