The stability is investigated of the swirling flow between two concentric cylinders in the presence of stable axial linear density stratification, for flows not satisfying the well-known Rayleigh criterion for inviscid centrifugal instability, d(Vr)2/dr < 0. We show by a linear stability analysis that a sufficient condition for non-axisymmetric instability is, in fact, d(V/r)2/dr < 0, which implies a far wider range of instability than previously identified. The most unstable modes are radially smooth and occur for a narrow range of vertical wavenumbers. The growth rate is nearly independent of the stratification when the latter is strong, but it is proportional to it when it is weak, implying stability for an unstratified flow. The instability depends strongly on a non-dimensional parameter, S, which represents the ratio between the strain rate and twice the angular velocity of the flow. The instabilities occur for anti-cyclonic flow (S < 0). The optimal growth rate of the fastest-growing mode, which is non-oscillatory in time, decays exponentially fast as S (which can also be considered a Rossby number) tends to 0. The mechanism of the instability is an arrest and phase-locking of Kelvin waves along the boundaries by the mean shear flow. Additionally, we identify a family of (probably infinitely many) unstable modes with more oscillatory radial structure and slower growth rates than the primary instability. We determine numerically that the instabilities persist for finite viscosity, and the unstable modes remain similar to the inviscid modes outside boundary layers along the cylinder walls. Furthermore, the nonlinear dynamics of the anti-cyclonic flow are dominated by the linear instability for a substantial range of Reynolds numbers.