In this paper it is proved that the index of a Fredholm operator between $p$-adic Banach spaces is preserved under compact perturbations. A case of special interest is provided when the ground field is nonspherically complete. In this case the classical techniques are no longer valid and the relation between the kernels of a Fredholm operator and that of a small compact perturbation turn out to be in general much richer than in the complex context.