To affirm the morality of human rights is to affirm the twofold claim that each and every (born) human being has inherent dignity and is inviolable (not-to-be-violated). This is true no matter what ground one has, or thinks one has, for affirming the morality of human rights. Indeed, this is true even if one has no ground, religious or non-religious, for affirming the morality of human rights. (“I have reached bedrock and this is where my spade is turned.”) We who affirm the morality of human rights, because we affirm it, should do what we can, all things considered – we have conclusive reason to do what we can, all things considered – to try to prevent human beings, including government offficials, from doing things (even if the doing is a not-doing) that “violate” human beings, in the sense of denying, implicitly if not explicitly, that one or more human beings lack inherent dignity.

Of course, the “all things considered” will be, in many contexts, indeterminate. What Amartya Sen, borrowing from Immanuel Kant, calls the distinction between “perfect” and “imperfect” duties is relevant here – though I'd mark the distinction by different terms: “determinate” and “indeterminate” duties.