We develop a stochastic model of population viability which explicitly links demography and genetics in order to examine the consequences for extinction dynamics of different levels of heritable fitness variance within a population. We particularly focus on situations in which a local small population is artificially built with individuals that were taken from several large source populations. Our results suggest that different levels of fitness variability within a population (due to partially recessive deleterious alleles rather than local adaptation) have a large influence on its viability. Moreover, the optimal level of fitness variance for maximizing population persistence is a function of the species life-cycle. Two mechanisms with opposite effects are mainly responsible for the different patterns of extinction obtained depending on the life-cycle, (1) purging of deleterious alleles, (2) demographic stochasticity. For high growth rate or long-lived species, a high fitness variance decreases short-term viability and increases long-term viability. In contrast, for other cases, a high fitness variance increases both short- and long-term viability.