Hostname: page-component-7bb8b95d7b-nptnm Total loading time: 0 Render date: 2024-09-15T08:20:28.288Z Has data issue: false hasContentIssue false
  • English
  • Français

Uncertainty quantification and confidence intervals for naive rare-event estimators

Part of: Foundational and philosophical topics General and miscellaneous specific topics

Published online by Cambridge University Press:  02 September 2024

Yuanlu Bai  and
Henry Lam
Yuanlu Bai*
Affiliation:
Columbia University
Henry Lam*
Affiliation:
Columbia University
*
*Postal address: 500 West 120th Street, New York, NY, USA.
*Postal address: 500 West 120th Street, New York, NY, USA.
Get access Rights & Permissions [Opens in a new window]

Abstract

We consider the estimation of rare-event probabilities using sample proportions output by naive Monte Carlo or collected data. Unlike using variance reduction techniques, this naive estimator does not have an a priori relative efficiency guarantee. On the other hand, due to the recent surge of sophisticated rare-event problems arising in safety evaluations of intelligent systems, efficiency-guaranteed variance reduction may face implementation challenges which, coupled with the availability of computation or data collection power, motivate the use of such a naive estimator. In this paper we study the uncertainty quantification, namely the construction, coverage validity, and tightness of confidence intervals, for rare-event probabilities using only sample proportions. In addition to the known normality, Wilson, and exact intervals, we investigate and compare them with two new intervals derived from Chernoff’s inequality and the Berry–Esseen theorem. Moreover, we generalize our results to the natural situation where sampling stops by reaching a target number of rare-event hits. Our findings show that the normality and Wilson intervals are not always valid, but they are close to the newly developed valid intervals in terms of half-width. In contrast, the exact interval is conservative, but safely guarantees the attainment of the nominal confidence level. Our new intervals, while being more conservative than the exact interval, provide useful insights into understanding the tightness of the considered intervals.

Keywords

Rare-event estimationconfidence intervalrelative errorsample proportion

MSC classification

Primary: 62A01: Foundations and philosophical topics
Secondary: 00A72: General methods of simulation
Type
Original Article
Information
Journal of Applied Probability , First View , pp. 1 - 27
Check for updates
Copyright
© The Author(s), 2024. Published by Cambridge University Press on behalf of Applied Probability Trust

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Agresti, A. and Coull, B. A. (1998). Approximate is better than ‘exact’ for interval estimation of binomial proportions. Amer. Statistician 52, 119–126.Google Scholar
Arief, M. et al. (2021). Deep probabilistic accelerated evaluation: A robust certifiable rare-event simulation methodology for black-box safety-critical systems. In International Conference on Artificial Intelligence and Statistics, eds A. Banerjee and K. Fukumizu. Proceedings of Machine Learning Research, pp. 595–603.Google Scholar
Asmussen, S. and Albrecher, H. (2010). Ruin Probabilities (Adv. Ser. Statist. Sci. Appl. Pron. 14). World Scientific, Singapore.Google Scholar
Asmussen, S. et al. (1985). Conjugate processes and the simulation of ruin problems. Stoch. Process. Appl. 20, 213–229.CrossRefGoogle Scholar
Asmussen, S. and Glynn, P. W. (2007). Stochastic Simulation: Algorithms and Analysis. Springer, New York.CrossRefGoogle Scholar
Au, S.-K. and Beck, J. L. (2001). Estimation of small failure probabilities in high dimensions by subset simulation. Prob. Eng. Mechanics 16, 263–277.CrossRefGoogle Scholar
Bai, Y., Huang, Z., Lam, H. and Zhao, D. (2022). Rare-event simulation for neural network and random forest predictors. ACM Trans. Model. Comput. Simul. 32, 1–33.CrossRefGoogle Scholar
Bai, Y. and Lam, H. (2020). On the error of naive rare-event Monte Carlo estimator. In 2020 IEEE Winter Simulation Conf. (WSC), pp. 397–408.CrossRefGoogle Scholar
Blanchet, J., Glynn, P. and Lam, H. (2009). Rare event simulation for a slotted time M/G/S model. Queueing Systems 63, 33–57.CrossRefGoogle Scholar
Blanchet, J. and Lam, H. (2012). State-dependent importance sampling for rare-event simulation: An overview and recent advances. Surv. Operat. Res. Manag. Sci. 17, 38–59.Google Scholar
Blanchet, J. and Lam, H. (2014). Rare-event simulation for many-server queues. Math. Operat. Res. 39, 1142–1178.CrossRefGoogle Scholar
Bucklew, J. (2004). Introduction to Rare Event Simulation. Springer, New York.CrossRefGoogle Scholar
Clopper, C. J. and Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26, 404–413.CrossRefGoogle Scholar
Collamore, J. F. (2002). Importance sampling techniques for the multidimensional ruin problem for general Markov additive sequences of random vectors. Ann. Appl. Prob. 12, 382–421.CrossRefGoogle Scholar
Davison, A. C. and Smith, R. L. (1990). Models for exceedances over high thresholds. J. R. Statist. Soc. B 52, 393–425.CrossRefGoogle Scholar
Dupuis, P., Leder, K. and Wang, H. (2009). Importance sampling for weighted-serve-the-longest-queue. Math. Operat. Res. 34, 642–660.CrossRefGoogle Scholar
Embrechts, P., Klüppelberg, C. and Mikosch, T. (1997). Modelling Extremal Events for Insurance and Finance. Springer, Berlin.CrossRefGoogle Scholar
Glasserman, P. (2004). Monte Carlo Methods in Financial Engineering (Stoch. Model. Appl. Prob. 53). Springer, New York.Google Scholar
Glasserman, P., Heidelberger, P., Shahabuddin, P. and Zajic, T. (1999). Multilevel splitting for estimating rare event probabilities. Operat. Res. 47, 585–600.CrossRefGoogle Scholar
Glasserman, P., Kang, W. and Shahabuddin, P. (2008). Fast simulation of multifactor portfolio credit risk. Operat. Res. 56, 1200–1217.CrossRefGoogle Scholar
Glasserman, P. and Li, J. (2005). Importance sampling for portfolio credit risk. Manag. Sci. 51, 1643–1656.CrossRefGoogle Scholar
Heidelberger, P. (1995). Fast simulation of rare events in queueing and reliability models. ACM Trans. Model. Comput. Sim. 5, 43–85.CrossRefGoogle Scholar
Huang, Z., Lam, H., LeBlanc, D. J. and Zhao, D. (2017). Accelerated evaluation of automated vehicles using piecewise mixture models. IEEE Trans. Intellig. Transport. Syst. 19, 2845–2855.CrossRefGoogle Scholar
Juneja, S. and Shahabuddin, P. (2006). Rare-event simulation techniques: An introduction and recent advances. In Simulation (Handbooks Operat. Res. Manag. Sci. 13), eds S. G. Henderson and B. L. Nelson. Holland, North, Amsterdam, pp. 291–350.CrossRefGoogle Scholar
Kroese, D. P. and Nicola, V. F. (1999). Efficient estimation of overflow probabilities in queues with breakdowns. Performance Evaluation 36, 471–484.CrossRefGoogle Scholar
McNeil, A. J., Frey, R. and Embrechts, P. (2015). Quantitative Risk Management: Concepts, Techniques and Tools. Princeton University Press.Google Scholar
Nicola, V. F., Nakayama, M. K., Heidelberger, P. and Goyal, A. (1993). Fast simulation of highly dependable systems with general failure and repair processes. IEEE Trans. Computers 42, 1440–1452.CrossRefGoogle Scholar
Nicola, V. F., Shahabuddin, P. and Nakayama, M. K. (2001). Techniques for fast simulation of models of highly dependable systems. IEEE Trans. Reliab. 50, 246–264.CrossRefGoogle Scholar
O’Kelly, M. et al. (2018). Scalable end-to-end autonomous vehicle testing via rare-event simulation. In Proc. 32nd Int. Conf. Neural Inf. Proc. Syst., eds S. Bengio et al. Curran Associates, Inc., Red Hook, NY, pp. 9849–9860.Google Scholar
Ridder, A. (2009). Importance sampling algorithms for first passage time probabilities in the infinite server queue. Europ. J. Operat. Res. 199, 176–186.CrossRefGoogle Scholar
Rubinstein, R. Y. and Kroese, D. P. (2016). Simulation and the Monte Carlo Method. John Wiley, Hoboken, NJ.CrossRefGoogle Scholar
Sadowsky, J. S. (1991). Large deviations theory and efficient simulation of excessive backlogs in a GI/GI/M queue. IEEE Trans. Automatic Control 36, 1383–1394.CrossRefGoogle Scholar
Sadowsky, J. S. and Bucklew, J. A. (1990). On large deviations theory and asymptotically efficient Monte Carlo estimation. IEEE Trans. Inf. Theory 36, 579–588.CrossRefGoogle Scholar
Shao, Q.-M. and Wang, Q. (2013). Self-normalized limit theorems: A survey. Prob. Surv. 10, 69–93.CrossRefGoogle Scholar
Siegmund, D. (1976). Importance sampling in the Monte Carlo study of sequential tests. Ann. Statist. 4, 673–684.CrossRefGoogle Scholar
Smith, R. L. (1984). Threshold methods for sample extremes. In Statistical Extremes and Applications, ed. de Oliveira, J. T.. Springer, Dordrecht, pp. 621–638.CrossRefGoogle Scholar
Szechtman, R. and Glynn, P. W. (2002). Rare-event simulation for infinite server queues. In Proc. IEEE Winter Simul. Conf., Vol. 1, pp. 416–423.CrossRefGoogle Scholar
Tuffin, B. (2004). On numerical problems in simulations of highly reliable Markovian systems. In First IEEE Int. Conf. Quant. Eval. Syst., pp. 156–164.CrossRefGoogle Scholar
Villén-Altamirano, M. and Villén-Altamirano, J. (1994). Restart: A straightforward method for fast simulation of rare events. In Proc. IEEE Winter Simul. Conf., eds J. D. Tew, M. S. Manivannan, D. A. Sadowski, and A. F. Seila, pp. 282–289.CrossRefGoogle Scholar
Wang, Q. and Hall, P. (2009). Relative errors in central limit theorems for Student’s t statistic, with applications. Statistica Sinica 19, 343–354.Google Scholar
Wang, Q. and Jing, B.-Y. (1999). An exponential nonuniform Berry–Esseen bound for self-normalized sums. Ann. Prob. 27, 2068–2088.Google Scholar
Webb, S., Rainforth, T., Teh, Y. W. and Kumar, M. P. (2018). A statistical approach to assessing neural network robustness. Preprint, arXiv:1811.07209.Google Scholar
Weng, T.-W. et al. (2018). Evaluating the robustness of neural networks: An extreme value theory approach. Preprint, arXiv:1801.10578.Google Scholar
Zhao, D. et al. (2017). Accelerated evaluation of automated vehicles in car-following maneuvers. IEEE Trans. Intellig. Transport. Syst. 19, 733–744.CrossRefGoogle Scholar
Zhao, D. et al. (2016). Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques. IEEE Trans. Intellig. Transport. Syst. 18, 595–607.CrossRefGoogle ScholarPubMed