A Bayesian Methodology for Estimating Uncertainty of Decisions in Safety-Critical Systems

📝 Abstract

Uncertainty of decisions in safety-critical engineering applications can be estimated on the basis of the Bayesian Markov Chain Monte Carlo (MCMC) technique of averaging over decision models. The use of decision tree (DT) models assists experts to interpret causal relations and find factors of the uncertainty. Bayesian averaging also allows experts to estimate the uncertainty accurately when a priori information on the favored structure of DTs is available. Then an expert can select a single DT model, typically the Maximum a Posteriori model, for interpretation purposes. Unfortunately, a priori information on favored structure of DTs is not always available. For this reason, we suggest a new prior on DTs for the Bayesian MCMC technique. We also suggest a new procedure of selecting a single DT and describe an application scenario. In our experiments on the Short-Term Conflict Alert data our technique outperforms the existing Bayesian techniques in predictive accuracy of the selected single DTs.

💡 Analysis

Uncertainty of decisions in safety-critical engineering applications can be estimated on the basis of the Bayesian Markov Chain Monte Carlo (MCMC) technique of averaging over decision models. The use of decision tree (DT) models assists experts to interpret causal relations and find factors of the uncertainty. Bayesian averaging also allows experts to estimate the uncertainty accurately when a priori information on the favored structure of DTs is available. Then an expert can select a single DT model, typically the Maximum a Posteriori model, for interpretation purposes. Unfortunately, a priori information on favored structure of DTs is not always available. For this reason, we suggest a new prior on DTs for the Bayesian MCMC technique. We also suggest a new procedure of selecting a single DT and describe an application scenario. In our experiments on the Short-Term Conflict Alert data our technique outperforms the existing Bayesian techniques in predictive accuracy of the selected single DTs.

📄 Content

1Corresponding Author: Vitaly Schetinin, Department of Computing and Information Systems, University of Luton, Luton, LU1 3JU, The UK; E-mail: vitaly.schetinin@luton.ac.uk. A Bayesian Methodology for Estimating Uncertainty of Decisions in Safety-Critical Systems
Vitaly SCHETININa,1, Jonathan E. FIELDSENDb, Derek PARTRIDGEb, Wojtek J. KRZANOWSKIb, Richard M. EVERSONb, Trevor C. BAILEYb and Adolfo HERNANDEZb aDepartment of Computing and Information Systems, University of Luton, LU1 3JU, UK

bSchool of Engineering, Computer Science and Mathematics, University of Exeter, EX4 4QF, UK
Abstract. Uncertainty of decisions in safety-critical engineering applications can be estimated on the basis of the Bayesian Markov Chain Monte Carlo (MCMC) technique of averaging over decision models. The use of decision tree (DT) models assists experts to interpret causal relations and find factors of the uncertainty. Bayesian averaging also allows experts to estimate the uncertainty accurately when a priori information on the favored structure of DTs is available. Then an expert can select a single DT model, typically the Maximum a Posteriori model, for interpretation purposes. Unfortunately, a priori information on favored structure of DTs is not always available. For this reason, we suggest a new prior on DTs for the Bayesian MCMC technique. We also suggest a new procedure of selecting a single DT and describe an application scenario. In our experiments on real data our technique outperforms the existing Bayesian techniques in predictive accuracy of the selected single DTs.
Keywords. Uncertainty, decision tree, Bayesian averaging, MCMC. Introduction The assessment of uncertainty of decisions is of crucial importance for many safety- critical engineering applications [1], e.g., in air-traffic control [2]. For such applications Bayesian model averaging provides reliable estimates of the uncertainty [3, 4, 5]. In
theory, uncertainty of decisions can be accurately estimated using Markov Chain Monte Carlo (MCMC) techniques to average over the ensemble of diverse decision models. The use of decision trees (DT) for Bayesian model averaging is attractive for experts who want to interpret causal relations and find factors to account for the uncertainty [3, 4, 5].
Bayesian averaging over DT models allows the uncertainty of decisions to be estimated accurately when a priori information on favored structure of DTs is available as described in [6]. Then for interpretation purposes, an expert can select a single DT

2 model which provides the Maximum a Posteriori (MAP) performance [7]. Unfortunately, in most practical cases, a priori information on the favored structure of DTs is not available. For this reason, we suggest a new prior on DT models within a sweeping strategy that we described in [8].
We also suggest a new procedure for selecting a single DT, described in Section 3. This procedure is based on the estimates obtained within the Uncertainty Envelope technique that we described in [9]. An application scenario, which can be implemented within the proposed Bayesian technique, is described in Section 5. In this Chapter we aim to compare the predictive accuracy of decisions obtained with the suggested Bayesian DT technique and the standard Bayesian DT techniques. The comparison is run on air-traffic control data made available by the National Air Traffic Services (NATS) in the UK. In our experiments, the suggested technique outperforms the existing Bayesian techniques in terms of predictive accuracy.

1. Bayesian Averaging over Decision Tree Models
   In general, a DT is a hierarchical system consisting of splitting and terminal nodes. DTs are binary if the splitting nodes ask a specific question and then divide the data points into two disjoint subsets [3]. The terminal node assigns all data points falling in that node to the class whose points are prevalent. Within a Bayesian framework, the class posterior distribution is calculated for each terminal node, which makes the Bayesian integration computationally expensive [4].
   To make the Bayesian averaging DTs a feasible approach, Denison et al. [5] have suggested the use of the MCMC technique, taking a stochastic sample from the posterior distribution. During sampling, the parameters θ of candidate-models are drawn from the given proposal distributions. The candidate is accepted or rejected accordingly to Bayes rule calculated on the given data D. Thus, for the m-dimensional input vector x, data D and parameters θ , the class posterior distribution ) , | ( D x y p is
        N i i y p N d p y p y p 1 ) ( ) , , | ( 1 ) | ( ) , , | ( ) , | ( D θ x θ D θ D θ x D x , where ) | ( D θ p is the posterior distribution of parameters θ conditioned on data D, and N is the number of samples taken from the posterior distribution. Sampling across DT models of variable dimensionalit

This content is AI-processed based on ArXiv data.