RINA - Démonstrateur d’utilisation de l’intelligence artificielle pour la gestion opérationnelle des risques naturels gravitaires géologiques | FEREC 2020

DNN actual 0 1 predicted 0 780 82 1 158 60 Figure 33: confusion matrix representing result of DNN model with the rain of the previous 10 days for the test data DNN Operational accuracy 0.78 0.84 balanced accuracy 0.63 0.60 precision 0.28 0.35 recall 0.42 0.27 F1 0.33 0.30 Table 7: Indices calculated based on the confusion matrix to quantify performance of the DNN and the operation model for the test set. This model produced by the DNN is safer than the operational model with a higher recall. It is capable of identifying almost half of all the events. But, this comes with a lower precision. The model is capable of predicting more rock falls making it a safer alternative, but it will also predict more frequently an event will occur while no rock falls. This model would lead the road to be closed 20.2% of the time. With the objective of better understanding the data a third class was added. Class 0: no event, class 1: 1 rock fall, class 2: more than one event. It is expected that when we have more than one rock fall the same day it will be related to more important rains. In order to more easily compare to the other models and also to have a more operational result classes 1 and 2 are combined and a final confusion matrix (Figure 34) is produced. Comparing the overall result of the model using 3 classes and 2 classes we observe they are similar. With 3 classes the model was slightly safer predicting 2 additional rock falls, but also decreased the precision closing the road 38 additional days. The confusion matrix with the 3 classes conforms to the hypothesis that days with more than one instance will be easier to predict. Only 14 of the 43 days (33%) with more than one event are classified as a day with no event while two thirds of the days with only one event were classified as days no rocks would fall. In order to improve predictions a model was constructed including not only the rain as a variable but also the number of blocks that reached the road on the previous days. When a block 42