RINA - Démonstrateur d’utilisation de l’intelligence artificielle pour la gestion opérationnelle des risques naturels gravitaires géologiques | FEREC 2020

26 CHAPTER 3. EXPERIMENTS period Nb events Nb sample 0 1 2 train 2000-2004 281 1512 81% 13% 5% val 2005-2007 142 1046 87 % 9% 4% X 2008-2018 134 3462 97 % 2% 1% Table 3.1: 3-class Distribution of labels in 2000-2007 dataset in train/val set and in 2008-2018 dataset • 3 classes with ’0’ no event, ’1’ one event, ’2’ two events: a bit more accurate representation of data Therefore, targets Y t for classification networks belong to { 0 , 1 } or { 0 , 1 , 2 } More groups could be considered with more available data. On other hand, we could also consider a quantitative output with regression task (labels are bounded integer and could be normalise by K max to give a probability) 3.1.3 Metrics for classification model First, let recall classical metrics for classification: • confusion matrix (see example in Table 3.2) to have all information on well , missed classifications. • accuracy which measure the proportion of correct classification among all, acc = TP + TN TP + TN + FN + FP , • precision of each class which measure the proportion of well classified samples among all predicted in this class, precision = TP TP + FP . Poor class precision means that there are many over-detection ( FP for binary) in this class, • recall of each class which measure the proportion of well classified samples among all real labelled in this class. In binary classification, recall usually indicate recall of posive class recall = TP TP + FN . Poor class recall means that there are miss-detection ( FN for binary) in this class. When classes are unbalanced, we prefer other metrics. In our binary case, negative samples (0 rockfall) is much more frequent in our dataset. • balanced accuracy is the mean of positive and negative recall, ba = TP TP + FN + TN TN + FP / 2 , • F β score , F β = (1+ β 2 ) precision × recall β 2 ( precision + recall ) . When β = 1 , F 1 which weights precision and recall equally, is the variant most often used when learning from imbalanced data. For β > 1 , F β score put less more weight on recall and more weight on precision if 0 < β < 1