Simple techniques work surprisingly well for neural network test prioritization and active learning (replicability study)

Test Input Prioritizers (TIP) for Deep Neural Networks (DNN) are an important\ntechnique to handle the typically very large test datasets efficiently, saving\ncomputation and labeling costs. This is particularly true for large-scale,\ndeployed systems, where inputs observed in production are recorded to serve as\npotential test or training data for the next versions of the system. Feng et.\nal. propose DeepGini, a very fast and simple TIP, and show that it outperforms\nmore elaborate techniques such as neuron- and surprise coverage. In a\nlarge-scale study (4 case studies, 8 test datasets, 32'200 trained models) we\nverify their findings. However, we also find that other comparable or even\nsimpler baselines from the field of uncertainty quantification, such as the\npredicted softmax likelihood or the entropy of the predicted softmax\nlikelihoods perform equally well as DeepGini.\n