An automatic approach to detect problems in Android builds through screenshot analysis
Lorena P. de Figueiredo, Jonathan P. Gomes, Flávia de Souza Santos, Guilherme M. Queiroz, Felipe T. Giuntini, Juliano E. Sales
Abstract
The number of Android smartphones has grown in recent years, generating more than twenty thousand builds daily in our industry. Expert testers manually evaluate system builds to ensure Google's approval before entering in mass-production. In this context, automated approaches are needed to speed up this manual process. This paper presents a machine-learning-based approach to automate the validation of builds through the analysis of screenshots. Our approach uses screenshots and input files to identify which rules need to be applied as a first step. Then, it performs an analysis to recognize elements present in the screenshots using Optical Character Recognition (OCR) and Support Vector Machine (SVM). Finally, it uses a rule-based algorithm to classify the validation points as correct, incorrect, or not applicable. At the end, the labeled validation points are presented on a monitor, enabling experts to review them manually. To assess our approach, we compared our method with a manually labelled data set generated by three guest reviewers, and also with ground-truth labels obtained after Google's validation. Our approach achieved an average F1-score of 0.99. The results achieved by the six guest reviewers were, in general, worse than the result of our approach. Additionally, the approach helps the reviewer to save in average 90.86% of their work time, when compared with a purely human review.