Factors and conditions that affect the goodness of machine learning models for predicting the success of learning
László Bognár, Tibor Fauszt
Abstract
The process for building effective machine learning models that predict the learning success of university students, the competences of the actors involved in model building, and the main factors and conditions that influence the reliability of the predictions are reviewed in this paper. It is shown that, in addition to the site-level and course-level indicators commonly used in the literature for prediction, significantly more accurate predictions can be made by introducing so-called chapter-level indicators. These chapter-level indicators are closely linked to the content structure of the subject under study, the hierarchy of its chapters and the learning resources and student activities used in them. Specifically, we make suggestions to the course instructor about the conditions under which there is a hope of obtaining reliable predictions of the student's chances of success or failure. We show how the use of a previously trained model for a newly started similar course requires caution. Even relatively small differences, such as a change in the minimum score to be achieved, a difference in the form of study (full-time or correspondence course), or a difference in the number of compulsory mid-term tests, can cast doubt on the validity of our predictions. We also discuss details of model training that affect the goodness of fit of machine learning models.