Quantitative prediction of the risk of failure using learning analytics in virtual engineering courses

Abstract

The expansion of online education in engineering programs has been accompanied by persistently high failure rates, making data-driven early warning systems urgently needed. This article presents and evaluates a quantitative, interpretable, and reproducible methodological framework for predicting the risk of failing introductory engineering courses delivered online, based on learning analytics obtained from the Learning Management System (LMS) and the academic system. The approach was applied to a group of 200 students and included variables such as platform activity (number of active days, logins, resource views, and submissions); performance at a later stage (cumulative GPA and percentage of assessments submitted); and prior academic performance (GPA and number of course re-enrollments). Supervised logistic regression, random forest, and gradient reinforcement models were compared and evaluated based on AUC, sensitivity, specificity, F1 score, and Brier score. The probabilities of failure were transformed into a risk score (low/medium/high). The best-performing model showed an AUC of 0.84 at week 5 for the logistic regression model and approached 0.90 for gradient boosting with very good calibration. Risk categorization was able to accurately group approximately 60% of all students who ultimately failed into the high-risk category. (This high-risk category represented only about 25% of the cohort.) These results demonstrate the potential operational capability of the early warning system. The framework provides an interpretable tool for faculty and administrators to categorize risk levels in an effort to improve early intervention practices based on student performance data, as well as to enhance data-driven decision-making in the field of virtual engineering education.