Detection of Non-Formative Use of Generative AI in Computer Science 1: Study Based on Automated Rubric and Individual Oral Assessments

Abstract

The use of generative artificial intelligence (GenAI) tools in introductory computer science courses can support learning, but it also entails the risk that these tools may be used as a substitute for students’ own reasoning. This paper presents, as its main contribution, an analysis of the detection of non-formative use of GenAI in a mandatory assignment of a Computer Science 1 course. The course included the use of explicit instructional guidelines, recommendations, and activities for formative GenAI use. The study was conducted during the second semester of 2025. Throughout the course, activities integrating GenAI were implemented with a focus on formative roles. An automated rubric was designed to identify coding patterns that are not expected in first-semester students. The results of the automated analysis were contrasted with students’ performance in oral assessment sessions. The results show that 42% (5 out of 12) of the submitted assignments exhibited strong indications of non-formative GenAI use, evidenced by a mismatch between the characteristics and complexity of the submitted code and the level of understanding demonstrated during the corresponding oral assessment.