When AI Mentorship Scaffolds or Substitutes: Iterative Design and Evaluation of a Hybrid Validation System for Deep Tech Entrepreneurship Education

Abstract

Deep technology entrepreneurship education, requiring independent judgment under uncertainty, is either advanced or undermined by AI mentoring, depending on system architecture. Current AI systems have three flaws: unvalidated feedback, lack of session memory, and relying on pre-training data rather than entrepreneurial precedents. This article details the first Design Science Research (DSR) cycle of ProjectIA, designed to address these limitations. ProjectIA uses two complementary artifacts: Eight structured validation frameworks operationalizing the Technology Startup Roadmap (TRL 1-5), specifying evidence and experiment protocols; and an AI platform constrained to Socratic interrogation and evidence-grounded feedback against these frameworks. Evaluation with 13 professionals over 16 weeks tested three hypotheses: H1 (frameworks surface missed risks), H2 (Socratic AI complements human mentoring), and H3 (LLM fidelity failures are the main trust barrier). H1 and H2 were confirmed, with frameworks achieving unanimous utility (5.0/5.0) and the AI platform scoring high on complementarity (4.69/5.0). H3 was confirmed and refined: hallucination control (3.23/5.0) and session memory (3.31/5.0) scored lowest. Their consequences were expertise-asymmetric, recoverable for experts but potentially corrupting for novices. Whether the system develops or substitutes for independent judgment is an open question; this cycle measured perceived utility and fidelity, not unassisted performance post-intervention. Unassisted performance measurement is the primary design requirement for Cycle 2.