TEACHING CLINICAL REASONING TO MEDICAL STUDENTS USING AI-GENERATED CLINICAL SCENARIO TESTS: A MIXED-METHODS FORMATIVE EVALUATION

Abstract

The integration of artificial intelligence (AI) into medical education is rapidly evolving, offering new tools to enhance the teaching and assessment of clinical reasoning. One such tool is the Script Concordance Test (SCT), designed to evaluate clinical reasoning under conditions of uncertainty. Traditionally, developing SCT scoring systems and providing feedback requires the involvement of expert panels, which is time-consuming and resource-intensive. Recent advances in generative AI and large language models (LLMs) offer the potential to simulate expert judgment, yet this capability remains underexplored. This study investigated the feasibility of using LLMs to emulate expert clinical judgment in the development, scoring, and feedback of SCTs in cardiology and pulmonology. Fifteen third-year medical students completed a 32-item test generated by ChatGPT-4o. Six LLMs were used as simulated experts, three of which were trained on course materials and three untrained. Students answered test items, rated perceived difficulty, and selected the most helpful feedback explanations. The average score was 22.8 out of 32. Trained models showed higher concordance with student responses (ρ = 0.64) compared to untrained models (ρ = 0.41). AI-generated feedback was considered most useful in 62.5% of cases, particularly when using trained models. These results indicate that trained generative AI models can reliably emulate expert clinical reasoning within the SCT framework. The use of such technology could simplify SCT development while maintaining educational value in feedback. Further research is needed to assess the long-term impact of these tools on the development of clinical reasoning and to determine the optimal balance between expert involvement and AI systems in medical education.