From rules to language models: a comparative study of chatbot learning assistants

Abstract

Abstract

Introduction As AI–powered learning assistants become increasingly integrated into higher education, understanding their pedagogical impact across different technological designs is crucial. This study investigated the impact of a mobile chatbot learning assistant in a first–year university course, with attention to both its overall educational contribution and the effects of evolving chatbot technology. Methods This quasi–experimental, multi–cohort study followed five iterations of a first-year university course over a five-year period. A total of 240 students were naturally assigned by course enrollment to either a control group ( n = 113), which used traditional learning methods, or an experimental group ( n = 127), which received the same instruction supplemented by a mobile chatbot. Over the study period, the chatbot technology evolved from an initial decision–tree–based system to a more advanced Large Language Model (LLM)–driven version. The study therefore examined both the overall effects of chatbot support on learning outcomes, student perceptions, and motivation, and the differences associated with the two chatbot designs. Results The findings showed no statistically significant improvement in academic performance among students who used chatbot support compared with those in the control group. Nevertheless, students evaluated the chatbots positively, particularly in terms of usability, accessibility, and support outside regular school hours. Compared with the decision tree–based version, the LLM–driven chatbot showed a modest advantage in promoting engagement and offering more personalized support. Discussion Although chatbot integration did not lead to measurable gains in academic achievement, it was perceived as a valuable supplementary learning tool. The results suggest that AI chatbots may contribute more strongly to student experience and perceived support than to direct performance outcomes. In addition, the slight benefits of the LLM–driven version suggest that advances in chatbot design may enhance engagement and personalization in educational settings.