Evaluating the Effectiveness of Voice and Text Software Chatbots on Students' Programming Skills
Supervisor Name
Mamoun Nawahdah
Supervisor Email
mnawahdah@birzeit.edu
University
Birzeit University
Research field
Computer Science
Bio
Dr. Mamoun Nawahdah is an Assistant Professor of Computer Science at Birzeit University, Palestine. He earned his Ph.D. in Computer Science from the University of Tsukuba, Japan, where he was awarded the prestigious Japanese Government Scholarship (MEXT). His research focuses on human-computer interaction, educational technologies, and AI-driven learning systems. Dr. Nawahdah has published widely in international journals and conferences and actively contributes as a reviewer for leading venues in the field.
Traditional programming courses often struggle to provide continuous, personalized, and interactive feedback, which is essential for developing strong coding skills. With advancements in AI-driven chatbots, real-time assistance has emerged as a promising tool for enhancing programming education. However, the effectiveness of voice-based and text-based chatbots in improving students' programming skills remains underexplored. In our previous research [1][2], we identified key challenges that limit the effectiveness of chatbot-driven learning, including linguistic diversity, emotional detection, handling complex queries, and students’ over-reliance on chatbot assistance instead of critical thinking. Additionally, existing chatbot-based learning solutions often fail to provide a seamless and adaptive learning experience. To address these challenges, we will explore an innovative solution through different chatbot-enabled learning setups and evaluate its effectiveness through a practical experiment. We designed an experimental study involving second-year computer science students who possess object-oriented programming (OOP) skills. Participants will be assigned to one of three learning setups: Traditional Learning (Control Group): Students receive instruction and support exclusively from a human teacher in a traditional classroom setting. This serves as the baseline for comparison with chatbot-enabled learning environments. Text Chatbot-Enabled Learning: Students solve the task using a text-based chatbot as their programming assistant. The chatbot provides continuous feedback, answers questions, guides them through the solution, and explains difficult concepts through written responses, acting like a direct assistant. Voice Chatbot-Enabled Learning: Students solve the task using a voice-based chatbot as their programming assistant. The voice chatbot offers continuous feedback, answers questions, assists in problem-solving, and clarifies challenging concepts through spoken responses, serving as a direct assistant throughout the process. Each setup will have a diverse group of students to ensure comprehensive analysis. The programming task will require students to complete a problem that tests their OOP skills within a fixed time limit, simulating real-world coding challenges. To evaluate the effectiveness of the chatbot-enabled setups, we will analyze multiple key factors, including student performance, code accuracy, programming skill development, problem-solving ability, code quality, satisfaction, and usability. Our data collection methodology includes both quantitative and qualitative approaches. We will use pre-surveys and post-surveys to measure knowledge improvement, while direct observations will assess student engagement. Additionally, interviews will provide qualitative insights into student experiences. Statistical techniques will be applied to evaluate performance differences across the three setups, ensuring a comprehensive analysis of the data. By comparing the results, this study aims to determine if chatbot-enabled learning improves programming education and whether voice or text interactions are more effective. It contributes to AI-driven educational tools by providing empirical evidence on chatbots' role in programming education. The findings will help educators create more interactive and adaptive learning environments, enhancing student engagement, coding skills, and problem-solving abilities. [1] H. Sawalha, M. Nawahdah, H. Jebara (2025) GUIDELINES FOR DEVELOPING PERSONALIZED AND EFFECTIVE VOICE CHATBOTS IN EDUCATION, INTED2025 Proceedings, pp. 5921-5929. [2] H. Jebara, M. Nawahdah, H. Sawalha (2025) EMOTION RECOGNITION IN EDUCATIONAL CHATBOTS: GUIDELINES FOR ENHANCING ENGAGEMENT AND PERSONALIZATION, INTED2025 Proceedings, pp. 5930-5937.
