Understanding College Students’ Satisfaction With ChatGPT: An Exploratory And Predictive Machine Learning Approach Using Feature Engineering

Authors

DOI:

https://doi.org/10.52634/mier/2024/v14/i1/2568

Keywords:

ChatGPT, Machine Learning, Area Under Curve, Prediction, Support Vector Classifier, Feature Engineering

Abstract

Artificial Intelligence (AI) technologies are continually improving and becoming more pervasive in many facets of our lives. ChatGPT is one such cutting-edge artificial intelligence application, and it has received a lot of worldwide media attention, specifically from educationists, technologists, and learners. It is imperative to understand and evaluate the impact of ChatGPT on computer science students as it directly and holistically influences them. A quantitative instrumental case study explores ChatGPT’s impact on early adopters in education. A survey of undergraduate computer science students at a state university of Delhi was conducted to get insight into their opinion on adopting this revolutionising technology for their education, career, and overall satisfaction. An end-to-end data science approach is applied to encompass exploratory and predictive modelling with feature engineering solutions. Results reveal the most influential features contributing to students’ satisfaction in adopting ChatGPT for their day-to-day chores concerning their social life, education, and career. The Linear Support Vector classifier, a machine learning algorithm for predicting the satisfaction or dissatisfaction in students’ shows an accuracy score of 72.73% and 97.72%, respectively. The AUC for this multiclass prediction model is convincing and is 0.74, 0.71, and 0.96 for satisfied, neutral, and dissatisfied classes, respectively.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

Downloads

Published

2024-05-08

How to Cite

Pabreja, K., & Pabreja, N. (2024). Understanding College Students’ Satisfaction With ChatGPT: An Exploratory And Predictive Machine Learning Approach Using Feature Engineering. MIER Journal of Educational Studies Trends and Practices, 14(1), 37–63. https://doi.org/10.52634/mier/2024/v14/i1/2568

Issue

Volume 14, Issue 1, May 2024

Section

Articles

License

Copyright (c) 2024 Dr Kavita Pabreja, Ms.

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

The articles published in the MIER Journal of Educational Studies, Trends and Practics (MJESTP) are distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

  • Copyright on any open access article in the MIER Journal of Educational Studies, Trends and Practics (MJESTP) published by Model Institute of Education and Research (MIER) is retained by the author(s).
  • Author(s) grant MIER a license to publish the article and identify himself/herself/themselves as the original publisher.
  • Authors also grant any third party the right to use the article freely as long as its integrity is maintained and its original authors, citation details and publisher are identified.
  • The Creative Commons Attribution License 4.0 formalizes these and other terms and conditions of publishing articles.
Crossref
Scopus
Google Scholar
Europe PMC

References

Adamopoulou, E., & Moussiades, L. (2020). An Overview of Chatbot Technology. IFIP Advances in Information and Communication Technology,584. https://doi.org/10.1007/978-3-030-49186-4_31 DOI: https://doi.org/10.1007/978-3-030-49186-4_31

Almeida, F. (2017). Introduction to AUC and Calibrated Models with Examples using Scikit-Learn. Retrieved from https://queirozf.com/entries/introduction-to-auc-and-calibrated-models-with-examples-using-scikit-learn

Alshater, M. (2023). Exploring the Role of Artificial Intelligence in Enhancing Academic Performance: A Case Study of ChatGPT. SSRN ElectronicJournal. https://doi.org/10.2139/ssrn.4312358 DOI: https://doi.org/10.2139/ssrn.4312358

Bhandari, A. (2020). Guide to AUC ROC Curve in Machine Learning: What Is Specificity? Retrieved from https://www.analyticsvidhya.com/blog/2020/06/auc-roc-curve-machine-learning/

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... Amodei, D. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020). NeurIPS. Retrieved from https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html

Brownlee, J. (2018). How and When to Use a Calibrated Classification Model with Scikit-learn. Machine Learning Mastery-Probability, 1-40.

Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023). ChatGPT Goes to Law School. SSRN Electronic Journal, 1-16. https://doi.org/10.2139/ssrn.4335905 DOI: https://doi.org/10.2139/ssrn.4335905

Cooper, K. (2021). OpenAI GPT-3: Everything You Need to Know. Retrieved from https://www.springboard.com/blog/data-science/machine-learning-gpt-3-open-ai/

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol. 1, p. 4171-4186).

Ertan, H. (2020). Which features to use in your model? Retrieved from https://medium.com/@hertan06/which-features-to-use-in-your-model-350630a1e31c

Floridi, L., & Chiriatti, M. (2020). GPT-3: Its Nature, Scope, Limits, and Consequences. Minds and Machines, 30(4), 681-694. https://doi.org/10.1007/s11023-020-09548-1 DOI: https://doi.org/10.1007/s11023-020-09548-1

Folstad, A., & Brandtzaeg, P. B. (2017). Chatbots and the New World of HCI. Interactions, 24(4), 38-42. https://doi.org/10.1145/3085558 DOI: https://doi.org/10.1145/3085558

Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023). How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Medical Education, 9(e45312). https://doi.org/10.2196/45312 DOI: https://doi.org/10.2196/45312

Gupta, K. (2022). ChatGPT vs Chatbot-What’s the Difference ? Retrieved from https://www.hellotars.com/blog/chat-gpt-vs-chatbot-whats-the-difference/

Haque, M. U., Dharmadasa, I., Sworna, Z. T., Rajapakse, R. N., & Ahmad, H. (2022). I think this is the most disruptive technology”: Exploring Sentiments of ChatGPT Early Adopters using Twitter Data. Retrieved from http://arxiv.org/abs/2212.05856

Hu, L. G. A., & R, F. (2023). Generative AI and Future. Retrieved from https://pub.towardsai.net/generative-ai-and-future-c3b1695876f2

Huang, W., Hew, K. F., & Fryer, L. K. (2021). Chatbots for language learning-Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1), 1-325. DOI: https://doi.org/10.1111/jcal.12610

India Today. (2023). How ChatGPT and AI are transforming education. Retrieved from https://www.indiatoday.in/education-today/featurephilia/story/how-chatgpt-and-ai-are-transforming-education-2355138-2023-04-04

Java T Point. (2016). Support Vector Machine Algorithm. Retrieved from https://www.javatpoint.com/machine-learning-support-vector-machine-algorithm

Jovanovic, M., & Campbell, M. (2022). Generative Artificial Intelligence: Trends and Prospects. Computer, 55(10), 107-112. https://doi.org/10.1109/MC.2022.3192720 DOI: https://doi.org/10.1109/MC.2022.3192720

Kim, N. Y., Cha, Y., & Kim, H. S. (2019). Future English learning: Chatbots and artificial intelligence. Multimedia-Assisted Language Learning, 22(3), 32-53.

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. Retrieved from https://arxiv.org/abs/1907.11692

Ministry of Statistics and Programme Implementation, Government of India. (2022). Youth in India. Retrieved from https://mospi.gov.in/sites/default/files/publication_reports/Youth_in_India_2022.pdf

Openai. (2022). Introducing ChatGPT. Retrieved from https://openai.com/blog/chatgpt

Pavlik, J. V. (2023). Collaborating With ChatGPT: Considering the Implications of Generative Artificial Intelligence for Journalism and Media Education. Journalism & Mass Communication Educator, 78(1). https://doi.org/10.1177/10776958221149577 DOI: https://doi.org/10.1177/10776958221149577

Pupale, R. (2018). Support Vector Machines ( SVM ) Support Vector Machines ( SVM ). Towards Data Science, 23, 349-361. https://doi.org/10.1007/978-0-387-30162-4_415 DOI: https://doi.org/10.1007/978-0-387-30162-4_415

Qadir, J. (2023). Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education. IEEE Global Engineering Education Conference, EDUCON. https://doi.org/10.1109/EDUCON54358.2023.10125121 DOI: https://doi.org/10.36227/techrxiv.21789434

Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2023). Improving Language Understanding by Generative Pre-Training. Homology, Homotopy and Applications, 9(1), 399-438. https://doi.org/10.4310/HHA.2007.v9.n1.a16 DOI: https://doi.org/10.4310/HHA.2007.v9.n1.a16

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... Liu, P. J. (2020). T5: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21, 1-67.

Samarth, V. (2023, December 14). Large Language Models: A Guide on its Benefits, Limitations, and Future. Retrieved from https://emeritus .org/in/learn/ai-and-ml-large-language-models/

Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook messenger. Computers and Education, 151(103862). https://doi.org/10.1016/j.compedu.2020.103862 DOI: https://doi.org/10.1016/j.compedu.2020.103862

Tate, T., Doroudi, S., Ritchie, D., Xu2, Y., & Warschauer, M. (2023). Educational Research and AI-Generated Writing: Confronting the Coming Tsunami. Retrieved from https://doi.org/10.35542/osf.io/4mec3 DOI: https://doi.org/10.35542/osf.io/4mec3

Tech Desk. (2023). ChatGPT hit 1 million users in 5 days: Here’s how long it took others to reach that milestone. Retrieved from https://indianexpress.com/article/technology/artificial-intelligence/chatgpt-hit-1-million-users-5-days-vs-netflix-facebook-instagram-spotify-mark-8394119/

Terwiesch, C. (2023). Would Chat GPT3 Get a Wharton MBA? Retrieved from https://mackinstitute.wharton.upenn.edu/wp-content/uploads/2023/01/Christian-Terwiesch-Chat-GTP-1.24.pdf

Topsakal, O., & Topsakal, E. (2022). Framework for A Foreign Language Teaching Software for Children Utilizing AR, Voicebots and ChatGPT (Large Language Models). The Journal of Cognitive Systems,7(2). https://doi.org/10.52876/jcs.1227392 DOI: https://doi.org/10.52876/jcs.1227392

Vanian, J. (2022). ChatGPT is a new AI chatbot that can answer questions and write essays. Retrieved from https://www.cnbc.com/2022/12/13/chatgpt-is-a-new-ai-chatbot-that-can-answer-questions-and-write-essays.html

West, C. G. (2023). AI and the FCI: Can ChatGPT Project an Understanding of Introductory Physics? Retrieved from http://arxiv.org/abs/2303.01067

Williams, C. (2023). Hype or the future of learning and teaching? 3 limits to AI’s ability to write students’ essays. LSE: Impact of Social Sciences, 40(2), 366-374.

Wu, E. H. K., Lin, C. H., Ou, Y. Y., Liu, C. Z., Wang, W. K., & Chao, C. Y. (2020). Advantages and constraints of a hybrid model K-12 E-Learning assistant chatbot. IEEE Access, 8, 77788-77801. https://doi.org/10.1109/ACCESS.2020.2988252 DOI: https://doi.org/10.1109/ACCESS.2020.2988252

Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019). XLNet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems, 32(NeurIPS), 1-18.

Zhou, L., Gao, J., Li, D., & Shum, H. Y. (2020). The design and implementation of xiaoice, an empathetic social chatbot. Computational Linguistics,46(1),53-93. https://doi.org/10.1162/COLI_a_00368 DOI: https://doi.org/10.1162/coli_a_00368