Understanding College Students’ Satisfaction With ChatGPT: An Exploratory And Predictive Machine Learning Approach Using Feature Engineering

Artificial Intelligence (AI) technologies are continually improving and becoming more pervasive in many facets of our lives. ChatGPT is one such cutting-edge artificial intelligence application, and it has received a lot of worldwide media attention, specifically from educationists, technologists, and learners. It is imperative to understand and evaluate the impact of ChatGPT on computer science students as it directly and holistically influences them. A quantitative instrumental case study explores ChatGPT’s impact on early adopters in education. A survey of undergraduate computer science students at a state university of Delhi was conducted to get insight into their opinion on adopting this revolutionising technology for their education, career, and overall satisfaction. An end-to-end data science approach is applied to encompass exploratory and predictive modelling with feature engineering solutions. Results reveal the most influential features contributing to students’ satisfaction in adopting ChatGPT for their day-to-day chores concerning their social life, education, and career. The Linear Support Vector classifier, a machine learning algorithm for predicting the satisfaction or dissatisfaction in students’ shows an accuracy score of 72.73% and 97.72%, respectively. The AUC for this multiclass prediction model is convincing and is 0.74, 0.71, and 0.96 for satisfied, neutral, and dissatisfied classes, respectively.


Introduction
ChatGPT (Chat Generative Pre-Trained Transformer), a technology and research project based on Artificial Intelligence, was developed by Ope-nAI (Openai, 2022;Vanian, 2022) and released worldwide on November 30, 2022.It is the latest development in the world of generative AI, based on "Generative Pretrained Transformer 4 (GPT-4)".According to Samarth (2023), ChatGPT is an example of a "large language model (LLM)" that produces natural-sounding sentences by mimicking the linguistic statistical patterns seen in a significant body of online-sourced literature.The users can ask questions or request to develop a program code, an itinerary, technical support, prescribe medicine, and many more types of questions, and the ChatGPT responds within seconds.
Hence, ChatGPT is a useful tool for chatbots, academia, customer assistance, the medical field, and a variety of other applications because of its capacity to reply to a wide range of themes and subjects (Gilson et al., 2023).Also, this is the reason ChatGPT became popular quickly and grasped one million users globally just within five days after its initial launch (Tech Desk, 2023).Early adopters such as students have given it a lot of attention, and have adopted it diligently in their day-to-day study and other activities (Haque et al., 2022).With the increase in low-cost availability of the Internet, mobile networks, and phones, ChatGPT has gained instant acceptance by Generation Z which is already well-conversant with personal digital assistants like Alexa, Siri, Cortana, and Google Assistant (Smutny & Schreiberova, 2020;Zhou et al., 2020).
According to the Ministry of Statistics and Programme Implementation, the Government of India, 27.3% of the country's population, or 371.4 million individuals, are in the age range of 15 to 29 (Ministry of Statistics and Programme Implementation, Government of India, 2022).India is witnessing a revolution in the field of teaching and learning (India Today, 2023) due to widespread acceptance of ChatGPT by students, hence it becomes imperative to understand what are the attitudes and intentions of learners that make them feel satisfied with this machine learning-based AI tool.
This study is a novel piece of research to get deeper insights into the facilities offered by ChatGPT that result in the overall satisfaction of the computer science college students and motivate them to adopt this AI-based technology product.The authors have made use of exploratory as well as predictive modelling techniques of machine learning technology to extract the significant features that impact the satisfaction level of the students in adopting the Generative Pretrained Transformer, named ChatGPT for their day-to-day life specifically related to education and career.Also, an advanced machine learning classifier based on the Support Vector algorithm has been trained to predict the satisfaction level of the students depending upon their response to a minimal set of questions.
The paper is organized as follows.The next section details the review of the literature, followed by the Research Objectives, Methodology, Data Collection, Exploratory Data Analysis, Data pre-processing and Feature Engineering.The next sections are for the Development of a Machine Learning model to predict the satisfaction of students in using ChatGPT followed by Results and discussions on its performance.This is followed by another section on Feature Selection for an improved version of the Predictive model and the development of a predictive machine learning model with selected six features.Lastly, Conclusions and Future scope are explained.

Background of the Study
Two significant players are garnering attention in the modern digital world.One is the chatbot, which is an expert at providing predetermined answers and is renowned for being clear and dependable.The other is ChatGPT, a brand-new chatbot that can converse with you like a human and comprehend what you are saying (Gupta, 2022).
Studies investigating the advantages of employing chatbots in educational settings have already surfaced, despite the widespread use of chatbots around the world.These advantages include giving users a positive learning experience by enabling real-time interaction, improving peer communication skills, increasing learner learning efficiency (Wu et al., 2020), and aiding teachers in managing significant in-class activities.Because chatbots could be produced using machine learning and natural language processing, their use in education became a new area of academic study, in 2017 (Folstad & Brandtzaeg, 2017).Smutny and Schreiberova (2020) highlight the potential for chatbots to develop into smart teaching assistants in their study on educational chatbots for Facebook Messenger to promote learning.Chatbots have been the subject of other studies looking at language learning.According to Huang et al. (2021), who conducted a review of 25 empirical studies, educational chatbots can promote students' language acquisition through interactive activities supported by specified learning objectives.According to a similar study Kim et al. (2019), chatbots help students communicate more effectively by boosting the number of interactions they have, motivating them, and capturing their interest in the subject matter.Following Chatbots, there has been the development of several large language models, including "Generative Pre-trained Transformer"(GPT) (Radford et al., 2023), "Bidirectional Encoder Representations from Transformers" (BERT) (Devlin et al., 2019), "an autoregressive Transformer" (XLNet) (Yang et al., 2019), "Text-to-Text-Transfer-Transformer" (T5) (Raffel et al., 2020), and "Robustly Optimized BERT Pre-training Approach" (RoBERTa) (Liu et al., 2019).
"Generative Pre-trained Transformer" (GPT)-3, is a more advanced version of GPT that has recently been created (Brown et al., 2020) and is currently being used the most (Floridi & Chiriatti, 2020).With just one pre-training and fine-tuning pipeline, these models, which are built on transformer architecture, can generate text that resembles human speech, respond to inquiries, help with translation and summarization, and carry out a variety of NLP tasks.The GPT-3 algorithm is based on an unsupervised or partly supervised machine learning framework with 175 billion arguments (Brown et al., 2020), which creates artefacts from human activity by using statistics, probability, and other mathematical concepts (Hu & R, 2023;Jovanovic & Campbell, 2022).Forty-five terabytes of text were used to train the GPT-3 algorithm (Cooper, 2021).
The data sources of ChatGPT include books (Book1 & Book 2 are two internet-based books corpora), Wikipedia (Brown et al., 2020), Common Crawl (a non-profit organization that crawls the web and freely makes its archives and datasets available to the public), WebText2, and all outbound Reddit links from posts with more than three upvotes.According to Brown et al. (2020), "GPT-3" is ten times more advanced than any prior non-sparse language model.
ChatGPT could be a useful tool for chatbots, academia, customer assistance, and a variety of other applications because of its capacity to reply to a wide range of themes and subjects (Gilson et al., 2023).As a result, the platform has drawn a lot of interest from early adopters (such as students and academics) and has even been referred to as a disruptive technology in a variety of industries, including academia and education (Haque et al., 2022).Users were astounded by ChatGPT's ability to produce content like short stories and dialogues based on the user's straightforward directions once it was released (Topsakal & Topsakal, 2022).ChatGPT has already demonstrated that it can pass the medical licensing exam (Gilson et al., 2023), a test for admission to law school (Choi et al., 2023), and a standard examination for introductory physics courses (West, 2023).
Such widespread acceptance and popularity of ChatGPT have fuelled the motivation for this research.The authors have executed this study by applying machine learning techniques to understand the reasons for the acceptance and satisfaction of the computer science students for this AI-based Generative Pretrained Transformer.This study applies feature engineering and feature selection techniques for the identification of the most influential characteristics of ChatGPT that impact the satisfaction level of the students in terms of adapting ChatGPT for their day-to-day academic, career, and social life.

Objectives of the Study
Our focus is to understand the influence of ChatGPT on the tech-savvy youth of India specifically university students of the computer science stream.Specifically, an attempt has been made to get the answer to "How do the computer science undergraduate students perceive the ChatGPT's impact on productivity, creativity, reliance, and concerns regarding future dependency or insecurity in their educational endeavours and can a Machine learning predictive model be developed that predicts the satisfaction levels with ChatGPT based on a limited set of identified features?" To obtain an answer to the mentioned research question, the following objectives have been formulated.

Research Methodology
The adopted methodology for this novel research has been shown as a flowchart in Figure 1 and the same has been explained step by step in the following sections.

Data Collection
Real datasets have been collected by surveying the students of undergraduate courses at a state university in Delhi via non-probability convenience sam- pling.The sample has been confined only to computer science students as the focus is to gain insights into their productivity, purpose, support for code development, etc.There are a total of sixteen questions which include 2 demographic, 8 dichotomous, 1 multiple choices, and 5 Likert scale-type questions.The detailed questionnaire is mentioned in Table 1.We have used Google Forms and the respondents were assured of confidentiality to gather honest inputs.The respondents include 117 males (81%) and 28 females (19%).The data collection was done in April 2023 as by then the students had used Chat-GPT exhaustively, especially for their examination preparation.

Exploratory Data Analysis
Generation Z learners are technology-oriented and to observe whether they are harvesting the power of the AI-enabled ChatGPT tool, their answers to various questions have been analysed using the visualization facility of Python language.We have used the Jupyter Notebook application (Jupyter, 2023) for the development of complete code for exploratory data analysis in Python language.All graphs are shown in Figures 2 to 17 and the findings are mentioned below: - • Figure 2 depicts that there are more males than females as far as the satisfaction in using ChatGPT is concerned.68.4% of males are satisfied as compared to 53.6% of females.However, 29.1% of males are neutral as compared to 42.9% of females.We have taken percentages for gender-wise comparison as our dataset is skewed.There are more males in the computer science field as compared to females and hence we could collect responses from 117 males and 28 females.The percentage of neutral respondents also reveals that though females are using ChatGPT they are not very clear about their viewpoint about the facilities offered by ChatGPT, that's why 42.9% of females are neutral about their satisfaction level.
• Figure 3 reveals the various purposes for which the students use Chat-GPT.The most prevalent application of this AI GPT has been identified as "Gaining Knowledge" followed by "Completing Assignments".
• Figure 4 elaborates that there are more male members in the respondents list.We found that there are 80.7% males in comparison to 19.3% females.This also advocates the fact that in India there are more males in the computer education field as compared to females.
• Figure 5 gives insight into the overall satisfaction level of the respondents.65.5% of the respondents are satisfied, 31.7% are neutral whereas only 2.8% are dissatisfied with ChatGPT.This shows clearly that Gen Z is enormously adept at deploying the latest AI-based technologies.
• Figure 6 depicts that within 4 months of the release of ChatGPT, the computer science students in India have adapted to it well, and more than 80% of them have been using it quite frequently for various purposes like "Gaining Knowledge", "Completing Assignments", "Content/blog writing", "Writing/ debugging code", "Getting ideas for social life (deciding a gift for a friend, planning a party, etc.)", etc.
• Figure 7 explains the reason for usage by the students as 83.4% of them feel that ChatGPT is saving their time and increasing their productivity.
• Figure 8 depicts that 56.6% of the participants feel that ChatGPT is killing the intelligence and creativity of the users.This indicates that the students have realized that continuous use would limit their mental development as well, as 69% of the respondents agree/ strongly agree with this.The same is visible in Figure 9.
• Figure 10 shows that 49% of students have observed that ChatGPT gives correct answers often and 25.5% of students found that they obtained correct answers always.Also, since many of the students have found accurate answers, hence 80.7% of the students are in favour of recommending ChatGPT for educational purposes to others, as depicted in Figure 11.The study reveals that students were still more confident with Google Search and 56.6% of them felt that ChatGPT cannot replace Google Search.
• Figure 14 depicts that 68.3% of the respondents are confident that Chat-GPT will not be able to replace programmers, and their future will not be at risk because of AI technology-based products.
• Figure 15 shows that about 60% of respondents feel that they would be dependent on the GPT as they have been utilizing it for numerous purposes to make their day-to-day tasks easier as discussed above.
• Figure 16 elaborates that 60% of the surveyed computer science students have used ChatGPT to the extent that they have observed that "ChatGPT produces same results for similar-looking questions" and hence it cannot replace humans.
• Figure 17 reveals that students are using ChatGPT as long as it is available for free.Less than 5% of the respondents have shown their willingness to buy the paid subscription.Data Pre-processing and Feature Engineering Since all sixteen questions were marked compulsory in Google form, so there were no missing values.This study is exploratory as well as predictive in nature where advanced feature engineering techniques have been applied.The response to most of the questions is in the Likert scale or dichotomous and both of these are considered categorical data by Python.Categorical data must be turned into numerical data for machine learning models to accept them because they do not work with them.For this conversion purpose, the label encoding technique of feature engineering has been used.In its simplest form, feature engineering refers to the approaches used to handle features such that a specific machine-learning model can use them.Overfitting and Underfitting are reduced with the aid of feature engineering.
Out of sixteen questions, we have dropped two features from the datasets to be analysed further.First is the name of the respondents.The second feature that was dropped is "The purpose for which the students use ChatGPT".This was only utilized for exploratory analysis and dropped for feature engineering and development of the Machine Learning (ML) model.Since this question had multiple options to choose from a pool of twelve choices and the ML algorithms for classification work only on categorical data.Converting this to a category would increase the feature enormously and would result in a curse of dimensionality problem.From the remaining fourteen features, we have considered "overall satisfaction" as the output feature (target variable) which is dependent on the rest of the thirteen features (questions).The list of these thirteen features is mentioned in Table 1.

Development of a Machine Learning Model To Predict The Satisfaction Of Students In Using ChatGPT
To predict whether the students are satisfied, neutral, or dissatisfied with ChatGPT, we have developed a machine learning classification-based model based on their responses to thirteen questions.We have used the Scikit-learn (Sklearn) library of Python, which is the most effective and reliable machinelearning library that offers a range of powerful machine-learning technologies.
A linear support vector classifier (SVC) has been selected for the study as it seeks to identify the decision border with the greatest margin of separation between classes.The support vector refers to the data points close to the boundary, while the margin is the separation between the dividing boundary and its nearest data points (Pupale, 2018).A linear support vector classifier can discover the "best fit" hyperplane to classify given some linearly separable input data.Since just a small sample of data points that are close to the boundary decides the boundary, the advantage of linear SVC is that it reduces the impact of outliers (Java T Point, 2016).After dividing the multi-classification problem into numerous binary classification problems, the same method is applied to multi-class classification.Utilizing Linear SVC has the advantage of producing quick predictions with a sizable dataset that can be linearly separated and is reasonably simple to analyse.The entire dataset has been split into train and test in the ratio of 80:20.The Linear SVC model has been trained using 80% of records and tested with the balance 20% of the records.

Findings On the Development Of SVC-The Machine Learning Classifier
After running the SVC model, impressive results have been found.The model has resulted in an accuracy of 79.21% on train data and 77.27% accuracy on test data.This depicts that the model is neither overfitting nor underfitting as the results of accuracy for train and test data are quite close.Hence, the model does not suffer from high bias or high variance.Since our dataset comprises three different output classes viz.Dissatisfied, Neutral and Satisfied; our focus is to find the accuracy for the classification of all three classes separately.We have found that the individual accuracy for each output class is also very high (more than 77%) and convincing, and the same has been given in Table 2. Though the accuracy of a classifier is always the key performance indicator, this is only true for data with a uniform distribution.Since our dataset is extremely imbalanced (65.5% of the respondents are satisfied, 31.7% are neutral whereas only 2.8% are dissatisfied with ChatGPT), ROC (Receiver Operating Characteristics) AUC (Area Under the Curve) is a more important indicator of the performance of the classification algorithm.ROC, a probability curve, basically separates the "signal" from the "noise" by plotting the True Positive Rate (TPR) versus the False Positive Rate (FPR) at different threshold values.In other words, it displays how well a categorization model performs across the board.In other words, it displays how well a categorization model performs at all classification thresholds (Bhandari, 2020).The ROC curve is summarised using AUC which represents the degree or amount of separability between classes.AUC reveals how well the model can differentiate across classes.The higher the AUC, the better the model's performance at discriminating between the positive and negative classes.In our dataset, the classes dissatisfied, neutral, and satisfied have been encoded as 0, 1, and 2, respectively.
To draw a ROC curve, we need to predict the probability of a sample belonging to a particular class.Since we have used a Linear Support Vector classifier that cannot predict this probability, so we have wrapped our ML model with a Calibrated Classifier CV.This classifier uses cross-validation to both estimate the parameters of a classifier and subsequently calibrate a classifier to predict the probability of a sample belonging to a particular class (Almeida, 2017;Brownlee, 2018).Our case study comprises three different classes viz.dissatisfied, neutral, and satisfied.We need to plot three AUC ROC curves and the concept used is One vs. Rest of the classes.The first ROC is for class 0 classified against classes 1 and 2, the second ROC is for class 1 classified against 0 and 2, and the third one of 2 classified against 0 and 1.These are visualized in Figure 18, 19, and 20 respectively.

Feature Selection For An Improved Version Of The Predictive Model
A subset of pertinent features (variables, predictors, etc.) are chosen via feature selection and used to build models.Feature selection offers numerous benefits viz.reduction in training time and storage requirement; improvement in interpretability of model; development of compact and visually appealing visualizations; and simpler and easier-to-understand models (Tung M Phung, 2019).
Python's Scikit-learn API provides a wrapper function named SelectKBest that uses numerous pre-defined score functions to extract the best features on which the output variable depends (Ertan, 2020).mutual_info_classif is the score function used in the code and this function computes the mutual information (Information gain) with the target.The KBest scores for the top 6 features as returned by the mentioned function are written in Table 3 and the bar graph for the same is visualized in Figure 21.These significant features reveal the facts that determine the satisfaction level of university students while using ChatGPT.It is observed that the frequency of usage, comparison with Google, getting the correct answer, increase in productivity, and becoming dependent are the top features that determine the satisfaction and hence adaptation of this innovative AI-based tool by the computer science students at the university.Predictive Machine Learning Model With Selected Six Features The dataset has been reduced in terms of the number of dimensions and the predictive model has been developed with selected six input variables only.Impressive results in terms of the accuracy of the model and Receiver Operating characteristics of this new model have been obtained.Accuracy for the   classification of all three classes separately is given in Table 4 and this is quite close to the previous value of the accuracy with all features considered for the development of the model.As depicted in Figures 22, 23, 24 the area under ROC curves for the three classes viz.Dissatisfied, Neutral and Satisfied respectively, is large enough and our model with selected features is capable enough to differentiate across classes.

Conclusions
The nucleus of this study lies in revealing the perception, attitude, purpose, and satisfaction of the students in adapting ChatGPT to their learning, socializing, and other day-to-day tasks.With this exploratory and predictive machine learning classification model, we have drawn many conclusions.Students find ChatGPT innovative, user-friendly, problem-solving and helpful in their academic and social lives.ChatGPT has numerous benefits and strengths and the same is reflected in the high percentages of the students who are satisfied with it.Only 3% of the respondents have shown dissatisfaction with ChatGPT, 66% are satisfied and the rest are neutral.This convinces us to accept that ChatGPT is here to stay as the respondents feel that it is increasing their productivity by saving time, and giving correct answers, and they want to recommend ChatGPT to others as well.On the negative side, the students feel that Chat-GPT would limit their mental development and hinder their creativity and intelligence.
For the development of a predictive machine learning classifier, we have done feature engineering that includes label encoding techniques to convert categorical variables into numeric ones so that the classification models can be created.We developed a predictive algorithm using the Linear Support Vector technique based on the student's responses to sixteen questions.These responses have been used as independent features to predict the output class, which is satisfied, dissatisfied, or neutral.The accuracy of the classifier is 79.55%, 97.72%, and 77.27% for the classes satisfied, dissatisfied, and neutral, respectively.To assess the model's performance at discriminating between the three classes, the receiver operating characteristics have been visualized and for that, we have wrapped our ML model with a CalibratedClassifierCV to get the probability of class prediction.The Area Under the Curve is 0.82, 0.98, and 0.78 for the classes satisfied, dissatisfied, and neutral, respectively.
For the development of a compact model that does not suffer from the curse of dimensionality, reduces overfitting, and collinearity, and enhances interpretability, we have selected a subset of six relevant features by applying a wrapper method named SelectKBest.These features play an important role in determining the satisfaction level of the students in adapting ChatGPT for their academic pursuits.These features in the form of questions are mentioned below: -1.How often do you use ChatGPT?
2. Do you think ChatGPT may replace Google search? 3. How often do you find answers given by ChatGPT to be right?4. Do you think ChatGPT is increasing your productivity and is a time saver? 5. Do you believe that ChatGPT will make you dependent in the near future?6.Which one do you think is better and more accurate for getting answers to your questions?
Using only these six features, the SVC machine learning model was developed again.The model was trained and tested and it produced convincing results.The accuracy of this classifier is quite close to the accuracy of the previous one with sixteen features as input and it stands at 72.73%, 97.72%, and 70.45% for the classes satisfied, dissatisfied, and neutral, respectively.Also, the AUC is 0.74, 0.96, and 0.71 for the classes satisfied, dissatisfied, and neutral, respectively.Hence, we have developed a model with just six features that have produced convincing performance parameters.
It can be concluded that there are two dimensions to our findings from this piece of research.First is that ChatGPT is here to stay as a majority of the respondents have shown satisfaction with this AI-based solution.Secondly, we have extracted the important factors that are driving the satisfaction of the students with using ChatGPT.

Future Scope
ChatGPT is a primogenitor of the Artificial Intelligence deluge that is going to hit global education.Right now, there is no law governing its usage in the education sector though students have been using it for getting solutions to their assignments, programming codes, and PowerPoint presentations.The point of concern is that their natural creativity may be hindered as they become more and more dependent on large language models (LLM) like ChatGPT.Hence, it is important to embrace this new era with innovation and effective assessment methodologies as the introduction of AI into education is a reality that cannot be ignored.The onus is now on the education policymakers to adapt the education system to harvest the power of ChatGPT and other LLMs.Policymakers should formalize the ethical boundaries that should be accepted in educational setups so that academic integrity is not violated.
In the future, a qualitative as well as quantitative approach can be applied to understand the impact of ChatGPT, Google Bard, and other LLM-based tools on various stakeholders and statutory bodies of education in India like Central Board of Secondary Education (CBSE), University Grants Commission (UGC), All India Council for Technical Education (AICTE), National Assessment and Accreditation Council (NAAC), National Council for Teacher Education (NCTE) etc.

Figure 1 .
Figure 1.Flowchart of the Methodology Followed in the Study.

•
Figures 12 and 13 throw light on the comparison of ChatGPT with Google search.(This data was collected in April and Google Bard was launched on 15 May 2023 in India, hence the comparison was done with Google search only).

Figure 2 .
Figure 2. Visualization of satisfaction level according to Gender.

Figure 3 .
Figure 3. Numerous purposes for which ChatGPT is used by computer science students.

Figure 10 .
Figure 10.Does ChatGPT give the correct answer?

Figure 14 .
Figure 14.Will ChatGPT replace programmers in the near future?

Figure
Figure 16.ChatGPT Produces the Same Results for Similar-Looking Questions.

Figure 22 .
Figure 22.Class 0 -ROC Curve For "Dissatisfied" Category for Classifier Developed with Selected Features.

Figure 23 .
Figure 23.Class 1-ROC Curve For "Neutral" Category for Classifier Developed with Selected Features.