1 Introduction

Artificial Intelligence in Education (AIED) is an emerging technology and a rapidly expanding project in recent years, involving numerous ethical issues, including data privacy breaches, intellectual property rights, algorithmic bias, and the generation of false content. These issues have profound impacts on the fairness of education, students’ mental health, and social inclusion. As a result, education policymakers view the ethical regulation of AIED as a major challenge in today’s educational sector (Bommasani et al., 2022; Cardona et al., 2023).

Since 2024, several Chinese universities have issued notifications requiring graduation theses to undergo detection for AI-Generated Content (AIGC) to prevent students from overly relying on AI-generated content. U.S. higher education institutions have also begun checking students’ assignments and theses for AIGC since 2020. However, the use of AIGC detection tools has raised concerns about data privacy and security, bringing widespread attention to the ethical issues in AIED (Chen et al., 2023; Wu et al., 2023). This includes questions about who should oversee the ethical standards for the development of AIED applications and whether these applications can be released into the market without inspection and approval from relevant authorities.

Regarding the ethical issues raised by artificial intelligence, Buitem (2019) suggests enhancing the transparency of input data, algorithm testing, and decision-making models to make machine learning algorithms more interpretable (Buitem, 2019). Gordon (2021) calls for the establishment of regulations to address issues such as machine bias, legal decisions, and legal responsibilities (Gordon, 2021). In May 2023, the U.S. Department of Education issued four urgent recommendations on AIED ethical standards, including: (1) using automation technology to advance learning outcomes while protecting human decision-making and judgment; (2) reviewing the quality of foundational data in AI models to ensure accurate, contextually appropriate information is used in educational applications for fair and unbiased pattern recognition and decision-making; (3) examining specific AI technologies, such as those used in large educational technologies or systems, to determine if they enhance or undermine student fairness; (4) implementing measures to safeguard and promote fairness, including providing human checks and balances and restricting any AI systems and tools that diminish fairness (Cardona et al., 2023). In August 2024, UNESCO (the United Nations Educational, Scientific and Cultural Organization) released guidelines on AI competence in education, highlight the importance on the ethics of AI. It declared that educators must understand and apply the essential ethical standards, guidelines, regulations, institutional structures, and practical moral principles outlined by AI. These principles originate from the swiftly growing domain of AI ethics and its implications for the education industry (Cukurova & Miao, 2024).

2 Methodology

Since current research on the ethics of AI is still insufficient, we can draw upon and reference technology ethics and utilitarian ethics as our methodological framework. The four principles of technology ethics are proper use of technology, responsibility, fairness, and cost. Among them, “proper use of technology” is the cornerstone of technology ethics; “responsibility” is key to ensuring proper use of technology, focusing on the accountability of participants; “fairness” refers to the responsibility and requirement for the reasonable distribution of technology; and “cost” encompasses the technological risks and negative effects that must be borne when using technology (Milano et al., 2023; Xiao, 2023).

AI is fundamentally a technology, and the principles of technology ethics apply to it as they do to any other technological advancement. “Proper use of technology” in the context of AI means that its development needs to align with the goal of promoting good, including adherence to ethical standards and transparency principles. This requires the joint efforts of the government, tech companies, educators, ethicists, and other stakeholders to ensure that AI development meets the standards. “Responsibility” pertains to the obligations of AI developers to society and users, including the impact on their rights during development, application, and promotion, and the implementation of measures to protect their rights and safety. “Fairness” ensures that AI applications do not contradict the current societal advocacy for educational fairness, such as embedding gender, racial, ethnic, or economic discrimination into AI algorithms. “Cost” includes the technological risks and negative effects that AI developers should bear, as well as the consequences of technical errors. Developers should not transfer these costs to users or try to evade responsibility through legal loopholes in user agreements. Users also bear corresponding risks and costs when using AI, which necessitates ethical standards and regulations to reduce these risks (Nasr et al., 2018; Song et al., 2017).

Utilitarianism is a theory widely applied in the field of ethics, whose core tenet is that the correctness of actions depends on whether they produce the greatest happiness or benefit (Mill, 1863). It emphasizes results, advocating for the moral value of actions based on their overall happiness. The development and use of AI systems should serve humans, aiming to enhance the maximum happiness or benefit of human groups, rather than prioritizing the interests of machines. Utilizing utilitarian ethics can act as a constraint on machines in the face of human-machine conflicts. Mill also pointed out a potential risk of utilitarianism in his book, namely the tyranny of the majority. If the principles of utilitarianism are abused, it may lead to the majority using their dominant position to oppress the minority, thereby undermining social fairness and justice. Therefore, citation should be used judiciously to ensure that the principles of utilitarianism are applied in a manner that respects the rights and interests of all individuals (Mill, 1863).

In AIED, evaluating whether an AIED application improves educational quality, increases learning efficiency, and expands educational opportunities, all while aiming for the greatest benefit for the primary users (students, teachers, education administrators, and parents), rather than prioritizing the interests of machines or the companies or individuals controlling them. While there are internal conflicts among students, teachers, education administrators, and parents, their educational goals align—focusing on the comprehensive development and success of students. Therefore, AIED ethics can combine the basic principles of technology ethics—proper use of technology, responsibility, fairness, and cost—with utilitarian ethics to ensure that technological applications not only meet educational goals but also adhere to ethical standards that prioritize the maximum benefit for the primary users of AIED.

3 Current ethical issues in AI in education

At present, artificial intelligence companies typically use vast amounts of data to train large models. Most of this data comes from publicly available internet sources, with a smaller portion consisting of preprocessed datasets and private data. While internet data contains positive and useful information, it also includes a significant amount of harmful and biased content. The expectation of tech companies and stakeholders is for AI models to be launched and monetized quickly. This is why the ChatGPT-3 model sometimes provides biased, inaccurate, or inappropriate responses. These issues arise because ChatGPT-3 was trained on vast amounts of text that may contain biased and inaccurate information. The capabilities of ChatGPT-3 have raised concerns about the ethical implications and potential misuse of such powerful language models, including the generation of false data and fake news (Kietzmann et al., 2020; Mubarak et al., 2023; Wu et al., 2023).

For example, current AI technology can simulate historical narratives, but its primary goal is to generate content that feels real, not necessarily factual. If AIED applications are trained on large amounts of inaccurate or fictional data, such as a history teaching AIED application using historical novels instead of accurate historical records, the result could be that students believe the novel’s descriptions are true facts, leading to misinformation and hindering proper historical knowledge construction. Once AI models develop “biased values,” tech companies implement a series of remedial measures, commonly including setting filters and prohibition commands within the system to prevent the model from outputting biased or harmful information.

This raises another ethical issue: who is responsible for filtering and reviewing the AI’s output, and whether the ethical standards and values of AI tech companies are fair and unbiased. When AIED determines that certain information conflicts with its “filter” mechanism, such as topics related to “war” in history lessons, it may lead to the output of misleading or even falsified historical information to students.

Secondly, there is the question of whether AI companies can use user input to continue training models, which raises concerns about privacy breaches or the misuse of confidential user information. Currently, technology companies in the market are divided into two camps. Some companies use user input to further train their models, a method that significantly enhances the learning efficiency and output quality of AI systems. However, this approach can involve sensitive information, and if mishandled, it could lead to privacy breaches or misuse. To reassure users, some tech companies choose not to use user input data for training AI models, thereby avoiding potential privacy risks. This method typically relies on pre-prepared, depersonalized datasets or simulated data. The downside is that it may limit the learning effectiveness and application scope of the AI model, as the model cannot learn and iterate from real-time user feedback.

In the U.S., some tech companies prohibit employees from uploading company-related information to large models like ChatGPT. For example, Microsoft implemented a new policy banning its employees from using ChatGPT or any other third-party chatbots for work-related purposes. This move is driven by concerns about the potential leakage of internal confidential information. Microsoft worries that employees might inadvertently input queries containing sensitive company information, leading to these data being recorded or misused by external systems (Novet, 2023). Similarly, schools could face data leakage risks if they upload internal documents or sensitive information into unvetted AIED applications.

With Meta (formerly Facebook) first open-sourcing the code for its large AI model Llama on the internet in 2023, AI industry has also entered an era of rapid development, with various new AIED applications emerging. On January 19, 2024, the European Commission, European Parliament, and Council of the European Union finalized the Artificial Intelligence Act, which states that “AI systems used in education or vocational training, particularly those determining admissions or placements, assigning individuals to various levels of education and training institutions or programs, assessing individuals’ learning outcomes, determining the appropriate level of education for individuals, and impacting the education and training levels individuals will receive or can receive, as well as those monitoring and detecting student misconduct in exams, should be classified as high-risk AI systems. This is due to their potential to decide on a person’s educational and professional career, thereby affecting their livelihood” (European Commission, 2024).

Therefore, to safeguard the interests and information security of students, parents, teachers, and schools, it is crucial to pay special attention to high-risk AIED systems that may pose privacy risks. Establishing ethical norms and regulatory measures in education is both urgent and necessary.

4 Examples of AI education applications

With the widespread use of AI tools like ChatGPT in student assignments and exams, American universities have started employing AI plagiarism detection tools such as Turnitin and ZeroGPT to ensure academic integrity and uphold academic standards. This study compares major AIED applications on the market, including ChatGPT, Turnitin, and ZeroGPT, and delves into the underlying AI and foundational models behind them for further analysis and comparison.

4.1 ChatGPT

ChatGPT is an AI-based conversational generation model designed to simulate natural language communication and generate contextually relevant dialogue (Brown et al., 2020). Unlike other language models, ChatGPT focuses on generating conversational text, offering unique advantages and challenges in the educational field. ChatGPT operates using a transformer architecture similar to other language models, learning from vast amounts of conversational data to understand language context and dialogue logic, thereby generating coherent conversation content. Its training process typically involves extensive dialogue data, such as social media conversations or online chat logs. In education, ChatGPT is mainly used for simulating dialogue scenarios and providing personalized learning experiences. Educators can use ChatGPT to create virtual teaching assistants or intelligent tutors to help students solve problems, answer questions, or offer personalized learning advice. ChatGPT can also be used to simulate conversational learning environments, helping students improve their language communication skills (Baidoo-Anu & Ansah, 2023; Kasneci et al., 2023).

However, ChatGPT’s use also poses ethical and educational challenges. Since its generated content might be automated, there are concerns about the accuracy and reliability of the information. In educational settings, ChatGPT-generated dialogues might mislead students with inaccurate information, affecting their learning outcomes. Additionally, ChatGPT’s generated content might be influenced by biases, leading to unfair or discriminatory dialogues. While ChatGPT can offer personalized learning experiences and real-time teaching support, it might also reduce students’ interaction with real teachers or peers, impacting their social skills and emotional development, and increasing their dependency on technology. Furthermore, ChatGPT’s potential use in completing assignments and exams has raised widespread ethical concerns. Students might use ChatGPT to complete their work, which, although improving short-term grades, undermines the authenticity and value of the learning process, as students fail to genuinely grasp knowledge and skills. Teachers might find it difficult to assess whether submitted assignments truly reflect students’ abilities, posing a challenge to the fairness and accuracy of educational assessments. Relying on AI-generated tools could limit students’ development of independent thinking and problem-solving skills, which are core educational goals. Therefore, despite ChatGPT’s potential in simulating dialogue scenarios and providing personalized learning, educators and policymakers need to carefully consider its use and develop appropriate guidelines and educational strategies. Strengthening real dialogue and interaction between students, teachers, and peers is essential to promote comprehensive development and effective learning (Ray, 2023).

4.2 Turnitin

Turnitin is a cloud-based tool used to detect plagiarism in academic papers and other written works. Its core function is to detect plagiarism through its extensive database, which includes academic articles, books, student-submitted papers, and web content. When students submit assignments, Turnitin’s algorithm compares the submitted text with the database content to find similarities. This comparison is typically based on aspects such as text grammar, vocabulary, and structure. If similarities are found, Turnitin generates a similarity report indicating potential plagiarism.

Turnitin employs pattern-matching technology to identify plagiarism in texts. This technology does not require traditional “training datasets” but continuously updates its database to include new source materials. Each time a new document is added to the database, the system indexes it for future comparisons with submitted assignments (Buckley & Cowap, 2013). However, Turnitin is not without flaws. Its algorithm may produce false positives, mistaking legitimate citations or shared texts for plagiarism. Additionally, there are privacy and copyright concerns regarding Turnitin’s database. Student submissions might be permanently stored in the database, and unauthorized use could infringe on their privacy and intellectual property rights (Zaza & McKenzie, 2018). Despite Turnitin’s role in maintaining academic integrity, it faces ethical and educational issues. The opacity of Turnitin’s algorithm and database content prevents users from knowing how their work is processed and compared. Over-reliance on such tools by schools might stifle students’ creative thinking, just as excessive supervision might limit their ability to explore and learn independently. Educational institutions should use Turnitin while also providing educational guidance, emphasizing not only the consequences of plagiarism but also teaching proper citation practices and independent thinking to ensure students understand the importance of academic integrity. Turnitin should be part of a broader academic integrity education plan rather than the sole regulatory tool.

4.3 ZeroGPT

ZeroGPT is a relatively new AI detection tool designed to identify whether texts are generated by AI, such as ChatGPT. It works by analyzing the linguistic features and patterns of texts to determine if they deviate from typical human writing (Liu et al., 2023). ZeroGPT is trained by comparing human-written texts with AI-generated texts, focusing on metrics like “perplexity” and “burstiness.” Low perplexity indicates common text within language models, while high burstiness indicates vocabulary variations not typical in human writing.

ZeroGPT’s architecture typically involves several key steps. First, it analyzes the text to detect linguistic features such as vocabulary choice, grammatical structure, and logical coherence. ZeroGPT then compares these features with pre-trained AI models to determine similarity with AI-generated texts. Finally, ZeroGPT generates a confidence score indicating the likelihood that the text was generated by AI. In education, ZeroGPT can be used to detect whether student assignments or papers contain plagiarized or AI-generated content. Educators can use ZeroGPT to better identify and prevent academic misconduct, maintaining academic integrity and ethics. However, ZeroGPT faces challenges and ethical issues. Its accuracy can be affected by the model’s training data and algorithm design, leading to potential misjudgments or omissions. Additionally, the use of ZeroGPT might raise privacy and data security concerns, especially when analyzing personal or sensitive information. Moreover, ethical dilemmas might arise, such as balancing academic integrity with protecting student privacy rights.

5 Analysis of foundational models in AI education applications

In the field of education, the use of AI in Education (AIED) is becoming increasingly prevalent. BERT and GPT, as two fundamental models of AIED, play a crucial role in the development of machine learning and artificial intelligence by simulating the way humans learn language and process information (Bommasani et al., 2022; Brown et al., 2020; Devlin et al., 2018). Despite the emergence of more advanced and modern models, BERT and GPT remain significant. Analyzing these two “foundational” models is essential due to their high recognition and widespread use in the industry. By comparing and analyzing the architectures of BERT and GPT, we can better understand their applications in education. This section systematically dissects the principles and structures of artificial intelligence by drawing parallels between AIED and human development and education, helping educators quickly grasp the principles behind AIED and laying the foundation for constructing an ethical framework for AIED.

5.1 The transformer: The fundamental model of AI

Most foundational AI models are based on the Transformer architecture. The Transformer is a deep learning model that uses an attention mechanism to weigh the importance of different parts of the input data. This process is akin to how teachers identify which students need extra attention and support and which teaching content is most crucial for the students. The attention mechanism enables the model to focus on the most informative parts of the input data, similar to how teachers highlight key points or difficult concepts during complex explanations to ensure students focus on the most critical information. This model is widely used in natural language processing and computer vision. Since its introduction in 2017 through the paper “Attention is All You Need,” Transformers have been extensively studied. They are seen as an upgrade to recurrent neural networks and long short-term memory architectures, offering advantages like parallel processing (enhancing performance and scalability) and bidirectionality (aiding in understanding ambiguous words and references) (Hochreiter & Schmidhuber, 1997; Staudemeyer & Morris, 2019; Sutskever et al., 2011; Vaswani et al., 2017).

Why are Transformers so important in AI models? One can describe the ethical issues involved in training, processing, and filtering large language models using an educational analogy. Training a large language model is like raising a baby, an analogy that helps understand the complexities involved in the architecture of artificial intelligence. A baby’s genetic code and the foundational Transformer architecture in an LLM share a profound similarity: both determine future potential and direction.

A baby’s genetic code carries the characteristics of the parents, determining not only physical traits like eye color and hair type but also potentially influencing future health, intellectual development, and personality tendencies. Genetic code can be seen as a blueprint for the baby’s growth and development, setting certain biological parameters and potentials before birth. Similarly, the Transformer architecture used in the foundational model of an LLM plays a decisive role in the model’s functionality and efficiency. How the Transformer architecture is designed determines the model’s ability to process language, learning patterns, problem-solving capabilities, and efficiency in handling long-range dependencies (a method of considering relationships between distant elements). This architecture includes multiple layers of attention mechanisms and feed-forward networks. Its depth (number of layers), width (number of hidden units), and number of attention heads are akin to the “genetic code” of the large language model, presetting the model’s processing capabilities and learning potential. Just as a baby’s genetic code predefines their physical and psychological potential, the Transformer architecture determines the model’s information processing ability and learning depth. The design of the Transformer impacts the model’s performance, applicability, and scalability.

5.2 Common features of BERT and GPT: The attention mechanism

BERT and GPT are two widely applied foundational Transformer models, sharing several common features: (1) Both are pre-trained on large unlabelled datasets, which is similar to how children naturally learn language in a family environment before entering the school system by listening to parents’ conversations, watching TV shows, and daily interactions. This stage lacks a formal curriculum structure, and children accumulate knowledge through vast auditory and visual information (Brown et al., 2020; Devlin et al., 2018; Qiu et al., 2020). (2) Both adopt a self-supervised learning mechanism, which in the field of artificial intelligence means that the model can guide its own learning without the need for externally provided correct answers (Liu et al., 2023). This is similar to how children learn through trial and error during play and daily activities, understanding which behaviors are encouraged and which are not by observing the actions, reactions, and results of adults and peers. (3) Both can apply the knowledge learned to various downstream tasks, such as text classification, sentiment analysis, and content generation, similar to how children apply language and social skills in different life scenarios, such as solving problems at school, communicating with friends, or helping parents at home. (4) The transformer architecture allows them to handle complex data inputs and maintain the contextual relationship of information, which is crucial for understanding and generating language. This is similar to how children build cognitive models through continuous interaction, learning to understand complex instructions or abstract concepts based on context and applying them to real situations.

The original Transformer architecture includes both an encoder and a decoder. Both rely on the attention mechanism. Simply put, the attention mechanism allows the model to focus on important parts of the information while ignoring less relevant content (Bahdanau et al., 2015; Chorowski et al., 2014; Wang et al., 2018). This is similar to how teachers guide students to focus on the most critical information in a lesson. For example, when explaining complex scientific concepts, teachers might use experiments or specific examples to highlight key points, helping students concentrate and avoid being distracted by irrelevant details.

In Transformer models, the mechanism of “queries,” “keys,” and “values” is similar to how students search for and connect information during reading comprehension. When asked specific questions about reading material (queries), students search the text for relevant information (keys) and grasp the meaning of this information (values). For instance, if the question is “Why is the protagonist sad?” students scan the story for emotional descriptions, akin to how the model evaluates the relevance of information and focuses on the most crucial parts.

More specifically, the attention mechanism allows the model to consider the influence of other words in a sentence when processing tasks like language translation or content generation. This is similar to how students need to focus on key numbers and operators in a math problem while considering how these elements interact to find the correct answer. Additionally, the model optimizes its ability to predict and generate text by learning the relationships between words. This is comparable to how students gradually understand the meaning and usage of words by seeing them used in different contexts multiple times. For example, students might notice that “restaurant” often appears with “eat,” helping them associate “restaurant” with relevant contexts in future encounters.

5.3 Differences between BERT and GPT: Encoder and decoder applications

Not all foundational models use the complete encoder and decoder architecture. In the architectures of BERT and GPT, BERT uses only the encoder part, while GPT uses only the decoder part (Brown et al., 2020; Devlin et al., 2018; Zhong et al., 2023). The encoder is responsible for understanding and processing input information, similar to how students absorb knowledge through listening to lessons, reading, or observing. BERT uses the encoder to deeply analyze and understand the semantic and grammatical structure of input text, similar to how teachers help students understand the main ideas and details of an article during reading comprehension. The decoder’s role is more like the output stage in a student’s learning process, where they need to express the acquired knowledge through assignments, tests, or other forms. GPT focuses on using the decoder to generate coherent text, just as students create stories or essays in writing class based on what they have learned.

BERT’s Transformer architecture’s encoder part allows it to understand the semantic and grammatical information of the text. BERT’s output is an embedded representation, not a direct prediction result. To utilize these embedded representations, additional layers, such as those for text classification or question-answering tasks, need to be added on top of BERT. This process is similar to how teachers guide students to understand sentences and paragraphs during reading comprehension by identifying key words, phrases, and grammatical structures to help them grasp the deep meaning and contextual connections of the text. This is akin to how BERT processes and parses text through the encoder. During this process, the model is pre-trained by randomly masking certain words in the input sentences (using [MASK] tokens) and then predicting these masked words. This is similar to cloze tests, where teachers remove certain words from a text and ask students to fill in the blanks based on context clues, practicing and deepening their understanding and application of the language. This method enables BERT to be effectively trained on large amounts of unlabelled data.

On the other hand, GPT, with its billions of parameters, performs exceptionally well in handling language tasks. BERT and GPT have different application scenarios. BERT is more suitable for tasks requiring deep content understanding, such as sentiment analysis, question answering, summarization, and named entity recognition, similar to students’ analytical or comprehension learning in the classroom. GPT excels at tasks like translation, text generation, and story creation, akin to students exercising creativity in art and writing classes (Zhong et al., 2023).

The outputs of these two models also have distinct characteristics. BERT’s output includes attention information embedded representations that can be further used for other tasks, while GPT directly outputs the probability distribution of the next word. Since these models are pre-trained, they can be easily reused and extended in different fields and tasks. The focus of pre-trained models is crucial because training these models requires extensive computational resources and time, which only a few companies can afford. Pre-trained models can be customized and extended for specific fields and tasks. Sometimes, adding a few extra “layers” on top of the model suffices without modifying its internal structure. However, in some cases, internal layer modifications might be necessary, requiring more training and usually increasing costs. This customization technique, known as transfer learning, allows a general model to be easily applied to other fields.

The term “large” in large language models refers to the significant expansion in parameter scale compared to early small models. Early small models were trained with a limited number of parameters and smaller datasets, making supervision and review relatively easy. Although large language models use network architectures and training methods similar to those of smaller pre-trained language models, they achieve impressive performance improvements by expanding the scale of parameters, datasets, and computational resources. These models have demonstrated for the first time the ability of a single model to effectively solve many complex tasks.

Currently, there is no clear standard for the minimum parameter scale, but it is generally believed that models with hundreds of billions, tens of billions, or even trillions of parameters are considered large language models. Some research suggests that models with billions of parameters after pre-training can also be classified as large language models. This expansion raises regulatory issues, as models with hundreds of billions or trillions of parameters make supervision and review almost impossible, which is one of the reasons why AI generates fear among people.

To address this issue, we have outlined three major directions for AIED (AI in Education) regulation and developed a multi-level AIED ethical framework based on the creation steps of AI.

The three major directions for AIED regulation are: (1) National-level education administrators should have a comprehensive understanding of the model frameworks of large AIED applications on the market, ensuring that the model frameworks meet the development needs of our country’s AIED and understanding their potential risks. (2) National-level education administrators should control the data sources, quality, and preprocessing methods of AIED models, especially those AIED applications widely used in schools. Educators (teachers and schools) should also fully understand what data was used to train the AIED they are using. (3) National-level education administrators should control the self-supervised learning mechanisms (deep training) of AIED models, ensuring that AIED applications can exhibit reasonable behavior and decision-making abilities when dealing with complex situations.

6 Steps for constructing an ethical framework for AI in education

Based on the ethical concern for AIED development, we have designed a multi-level ethical framework to guide the application of artificial intelligence in education. This framework includes protecting student privacy and data security, ensuring the transparency and interpretability of AI tools, and respecting human values and moral principles throughout the educational process. Ensuring that AIED applications meet ethical and moral standards helps to enhance the quality and fairness of education, rather than causing negative impacts. Starting from the selection of the AIED Transformer, the following are the five main steps and an overview of each step:

  • Step 1: Supervising the “Baby Genes” of AI Models - Oversight of Large Language Model (LLM) Framework Selection

This step emphasizes the rigorous evaluation required when selecting foundational models for AI, whether it be BERT, GPT, or any other base model, ensuring that the chosen framework meets the needs for future AIED development. Choosing the appropriate foundational model is crucial for determining AI’s future performance and potential risks. This process is akin to selecting a spouse to shape ideal offspring. Just as a spouse provides the genetic material that affects the future child’s health, intelligence, behavior, and personality, the large language model framework acts as the “genes” of artificial intelligence, determining its potential capabilities, performance, and risks.

When humans choose a spouse, they consider genetic traits with the hope of passing on desirable genes to their children. They also consider the spouse’s character, morality, sense of responsibility, and life values—ethical concerns that are equally important. Similarly, when selecting a large language model framework, developers are essentially choosing the focus and risk profile for AIED development. Therefore, in the field of AIED, it is crucial to supervise the “baby genes” of AI from the outset to ensure that early-stage technical development aligns with human ethical principles and does not introduce potential risks. Additionally, incorporating logging technology in AIED development is essential. Logging can provide detailed records of AIED development processes and upload data to cloud servers for subsequent auditing and accountability. This ensures transparency and responsibility throughout the development lifecycle.

  • Step 2: Supervising the “Kindergarten” Phase of AI Learning Basic Rules - Monitoring AI Model Data Sources and Training

This phase emphasizes strict supervision of the data sources, quality, and processing methods used for training AIED models. Ensuring the quality and suitability of training data is crucial, as the data selected directly impacts the behavior and performance of AIED. Just as careful selection and oversight of the “baby genes” of AI are critical, so is the data used during training. The data influences not only the AI model’s behavior and performance but also its ethical conduct and real-world application effectiveness. Therefore, rigorous supervision of the sources and quality of AI training data is indispensable.

Microsoft’s study, “Textbook is All You Need,” highlighted the importance of data quality over quantity. Microsoft trained a model named “phi-1” with only 1.3 billion parameters using “textbook-quality” datasets and generated textbook-like exercises using GPT-3.5. They then assessed “phi-1’s” coding ability through the HumanEval test, which showed that “phi-1” outperformed other open-source models that were ten times larger and used a hundred times more data. This demonstrates that data quality can be more important than data quantity or model parameters. Microsoft suggested that just as a comprehensive, well-crafted textbook provides students with the necessary knowledge to master a new subject, high-quality data significantly enhances the proficiency of language models in code generation tasks (Gunasekar et al., 2023).

Datasets, much like student textbooks, provide examples and experiences that teach AI models how to understand and handle various tasks. A prompt extracted from a source dataset serves as a question or command for the AI, guiding its responses or actions. Annotators play a role similar to teachers in the AI’s creation process by providing manual annotations or using semi-automated tools to offer guidance, clearly indicating the desired responses or behaviors. These annotated data points are then used to train the model through supervised learning, helping it learn the correct answers provided by the annotators and fine-tuning its parameters to better adapt to specific tasks or data.

Selecting a high-quality kindergarten teacher for children mirrors the need for patience, wisdom, and strategy in educating AIED models during their formative training phase to help them grow and reach their full potential. During AI training, demonstrating the correct behavior (data) and teaching the model to respond appropriately in various situations through supervised learning (annotators’ expected outputs) is similar to kindergarten children learning basic skills through imitation. It is essential to ensure that demonstrated behaviors are positive and ethically sound and that annotators (“kindergarten teachers”) are certified. Additionally, incorporating logging technology is crucial to prevent AI models from learning inappropriate behaviors.

  • Step 3: Supervising the “Elementary School” Phase of AI - Oversight of AI Training and Reward Mechanisms

After ensuring that the training data for AI models meets ethical and quality standards, the next step involves gradually guiding the models from simple to complex tasks to optimize their responses and behaviors. This phase focuses on providing more in-depth education and training to AIED models to ensure they exhibit appropriate behavior and decision-making skills in complex situations. The optimization training of AIED models involves step-by-step guidance from annotators (“elementary school teachers”), progressing from simple to complex tasks. Annotators evaluate the output of AIED models, ranking the quality of responses from best to worst (similar to grading student assignments on a scale from 0 to 100).

These ranked data are then used to train a reward model, a tool for assessing AI performance. The reward model scores each response or behavior of the AI, with the scores directly reflecting the quality of its performance. Through this gradual training process, AI models can learn to handle complex tasks. Just as elementary school students improve through teacher feedback on assignments, AI uses the reward model to score its performance, identifying which behaviors are good and which need improvement to drive self-optimization. This process closely mirrors how students enhance their learning through assignment evaluation and reflection.

This phase is crucial for AIED models to learn and adapt to social environments, ensuring that the guidance they receive is beneficial and ethically sound. It is also essential to ensure that evaluation standards are fair and do not reinforce any potential biases. Annotators (“elementary school teachers”) must be certified, and all operations should be logged and uploaded to the cloud for transparency and accountability.

  • Step 4: Supervising the “High School” Phase of AI - Guidance and Monitoring of AI Self-Adjustment and Optimization

As AIED models mature, they will face more complex challenges. This step involves regulating the behavior of mature AIED models to ensure they make reasonable decisions in various situations. After successfully training the reward model, annotators can distinguish the quality of AI outputs and effectively reward behaviors that meet expectations. However, training the reward model alone is insufficient for continuous optimization of AI behavior. The next step is to apply these rewards in a dynamic learning process to further adjust and optimize the AI’s behavior strategy.

This brings us to Step 4: using Proximal Policy Optimization (PPO) reinforcement learning algorithms to optimize AI behavior based on feedback from the reward model. The PPO algorithm helps adjust and refine AI behavior strategies so that future actions achieve higher rewards (Schulman et al., 2017). This process is akin to teachers adjusting their teaching methods and content based on students’ learning performance to ensure that teaching strategies effectively promote student progress. The optimization process is continuous and iterative, with each strategy update building on previous learning experiences, gradually approaching an optimal strategy.

In education, this mirrors the ongoing process of teaching improvement, where teachers continuously optimize their lesson plans based on student feedback and learning outcomes. The PPO algorithm enables AI models to optimize their behavior through trial and error and continuous learning in complex environments, ultimately achieving more efficient decision-making. Reinforcement learning algorithms allow AI to adjust strategies based on reward model feedback, optimizing behavior similar to how high school students tackle advanced problems in mathematics, learning to use different methods and techniques through practice and feedback.

When using reinforcement learning, it is crucial to ensure that the reward system does not reinforce unethical behavior. Therefore, AI decisions must align with social ethical standards. Licensed annotators (“high school teachers”) should guide, supervise, and log all operations, uploading the information to the cloud for transparency and accountability.

  • Step 5: Supervising adult AI Self-Adjustment and Final Outcome Review-ensuring AI serves Humanity’s Best Interests

Upon reaching full maturity, AIED models should possess the ability for self-adjustment and self-supervision to adapt to ever-changing external environments and new challenges. At this stage, AIED models should independently handle complex problems and self-correct when necessary. When reviewing AIED models, it is crucial to ensure that the development and application of these AI products adhere to technological ethics and the core principle of utilitarian ethics—promoting the greatest happiness and benefit for humanity, rather than prioritizing the interests of the machines.

AIED tools should be designed to enhance educational quality, ensure educational equity, and provide personalized learning opportunities for all students. For example, an AI-assisted personalized learning system should adapt to most students’ learning paces and styles, helping them effectively grasp knowledge while also paying attention to disadvantaged groups, ensuring that technology does not exacerbate existing educational inequalities. Additionally, these systems must strictly protect student data privacy and prevent data misuse during their design.

Comprehensive reviews ensure that AI educational tools genuinely serve to improve the overall educational level and quality of society. When AIED models are deployed, they should record feedback data from humans and society, including user feedback and societal feedback. User feedback includes data such as code, score values, input-output pairs, and document contents. Societal feedback comprises code, annotated data, documents, media comments, and public opinion analysis. If an AIED application continuously updates and learns from user and societal feedback, it becomes extremely powerful and potentially unpredictable.

Returning to a previously mentioned point, using user input for continued model training can improve AIED learning efficiency and output quality but may involve risks such as privacy breaches, biased information, and the generation of false information. On the other hand, not using user input data for AI model training can avoid potential risks but may limit the learning effectiveness and application scope, as the model cannot learn and iterate from real-time user feedback. Therefore, educators must consider their needs and actual circumstances when choosing an appropriate AIED application. Figure 1 illustrates the construction steps of an AIED ethical model based on GPT LLM.

Fig. 1
figure 1

AIED model construction steps and ethical norms detailed flowchart

Full size image

7 Discussion and conclusion

Large Language Models (LLMs) and annotators (humans) play crucial roles in the creation of AI in Education (AIED). The construction of LLMs determines the potential capabilities and risks of AIED at its inception, while the ethics and morals of subsequent annotators determine whether the AIED model will benefit or harm human society. If annotators display cultural, gender, or racial biases when organizing and labeling educational content, these biases will directly affect the content of the AIED teaching platform. For example, if an AI for history education is based on biased annotation data, it may present skewed descriptions of certain historical events or figures, directly impacting students’ learning and worldview.

In automatic grading systems, inconsistent standards or personal preferences of annotators can affect the consistency and fairness of the model’s grading, thereby influencing students’ motivation and educational opportunities. When annotators handle data involving students with different learning abilities and needs, they must possess a high degree of sensitivity and expertise to ensure that AI education tools can fairly serve all students, including those with special needs. Additionally, protecting student privacy is paramount in processing educational data. Annotators must strictly adhere to privacy protection laws and ethical standards to prevent sensitive information from being disclosed or misused.

Due to the complexity of AI technology, principals, teachers, students, and parents find it challenging to assess whether an educational AI application meets ethical standards. This necessitates joint regulation of AIED ethics by national and industry levels. First, an AI Education Ethics Committee should be established at the industry level, consisting of industry practitioners and educators, to develop a set of AIED ethical standards and annually nominate companies that meet these standards. Second, similar to AI accountants, third-party companies should conduct ethical audits of nominated AIED companies, issuing certifications to those that pass. Third, the state should regulate third-party AI audit companies and hold them accountable for problematic audits. By coordinating the “invisible hand” and the “visible hand” of economics (Smith, 2008; Keynes, 2017), a healthy development environment for the AIED industry can be created. Furthermore, it is crucial to train auditors and annotators who possess knowledge of “artificial intelligence moral and ethical norms” and meet “artificial intelligence moral and ethical certification” standards, to maximize the avoidance of ethical risks in AIED.

Today, artificial intelligence is categorized in various ways: AlphaGo is considered weak AI, ChatGPT is seen as strong AI, and the sentient machines in science fiction movies are regarded as superintelligent AI or Artificial General Intelligence (AGI). AI thinker Nick Bostrom defines AGI as an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom, and social skills (Bostrom, 2014).

AGI is predicted to possess intelligence surpassing that of humans. Such AGI would not only perform specific tasks but also have the capability for self-learning and innovation, understanding and generating natural language, and solving complex problems. Furthermore, it might have consciousness and emotions. AGI could excel in various fields, including scientific research, artistic creation, and decision-making, profoundly impacting society, the economy, and ethics, potentially leading to numerous ethical issues. These include (1) Autonomy and Control: Once AGI surpasses human intelligence, will it still be under human control? How can we ensure AGI always prioritizes human interests? (2) Values and Moral Standards: AGI might develop its own values and moral systems, which could conflict with human values. How can we reconcile these differences? (3) Employment and Social Impact: AGI could replace many human jobs, exacerbating employment issues and social inequality. How do we address the employment disruptions and social changes brought about by AI? (4) Privacy and Security: AGI’s ability to handle vast amounts of data and its powerful capabilities could be misused for privacy invasions and opinion manipulation. How can we prevent the misuse and abuse of AI?

In the future, there may be a game between AI (machines) and humans. This brings us back to the ethical discussion on AIED. Using utilitarian ethics on top of technological ethics emphasizes outcomes centered on human well-being. During AI system development, the focus should be on maximizing human happiness and benefit, integrating ethical considerations into every aspect of AI systems, rather than prioritizing the interests of machines or those who control them. The development of AI technology is not science fiction. This double-edged sword of AI presents both opportunities and challenges for humanity’s future. By understanding and managing AI’s “black box” and annotations, we can trace AI’s actions, ensuring system transparency and accountability. Mastering the fundamental principles and logic of AI technology is crucial for guiding the future development of AIED, which is the core intent of this research and our sincere hope for human prosperity.