Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT

Andreas Scholl, Daniel Schiffner, Natalie Kiesler·May 29, 2024

Summary

This research paper investigates novice computing students' use of ChatGPT-3.5 in an introductory programming course at a German university. The study, based on 213 students' chat logs with 2,335 prompts, reveals diverse interactions, from supportive to concerning, in students' learning process. It examines how students engage with the AI for problem-solving, understanding, and feedback, revealing three main patterns: those seeking quick solutions, seeking clarification, and using the tool for tutoring-like support. The findings highlight the potential benefits and challenges for educators, emphasizing the need for guidelines on responsible use, addressing limitations, and adapting to the evolving role of AI in education. The study contributes to the ongoing discussion on the impact of AI tools on programming education, assessment, and the balance between human guidance and AI-generated support.

Key findings

2

Paper digest

What problem does the paper attempt to solve? Is this a new problem?

The paper aims to investigate students' use of ChatGPT as a tool to support problem-solving in the early stages of programming education at the university level . This study addresses the interaction patterns of students with ChatGPT, revealing different groups of students seeking various types of assistance, such as immediate solutions, help with problem understanding, debugging, syntax errors, and more . While the use of Large Language Models (LLMs) like ChatGPT in educational settings is a relatively new development, the specific focus on students' authentic interactions with LLMs in a curricular course setting is a novel aspect of this research .


What scientific hypothesis does this paper seek to validate?

This paper aims to validate the hypothesis related to students' use of ChatGPT as a tool to support problem-solving in the early stages of programming education at the university level. The study explores students' interactions with ChatGPT by analyzing their chat protocols quantitatively and qualitatively to understand how students utilize the tool for problem-solving . The research investigates various aspects such as students seeking immediate solutions, help with problem understanding, debugging, syntax errors, and more, to characterize the diverse patterns of interaction with ChatGPT . The study provides insights into the implications for teaching and learning, highlighting the need for educators to offer necessary tools, guidance on successful tool utilization, and communication of the tool's limitations to students to prevent potential frustrations .


What new ideas, methods, or models does the paper propose? What are the characteristics and advantages compared to previous methods?

The paper "Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT" proposes several innovative ideas, methods, and models related to the use of Large Language Models (LLMs) in programming education .

  1. Use of LLMs in Programming Education: The paper explores the potential of LLMs, such as ChatGPT, to support novice learners in programming tasks and courses . It discusses how LLMs can enhance programming error messages, generate formative feedback, and assist students in solving programming tasks .

  2. Student Interaction with LLMs: The study investigates how students interact with LLMs like ChatGPT in the context of introductory programming course assignments . By analyzing chat protocols generated by students while using ChatGPT-3.5 for programming tasks, the paper provides insights into authentic student interactions with LLMs in a curricular setting .

  3. Educator Considerations: The insights from students' interactions with LLMs have implications for educators considering the integration of LLMs in introductory programming courses . These insights can inform the development of new instructional methods aimed at promoting a conscious and critical use of LLMs in programming education .

  4. Research Focus: The paper highlights the importance of ongoing research to keep pace with the rapid advancements in Generative AI tools like ChatGPT . It emphasizes the need for studies focusing on students' perspectives, trust, and actual use of LLMs in educational settings to better support and guide them in utilizing LLMs effectively .

In summary, the paper introduces innovative approaches to leveraging LLMs in programming education, explores student interactions with LLMs like ChatGPT, and underscores the significance of considering ethical and pedagogical aspects when incorporating LLMs in educational settings. The paper "Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT" highlights several characteristics and advantages of using Large Language Models (LLMs) like ChatGPT compared to previous methods in programming education .

  1. Enhanced Code Explanations: LLMs have shown effectiveness in generating code explanations, which can be beneficial for novice learners in understanding programming concepts and tasks . For instance, students have rated code explanations generated by LLMs more favorably compared to traditional explanations .

  2. Improved Error Messages: LLMs can enhance programming error messages, providing students with clearer feedback on their mistakes and guiding them towards error resolution . This feature can help students in debugging their code and improving their programming skills.

  3. Formative Feedback Generation: LLMs have the capability to generate formative feedback for learners, offering personalized support and guidance in programming exercises . This personalized feedback can cater to the diverse needs of students, enhancing their learning experience.

  4. Diverse Interaction Patterns: The study identified various interaction patterns of students with LLMs like ChatGPT, including seeking immediate solutions, help with problem understanding, debugging, syntax errors, and more . This diversity reflects the different support needs of students and the potential for personalized assistance through LLMs.

  5. Potential for Personalization: LLMs, such as ChatGPT, have the potential to offer personalized support to students based on their individual learning requirements . This personalized assistance can cater to the diverse needs of learners, enhancing their problem-solving skills and understanding of programming concepts.

  6. Educator Guidance: While LLMs offer advantages in programming education, educators play a crucial role in providing students with the necessary tools, advice, and guidance on how to effectively utilize LLMs in their learning process . Educators need to communicate the limitations and potential pitfalls of LLMs to students to ensure their successful and ethical use in educational settings.

In summary, the characteristics and advantages of using LLMs like ChatGPT in programming education include enhanced code explanations, improved error messages, formative feedback generation, diverse interaction patterns, potential for personalization, and the importance of educator guidance in facilitating effective LLM utilization by students.


Do any related researches exist? Who are the noteworthy researchers on this topic in this field?What is the key to the solution mentioned in the paper?

Several related research studies exist in the field of using ChatGPT for programming education. Noteworthy researchers in this area include Prather, Denny, Leinonen, Becker, Albluwi, Craig, Keuning, Kiesler, Kohn, Luxton-Reilly, MacNeil, Petersen, Pettit, Reeves, Savelka, and many others . These researchers have contributed to exploring the use of Generative AI tools like ChatGPT in educational settings, particularly in programming education.

The key to the solution mentioned in the paper involves analyzing students' interactions with ChatGPT during programming tasks. The study focused on students' use of ChatGPT as a tool to support problem-solving in the early stages of programming education at the university level. Students' chat protocols were analyzed quantitatively and qualitatively to understand their interaction patterns, needs for support, and feedback preferences . The study revealed different groups of students interacting with the tool in various ways, such as seeking immediate solutions, help with problem understanding, debugging, syntax errors, and more. The diversity in students' use of ChatGPT highlighted the heterogeneity of students' needs for support and feedback in programming education .


How were the experiments in the paper designed?

The experiments in the paper were designed as follows:

  • The study involved an exploratory analysis of students' use of ChatGPT as a problem-solving tool in the early stages of programming education at the university level .
  • A total of 360 students completed the exercise sheet, with 305 students engaging with ChatGPT-3.5 as part of the exercise. The final dataset for analysis comprised 213 students, providing 2335 prompts for detailed examination .
  • Students were instructed to use ChatGPT without specific guidance on how to do so, allowing for natural interactions with the tool .
  • The exercise sheet focused on concepts like recursion, functions, lists, conditionals, string manipulation, and documentation, with tasks involving code interpretation, function implementation, and problem-solving approaches .
  • The study aimed to analyze students' interactions with ChatGPT-3.5 quantitatively and qualitatively, examining aspects such as the number of prompts per student, the number of words per prompt, follow-up interactions, solution requests, and types of interactions .
  • The analysis of students' chat protocols aimed to identify patterns in how students sought assistance from ChatGPT, the issues addressed in their prompts, and the overall strategies employed during problem-solving interactions .

What is the dataset used for quantitative evaluation? Is the code open source?

The dataset used for quantitative evaluation in the context of analyzing chat protocols of novice programmers solving introductory programming tasks with ChatGPT is not explicitly mentioned in the provided excerpts . Regarding the open-source status of the code, the information about whether the code is open source is not directly addressed in the context provided. Therefore, it is unclear from the given content whether the code used for the evaluation is open source or not.


Do the experiments and results in the paper provide good support for the scientific hypotheses that need to be verified? Please analyze.

The experiments and results presented in the paper provide substantial support for the scientific hypotheses that needed verification. The study conducted an exploratory analysis of students' interactions with ChatGPT in the context of programming education at the university level . The research involved a significant number of participants (n=213), allowing for robust conclusions to be drawn regarding students' use and interaction patterns with the tool .

The findings of the study revealed distinct groups of students with varying approaches to utilizing ChatGPT for problem-solving, such as seeking immediate solutions, help with problem understanding, debugging, syntax errors, and more . This detailed characterization of students' interactions with the tool provides valuable insights into the diverse needs and preferences of students when using AI tools like ChatGPT in programming education .

Moreover, the study identified new interaction patterns among students, such as starting with Task Description Prompts and progressing to more specific requests, highlighting the nuanced ways in which students engage with ChatGPT . These novel insights contribute to a deeper understanding of how students leverage AI tools for support in programming tasks, thereby reinforcing the scientific hypotheses under investigation .

Overall, the comprehensive analysis of students' chat protocols, the identification of distinct user groups, and the exploration of new interaction patterns collectively strengthen the scientific hypotheses being tested in the study. The results offer valuable empirical evidence to support the effectiveness and implications of integrating AI tools like ChatGPT in programming education at the university level .


What are the contributions of this paper?

The paper provides valuable insights into students' interactions with ChatGPT in the context of programming education at the university level. It offers the following key contributions:

  • An exploratory study on students' use of ChatGPT as a problem-solving tool in early programming education .
  • Quantitative and qualitative analysis of students' chat protocols to characterize their usage patterns and interactions with the tool .
  • Identification of different groups of students with varying approaches to using ChatGPT, such as seeking immediate solutions, help with problem understanding, debugging, syntax errors, and more .
  • Revelation of new interaction patterns among students, including the use of Task Description Prompts and specifying requests over chat .
  • Highlighting the importance of providing personalized support and feedback to students using Generative AI tools like ChatGPT .
  • Emphasizing the need for educators to guide students on successful tool utilization, communicate limitations, and discuss ethical considerations related to AI tools in education .

What work can be continued in depth?

To further advance the research in the field of using Large Language Models (LLMs) like ChatGPT in programming education, several areas of work can be continued in depth based on the provided context:

  1. Specialized AI Tools for Coding: Exploring how a specialized AI tool specifically designed for coding tasks can enhance support for learners in programming education .

  2. Replication Studies: Conducting replication studies at different institutions or countries to validate and strengthen the results obtained from using LLMs like ChatGPT in educational settings .

  3. Understanding Student Interactions: Delving deeper into how students interact with LLMs like ChatGPT during their coursework, particularly in the context of solving programming tasks. This includes analyzing the nature of student-chatbot interactions, the types of prompts used, and the strategies students employ when seeking assistance .

  4. Impact on Learning Outcomes: Investigating the impact of LLMs on learning outcomes, such as the effectiveness of code explanations generated by AI tools, the quality of programming error messages, and the provision of feedback to students .

  5. Ethical Considerations: Addressing ethical considerations related to the use of LLMs in education, such as ensuring the accuracy and reliability of information provided by AI tools, mitigating biases, and maintaining academic integrity .

  6. Pedagogical Approaches: Exploring new forms of instructional methods that leverage LLMs to foster critical thinking and conscious use of AI tools in programming courses .

By focusing on these areas, researchers can contribute to a deeper understanding of the implications, challenges, and benefits of integrating LLMs into programming education, ultimately enhancing the learning experience for novice programmers.

Tables

1

Introduction
Background
Emergence of AI in education
ChatGPT-3.5 as a learning aid
Objective
To explore student interactions with ChatGPT-3.5
To identify learning patterns and implications for education
Method
Data Collection
Sample description (German university, introductory programming course)
Data source: Students' chat logs and prompts
Data collection period
Data Preprocessing
Log analysis techniques
Identifying relevant interactions
Categorizing prompts and responses
Main Findings
Student Engagement Patterns
Quick Solution Seekers
Frequency and types of questions
Impact on problem-solving independence
Clarification Seekers
Seeking understanding through AI assistance
Role in enhancing comprehension
Tutoring-like Support
AI as a supplementary teaching tool
Limitations and benefits of this approach
Implications for Education
Guidelines for Responsible Use
Ethical considerations
Promoting critical thinking
Setting boundaries for AI assistance
Addressing Limitations
AI-generated feedback accuracy
Overreliance on AI vs. human guidance
Mitigating potential biases
Adapting to the Evolving Role
Integrating AI in assessment and learning
Balancing human- vs. AI-driven instruction
Future directions for AI in programming education
Conclusion
Summary of key findings
The role of educators in navigating AI integration
Research implications and future studies
References
Cited works on AI in education and programming education
Studies on the impact of ChatGPT in learning environments
Basic info
papers
artificial intelligence
Advanced features
Insights
How many students and chat prompts were involved in the investigation?
What three main patterns did the study identify in students' interactions with ChatGPT-3.5?
In which course setting was ChatGPT-3.5 used in the study?
What is the primary focus of the research paper?

Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT

Andreas Scholl, Daniel Schiffner, Natalie Kiesler·May 29, 2024

Summary

This research paper investigates novice computing students' use of ChatGPT-3.5 in an introductory programming course at a German university. The study, based on 213 students' chat logs with 2,335 prompts, reveals diverse interactions, from supportive to concerning, in students' learning process. It examines how students engage with the AI for problem-solving, understanding, and feedback, revealing three main patterns: those seeking quick solutions, seeking clarification, and using the tool for tutoring-like support. The findings highlight the potential benefits and challenges for educators, emphasizing the need for guidelines on responsible use, addressing limitations, and adapting to the evolving role of AI in education. The study contributes to the ongoing discussion on the impact of AI tools on programming education, assessment, and the balance between human guidance and AI-generated support.
Mind map
Limitations and benefits of this approach
AI as a supplementary teaching tool
Role in enhancing comprehension
Seeking understanding through AI assistance
Impact on problem-solving independence
Frequency and types of questions
Future directions for AI in programming education
Balancing human- vs. AI-driven instruction
Integrating AI in assessment and learning
Mitigating potential biases
Overreliance on AI vs. human guidance
AI-generated feedback accuracy
Setting boundaries for AI assistance
Promoting critical thinking
Ethical considerations
Tutoring-like Support
Clarification Seekers
Quick Solution Seekers
Categorizing prompts and responses
Identifying relevant interactions
Log analysis techniques
Data collection period
Data source: Students' chat logs and prompts
Sample description (German university, introductory programming course)
To identify learning patterns and implications for education
To explore student interactions with ChatGPT-3.5
ChatGPT-3.5 as a learning aid
Emergence of AI in education
Studies on the impact of ChatGPT in learning environments
Cited works on AI in education and programming education
Research implications and future studies
The role of educators in navigating AI integration
Summary of key findings
Adapting to the Evolving Role
Addressing Limitations
Guidelines for Responsible Use
Student Engagement Patterns
Data Preprocessing
Data Collection
Objective
Background
References
Conclusion
Implications for Education
Main Findings
Method
Introduction
Outline
Introduction
Background
Emergence of AI in education
ChatGPT-3.5 as a learning aid
Objective
To explore student interactions with ChatGPT-3.5
To identify learning patterns and implications for education
Method
Data Collection
Sample description (German university, introductory programming course)
Data source: Students' chat logs and prompts
Data collection period
Data Preprocessing
Log analysis techniques
Identifying relevant interactions
Categorizing prompts and responses
Main Findings
Student Engagement Patterns
Quick Solution Seekers
Frequency and types of questions
Impact on problem-solving independence
Clarification Seekers
Seeking understanding through AI assistance
Role in enhancing comprehension
Tutoring-like Support
AI as a supplementary teaching tool
Limitations and benefits of this approach
Implications for Education
Guidelines for Responsible Use
Ethical considerations
Promoting critical thinking
Setting boundaries for AI assistance
Addressing Limitations
AI-generated feedback accuracy
Overreliance on AI vs. human guidance
Mitigating potential biases
Adapting to the Evolving Role
Integrating AI in assessment and learning
Balancing human- vs. AI-driven instruction
Future directions for AI in programming education
Conclusion
Summary of key findings
The role of educators in navigating AI integration
Research implications and future studies
References
Cited works on AI in education and programming education
Studies on the impact of ChatGPT in learning environments
Key findings
2

Paper digest

What problem does the paper attempt to solve? Is this a new problem?

The paper aims to investigate students' use of ChatGPT as a tool to support problem-solving in the early stages of programming education at the university level . This study addresses the interaction patterns of students with ChatGPT, revealing different groups of students seeking various types of assistance, such as immediate solutions, help with problem understanding, debugging, syntax errors, and more . While the use of Large Language Models (LLMs) like ChatGPT in educational settings is a relatively new development, the specific focus on students' authentic interactions with LLMs in a curricular course setting is a novel aspect of this research .


What scientific hypothesis does this paper seek to validate?

This paper aims to validate the hypothesis related to students' use of ChatGPT as a tool to support problem-solving in the early stages of programming education at the university level. The study explores students' interactions with ChatGPT by analyzing their chat protocols quantitatively and qualitatively to understand how students utilize the tool for problem-solving . The research investigates various aspects such as students seeking immediate solutions, help with problem understanding, debugging, syntax errors, and more, to characterize the diverse patterns of interaction with ChatGPT . The study provides insights into the implications for teaching and learning, highlighting the need for educators to offer necessary tools, guidance on successful tool utilization, and communication of the tool's limitations to students to prevent potential frustrations .


What new ideas, methods, or models does the paper propose? What are the characteristics and advantages compared to previous methods?

The paper "Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT" proposes several innovative ideas, methods, and models related to the use of Large Language Models (LLMs) in programming education .

  1. Use of LLMs in Programming Education: The paper explores the potential of LLMs, such as ChatGPT, to support novice learners in programming tasks and courses . It discusses how LLMs can enhance programming error messages, generate formative feedback, and assist students in solving programming tasks .

  2. Student Interaction with LLMs: The study investigates how students interact with LLMs like ChatGPT in the context of introductory programming course assignments . By analyzing chat protocols generated by students while using ChatGPT-3.5 for programming tasks, the paper provides insights into authentic student interactions with LLMs in a curricular setting .

  3. Educator Considerations: The insights from students' interactions with LLMs have implications for educators considering the integration of LLMs in introductory programming courses . These insights can inform the development of new instructional methods aimed at promoting a conscious and critical use of LLMs in programming education .

  4. Research Focus: The paper highlights the importance of ongoing research to keep pace with the rapid advancements in Generative AI tools like ChatGPT . It emphasizes the need for studies focusing on students' perspectives, trust, and actual use of LLMs in educational settings to better support and guide them in utilizing LLMs effectively .

In summary, the paper introduces innovative approaches to leveraging LLMs in programming education, explores student interactions with LLMs like ChatGPT, and underscores the significance of considering ethical and pedagogical aspects when incorporating LLMs in educational settings. The paper "Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT" highlights several characteristics and advantages of using Large Language Models (LLMs) like ChatGPT compared to previous methods in programming education .

  1. Enhanced Code Explanations: LLMs have shown effectiveness in generating code explanations, which can be beneficial for novice learners in understanding programming concepts and tasks . For instance, students have rated code explanations generated by LLMs more favorably compared to traditional explanations .

  2. Improved Error Messages: LLMs can enhance programming error messages, providing students with clearer feedback on their mistakes and guiding them towards error resolution . This feature can help students in debugging their code and improving their programming skills.

  3. Formative Feedback Generation: LLMs have the capability to generate formative feedback for learners, offering personalized support and guidance in programming exercises . This personalized feedback can cater to the diverse needs of students, enhancing their learning experience.

  4. Diverse Interaction Patterns: The study identified various interaction patterns of students with LLMs like ChatGPT, including seeking immediate solutions, help with problem understanding, debugging, syntax errors, and more . This diversity reflects the different support needs of students and the potential for personalized assistance through LLMs.

  5. Potential for Personalization: LLMs, such as ChatGPT, have the potential to offer personalized support to students based on their individual learning requirements . This personalized assistance can cater to the diverse needs of learners, enhancing their problem-solving skills and understanding of programming concepts.

  6. Educator Guidance: While LLMs offer advantages in programming education, educators play a crucial role in providing students with the necessary tools, advice, and guidance on how to effectively utilize LLMs in their learning process . Educators need to communicate the limitations and potential pitfalls of LLMs to students to ensure their successful and ethical use in educational settings.

In summary, the characteristics and advantages of using LLMs like ChatGPT in programming education include enhanced code explanations, improved error messages, formative feedback generation, diverse interaction patterns, potential for personalization, and the importance of educator guidance in facilitating effective LLM utilization by students.


Do any related researches exist? Who are the noteworthy researchers on this topic in this field?What is the key to the solution mentioned in the paper?

Several related research studies exist in the field of using ChatGPT for programming education. Noteworthy researchers in this area include Prather, Denny, Leinonen, Becker, Albluwi, Craig, Keuning, Kiesler, Kohn, Luxton-Reilly, MacNeil, Petersen, Pettit, Reeves, Savelka, and many others . These researchers have contributed to exploring the use of Generative AI tools like ChatGPT in educational settings, particularly in programming education.

The key to the solution mentioned in the paper involves analyzing students' interactions with ChatGPT during programming tasks. The study focused on students' use of ChatGPT as a tool to support problem-solving in the early stages of programming education at the university level. Students' chat protocols were analyzed quantitatively and qualitatively to understand their interaction patterns, needs for support, and feedback preferences . The study revealed different groups of students interacting with the tool in various ways, such as seeking immediate solutions, help with problem understanding, debugging, syntax errors, and more. The diversity in students' use of ChatGPT highlighted the heterogeneity of students' needs for support and feedback in programming education .


How were the experiments in the paper designed?

The experiments in the paper were designed as follows:

  • The study involved an exploratory analysis of students' use of ChatGPT as a problem-solving tool in the early stages of programming education at the university level .
  • A total of 360 students completed the exercise sheet, with 305 students engaging with ChatGPT-3.5 as part of the exercise. The final dataset for analysis comprised 213 students, providing 2335 prompts for detailed examination .
  • Students were instructed to use ChatGPT without specific guidance on how to do so, allowing for natural interactions with the tool .
  • The exercise sheet focused on concepts like recursion, functions, lists, conditionals, string manipulation, and documentation, with tasks involving code interpretation, function implementation, and problem-solving approaches .
  • The study aimed to analyze students' interactions with ChatGPT-3.5 quantitatively and qualitatively, examining aspects such as the number of prompts per student, the number of words per prompt, follow-up interactions, solution requests, and types of interactions .
  • The analysis of students' chat protocols aimed to identify patterns in how students sought assistance from ChatGPT, the issues addressed in their prompts, and the overall strategies employed during problem-solving interactions .

What is the dataset used for quantitative evaluation? Is the code open source?

The dataset used for quantitative evaluation in the context of analyzing chat protocols of novice programmers solving introductory programming tasks with ChatGPT is not explicitly mentioned in the provided excerpts . Regarding the open-source status of the code, the information about whether the code is open source is not directly addressed in the context provided. Therefore, it is unclear from the given content whether the code used for the evaluation is open source or not.


Do the experiments and results in the paper provide good support for the scientific hypotheses that need to be verified? Please analyze.

The experiments and results presented in the paper provide substantial support for the scientific hypotheses that needed verification. The study conducted an exploratory analysis of students' interactions with ChatGPT in the context of programming education at the university level . The research involved a significant number of participants (n=213), allowing for robust conclusions to be drawn regarding students' use and interaction patterns with the tool .

The findings of the study revealed distinct groups of students with varying approaches to utilizing ChatGPT for problem-solving, such as seeking immediate solutions, help with problem understanding, debugging, syntax errors, and more . This detailed characterization of students' interactions with the tool provides valuable insights into the diverse needs and preferences of students when using AI tools like ChatGPT in programming education .

Moreover, the study identified new interaction patterns among students, such as starting with Task Description Prompts and progressing to more specific requests, highlighting the nuanced ways in which students engage with ChatGPT . These novel insights contribute to a deeper understanding of how students leverage AI tools for support in programming tasks, thereby reinforcing the scientific hypotheses under investigation .

Overall, the comprehensive analysis of students' chat protocols, the identification of distinct user groups, and the exploration of new interaction patterns collectively strengthen the scientific hypotheses being tested in the study. The results offer valuable empirical evidence to support the effectiveness and implications of integrating AI tools like ChatGPT in programming education at the university level .


What are the contributions of this paper?

The paper provides valuable insights into students' interactions with ChatGPT in the context of programming education at the university level. It offers the following key contributions:

  • An exploratory study on students' use of ChatGPT as a problem-solving tool in early programming education .
  • Quantitative and qualitative analysis of students' chat protocols to characterize their usage patterns and interactions with the tool .
  • Identification of different groups of students with varying approaches to using ChatGPT, such as seeking immediate solutions, help with problem understanding, debugging, syntax errors, and more .
  • Revelation of new interaction patterns among students, including the use of Task Description Prompts and specifying requests over chat .
  • Highlighting the importance of providing personalized support and feedback to students using Generative AI tools like ChatGPT .
  • Emphasizing the need for educators to guide students on successful tool utilization, communicate limitations, and discuss ethical considerations related to AI tools in education .

What work can be continued in depth?

To further advance the research in the field of using Large Language Models (LLMs) like ChatGPT in programming education, several areas of work can be continued in depth based on the provided context:

  1. Specialized AI Tools for Coding: Exploring how a specialized AI tool specifically designed for coding tasks can enhance support for learners in programming education .

  2. Replication Studies: Conducting replication studies at different institutions or countries to validate and strengthen the results obtained from using LLMs like ChatGPT in educational settings .

  3. Understanding Student Interactions: Delving deeper into how students interact with LLMs like ChatGPT during their coursework, particularly in the context of solving programming tasks. This includes analyzing the nature of student-chatbot interactions, the types of prompts used, and the strategies students employ when seeking assistance .

  4. Impact on Learning Outcomes: Investigating the impact of LLMs on learning outcomes, such as the effectiveness of code explanations generated by AI tools, the quality of programming error messages, and the provision of feedback to students .

  5. Ethical Considerations: Addressing ethical considerations related to the use of LLMs in education, such as ensuring the accuracy and reliability of information provided by AI tools, mitigating biases, and maintaining academic integrity .

  6. Pedagogical Approaches: Exploring new forms of instructional methods that leverage LLMs to foster critical thinking and conscious use of AI tools in programming courses .

By focusing on these areas, researchers can contribute to a deeper understanding of the implications, challenges, and benefits of integrating LLMs into programming education, ultimately enhancing the learning experience for novice programmers.

Tables
1
Scan the QR code to ask more questions about the paper
© 2025 Powerdrill. All rights reserved.