LLM 기반 회로 분석 과제 채점 향상 파이프라인 GPT4o의 다단계 프롬프트와 데이터 증강 적용

📝 Abstract

This research full paper presents an enhancement pipeline for large language models (LLMs) in assessing homework for an undergraduate circuit analysis course, aiming to improve LLMs’ capacity to provide personalized support to electrical engineering students. Existing evaluations have demonstrated that GPT-4o possesses promising capabilities in assessing student homework in this domain. Building on these findings, we enhance GPT-4o’s performance through multi-step prompting, contextual data augmentation, and the incorporation of targeted hints. These strategies effectively address common errors observed in GPT-4o’s responses when using simple prompts, leading to a substantial improvement in assessment accuracy. Specifically, the correct response rate for GPT-4o increases from 74.71% to 97.70% after applying the enhanced prompting and augmented data on entry-level circuit analysis topics. This work lays a foundation for the effective integration of LLMs into circuit analysis instruction and, more broadly, into engineering education.

💡 Analysis

This research full paper presents an enhancement pipeline for large language models (LLMs) in assessing homework for an undergraduate circuit analysis course, aiming to improve LLMs’ capacity to provide personalized support to electrical engineering students. Existing evaluations have demonstrated that GPT-4o possesses promising capabilities in assessing student homework in this domain. Building on these findings, we enhance GPT-4o’s performance through multi-step prompting, contextual data augmentation, and the incorporation of targeted hints. These strategies effectively address common errors observed in GPT-4o’s responses when using simple prompts, leading to a substantial improvement in assessment accuracy. Specifically, the correct response rate for GPT-4o increases from 74.71% to 97.70% after applying the enhanced prompting and augmented data on entry-level circuit analysis topics. This work lays a foundation for the effective integration of LLMs into circuit analysis instruction and, more broadly, into engineering education.

📄 Content

Enhancing Large Language Models for Automated Homework Assessment in Undergraduate Circuit Analysis Liangliang Chen, Huiru Xie, Zhihao Qin, Yiming Guo, Jacqueline Rohde, Ying Zhang† School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, USA †Corresponding Author. E-mail: yzhang@gatech.edu Abstract—This research full paper presents an enhancement pipeline for large language models (LLMs) in assessing home- work for an undergraduate circuit analysis course, aiming to improve LLMs’ capacity to provide personalized support to electrical engineering students. Existing evaluations have demon- strated that GPT-4o possesses promising capabilities in assessing student homework in this domain. Building on these findings, we enhance GPT-4o’s performance through multi-step prompting, contextual data augmentation, and the incorporation of tar- geted hints. These strategies effectively address common errors observed in GPT-4o’s responses when using simple prompts, leading to a substantial improvement in assessment accuracy. Specifically, the correct response rate for GPT-4o increases from 74.71% to 97.70% after applying the enhanced prompting and augmented data on entry-level circuit analysis topics. This work lays a foundation for the effective integration of LLMs into circuit analysis instruction and, more broadly, into engineering education. Index Terms—large language model, automated homework assessment, circuit analysis I. INTRODUCTION Large language models (LLMs) are rapidly transforming a wide range of fields, including programming [1], robotics [2], [3], and education [4]. Pre-trained on vast corpora span- ning diverse domains, LLMs demonstrate impressive question- answering capabilities across numerous benchmarks [5]–[7]. Furthermore, their use of natural language inputs and outputs makes them highly accessible, particularly for users without specialized expertise. In the field of education, students’ diverse backgrounds and learning preferences [8] create a demand for personalized instruction. This, in turn, places a heavy workload on instructors striving to provide effective learning support [9]. LLMs are well-suited to address this challenge, as they can deliver instant, personalized feedback, explanations, and tutoring at scale. Motivated by this potential, this paper investigates the application of LLMs in engineering education, with a specific focus on automated homework assessment in undergraduate circuit analysis. Prior to the era of LLMs, the development of automated homework assessment tools had been a long-standing goal ©2025 IEEE. Accepted to 2025 Frontiers in Education (FIE) Conference. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. pursued by researchers for several decades. These tools pri- marily targeted disciplines or subjects where rule-based as- sessments could be readily implemented, such as program- ming [10], spreadsheets [11], or tasks in natural language processing, such as automated writing evaluation [12]. The advent of LLMs has made automated homework assessment more flexible and less dependent on rigid rule-based systems, allowing their application to a broader range of disciplines [13]. In fact, the potential of LLMs has been explored across various branches of education. For example, Yan et al. [14] developed a mixture-of-math-agent framework for multimodal mathematical error detection. Ref. [15] employed LLMs to foster hands-on problem-solving and programming skills in chemical engineering education, aiming to promote critical thinking and facilitate students’ deeper understanding of core subjects. In the domain of analog circuits, Skelic et al. [16] presented an LLM benchmark for circuit interpretation and reasoning. Using their constructed dataset, GPT-4o—identified as the best-performing model—achieved an accuracy of ap- proximately 48% when evaluated on final numerical answers, indicating that it remains insufficiently reliable for practical use in educational settings. Our work [7] investigated the application of LLMs in automated homework assessment for undergraduate circuit analysis. Three models—GPT-3.5 Turbo, GPT-4o, and Llama 3 70B—were evaluated using a dataset of reference solutions and real student submissions covering key circuit analysis topics. Due to current limitations of LLMs in interpreting handwritten or printed images, the evaluations in [7] relied on reference solutions as the ground truth. Five aspects of student work were assessed with a unified prompt: completeness, method, final answer, arithmetic accuracy, and unit correctness. Results show that GPT-4o and Llama 3 70B outperformed GPT-3.5 Turbo across

This content is AI-processed based on ArXiv data.