시각 기반 VLM을 활용한 CNC 가공 코드와 HMI 동시 검증

📝 Abstract

Manual generation of G-code is important for learning the operation of CNC machines. Prior work in G-code verification uses Large-Language Models (LLMs), which primarily examine errors in the written programming. However, CNC machining requires extensive use and knowledge of the Human-Machine Interface (HMI), which displays machine status and errors. LLMs currently lack the capability to leverage knowledge of HMIs due to their inability to access the vision modality. This paper proposes a few-shot VLM-based verification approach that simultaneously evaluates the G-code and the HMI display for errors and safety status. The input dataset includes paired G-code text and associated HMI screenshots from a 15-slant-PRO lathe, including both correct and error-prone cases. To enable few-shot learning, the VLM is provided with a structured JSON schema based on prior heuristic knowledge. After determining the prompts, instances of G-code and HMI that either contain errors or are error free are used as few-shot examples to guide the VLM. The model was then evaluated in comparison to a zero-shot VLM through multiple scenarios of incorrect G-code and HMI errors with respect to per-slot accuracy. The VLM showed that few-shot prompting led to overall enhancement of detecting HMI errors and discrepancies with the G-code for more comprehensive debugging. Therefore, the proposed framework was demonstrated to be suitable for verification of manually generated G-code that is typically developed in CNC training.

💡 Analysis

Manual generation of G-code is important for learning the operation of CNC machines. Prior work in G-code verification uses Large-Language Models (LLMs), which primarily examine errors in the written programming. However, CNC machining requires extensive use and knowledge of the Human-Machine Interface (HMI), which displays machine status and errors. LLMs currently lack the capability to leverage knowledge of HMIs due to their inability to access the vision modality. This paper proposes a few-shot VLM-based verification approach that simultaneously evaluates the G-code and the HMI display for errors and safety status. The input dataset includes paired G-code text and associated HMI screenshots from a 15-slant-PRO lathe, including both correct and error-prone cases. To enable few-shot learning, the VLM is provided with a structured JSON schema based on prior heuristic knowledge. After determining the prompts, instances of G-code and HMI that either contain errors or are error free are used as few-shot examples to guide the VLM. The model was then evaluated in comparison to a zero-shot VLM through multiple scenarios of incorrect G-code and HMI errors with respect to per-slot accuracy. The VLM showed that few-shot prompting led to overall enhancement of detecting HMI errors and discrepancies with the G-code for more comprehensive debugging. Therefore, the proposed framework was demonstrated to be suitable for verification of manually generated G-code that is typically developed in CNC training.

📄 Content

FEW-SHOT VLM-BASED G-CODE AND HMI VERIFICATION IN CNC MACHINING Yasaman Hashem Pour∗, Nazanin Mahjourian, Vinh Nguyen Department of Mechanical and Aerospace Engineering, Michigan Technological University, Houghton, MI 49931 ABSTRACT Manual generation of G-code is important for learning the operation of CNC machines. Prior work in G-code verification uses Large-Language Models (LLMs), which primarily examine errors in the written programming. However, CNC machining requires extensive use and knowledge of the Human-Machine In- terface (HMI), which displays machine status and errors. LLMs currently lack the capability to leverage knowledge of HMIs due to their inability to access the vision modality. This paper proposes a few-shot VLM-based verification approach that simultaneously evaluates the G-code and the HMI display for errors and safety status. The input dataset includes paired G-code text and as- sociated HMI screenshots from a 15-slant-PRO lathe, including both correct and error-prone cases. To enable few-shot learn- ing, the VLM is provided with a structured JSON schema based on prior heuristic knowledge. After determining the prompts, instances of G-code and HMI that either contain errors or are error free are used as few-shot examples to guide the VLM. The model was then evaluated in comparison to a zero-shot VLM through multiple scenarios of incorrect G-code and HMI errors with respect to per-slot accuracy. The VLM showed that few-shot prompting led to overall enhancement of detecting HMI errors and discrepancies with the G-code for more comprehensive de- bugging. Therefore, the proposed framework was demonstrated to be suitable for verification of manually generated G-code that is typically developed in CNC training. Keywords: Vision-Language Models, Large-Language Mod- els, Few-Shot Learning, Prompt Engineering, G-Code

1. INTRODUCTION While learning to write G-code is essential for understand- ing CNC machining, it can also be complex, error-prone, and time-consuming [1]. This is because G-code is a low-level lan- guage that also requires knowledge of the environment in which the code is hosted, such as the machine’s characteristics, pro- cess conditions, and the human-machine interface (HMI) display. CNC errors are typically caused by manual mistakes, user-defined toolpath routines, and high interpolation sequence codes that re- ∗Corresponding author: yhashemp@mtu.edu quire user correction[2, 3]. Even with advanced G-code gener- ation software, obtaining accurate and reliable G-code requires deep technical knowledge and manual debugging, including thor- ough verification [4, 5]. In practice, manual G-code debugging is vulnerable to extraneous variables, including human error, incor- rect coordinate systems, and missing tool calls. This is because troubleshooting these errors demands spatial and technical rea- soning [5]. Therefore, careful verification of the G-code is key to avoiding costly errors and facilitating safe and precise opera- tion. Intelligent verification systems can help analyze and detect potential G-code problems before machining begins [6, 7]. In recent research, many models including Large Language Models (LLMs) have been applied to analyze and correct G- code automatically [6, 8–10]. Various studies show considerable potential for LLMs interpreting and optimizing G-code within manufacturing applications. By leveraging pattern recognition and contextual understanding of programming syntax, these mod- els can analyze G-code and identify structural and logic errors [11]. Furthermore, the LLM approaches are also capable of self-correction and anomaly detection even in zero-shot settings, which leads to enhanced code reliability and minimizes reference errors [12]. Such models have been employed to generate opti- mized G-code. The optimized G-code can improve mechanical performance and manufacturing reliability in additive manufac- turing [13]. More recently, the developing body of literature has shifted toward exploring explainability and interpretability. This approach can help to detect manufacturing features and enhance process understanding and robustness through chain-of-thought prompting [14]. As a result of these strengths, LLM-based frame- works are capable of interpreting and analyzing low-level G- code structures and handling corrective tasks. However, despite their effectiveness in text-based reasoning, the implementation of LLMs in CNC machining faces significant limitations. Current LLM systems are mainly based on tokenized text and cannot pro- cess visual or sensory feedback that characterizes the machining environment [8, 14]. In addition, many systems simulate ma- chine signals instead of monitoring actual machine data, causing them to struggle with adapting to tool conditions and variations in cutting speed and load [13]. Hence, LLMs are incapable of aligning G-code with actual machine behavior since they operate 1 Copyright © 2025 by ASME offline without interf

This content is AI-processed based on ArXiv data.