CREME: Robustness Enhancement of Code LLMs via Layer-Aware Model Editing

Large language models (LLMs) have demonstrated impressive capabilities in code generation, where the natural language prompt plays a crucial role in conveying user intent to the model. However, prior studies have shown that LLMs are highly sensitive to prompt perturbations. Minor modifications in wording, syntax, or formatting can significantly reduce the functional correctness of generated code. As perturbations frequently occur in real-world scenarios, improving the robustness of LLMs to prompt perturbations is essential for ensuring reliable performance in practical code generation. In this paper, we introduce CREME (Code Robustness Enhancement via Model Editing), a novel approach that enhances LLM robustness through targeted parameter updates. CREME first identifies robustness-sensitive layers by comparing hidden states between an original prompt and its perturbed variant. Then, it performs lightweight parameter editing at the identified layer to reduce performance degradation. We evaluate CREME on two widely used code generation benchmarks (HumanEval and MBPP) along with their perturbed counterparts. Experimental results show that CREME improves Pass@1 accuracy by 63% on perturbed prompts while maintaining stable performance on clean inputs, with accuracy deviations within 1%. Further analysis reveals that robustness-sensitive layers are primarily concentrated in the middle and deeper layers of the network, and their locations vary across different model architectures. These insights provide a valuable foundation for developing future robustness-oriented editing strategies.

💡 Research Summary

The rapid advancement of Large Language Models (LLMs) in code generation has revolutionized software engineering. However, a critical vulnerability remains: extreme sensitivity to prompt perturbations. Minor alterations in wording, syntax, or formatting can lead to significant degradation in the functional correctness of the generated code. This instability poses a major challenge for deploying reliable AI-driven coding assistants in real-world, unpredictable environments.

To address this, the paper introduces CREME (Code Robustness Enhancement via Model Editing), a novel framework designed to enhance the robustness of Code LLMs through targeted, layer-aware parameter updates. Unlike traditional fine-tuning, which is computationally expensive and prone to catastrophic forgetting, CREME focuses on a surgical approach to model editing.

The CREME methodology operates in two distinct phases. First, the “Identification Phase” locates robustness-sensitive layers. The researchers achieve this by comparing the hidden states produced by an original prompt and its perturbed counterpart. By measuring the divergence in hidden representations, the framework can pinpoint exactly which layers are most susceptible to input noise. Second, the “Editing Phase” performs lightweight parameter updates specifically on these identified layers. This targeted approach aims to minimize the impact on the model’s original capabilities while maximizing its resistance to perturbations.

The experimental evaluation, conducted on the widely recognized HumanEval and MBPP benchmarks, demonstrates the extraordinary efficacy of CREME. The results show a massive 63% improvement in Pass@1 accuracy when dealing with perturbed prompts. Crucially, the model maintains its high performance on clean, original prompts, with accuracy deviations staying within a negligible 1% margin. This proves that CREME can bolster robustness without sacrificing the model’s fundamental coding intelligence.

Furthermore, the study provides profound architectural insights. The analysis reveals that robustness-sensitive layers are primarily concentrated in the middle and deeper layers of the transformer architecture. Interestingly, the paper also notes that the specific locations of these sensitive layers vary depending on the model architecture used. These findings are highly significant for the future of AI development, providing a foundational blueprint for designing and editing future-generation, robustness-oriented code models. Ultimately, CREME shifts the paradigm from reactive prompt engineering to proactive model-centric robustness enhancement.