A Dataset and Preliminary Study of Using GPT-5 for Code-change Impact Analysis

💡 Research Summary

The paper presents a comprehensive effort to advance code‑change impact analysis (CIA) by introducing a novel, richly annotated dataset and by evaluating the capabilities of the state‑of‑the‑art large language model GPT‑5 for this task. The authors first construct a dataset that captures the full context of software changes across 50 actively maintained open‑source projects. From these projects they extract over 100,000 commits, each accompanied by (1) the list of modified files, (2) the exact code diffs and function signatures, (3) the outcomes of the test suite executed after the commit, (4) related issue‑tracker entries, and (5) static‑analysis‑derived artifacts such as function‑call graphs and module‑dependency graphs. This multi‑modal information is stored in a line‑delimited JSON format, making it easy to feed both natural‑language descriptions and structured code data into a model.

To harness GPT‑5, the authors perform instruction‑tuning using a subset of the dataset, turning “change description → impacted files/functions” pairs into a supervised signal. The prompt design is deliberately hierarchical: it begins with a concise natural‑language summary of the change (e.g., “Refactor authentication module to use token‑based API”), followed by a list of changed file paths, a code snippet that includes the modified function plus five surrounding lines, and finally a textual representation of the relevant portion of the call graph. This design encourages the model to reason about both the developer’s intent and the structural dependencies of the codebase.

The evaluation compares the GPT‑5‑based approach against three baselines: two traditional static‑analysis tools (ChangeDistiller and ImpactScope) and a recent deep‑learning baseline (CodeBERT‑Impact). Metrics include precision, recall, F1‑score, and false‑positive rate, measured by checking whether the model’s predicted impacted files/functions correspond to actual test failures or bug reports triggered by the change. GPT‑5 achieves a precision of 0.84, recall of 0.78, and an F1‑score of 0.81, outperforming the baselines whose average precision, recall, and F1‑scores are 0.71, 0.66, and 0.68 respectively. Notably, the model shows a 12 % accuracy gain on commits involving complex refactorings (e.g., simultaneous function extraction and interface changes) and reduces the false‑positive rate to 9 % compared with the baseline average of 15 %.

Despite these promising results, the study identifies several practical limitations. First, GPT‑5’s context window imposes a token‑length ceiling, forcing the authors to prune large call graphs and select only the most relevant sub‑graphs for each query. Second, the model’s performance degrades when test coverage is sparse, because the training signal heavily relies on test outcomes. Third, the current system does not provide explainability; it outputs a list of impacted artifacts but does not articulate the reasoning chain that led to each prediction. To address these issues, the authors outline future work that includes (a) hierarchical prompting and graph‑summarization techniques to fit larger contexts within the token budget, (b) augmenting the training data with static‑analysis warnings and code‑quality metrics to reduce dependence on test results, and (c) integrating chain‑of‑thought prompting or graph‑based explanation generators to produce human‑readable impact paths. They also discuss model compression strategies such as knowledge distillation to make the approach feasible for real‑time CI/CD pipelines, where inference latency and computational cost are critical.

In summary, the paper makes two major contributions: (1) the release of a publicly available, multi‑modal dataset that captures the full semantic and structural context of code changes, and (2) an empirical demonstration that a fine‑tuned GPT‑5 model can surpass traditional static analysis tools in predicting the ripple effects of code modifications. By openly sharing the dataset and detailing the experimental setup, the authors provide a solid foundation for subsequent research in AI‑assisted software maintenance, while also candidly exposing the current technical challenges that must be overcome before such systems can be deployed at scale in industry settings.