Detecting Multiple Semantic Concerns in Tangled Code Commits

Code commits in a version control system (e.g., Git) should be atomic, i.e., focused on a single goal, such as adding a feature or fixing a bug. In practice, however, developers often bundle multiple concerns into tangled commits, obscuring intent and complicating maintenance. Recent studies have used Conventional Commits Specification (CCS) and Language Models (LMs) to capture commit intent, demonstrating that Small Language Models (SLMs) can approach the performance of Large Language Models (LLMs) while maintaining efficiency and privacy. However, they do not address tangled commits involving multiple concerns, leaving the feasibility of using LMs for multi-concern detection unresolved. In this paper, we frame multi-concern detection in tangled commits as a multi-label classification problem and construct a controlled dataset of artificially tangled commits based on real-world data. We then present an empirical study using SLMs to detect multiple semantic concerns in tangled commits, examining the effects of fine-tuning, concern count, commit-message inclusion, and header-preserving truncation under practical token-budget limits. Our results show that a fine-tuned 14B-parameter SLM is competitive with a state-of-the-art LLM for single-concern commits and remains usable for up to three concerns. In particular, including commit messages improves detection accuracy by up to 44% (in terms of Hamming Loss) with negligible latency overhead, establishing them as important semantic cues.

💡 Research Summary

The paper tackles the pervasive problem of tangled commits in version‑control systems, where a single commit bundles multiple semantic concerns (e.g., a feature addition together with a bug fix or a refactoring). While prior work has largely treated commit atomicity as a structural issue or has focused on single‑label classification, this study reframes the task as a multi‑label classification problem and investigates whether small language models (SLMs) can reliably detect multiple concerns under realistic constraints.

Dataset Construction – The authors start from the Zeng CCS‑labeled corpus of atomic commits, apply a refined Conventional Commits Specification (CCS) taxonomy that collapses overlapping types and removes ambiguous labels (perf, style, chore), leaving seven clean labels: feat, fix, refactor, docs, test, build, and ci. They filter out any commit that already exhibits more than one label, creating a pool of truly single‑concern commits. From this pool they synthesize tangled commits by randomly concatenating two or three single‑concern commits, preserving the original diff and message for each component. The resulting benchmark contains 1,750 artificially tangled commits with ground‑truth label sets ranging from one to three concerns, mirroring the distribution observed in real‑world studies (≈90 % of tangled commits involve ≤3 concerns).

Model and Training – The base model is Qwen‑3‑14B, a 14‑billion‑parameter transformer. To keep fine‑tuning affordable on consumer hardware, the authors employ LoRA (Low‑Rank Adaptation), a parameter‑efficient fine‑tuning (PEFT) technique that updates only a small subset of weights. They train the model on the synthetic dataset, comparing three input configurations: (1) diff only, (2) diff + commit message, and (3) diff truncated under a token budget while preserving the CCS header (the “header‑preserving truncation” policy). For baselines they use a state‑of‑the‑art large language model (LLM) such as GPT‑4, as well as the un‑fine‑tuned Qwen‑3‑14B.

Evaluation Metrics – Accuracy is measured with Hamming Loss (the fraction of incorrectly predicted labels), Exact Match Ratio, and micro‑averaged F1. Inference latency (milliseconds per commit) is also recorded to assess practical deployability.

Key Findings

1. Single‑Concern Performance – When a commit contains only one label, the fine‑tuned 14B SLM matches the LLM’s Hamming Loss (≈0.07) and outperforms the unfine‑tuned baseline by a large margin.

2. Impact of Concern Count – As the number of true concerns rises to two or three, the SLM’s Hamming Loss degrades modestly (to ≈0.12–0.15), still remaining within a competitive range compared to the LLM (which is only 5–8 % better).

3. Value of Commit Messages – Adding the commit message to the input yields the most dramatic improvement: Hamming Loss drops by up to 44 % and micro‑F1 rises from 0.78 to 0.92. This demonstrates that natural‑language cues carry essential semantic information that diff‑only representations miss.

4. Header‑Preserving Truncation – Limiting the total token budget from 1024 to 512 tokens (while keeping the CCS header intact) causes only a negligible accuracy loss (≤0.02 Hamming increase) but reduces inference latency from ~150 ms to ~90 ms. Hence, large context windows are not strictly necessary when the header and the initial diff lines are retained.

5. Inference Efficiency – The fine‑tuned SLM processes commits in 120–180 ms on a modern laptop GPU, roughly three to four times faster than the LLM baseline. Even under tighter token budgets the latency remains well below typical code‑review response times, confirming suitability for on‑device or CI‑pipeline deployment.

Contributions – The paper makes three primary contributions: (1) the first systematic empirical study of multi‑concern detection in tangled commits, including a publicly released benchmark and replication package; (2) evidence that a modestly sized, fine‑tuned SLM can achieve LLM‑comparable accuracy while offering substantially lower computational cost and preserving data privacy (since it can run locally); (3) a demonstration that commit messages and a simple header‑preserving truncation strategy are powerful levers for both accuracy and efficiency.

Limitations and Future Work – The synthetic nature of the dataset may not capture all the nuances of real‑world tangled commits, and the refined CCS taxonomy, while cleaner, still abstracts away domain‑specific concerns. Future research directions include (a) applying the approach to large, naturally tangled open‑source histories, (b) extending the label set with finer‑grained or project‑specific categories, and (c) integrating the model into an automated untangling pipeline that can suggest commit splits during pull‑request creation or CI runs.

Conclusion – By reframing tangled‑commit detection as a multi‑label classification task and leveraging a fine‑tuned 14‑billion‑parameter SLM, the authors show that high‑quality semantic analysis of commits is achievable without the heavy resource demands of large language models. The findings underscore the practical value of preserving commit messages and employing lightweight truncation, paving the way for privacy‑preserving, real‑time tooling that can help developers maintain cleaner, more atomic commit histories.