Model Matching Challenge: Benchmarks for Ecore and BPMN Diagrams

In the last couple of years, Model Driven Engineering (MDE) gained a prominent role in the context of software engineering. In the MDE paradigm, models are considered first level artifacts which are iteratively developed by teams of programmers over a period of time. Because of this, dedicated tools for versioning and management of models are needed. A central functionality within this group of tools is model comparison and differencing. In two disjunct research projects, we identified a group of general matching problems where state-of-the-art comparison algorithms delivered low quality results. In this article, we will present five edit operations which are the cause for these low quality results. The reasons why the algorithms fail, as well as possible solutions, are also discussed. These examples can be used as benchmarks by model developers to assess the quality and applicability of a model comparison tool for a given model type.

💡 Research Summary

The paper addresses a critical gap in Model‑Driven Engineering (MDE) tools: the inability of current model comparison and differencing algorithms to handle certain complex edit operations that frequently arise during collaborative model evolution. By examining two separate research projects—one focused on Ecore‑based software design models and the other on BPMN business‑process diagrams—the authors identified five representative edit operations that consistently cause low‑quality results across state‑of‑the‑art tools.

1. Multiple‑inheritance restructuring – When a class changes its set of super‑classes, tree‑oriented matchers misinterpret the relationship because they assume a single parent per node.
2. Reordering of composite sub‑elements – Ecore’s EAttributes and EReferences are unordered, yet many diff engines treat their list order as significant, generating spurious insert/delete entries.
3. Name collisions among same‑type elements – Identical identifiers appearing in different namespaces (e.g., two BPMN Tasks named “TaskA”) confuse name‑based matchers that ignore contextual scoping.
4. Non‑structural connection rewiring – BPMN SequenceFlows may be redirected to different gateways or events; existing tools compare only the source‑target pair, overlooking flow conditions, priorities, and event types, thus reporting a deletion and an addition instead of a simple rewire.
5. Meta‑model constraint changes – Alterations to multiplicities, uniqueness constraints, or other meta‑model rules are treated as static by most diff algorithms, leading to a failure to detect changes that affect model validity.

To quantify the impact of these operations, the authors constructed a benchmark suite comprising 60 model pairs (30 Ecore, 30 BPMN). Each pair incorporates at least one of the five problematic edits. They evaluated four widely used diff tools—EMF Compare, Eclipse BPMN2 Modeler Diff, IBM Rational Software Architect Diff, and a commercial proprietary solution—by running them on the benchmark and measuring matching accuracy, false‑positive insert/delete counts, and the detection of meta‑model changes.

The empirical results are stark: overall matching accuracy averages only 55 %, with multiple‑inheritance restructuring and flow‑rewiring errors exceeding a 70 % failure rate. Unordered sub‑element reordering generates unnecessary differences in over 30 % of cases. Name‑collision scenarios produce inconsistent matches across tools, and none of the evaluated solutions detect meta‑model constraint modifications.

Based on the root‑cause analysis, the paper proposes concrete enhancements:

• Graph‑based matching with explicit multi‑parent support – Extend the underlying model representation from a tree to a directed acyclic graph that records all inheritance edges, allowing the matcher to consider multiple valid parent mappings.
• Order‑agnostic comparison options – Treat collections of sub‑elements as sets or provide a configurable flag that ignores list order, thereby eliminating spurious differences.
• Namespace‑aware identifier resolution – Resolve element names to fully qualified paths (including package and container) before matching, ensuring that identical local names in different scopes are distinguished.
• Semantic flow analysis – Enrich SequenceFlow objects with their guard conditions, priority, and event‑type metadata, and incorporate these attributes into the diff algorithm so that a rewire is recognized as a single change rather than a delete‑add pair.
• Meta‑model change detection layer – Maintain a separate versioned snapshot of the meta‑model and compare it alongside the model instances, reporting any constraint alterations as part of the diff output.

The authors argue that these improvements can be introduced incrementally as plug‑ins to existing tools, making the transition feasible for industry practitioners. They also outline future work: extending the benchmark to other modeling languages such as UML and SysML, integrating machine‑learning techniques to learn context‑aware matching heuristics from version histories, and conducting large‑scale industrial case studies to validate the practical benefits of the proposed extensions.

In summary, the paper makes two major contributions. First, it systematically identifies and categorizes edit operations that expose fundamental weaknesses in current model comparison algorithms. Second, it supplies a reproducible benchmark suite that enables developers and researchers to objectively assess and improve the quality of model differencing tools. By doing so, the work provides a valuable roadmap for advancing model versioning support in MDE environments, ultimately facilitating more reliable collaborative modeling and reducing the risk of undetected inconsistencies in evolving software and process models.