Interactive Debugging of Knowledge Bases

Many AI applications rely on knowledge about a relevant real-world domain that is encoded by means of some logical knowledge base (KB). The most essential benefit of logical KBs is the opportunity to perform automatic reasoning to derive implicit knowledge or to answer complex queries about the modeled domain. The feasibility of meaningful reasoning requires KBs to meet some minimal quality criteria such as logical consistency. Without adequate tool assistance, the task of resolving violated quality criteria in KBs can be extremely tough even for domain experts, especially when the problematic KB includes a large number of logical formulas or comprises complicated logical formalisms. Published non-interactive debugging systems often cannot localize all possible faults (incompleteness), suggest the deletion or modification of unnecessarily large parts of the KB (non-minimality), return incorrect solutions which lead to a repaired KB not satisfying the imposed quality requirements (unsoundness) or suffer from poor scalability due to the inherent complexity of the KB debugging problem. Even if a system is complete and sound and considers only minimal solutions, there are generally exponentially many solution candidates to select one from. However, any two repaired KBs obtained from these candidates differ in their semantics in terms of entailments and non-entailments. Selection of just any of these repaired KBs might result in unexpected entailments, the loss of desired entailments or unwanted changes to the KB. This work proposes complete, sound and optimal methods for the interactive debugging of KBs that suggest the one (minimally invasive) error correction of the faulty KB that yields a repaired KB with exactly the intended semantics. Users, e.g. domain experts, are involved in the debugging process by answering automatically generated queries about the intended domain.

💡 Research Summary

The paper addresses the challenging problem of debugging logical knowledge bases (KBs) that violate essential quality criteria such as consistency. Traditional non‑interactive debugging approaches suffer from incompleteness (they cannot locate all possible faults), non‑minimality (they suggest removing overly large subsets of axioms), unsoundness (they may produce repaired KBs that still violate the required properties), and poor scalability due to the combinatorial nature of the diagnosis problem. Even when a non‑interactive system is complete, sound, and returns only minimal solutions, the number of candidate repairs grows exponentially, and selecting any one of them without guidance can lead to unintended entailments or loss of desired knowledge.

To overcome these limitations, the authors propose a comprehensive framework for interactive KB debugging that is complete, sound, and optimal with respect to minimal invasiveness. The core idea is to involve a domain expert (or any knowledgeable user) in a loop of query answering. The system maintains a set of candidate diagnoses—each a minimal hitting set of conflict sets (i.e., minimal subsets of axioms whose removal restores consistency). When at least two diagnoses remain, the system automatically generates a query: a minimal set of axioms whose truth value differs between the competing diagnoses. The user’s answer (whether the intended KB should entail the query or not) prunes the search space by discarding all diagnoses inconsistent with the response. Repeating this process until a single diagnosis remains yields a repaired KB that exactly matches the user’s intended semantics.

The technical contributions are extensive:

1. Theoretical Foundations – The paper formalizes non‑interactive debugging (conflict sets, diagnoses, hitting‑set trees) and then extends these notions to the interactive setting. It proves the existence of queries whenever more than one diagnosis is present, and shows that the iterative pruning process converges to the unique intended repair.

2. Diagnosis Computation – Minimal conflict sets are computed via a recursive tree algorithm; diagnoses are derived as minimal hitting sets using a breadth‑first Hitting Set Tree (HS‑Tree). Probabilistic information (fault likelihoods of axioms) is incorporated to rank diagnoses, enabling the system to focus on the most probable candidates first.

3. Query Generation and Minimization – The concept of Q‑Partitions captures how a query splits the set of leading diagnoses into three groups (positive, negative, and undecided). An algorithm builds a pool of candidate queries, then applies a minimization step that removes redundant axioms while preserving the partition. Soundness and completeness of this process are rigorously established.

4. Iterative Diagnosis Algorithms – Two families of algorithms are introduced:

   • StaticHS: Computes conflict sets once and reuses them throughout the interaction; the HS‑Tree is rebuilt only after each query answer.
   • DynamicHS: Updates conflict sets incrementally after each answer, pruning the HS‑Tree on‑the‑fly. This dramatically reduces the number of nodes explored, especially when many faults are present simultaneously.

5. Query Selection Strategies – Three strategies are evaluated:

   • Entropy‑based (maximizing expected information gain).
   • Split‑In‑Half (aiming to halve the candidate set).
   • RIO (Risk Optimization) – balances information gain against the risk of discarding the correct diagnosis. Empirical results show RIO consistently requires fewer queries and less overall time.

6. Empirical Evaluation – Experiments on synthetic and real‑world ontologies (including the well‑known Anatomy benchmark) cover propositional, description, and first‑order logics. Metrics include average number of queries, time to compute leading diagnoses, total debugging time, and correctness of the final repair. Both StaticHS and DynamicHS outperform non‑interactive baselines, with DynamicHS + RIO achieving the best performance, especially for high‑cardinality fault scenarios.

7. System Architecture – The paper sketches a complete interactive debugging system, integrating the diagnosis engine, query generator, user interface, and probabilistic model. It also discusses practical issues such as handling user errors, caching of entailments, and scalability to large KBs.

In conclusion, the work delivers a mathematically sound, algorithmically efficient, and practically viable solution for interactive knowledge‑base debugging. By guaranteeing minimal changes, completeness, and soundness while actively involving the user through carefully crafted queries, the framework bridges the gap between automated reasoning and human expertise. Future directions include extending the approach to non‑monotonic logics, distributed knowledge bases, and learning‑based query generation to further reduce user effort.