\textcolor{darkgreen}{\checkmark}
``` | 
``` math
\textcolor{darkgreen}{\checkmark}
``` | **Dual-Graph-based** | Sub-task Comp | **$`\textcolor{darkgreen}{\checkmark}`$** |

<div class="tablenotes">

$`\textcolor{darkgreen}{\checkmark}`$=Supported,
$`\textcolor{darkred}{\times}`$=Not supported. Cross-platform requires
simultaneous multi-device operations.

</div>

</div>

Existing work exhibits significant limitations in three key areas.
Currently, there is a lack of task datasets tailored for educational
agents, which hampers research and development in educational scenarios.
Existing knowledge graphs and GUI operation libraries are primarily
designed for general-purpose software and cannot provide structured
knowledge support for school-specific systems. Conventional metrics,
such as task completion rate and trajectory similarity, focus solely on
macroscopic outcomes. These metrics cannot quantify fine-grained issues
like backtracking operations or omissions of critical steps.

To address the the above issues, we propose Knowledge-Augmented
Dual-Graph Evaluator for Cross-Platform Educational Agent Benchmarking
with Multimodal Language Models, a cross-platform educational agent
benchmarking framework based on knowledge enhancement and dual-graph
evaluation. The overall framework is shown in
Fig. <a href="#fig:enter-label" data-reference-type="ref"
data-reference="fig:enter-label">1</a>.

**Table <a href="#tab:environments" data-reference-type="ref"
data-reference="tab:environments">[tab:environments]</a>** presents a
comparison between KGCE and existing benchmark frameworks. Key features
are categorized as: *Interactive Environment* (system’s operating
context: Web/GUI/Code); *Knowledge* (structured knowledge management
with knowledge graphs or dynamic reasoning supported by
$`\textcolor{darkgreen}{\checkmark}`$); *Cross-platform* (concurrent
multi-device operations requiring OS/device interoperability);
*Evaluation* (Goal-based: final state verification; Trajectory: action
sequence alignment; Graph-based: DAG checkpoint validation); *Task
Construction* (Template: predefined patterns; Manual: human-crafted;
Sub-task Comp: modular composition);*Educational Task* (curriculum
integration or pedagogical assessment via
$`\textcolor{darkgreen}{\checkmark}`$).

The main contributions are as follows:

- We constructed a dataset of 104 educational tasks spanning Windows,
  Android, and cross-platform collaboration. These tasks involve
  operations with private-domain software and multi-device coordination
  workflows, and its dependencies are modeled using a DAG.

- For private-domain software, we developed a structured JSON knowledge
  base. Knowledge from this base is dynamically retrieved and injected
  into model prompts, significantly improving execution efficiency and
  success rates in private-domain tasks.

- We propose a dual-graph evaluation framework consisting of a
  *Completeness Graph* and an *Efficiency Graph*. This framework
  introduces eight fine-grained metrics to assess task performance in
  detail.

- We validate the effectiveness of the knowledge base across several
  models, including Qwen-VL-Max-Latest, GPT-4o, and Gemini-2.0-Flash,
  revealing differences in their dependence on domain-specific
  knowledge.

# RELATED WORK

To comprehensively contextualize our research, we structure the related
work into four key dimensions: (1) cross-platform agents driven by large
models, (2) task modeling and agents in educational scenarios, (3)
knowledge-enhanced agent architectures, (4) agent evaluation
methodologies. These dimensions were selected because they collectively
address the challenges of building intelligent agents in educational
environments. By analyzing these aspects, we aim to highlight the gaps
in existing research and position our contributions accordingly.

## Large Model-Driven Cross-Platform Agents

In recent years, MLMs have demonstrated significant potential in the
field of cross-platform agents. CRAB introduced the first benchmark
framework supporting cross-environment tasks, enabling efficient
construction and evaluation of complex tasks through subtask composition
and a graph-based evaluator. AndroidWorld and OSWorld have established
dynamic Android environments and open computer environments,
respectively, providing diverse platform support for agent evaluation.
However, these cross-platform agent studies do not address the specific
support required for education-related private-domain software. Notably,
the directed acyclic graph-based task decomposition method proposed by
CRAB offers valuable inspiration for our task modeling. Nonetheless, its
coarse-grained trajectory-matching evaluation approach falls short of
capturing the nuanced execution differences critical to educational
scenarios.

## Task Modeling and Agents in Educational Scenarios

Research on educational agents faces the dual challenges of complex
environments and strong dependence on domain-specific knowledge.
EduAgent boosts task efficiency via multimodal interaction, yet only
targets general tools. GUICourse improves GUI grounding but ignores
cross-platform state sync. EduBench supplies real-world benchmarks, but
its task volume is small and lacks structured knowledge. Current studies
exhibit several key shortcomings: (1) Task datasets largely rely on
general educational platforms and lack support for school-customized
systems; (2) Evaluation metrics focus on macro-level metrics such as
task completion rate, which are insufficient for quantifying the
optimization level of execution paths. These limitations highlight the
critical need for a dedicated evaluation framework tailored to the
specific demands of educational scenarios.

## Knowledge-Augmented Agent Architectures

Knowledge base augmentation has emerged as an effective paradigm for
enhancing agent adaptability across domains. GAT proposes an attention
mechanism to dynamically calculate the importance of nodes in the graph,
which is applicable to the priority sorting of nodes in the knowledge
atlas and provides a theoretical basis for the priority call mechanism
of this research knowledge base. PLaG employed graph structures to
enhance task planning capabilities; however, its static knowledge
representations are ill-suited for the dynamically evolving nature of
educational software. Notably, existing knowledge enhancement methods
predominantly rely on general-purpose knowledge graphs, lacking
structured modeling tailored to private-domain educational software.
This essential yet underexplored factor currently constrains the
performance of educational agents.

## Agent Evaluation Methods

The development of agent evaluation systems is trending from
outcome-focused metrics toward process-oriented analysis. AgentBench
established a multidimensional evaluation benchmark, but its API-based
validation approach is poorly suited for GUI operation scenarios. DyVal
introduced a dynamic evaluation framework using a DAG structure to
capture task execution increments; however, its discrete state labeling
fails to quantify continuous metrics such as the backtracking rate.
Regarding fine-grained evaluation, although CRAB’s graph-based evaluator
can verify subtask completion states, it lacks the capability to analyze
execution path efficiency. Recent studies have shown that combining
structural analysis with process tracing provides a more accurate
reflection of an agent’s cognitive capabilities. These findings offer
theoretical support for the design of our dual-graph evaluation
framework.

# METHOD

This section provides a detailed introduction to the specific
implementation of the cross-platform educational agent benchmark, which
combines knowledge base enhancement with a double-layer graph evaluation
framework.

## Educational Dataset Construction

While benchmarks cover cross-environment tasks, they miss multi-device
collaboration in education. We introduce the first education-focused
task set, grounded in real activities at Central China Normal
University. Spanning proprietary software, it supports Windows, Android,
and cross-platform runs. Inspired by CRAB, we scale via decomposition,
templates, and composition: each complex task splits into atomic
subtasks, linked in a DAG. For example, a complex task such as *“Use
Xiaoya to check the assignments for the Big Data Technology course and
add the task in the Tasks app”* can be broken down into the following
subtasks: *“Open the Xiaoya app,” “Enter the Big Data Technology
course,” “View the assignment tasks,” “Switch to the Tasks app,”* and
*“Add the task.”* These subtasks are organized into a **DAG** that
models the logical dependencies between them. Each node represents a
subtask, and each edge denotes a dependency.

To instantiate concrete tasks, we design task templates using natural
language instruction patterns. These templates contain input attributes
such as `{app_name}`, `{feature}`, and `{action}`, which can be
dynamically replaced with specific values. For example, the template
“Open {feature} in {app_name}, perform {action}, and save it” enables
efficient generation of diverse task instances by substituting real
values. For complex scenarios, we compose multiple subtask templates to
create multi-step tasks. For instance, a cross-platform task may require
opening the One-Stop Service Platform on Windows, accessing the message
center, and then recording the message content in the Keep Notes app on
an Android device.

<figure id="fig:Knowledge-base" data-latex-placement="t">
<img src="/posts/2026/01/2026-01-04-190607-kgce__knowledge_augmented_dual_graph_evaluator_for/KGCE-Knowledge.png" style="width:100%" /> style="width:90.0%" />
<figcaption>The knowledge base module. Given a task description and
package names, the system identifies relevant packages, retrieves their
descriptions from the knowledge base, and constructs prompts for the
LLM. The knowledge base is organized by packages, pages, and elements,
with each element including its position, description, and
sub-elements.</figcaption>
</figure>

To guarantee task feasibility and validity, we rigorously verify each
subtask, ensuring it can be executed on the target platform and that its
input-output logic is consistent. Ultimately, we construct a dataset of
**104 tasks**, covering applications such as HuaShi XiaZi, Xiaoya
Assistant, Keep Notes, and MOOC platforms. This dataset balances task
diversity and complexity, laying a solid foundation for subsequent
experiments and evaluations.

## Knowledge Base Construction

Proprietary software such as Xiaoya Intelligent Assistant presents
unique interfaces and workflows that existing MLMs rarely master. To
close the gap, we manually interact with each target application to
collect software names, page descriptions, UI-element positions, and
functional explanations, then serialize the data into a uniform JSON
schema. At runtime, a **Knowledge Invocation Decision** module checks
the task description: if the software is mentioned, the corresponding KB
records are retrieved and injected into the prompt; otherwise, they are
omitted. Fig. <a href="#fig:Knowledge-base" data-reference-type="ref"
data-reference="fig:Knowledge-base">2</a> illustrates the full pipeline
from manual exploration to structured JSON and prompt augmentation.

## Dual-Graph Evaluation Framework

Current coarse-grained, goal-oriented or trajectory-matching metrics
inadequately capture the nuanced execution of agents in complex tasks.
KGCE proposes a dual-graph evaluation framework—Task Completeness Graph
(TCG) and Execution Efficiency Graph (EEG)—to provide fine-grained
metrics quantifying both completion quality and execution efficiency.

### Task Completeness Graph

TCG decomposes a task into sub-goals and builds a dependency graph where
each node $`v_i`$ represents a sub-goal. During execution, we annotate
real-time completion status with indicator $`I(v_i=1)`$. The overall
progress is captured by the **Completion Ratio**
$`\text{CR} = \sum_{v_i\in V} I(v_i=1)/|V|`$. Action-level performance
is measured by **Completeness per Action**
$`\text{CPA} = \sum_{a_j\in A} I(a_j=1)/|A|`$, $`I(a_j=1)`$ marks
whether action $`a_j`$ is completed.

### Execution Efficiency Graph

EEG evaluates execution efficiency. **Backtracking Ratio**
$`\text{BR} = \text{IO}/\text{ONU}`$ quantifies the frequency of
backtracking steps (IO) relative to total operations (ONU); lower BR
indicates more direct paths. **Precision** is $`\text{CAN}/\text{ANU}`$,
the ratio of completed actions (CAN) to attempted actions (ANU).
**Recall** measures coverage of essential key steps:
$`\text{Recall} = \sum_{k_m\in K} I(k_m=1)/|K|`$. Precision and Recall
are combined into an F1-score. Two exception metrics are recorded: **Out
of Range (OoR)** counts out-of-bounds errors, and **reach_max_step
(RMS)** tallies terminations due to reaching the maximum step limit.

## Dual-Graph Evaluation Framework

Current evaluation methods predominantly rely on goal-oriented or
trajectory-matching coarse-grained metrics, which are insufficient for
capturing the execution nuances of agents in complex tasks. KGCE
proposes a dual-graph evaluation framework, comprising the Task
Completeness Graph and the Execution Efficiency Graph. This framework
aims to provide fine-grained evaluation metrics that more accurately
quantify the agent’s execution efficiency and task completion quality.

### **Task Completeness Graph.**

The Task Completeness Graph evaluates an agent’s ability to achieve
individual subgoals within a task.. We decompose each task into multiple
subgoals, constructing a dependency graph where each node {vi}
represents a subgoal. Throughout execution, we monitor and annotate the
real-time completion status of each subgoal, using {I(vi=1)} to indicate
completion (1 for completed, 0 for incomplete). To quantify overall
progress and assess whether the agent achieves most subgoals, we define
the Completion Ratio (CR):
``` math
\begin{equation}
\text{CR} = \frac{\sum_{v_i \in V} I(v_i = 1)}{|V|}.
\label{eq:cr}
\end{equation}

\begin{equation}
\text{CPA} = \frac{\sum\limits_{a_j \in A} I(a_j = 1)}{|A|}.
\label{eq:cpa}
\end{equation}

\begin{equation}
Recall = \frac{\sum\limits_{k_m \in K} I(k_m = 1)}{|K|},
\label{eq:recall}
\end{equation}