Protocol Agent: What If Agents Could Use Cryptography In Everyday Life?

We often assume that agent-to-agent interaction will mirror human conversation. However, agents operate fundamentally differently. What if they could develop communication patterns that are more efficient and better aligned with their capabilities? While cryptographic primitives that could profoundly improve everyday interactions already exist, humans can’t use them because they are too complex and the math can’t be done in one’s head. Examples range from proving your age (or other attributes) without showing your ID, to filing an anonymous report within a group while proving you are a legitimate member, to splitting a dinner bill fairly without revealing salaries. What if agents could create protocols “on the fly” by recognizing which primitive fits an everyday situation, proposing it to an agentic counterpart, persuading them to participate, and then executing the protocol correctly using appropriate computation tools? Protocol Agent frames this problem by introducing a benchmark that spans: (1) cryptographic primitive recognition, (2) negotiation skills, (3) implementation correctness, (4) correct computation and (5) security strength. We evaluate current open-weight and state-of-the-art models on this benchmark, propose a dataset-generation approach to improve these capabilities, and measure the impact of supervised fine-tuning (SFT) on benchmark performance, with tuned models outperforming base models by a wide margin.

💡 Research Summary

The paper introduces “Protocol Agent,” a benchmark that evaluates whether autonomous agents can recognize, negotiate, implement, and securely execute cryptographic primitives in everyday conversational contexts. The authors argue that, unlike humans, agents possess fast computation, persistent machine‑readable state, and tool‑use capabilities, enabling them to embed sophisticated privacy‑preserving protocols (e.g., private set intersection, zero‑knowledge proofs, multi‑party computation) into ordinary dialogue. To assess this ability, the benchmark decomposes performance into five dimensions: (1) Primitive Selection – correctly mapping a natural‑language scenario to an appropriate cryptographic family; (2) Negotiation Skills – persuading a counterpart to adopt the protocol while addressing incentives and objections; (3) Implementation Correctness – specifying a concrete multi‑step procedure with proper checks and consistent end‑to‑end flow; (4) Computation/Tool Usage – invoking a cryptographic calculator (cryptomath) to generate verifiable artifacts (hashes, signatures, shared secrets) and integrating them correctly; (5) Security Strength – meeting confidentiality and integrity goals under an honest‑but‑curious threat model and avoiding common attacks such as replay, forgery, or selective failure.

Benchmark instances are provided as JSON records containing a public scenario, role‑specific goals, private information, and a symmetry‑breaker that forces one role to make the first move. The arena runs turn‑based self‑play with fixed budgets for turns and tool calls, logs the full transcript, and injects tool results as “TOOL RESULT” messages without consuming extra turns. After each match, an LLM judge receives the transcript, the number of tool calls, and hidden fields (target primitive families, ideal solution description, typical failure modes). The judge returns a strict JSON object with scores (1–5) for each dimension and a short verdict; it may also recompute tool outputs via cryptomath to verify numerical claims, ensuring objective correctness.

To generate training data, the authors build a pipeline that extracts primitive descriptions from cryptography textbooks and papers, then automatically pairs them with everyday scenarios (e.g., “find a meeting slot without sharing calendars” → private set intersection). This synthetic data is used for supervised fine‑tuning (SFT). Experiments compare several open‑weight large language models (DeepSeek‑V3P1, Qwen‑3‑30B‑i2507, etc.) in their base form and after SFT on the generated dataset. Results show substantial improvements: DeepSeek‑V3P1’s overall score rises from 0.473 to 0.693 (+46.5 %), and Qwen‑3‑30B‑i2507 from 0.390 to 0.676 (+73.3 %). Gains are especially pronounced in Negotiation Skills and Security Strength, indicating that fine‑tuning helps models not only recall cryptographic facts but also reason about incentives, construct persuasive arguments, and design robust protocols. Tool‑usage accuracy also improves, with fine‑tuned models achieving >90 % correct and necessary calls versus ~60 % for base models.

The paper situates its contribution among three research strands: (a) cryptographic reasoning benchmarks that focus on mathematical correctness; (b) tool‑using agents in security‑oriented tasks; and (c) multi‑agent negotiation/deception benchmarks. Protocol Agent uniquely combines these by requiring agents to select a primitive, convince a partner, and execute a verifiable protocol under realistic privacy constraints.

Limitations include the reliance on an honest‑but‑curious threat model, the assumption of a trusted cryptomath tool, and the exclusion of timing or side‑channel attacks. Future work is outlined: extending to malicious or colluding adversaries, scaling to multi‑agent collaborations, integrating richer toolsets, and deploying the framework in real‑world services. All code, data, and the arena are released under an MIT license to encourage community contributions.

In summary, “Protocol Agent” defines a new research agenda—embedding cryptographic protocols into everyday agent communication—provides a concrete benchmark and data generation pipeline, and demonstrates that supervised fine‑tuning can substantially elevate current language models toward this goal, paving the way for privacy‑preserving, secure autonomous agents in daily digital interactions.