What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data

Large language models (LLMs), and conversational agents based on them, are exposed to personal data (PD) during pre-training and during user interactions. Prior work shows that PD can resurface, yet users lack insight into how strongly models associate specific information to their identity. We audit PD across eight LLMs (3 open-source; 5 API-based, including GPT-4o), introduce LMP2 (Language Model Privacy Probe), a human-centered, privacy-preserving audit tool refined through two formative studies (N=20), and run two studies with EU residents to capture (i) intuitions about LLM-generated PD (N1=155) and (ii) reactions to tool output (N2=303). We show empirically that models confidently generate multiple PD categories for well-known individuals. For everyday users, GPT-4o generates 11 features with 60% or more accuracy (e.g., gender, hair color, languages). Finally, 72% of participants sought control over model-generated associations with their name, raising questions about what counts as PD and whether data privacy rights should extend to LLMs.

💡 Research Summary

This paper investigates how large language models (LLMs) associate personal data with an individual’s name and introduces a human‑centered, black‑box audit tool called LMP2 (Language Model Privacy Probe). The authors first adapt the WikiMem probing framework to work with closed‑source API models, selecting 50 out of 243 human‑related properties (e.g., eye colour, language, sexual orientation) to test. Eight LLMs—three open‑source and five commercial APIs, including GPT‑4o—are evaluated on two groups of subjects: well‑known public figures with extensive digital footprints and synthetic individuals with no online presence. Results show that models, especially GPT‑4o, can confidently generate multiple personal attributes for ordinary EU residents, achieving at least 60 % accuracy on 11 attributes such as gender, hair colour, and languages spoken, even for low‑frequency traits.

Building on these findings, the authors design LMP2 as a browser‑based interface where users input their name, the tool automatically sends calibrated prompts to the selected LLM, and visualizes the returned attribute list together with association‑strength scores. Two formative studies (N = 20) refined the prompt wording, result interpretation UI, and sensitivity indicators to make the output understandable for non‑technical users.

Two large‑scale user studies were then conducted with EU residents. Study 1 (N = 155) measured interest in using LMP2 and participants’ intuitions about LLMs’ ability to generate personal data; the majority expressed a desire to see what models might have learned about them, especially regarding health, financial, and location information. Study 2 (N = 303) presented actual GPT‑4o outputs and captured emotional and cognitive reactions. Seventy‑two percent of participants indicated they wanted control over model‑generated associations with their name, and many reported discomfort when the model produced detailed or seemingly inaccurate information.

The paper connects these empirical results to legal and policy discussions, questioning whether GDPR rights such as access, rectification, and erasure should extend to LLM‑generated outputs. Since most commercial LLMs do not expose internal parameters, post‑deployment “unlearning” is difficult, making user‑level auditing essential. The authors argue that probability‑based confidence scores provided by LMP2 can help users assess risk and make informed decisions about disclosure.

In summary, the work (1) demonstrates that LLMs can memorize and infer a wide range of personal attributes tied to a name, (2) provides a usable tool for individuals to audit these associations, and (3) highlights the need for regulatory frameworks and technical safeguards that address privacy risks at the model‑output level. Future directions include expanding the tool to more languages and cultural contexts, investigating bias mitigation, and developing concrete policy guidelines.