Campaign-2-PT-RAG: LLM-Guided Semantic Product Type Attribution for Scalable Campaign Ranking

E-commerce campaign ranking models require large-scale training labels indicating which users purchased due to campaign influence. However, generating these labels is challenging because campaigns use creative, thematic language that does not directly map to product purchases. Without clear product-level attribution, supervised learning for campaign optimization remains limited. We present Campaign-2-PT-RAG, a scalable label generation framework that constructs user-campaign purchase labels by inferring which product types (PTs) each campaign promotes. The framework first interprets campaign content using large language models (LLMs) to capture implicit intent, then retrieves candidate PTs through semantic search over the platform taxonomy. A structured LLM-based classifier evaluates each PT’s relevance, producing a campaign-specific product coverage set. User purchases matching these PTs generate positive training labels for downstream ranking models. This approach reframes the ambiguous attribution problem into a tractable semantic alignment task, enabling scalable and consistent supervision for downstream tasks such as campaign ranking optimization in production e-commerce environments. Experiments on internal and synthetic datasets, validated against expert-annotated campaign-PT mappings, show that our LLM-assisted approach generates high-quality labels with 78-90% precision while maintaining over 99% recall.

💡 Research Summary

The paper addresses a critical bottleneck in e‑commerce campaign ranking: the lack of large‑scale, high‑quality user‑campaign purchase labels. Campaigns are often expressed with creative, thematic, or metaphorical language that does not directly mention the concrete product types (PTs) they promote. Traditional labeling relies on manual analysts to read each campaign, infer the intended product categories, and then match post‑exposure purchases. This process is labor‑intensive, inconsistent, and infeasible at the scale required by modern platforms.

To solve this, the authors propose Campaign‑2‑PT‑RAG, a Retrieval‑Augmented Generation (RAG) pipeline that automatically infers the set of product types associated with any campaign and then generates binary purchase labels for users. The pipeline consists of four sequential components:

1. LLM‑based campaign interpretation – A large language model (e.g., GPT‑4) receives the raw campaign title and body and is prompted to produce a natural‑language summary s₍c₎ that captures both explicit and implicit product intent. This step enriches the downstream query with latent themes that are not present in the raw text.

2. Semantic retrieval of candidate PTs – Each PT node in the platform taxonomy (7 147 nodes) is represented as a concatenated string “Category | Family | Type” and embedded using a dense encoder. Using FAISS, the system retrieves all PTs whose cosine similarity with the embedding of s₍c₎ exceeds a low threshold τ, deliberately favoring high recall.

3. Optional cross‑encoder reranking – A BERT‑style cross‑encoder jointly encodes the original campaign text and each candidate PT description to produce finer‑grained relevance scores. This step reorders the candidate list, improving the signal for the final classifier while adding modest latency.

4. LLM‑based relevance classification – For every retrieved PT t, the LLM is given the pair ( s₍c₎, t ) and asked to label the relationship as “strongly relevant”, “weakly relevant”, or “irrelevant”. The model reasons over the hierarchical taxonomy (Category → Family → Type) and the thematic cues in the campaign, allowing it to distinguish, for example, that “wall‑mounts” are weakly relevant to a “large‑screen TV” promotion, while “wireless headphones” are irrelevant. Only PTs marked strong or weak are retained as the final coverage set PT(c).

The binary user‑campaign label is then defined as yᵤ,₍c₎ = 1 if the user’s purchased PT set Pᵤ intersects PT(c); otherwise 0. Consequently, the quality of the inferred PT coverage directly determines label fidelity.

Evaluation was performed on two fronts: (a) internal real‑world campaigns annotated by domain experts, and (b) synthetic campaigns where ground truth is known. Compared against strong baselines—BM25 lexical search, dense bi‑encoder retrieval, and BM25 + cross‑encoder reranking—Campaign‑2‑PT‑RAG achieved precision between 78 % and 90 % (15‑20 percentage points higher) while maintaining recall above 99 % (≈5 pp improvement). An LLM‑as‑judge protocol further validated that the semantic alignment judgments correlate well with human assessments. Importantly, the automated pipeline reduced manual labeling effort by over 90 % and could be integrated into the production stack with configurable latency (the reranking step can be toggled).

Key contributions include:

• Formal definition of the user‑campaign purchase labeling problem, previously under‑explored in recommendation literature.
• A novel RAG architecture that couples LLM‑driven semantic interpretation with dense retrieval and structured LLM reasoning over a product taxonomy.
• Empirical evidence that the approach yields high‑quality, scalable labels suitable for downstream campaign ranking models, enabling campaign‑aware learning at production scale.

Future directions suggested by the authors involve extending the framework to multimodal campaigns (incorporating images or video), modeling label uncertainty probabilistically to feed directly into ranking loss functions, and building an online feedback loop that continuously refines PT mappings as new user interaction data arrives. Such extensions would further generalize the method across diverse marketing channels and product domains.