[ email=giovannimaria.biancofiore@poliba.it, ]

[ email=vitowalter.anelli@poliba.it, ]

[ email=fedelucio.narducci@poliba.it, ]

[ email=tommaso.dinoia@poliba.it, ]

Introduction

Recommender systems (RSs) have long assisted users in discovering valuable information on the web by predicting their preferences and delivering personalized content. Over time, these systems have evolved from Matrix Factorization approaches to modern architectures that extend state-of-the-art Deep Learning models , originally developed for other domains such as time-series forecasting , natural language processing , and computer vision . Despite significant progress in improving accuracy, current research in the user modeling and personalization community increasingly emphasizes the importance of beyond-accuracy perspectives such as diversity, novelty, and popularity bias. These factors not only impact overall system effectiveness but also influence user satisfaction , long-term engagement , and fairness .

With the release of ChatGPT in November 2022 , Large Language Models (LLMs) have begun to reshape how recommendations can be delivered. Unlike traditional RSs that rely on carefully structured training data, LLMs can generate free-form text, potentially offering more nuanced explanations and broader item coverage by leveraging their vast knowledge. Consequently, the research community is now experimenting with LLM-driven recommendation pipelines , demonstrating notable successes in improving recommendation accuracy . However, most existing studies on ChatGPT-based recommender systems have emphasized improving accuracy while neglecting the beyond-accuracy dimensions that are critical for real-world impact .

Ignoring the beyond-accuracy behavior of ChatGPT creates a black box for researchers, making it difficult to determine whether it over-recommends popular items, reduces novelty, or offers less diverse recommendations, all of which may negatively impact user satisfaction and long-term personalization goals. Early investigations focused on how to use ChatGPT for re-ranking recommendations , while others began to study the serendipity of the generated recommendations  or explored how ChatGPT generates recommendations and whether its outputs align more closely with content-based or collaborative filtering approaches . Only investigates biases in ChatGPT-based recommender systems, with a specific focus on provider fairness. While a few works have begun examining potential biases related to sensitive attributes such as race, gender, and religion , aspects like recommendation diversity, novelty, and popularity bias in ChatGPT remain largely unexplored. Addressing these gaps is essential to ensure that personalization technologies are both effective and fair.

To this end, we analyze ChatGPT’s recommendation behavior, focusing on both ChatGPT-3.5 and ChatGPT-4 across multiple beyond-accuracy metrics. Specifically, we investigate whether ChatGPT generates diverse and novel recommendations or exhibits popularity bias, both under normal conditions and in user cold-start scenarios where users have interacted with only a few items. Our evaluation spans three distinct domains, Books, Movies, and Music, using the Facebook Books , MovieLens , and Last.FM  datasets as benchmarks, aiming to answer the following Research Questions (RQs):

1. Are ChatGPT’s recommendations diverse?

2. Are ChatGPT’s recommendations novel?

3. Is ChatGPT affected by popularity bias?

4. How effective is ChatGPT in user cold-start scenario across accuracy and beyond-accuracy dimensions?

Related Work

Diversity, Novelty, and Popularity Bias in Recommender Systems. Driven by the need for Recommender Systems (RSs) to enhance user engagement , this work focuses on beyond-accuracy measures of RSs, namely, diversity, novelty, and popularity bias, to investigate how these factors affect the recommendation lists provided by ChatGPT.

There was a moment in the evolution of RSs when researchers realized that evaluating recommendations solely based on accuracy metrics was insufficient. For instance, suggest that the performance of recommendations should be measured by their usefulness to the user. Similarly , in their survey on the evaluation of RSs, suggest that recommendations can be evaluated based on utility, novelty, diversity, unexpectedness, serendipity, and coverage. Karimi, Jannach, and Jugovac , in their review of state-of-the-art news RSs, identify diversity, novelty, and popularity as the most common quality factors for improving recommendations. Specifically, diversity and novelty are often considered quality factors that must be balanced with prediction accuracy , and the most discussed beyond-accuracy objectives in recommender system research .

As interest in studying RSs beyond accuracy metrics spread, more studies began to use these metrics as goals for improvement. For example, and focused on creating RSs that not only predict accurate items but also achieve a high level of diversity in recommendations. employed a graph-based approach to identify items with higher novelty, while proposed a method to mitigate popularity bias. Furthermore, the work by emphasizes the importance of evaluating RSs beyond accuracy, proposing a multi-objective evaluation approach.

Although many works use beyond-accuracy metrics to evaluate and improve RSs, prior literature lacks a unified framework that rigorously defines diversity, novelty, and bias, leading to vagueness and overlap among these measures. In our study, we define these concepts as follows: Diversity is the extent to which a recommender system suggests a wide range of items from the catalog ¹, as supported by and . Novelty is the degree to which recommended items expose users to relevant experiences they are unlikely to discover independently, based on . Popularity Bias refers to the tendency of recommender systems to favor popular items, those with many interactions, over less popular or niche items, aligning with and .
ChatGPT-based recommendation. A first example of ChatGPT for recommendation is proposed by , who introduced ChatREC, a ChatGPT-augmented recommender system that translates the recommendation task into an interactive conversation with users. The authors proposed a prompt template to convert user information and user-item interactions into a query for ChatGPT. However, the system was evaluated solely using accuracy metrics (i.e., Recall, Precision, nDCG). investigates ChatGPT’s performance in a multi-turn conversational recommendation setting, demonstrating its potential as a conversational recommender and showing that it outperforms traditional methods. focused their work on ChatGPT in zero-shot settings. They analyzed ChatGPT models by designing a dedicated prompting template and revealed that ChatGPT-4 achieved the highest ranking performance compared to other LLMs in the zero-shot recommendation task.

investigated the abilities of ChatGPT as a recommender systems for the Top-N recommendation task, aiming to identify the most effective prompting strategy for producing relevant recommendations. The authors concluded that the zero-shot setting yields the most relevant recommendation list, outperforming content-based baselines. However, their conclusions were based solely on nDCG as the evaluation metric, which limits the findings to only one dimension of RSs.

investigate ChatGPT’s abilities in suggesting items through rating prediction, pairwise recommendation, and re-ranking strategies using the prompting approach. Their experiments, conducted on four domains, demonstrate ChatGPT’s abilities to recommend items. Nonetheless, this study provide only an accuracy view of ChatGPTs’ capabilities in the Top-N recommendation.

focus on applying ChatGPT within the book recommendation scenario, designing BookGPT to address single-item and rating prediction tasks. However, the study does not provide a generalizable analysis of ChatGPT’s performance across multiple domains, as the authors focus only on the book domain.

Although all the presented works focus on using ChatGPT to improve the performance of recommender systems, they are primarily based on accuracy metrics. To address this gap, our work investigates the task of Top-N recommendation, moving beyond accuracy by evaluating ChatGPT’s performance in terms of diversity, novelty, and popularity bias, while also highlighting its beyond-accuracy capabilities in user cold-start scenarios.

Methodology

The following sections discuss the methodology used in our research, outline the design of the prompts employed to collect recommendations from ChatGPT, detail the datasets used in the experiments, present the baselines for comparison, and list the metrics used to assess diversity, novelty, and popularity bias.

Prompt Design

The introduction of GPT-3  demonstrated the ability of LLMs to perform diverse tasks when provided with clear, task-specific prompts, showing how prompts condition the model’s response and play a critical role in shaping its performance on a given task .

With the widespread diffusion of ChatGPT, the literature on prompt engineering has expanded, moving from basic prompts such as zero- and few-shot  to more complex prompts like Chain-of-Thought , Tree-of-Thoughts , Reflexion , or Graph-Prompting . Among the various prompt techniques , we hand-engineered Zero-Shot, Few-Shot, Chain-of-Thought, and Role-Playing (RP) prompting following the works of and , to identify the best approach for our investigation.

In the following, we present the hand-engineered prompts and explain the main reasons for selecting RP prompting as the primary technique for our investigation. Specifically, for all the tested prompts and for each user, the input consists of the user’s history, presented as a list of items formatted as follows: $`\{History\ of\ the\ User\}: Item_1, Item_2, \ldots, Item_{N}`$.

Zero-shot prompting . In zero-shot prompting, we directly provided the user’s history to ChatGPT and asked for 50 recommendations, as shown in the reference example (see fig. 1). However, $`\sim`$71% of the generated lists contained fewer than 50 items or included repeated entries, and $`\sim`$6% exhibited incorrect task execution.

Few-shot prompting . After zero-shot prompting, we tested few-shot prompting by providing a few demonstrations of recommendations to help the LLM better understand the task (see Fig. 2). While these contextual examples reduced execution errors, $`\sim`$44% of the generated lists contained duplicate items.

Chain-of-Thought (CoT) prompting . Using CoT, we attempted to break the recommendation task into explicit steps to force ChatGPT to reason step-by-step. As shown in Fig. 3, we explicitly defined the instructions, the user’s preferences, and the steps to identify the most suitable recommendations. This approach produced excessive tokens, reaching the context limit after generating $`\sim`$26 items.

Role-Playing prompting . Following the work of and , we also tested Role-Playing prompts, where ChatGPT impersonates a Recommender System and recommends items based on the user’s history (see Fig. 4). This strategy proved the most effective, eliminating duplicate recommendations.

After testing 30 hand-crafted prompts and aligning with studies on Role-Playing Prompting , we selected this approach for its ability to reduce duplicates and token usage. In this setup, ChatGPT acts as a Recommender System, generating 50 recommendations based on the user’s history (see Fig. 4).

Experimental Setup

This section outlines the experimental setup, including the datasets, baselines, and metrics used to assess the beyond-accuracy performance of ChatGPT’s recommendations, with a focus on diversity, novelty, and popularity bias.

Datasets. We evaluated ChatGPT on three well-known recommendation datasets, namely MovieLens100k , Last.FM , and Facebook Books ². To enhance data quality, we applied an iterative 10-core filtering strategy , retaining only users and items with at least ten interactions. Table 1 holds the dataset statistics after preprocessing.

Baseline Models. To measure the effectiveness of ChatGPT, we experimentally compare its performance with state-of-the-art baselines from three categories: Non-Personalized, Collaborative Filtering, and Content-Based Filtering methods. To ensure a fair comparison, we train the baselines and optimize their hyperparameters using the Elliot framework , and split the dataset into 80% training and 20% test sets, following the all unrated items evaluation protocol  . The code used for the experiments is publicly available at: https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt . Below, we describe the baselines, grouped by recommendation category. Non-Personalized. Random and Most Popular return random recommendations and the most popular recommendations, respectively, and are used as reference points. Collaborative Filtering. To compare the effect of ChatGPT recommendations on beyond-accuracy metrics, we selected the following collaborative filtering methods, each focusing on different aspects. Specifically, we selected $`\textrm{RP}^3_{\beta}`$  and LightGCN  for their demonstrated ability to maintain accuracy while preserving diversity . ItemKNN , UserKNN , and EASE$`^R`$  were chosen for their emphasis on relevance and personalization . Finally, MF2020  and NeuMF  were included as a tradeoff between model complexity and effectiveness. Content-Based Filtering. We further extend our comparison by including content-based models, which prioritize explicit feature representations and offer a meaningful contrast to collaborative models. This allows us to evaluate ChatGPT against the most appropriate model type for the dataset. Specifically, we include VSM  which represents items as vectors in a high-dimensional space, with each dimension corresponding to a feature, as well as AttributeItemKNN and AttributeUserKNN , which rely on TF-IDF-weighted attribute vectors to compute similarities and generate recommendations.
Ensuring Recommendation Consistency. ChatGPT models generate recommendations based solely on the user profile provided in the prompt, without being constrained to a predefined dataset. As a result, they may hallucinate  or suggest real items not present in the reference dataset, leading to discrepancies in item names and inconsistencies in evaluation.

To address this, we adopt a post-processing pipeline that uses Gestalt pattern matching  to identify the closest match in the dataset, accepting items with a similarity score above 90% (empirically determined). Unmatched items are flagged as External Items, originating from the LLM’s pre-trained knowledge, and excluded from evaluation to ensure a fair comparison with traditional recommenders by selecting in-catalogue items.

Since this final step could affect our evaluations, we verified that out-of-catalogue items consistently appeared beyond the top-10 positions in all recommendation lists, ensuring that rank-sensitive metrics remain unaffected and preserving the validity of our evaluation. In our configuration, ChatGPT placed these items only after rank 23, suggesting 2,740 out-of-catalogue items for Books, 870 for Music, and 234 for Movies.

Finally, to ensure a fair comparison, we evaluate all models and ChatGPT results at a cutoff of 10 (i.e., Top-10 recommendations per user), following widely accepted practices in recommendation .
Evaluation Metrics. While our primary focus is on the beyond-accuracy aspects of ChatGPT’s recommendations, it is also important to include accuracy metrics to assess whether the recommendations are relevant to users. For this purpose, we use two standard metrics: Precision and Recall . Higher values of Precision and Recall indicate that the recommender system provides a greater number of relevant items. Additionally, we evaluate the ranking quality of the recommendations using the Normalized Discounted Cumulative Gain (nDCG) , where higher values indicate better recommendation lists.

For beyond-accuracy metrics, we selected a set of measures to evaluate diversity, novelty, and popularity bias. The specific metrics considered are detailed in Table [tab:beyond_accuracy_table].

Experimental Results

ChatGPT Beyond-Accuracy Recommendation Performance

In this section, we discuss the empirical findings from Table [tab:combined_all], focusing on (RQ1.) the diversity of ChatGPT’s recommendations, (RQ2.) their novelty, and (RQ3.) the extent to which ChatGPT is affected by popularity bias. The evaluation comprises three datasets, Facebook Books, Last.FM, and MovieLens, and compares ChatGPT‑3.5 and ChatGPT‑4 against both Collaborative Filtering and Content‑Based Filtering baselines. Statistically significant differences (paired t‑tests at $`p<0.05`$) are noted where indicated in the table.
Preliminary Accuracy Analysis. Before examining diversity, novelty, and popularity bias, we first verify that ChatGPT’s recommendations fulfill the primary goal of offering relevant items. We use nDCG, Recall, and Precision as standard accuracy metrics. Higher values on these metrics imply better recommendation.

Overall, ChatGPT demonstrates a comparable level of accuracy in recommendation scenarios. Specifically, on Facebook Books, ChatGPT-4 attains the highest nDCG overall (0.0932), significantly outperforming the best baseline, AttributeItemKNN (0.0479), as well as ChatGPT-3.5 (0.0668). Recall and Precision follow a similar pattern to nDCG.

For Last.FM, while ChatGPT-4 (nDCG = 0.2832) does not surpass the best Collaborative Filtering (CF) approach ($`\textrm{RP}^3_{\beta}`$: 0.3147), it still ranks among the top-performing algorithms. ChatGPT-3.5 trails behind ChatGPT-4 but still outperforms some baselines (e.g., EASE$`^R`$, AttributeItemKNN).

For MovieLens, although ChatGPT-4 improves upon ChatGPT-3.5 across all accuracy metrics, raising nDCG from 0.1475 to 0.1815 and Precision from 0.1120 to 0.1551, certain CF algorithms (e.g., $`\textrm{RP}^3_{\beta}`$: 0.2827 nDCG, 0.2708 Precision) achieve significantly higher scores. Nonetheless, ChatGPT’s accuracy levels comfortably exceed those of some methods, such as VSM (0.0174 nDCG) and AttributeItemKNN (0.0326 nDCG).

These results demonstrate that both ChatGPT-3.5 and ChatGPT-4 achieve valid and reasonable performance on accuracy metrics. This preliminary evaluation ensures that the subsequent analysis of diversity, novelty, and popularity bias is based on recommendations that already meet the accuracy standard. In the following sections, our analysis is divided according to the research questions (RQs).
(RQ1.) Are ChatGPT’s recommendations diverse? We assess diversity using Gini and Item Coverage (ItemCV). A lower Gini indicates a higher concentration toward certain items, while higher coverage values indicate that more items from the catalog are recommended.

Facebook Books (Table [tab:combined_all]). ChatGPT-4 achieves a Gini of 0.1050 and an ItemCV of 1,004, outperforming ChatGPT-3.5 (Gini = 0.0713, ItemCV = 853) on both metrics. Although several baselines, such as ItemKNN (Gini = 0.5293, ItemCV = 2,141), still yield a better diversity score, both ChatGPT-4 and ChatGPT-3.5 generally rank above baselines such as MostPop and EASE$`^R`$. In terms of item coverage and Gini, ChatGPT models demonstrate a high concentration on specific items while covering nearly half of the total span (1,004 out of 2,234 items).

Last.FM (Table [tab:combined_all]). A similar trend emerges: ChatGPT-4 has a higher Gini (0.2023) than ChatGPT-3.5 (0.1927), indicating a lower concentration of recommendations on specific items. Additionally, GPT-4 covers 944 out of 1,507 items, whereas $`\textrm{RP}^3_{\beta}`$, which is designed to trade off diversity and accuracy, achieves a coverage value of 831 and a Gini of 0.1441, demonstrating ChatGPT’s strong ability to recommend diverse items.

MovieLens (Table [tab:combined_all]). ChatGPT-4 achieves a Gini of 0.0853, a slight improvement over ChatGPT-3.5 (0.0851). However, its item coverage spans 553 out of 1,862 items, which is comparatively lower than approaches such as $`\textrm{RP}^3_{\beta}`$(Gini = 0.1230, ItemCV = 744). These results highlight that, although the diversity score is lower than certain baselines, ChatGPT still presents a comparable diversity score on this dataset.
Summary (RQ1). ChatGPT’s recommendations are moderately diverse for Facebook Books and Last.FM, while exhibiting limited diversity on MovieLens, with GPT‑4 consistently outperforming GPT‑3.5. Although it does not match the highest-diversity baselines, it shows superior diversity compared to some CF and CBF approaches.
(RQ2.) Are ChatGPT’s recommendations novel? Novelty is measured via EPC (Expected Popularity Complement) and EFD (Expected Free Discovery), both interpreted such that higher values imply more novel recommendations.

Facebook Books (Table [tab:combined_all]). ChatGPT‑4 exhibits relatively high novelty (EPC=0.0353, EFD=0.3486), exceeding most baselines, including ChatGPT‑3.5 (EPC=0.0250, EFD=0.2480), and even surpassing all CF and CBF algorithms on these metrics.

Last.FM (Table [tab:combined_all]). Both ChatGPT versions rank above average in EPC and EFD, with CF and CBF methods (e.g., $`\textrm{RP}^3_{\beta}`$: EPC=0.2110, EFD=1.9970, VSM: EPC=0.1593, EFD=1.4845) performing at a comparable level. Still, the difference between ChatGPT‑4 (0.1918 EPC, 1.8663 EFD) and ChatGPT‑3.5 (0.1680 EPC, 1.6436 EFD) suggests GPT‑4 more effectively recommends less mainstream items.

MovieLens (Table [tab:combined_all]). On MovieLens, ChatGPT-4 (0.1453 EPC, 1.6010 EFD) outperforms ChatGPT-3.5 (0.1260 EPC, 1.3981 EFD) in terms of EPC and EFD values and places it on par with other methods (e.g., NeuMF: 0.1171 EPC, 1.2767 EFD), although lower than $`\textrm{RP}^3_{\beta}`$(EPC of 0.2421, EFD of 2.6613), the best model.
Summary (RQ2). ChatGPT’s recommendations exhibit above-average novelty in MovieLens and high novelty in Facebook Books and Last.FM, with GPT-4 generally surpassing GPT-3.5. The results suggest that ChatGPT, based on the user’s history, also recommends novel items for each user.
(RQ3.) Is ChatGPT affected by popularity bias? We examine popularity bias using APLT (Average Popularity of Long-Tail Items; higher indicates stronger inclination toward long-tail (less popular) items) and ARP (Average Rating-based Popularity; lower values imply less popularity bias).

Facebook Books (Table [tab:combined_all]). ChatGPT-3.5’s recommendations yield APLT = 0.1870 and ARP = 46, while ChatGPT-4 improves to APLT = 0.2424 and ARP = 40. With a higher APLT and lower ARP, GPT-4 demonstrates a better capability for recommending long-tail and less popular items than GPT-3.5. Although both models remain far from pure MostPop methods (ARP = 138), some baselines, such as AttributeItemKNN (APLT = 0.5879, ARP = 7) and VSM (APLT = 0.5761, ARP = 7), achieve better APLT and ARP values.

Last.FM (Table [tab:combined_all]). ChatGPT-3.5 has an APLT of 0.1391 and an ARP of 99, while GPT-4 has an APLT of 0.1267 and an ARP of 102, positioning it in the mid-range of models. This suggests that GPT-4 covers a smaller percentage of the long tail and tends to recommend more popular items. Although it outperforms certain baselines, such as $`\textrm{RP}^3_{\beta}`$(APLT = 0.0678, ARP = 153), it does not perform as well as other baselines, such as AttributeItemKNN (APLT = 0.3043, ARP = 87.8647).

MovieLens (Table [tab:combined_all]). ChatGPT shows an ARP of 90 for GPT-3.5 and 95 for GPT-4, which is lower than MostPop (ARP = 182) but higher than some graph-based methods (e.g., LightGCN: ARP = 43) or neighbor-based methods (e.g., AttributeItemKNN: ARP = 23). This indicates that its behavior is not as popularity-driven as MostPop but is still influenced by popular items. A similar trend is observed for APLT, further demonstrating that ChatGPT does not recommend items from the long tail and exhibits a degree of popularity bias.
Summary (RQ3). Although ChatGPT’s values are far from those obtained by MostPop, it still exhibits a tendency to recommend popular items, neglecting items in the long tail. In particular, GPT-4 demonstrates a lower ARP than ChatGPT-3.5, suggesting a tendency to recommend less popular items.
To conclude, ChatGPT models exhibit strong beyond-accuracy performance, achieving an optimal balance of novelty and diversity in the books domain, comparable results in the music domain, and suboptimal outcomes in the movie domain. Although it shows some inclination toward popular items, this bias is far less pronounced compared to MostPop or other strongly popularity-biased baselines. Furthermore, the improvements observed from GPT-3.5 to GPT-4 across all three datasets highlight the strength of GPT-4 for recommendations, particularly in balancing beyond-accuracy trade-offs.

These findings underscore the potential of ChatGPT as a recommender system while also highlighting areas for improvement, particularly in refining its ability to balance relevance, diversity, and novelty across domains.

User Cold-Start Scenario

We now examine user cold‐start recommendations, defined here as scenarios where each user has provided a maximum of ten interactions. Table [tab:coldstart_10] details these results across three datasets, Facebook Books, Last.FM, and MovieLens, comparing ChatGPT‐3.5 and ChatGPT‐4 to strong Collaborative Filtering (CF) and Content‐Based Filtering (CBF) baselines. Our central question is:
RQ4: How effective is ChatGPT in user cold-start scenario across accuracy and beyond-accuracy dimensions?

Accuracy under Cold‐Start. Despite limited user interactions, ChatGPT exhibits competitive to superior accuracy compared to traditional baselines. For Facebook Books, GPT-4 achieves higher nDCG (0.0538) and Recall (0.0873) than all baselines, including $`\textrm{RP}^3_{\beta}`$ (nDCG = 0.0346) and AttributeItemKNN (0.0335). GPT-3.5 also surpasses these baselines but is slightly behind GPT-4. For Last.FM, ChatGPT maintains robust performance ($`nDCG\geq0.2791`$, $`Recall\geq0.3423`$), outperforming MostPop (nDCG = 0.0529) and random baselines by a wide margin. Although $`\textrm{RP}^3_{\beta}`$ leads in nDCG (0.2389), GPT-4 often excels in Recall and Precision. For MovieLens, GPT-4 attains the highest nDCG (0.1405), surpassing both CF and CBF baselines, while ChatGPT-3.5 (0.1117) also remains competitive. These results underscore ChatGPT’s capacity to identify relevant items effectively from few interactions.

Beyond‐Accuracy in Cold‐Start.

Diversity. GPT‑4 generally surpasses GPT‑3.5 in Gini and item coverage across all three datasets (e.g., increasing from 0.0538 to 0.0846 in Gini on Facebook Books), indicating that GPT‑4’s recommendations span a broader set of items. Although baselines like $`\textrm{RP}^3_{\beta}`$achieve higher coverage in MovieLens and Facebook Books, GPT-4 performs best on Last.FM.

Novelty. ChatGPT’s EPC and EFD values exceed those of CF and CBF baselines across all datasets (e.g., GPT‑4’s EPC = 0.0186 vs. $`\textrm{RP}^3_{\beta}`$= 0.0115 on Facebook Books), implying a tendency to recommend novel items rather than relying on mainstream items.

Popularity Bias. ChatGPT exhibits a moderate inclination toward popular items compared to baselines across all datasets. Nonetheless, it remains far from MostPop (e.g., $`ARP\geq139`$ on Facebook Books) but is comparable to some baselines (e.g., AttributeUserKNN), indicating room for further mitigation strategies.
In summary, ChatGPT proves highly effective in cold-start scenarios by: (i) maintaining strong accuracy despite minimal user interactions, with GPT-4 often outperforming GPT-3.5; (ii) striking a balance among diversity, novelty, and popularity bias; (iii) demonstrating consistent improvements over baselines, underscoring ChatGPT’s capacity to infer user interests with limited interactions.

Conclusion

In this work, we explore the diversity, novelty, and popularity bias of ChatGPT recommendations. Our findings demonstrate that for the Facebook Books, Last.FM, and MovieLens datasets, ChatGPT models exhibit strong beyond-accuracy performance, achieving an optimal balance of novelty and diversity in Facebook Books, comparable results for Last.FM, and suboptimal outcomes for MovieLens.

Additionally, we show that while ChatGPT demonstrates a good balance between novelty and diversity, it also exhibits a tendency to recommend popular items, especially in the MovieLens dataset.

Finally, we extend our exploration to the user cold-start scenario, where ChatGPT proves highly effective by maintaining strong accuracy despite minimal user interactions, balancing diversity, novelty, and popularity bias, and demonstrating consistent improvements over baselines.

These findings underscore the beyond-accuracy capabilities of ChatGPT as a recommender system. Future research will include additional datasets to generalize the findings across domains, as well as experiments comparing ChatGPT with other LLMs such as Gemini, LLaMA, and DeepSeek.

Limitation

Nowadays, LLMs are used to augment the capabilities of recommender systems. However, these models are typically trained on vast internet-scale corpora, which may include portions of open datasets used for benchmarking. Recent work studying memorization in MovieLens-1M  shows that models like GPTs and LLaMA-3 can memorize such datasets, with larger models exhibiting higher memorization rates. For example, the reported memorization rate is 12.9% for LLaMA-3.1 405B and 80.76% for GPT-4. Further research should focus on understanding the correlation between improvements in recommendation quality and memorization capacity.