LLM-Generated Negative News Headlines Dataset: Creation and Benchmarking Against Real Journalism

📝 Abstract

This research examines the potential of datasets generated by Large Language Models (LLMs) to support Natural Language Processing (NLP) tasks, aiming to overcome challenges related to data acquisition and privacy concerns associated with real-world data. Focusing on negative valence text, a critical component of sentiment analysis, we explore the use of LLM-generated synthetic news headlines as an alternative to real-world data. A specialized corpus of negative news headlines was created using tailored prompts to capture diverse negative sentiments across various societal domains. The synthetic headlines were validated by expert review and further analyzed in embedding space to assess their alignment with real-world negative news in terms of content, tone, length, and style. Key metrics such as correlation with real headlines, perplexity, coherence, and realism were evaluated. The synthetic dataset was benchmarked against two sets of real news headlines using evaluations including the Comparative Perplexity Test, Comparative Readability Test, Comparative POS Profiling, BERTScore, and Comparative Semantic Similarity. Results show the generated headlines match real headlines with the only marked divergence being in the proper noun score of the POS profile test.

💡 Analysis

This research examines the potential of datasets generated by Large Language Models (LLMs) to support Natural Language Processing (NLP) tasks, aiming to overcome challenges related to data acquisition and privacy concerns associated with real-world data. Focusing on negative valence text, a critical component of sentiment analysis, we explore the use of LLM-generated synthetic news headlines as an alternative to real-world data. A specialized corpus of negative news headlines was created using tailored prompts to capture diverse negative sentiments across various societal domains. The synthetic headlines were validated by expert review and further analyzed in embedding space to assess their alignment with real-world negative news in terms of content, tone, length, and style. Key metrics such as correlation with real headlines, perplexity, coherence, and realism were evaluated. The synthetic dataset was benchmarked against two sets of real news headlines using evaluations including the Comparative Perplexity Test, Comparative Readability Test, Comparative POS Profiling, BERTScore, and Comparative Semantic Similarity. Results show the generated headlines match real headlines with the only marked divergence being in the proper noun score of the POS profile test.

📄 Content

1

LLM-Generated Negative News Headlines Dataset: Creation and Benchmarking Against Real Journalism

Olusola Babalola1*, Bolanle Ojokoh2, and Olutayo Boyinbode3

1. Department of Computer Science and Mathematics, Elizade University, Wuraola Ade-Ojo Avenue, P.M.B. 002, Ilara-Mokin, Ondo State, Nigeria; e-mail: olusola.babalola@elizadeuniversity.edu.ng
2. Department of Information Systems, Federal University of Technology, P.M.B. 704 Akure, Ondo State, Nigeria; e-mail: baojokoh@futa.edu.ng
3. Department of Information Technology, Federal University of Technology, P.M.B. 704 Akure, Ondo State, Nigeria; e-mail: okboyinbode@futa.edu.ng

Abstract This research examines the potential of datasets generated by Large Language Models (LLMs) to support Natural Language Processing (NLP) tasks, aiming to overcome challenges related to data acquisition and privacy concerns associated with real-world data. Focusing on negative valence text, a critical component of sentiment analysis, we explore the use of LLM-generated synthetic news headlines as an alternative to real-world data. A specialized corpus of negative news headlines was created using tailored prompts to capture diverse negative sentiments across various societal domains. The synthetic headlines were validated by expert review and further analyzed in embedding space to assess their alignment with real-world negative news in terms of content, tone, length, and style. Key metrics such as correlation with real headlines, perplexity, coherence, and realism were evaluated. The synthetic dataset was benchmarked against two sets of real news headlines using evaluations including the Comparative Perplexity Test, Comparative Readability Test, Comparative POS Profiling, BERTScore, and Comparative Semantic Similarity. Results show the generated headlines match real headlines with the only marked divergence being in the proper noun score of the POS profile test. Keywords: Synthetic data generation, Large Language Models (LLMs), Negative sentiment analysis, News headline synthesis, News content analysis

2

1.0. Introduction Our experiments aim to provide data and insights into Large Language Model (LLM)-generated negative content. We first generate synthetic news headlines using a chosen LLM and across specific and general categories; we assess how well each headline bears negative sentiment and analyze how negativity varies across different news categories; finally, we examine patterns and characteristics in the LLM-generated negative headlines. The experiments are expected to produce several useful results, including comparing how different sentiment analysis tools perform on negative news headlines, understanding the strengths and weaknesses of various sentiment analysis methods across different news categories, seeing how well current sentiment analysis tools detect negative sentiment in AI-generated content, finding any biases or patterns in how LLMs create negative news, and showing how these findings might affect the use of these tools in real-world applications, especially for news monitoring and analysis. News headlines play a crucial role in shaping public opinion and perception, and a lot of utility is being derived from the ability to use computers to accurately detect negative sentiment in news. Having a broader understanding of these sentiments as well is of great importance to various stakeholders, including media analysts, policymakers, and researchers. Using LLM-generated content as dataset, we position this research at the intersection of two rapidly evolving fields: sentiment analysis and large language models. This approach not only provides a controlled environment for testing sentiment analysis tools but also offers insights into the nature of AI-generated news content, a topic of increasing relevance as AI systems become more integrated into content creation pipelines. By focusing on negative sentiment detection in news headlines, this research addresses an important area of sentiment analysis with significant real-world implications. The findings from this research have the potential to inform the development and refinement of sentiment analysis tools, particularly for applications in news analysis. The insights gained about LLM- generated negative content could contribute to ongoing discussions about AI ethics, media literacy, and the potential impacts of AI-generated news on public discourse. We proceed to discuss negativity in news, synthetic datasets, and LLMs.
2.0 Related work Synthetic data, which is simply data generated by algorithms that is similar in every way to real world data but without direct derivation from actual, or naturally occurring objects or entities. They are therefore purely the creation of models which have some knowledge of the world the data is being 3

generated for. The primary use of synthetic data is for the training of machine learning systems. The li

This content is AI-processed based on ArXiv data.