An Impact Evaluation Framework of Personalized News Aggregation and Recommendation Systems

No Thumbnail Available
File version
Author(s)
Zhao, Yunwei
Wang, Can
Han, Han
Shu, Min
Wang, Wenlei
Griffith University Author(s)
Primary Supervisor
Other Supervisors
Editor(s)
Date
2020
Size
File type(s)
Location

Melbourne, Australia

License
Abstract

There are mounting concerns that personalized recommenders contribute to population differentiation and polarization by creating “information cocoons” that insulate people from opposing views. Despite the vast amount of research devoting to design new algorithms to improve the performance of recommendation systems, the evaluation mostly focuses on recommendation quality. Little attention is paid to the social impact of recommendation systems. On the other hand, it is noted in the literature that the users of recommendation systems suffer from “partial information blindness”, i.e. only exposed to the like-minded content information cocoon or filter bubble. Therefore, in this paper, we propose an impact evaluation framework of recommendation systems, comprising of five constructs, namely, diversity, objectivity, information cocoon, affect cocoon, and intentionality. The evaluation is conducted in a “black-box” way based on the empirical interaction data between the recommender and its individual users. To enable the individual level evaluation, we propose a lightweight agent-based simulation to collect individual's interaction data with recommenders in the complex real-world scenarios. Furthermore, we illustrate the effectiveness of the proposed methodology with a case study in Toutiao and Baidu News, which are the top two news aggregator apps in China. The results show that both applications demonstrate a periodic information cocoon of about 7-day time interval, the news recommended through these two apps bear distinct differences: Toutiao has better captured the user's reading preference and the quality of the recommended articles, while Baidu News demonstrate a higher affect cocoon and intentionality in the recommended articles, with average perceived difficulty of the recommended articles 2.9% higher than Toutiao, and average perceived persuasiveness 9.6% higher than Toutiao.

Journal Title
Conference Title

2020 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT)

Book Title
Edition
Volume
Issue
Thesis Type
Degree Program
School
Publisher link
Patent number
Funder(s)
Grant identifier(s)
Rights Statement
Rights Statement
Item Access Status
Note
Access the data
Related item(s)
Subject

Information systems

Persistent link to this record
Citation

Zhao, Y; Wang, C; Han, H; Shu, M; Wang, W, An Impact Evaluation Framework of Personalized News Aggregation and Recommendation Systems, 2020 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), 2020, pp. 893-900