Commercial providers of information access systems (such as Amazon or Google) usually evaluate the performance of their algorithms by observing how large numbers of their customers interact with different instances of their services. Unfortunately, due to the lack of access to users, university-based research is struggling to catch up with this large-scale evaluation methodology. NewsREEL, short for News Recommendation Evaluation Lab, aims to bridge this “evaluation gap” between Academia and Industry.
NewsREEL is organised as a campaign-style evaluation lab of CLEF 2017 and addresses the following information access task:
Whenever a visitor of an online news portal reads a news article on their side, the task is to recommend other news articles that the user might be interested in.
NewsREEL offers two tasks to study this use case. The first task implements the idea of a “living lab” where the provider of a recommendation service provides access to its infrastructure and user base. The second task replays a live setting using the recommender systems reference framework Idomaar.
By providing this service for millions of users, the recommendation scenario requires solutions for significant research challenges, such as processing information in real-time, handling of vast amount of data, and providing suitable recommendations. By providing access to the infrastructure of a company, we offer professionals and students the opportunity to develop skills that are in high demand in industry, while at the same time familiarizing themselves with the academic practice of evaluation of information access systems.