26-28 November, 2019, Vilnius
Conference is over! See you next year.
Carlos Manuel Duclos-Vergara
Informatics Engineer. Worked for Microsoft in Redmond for the Unified Communications Team. Moved to Norway to join Trolltech. Currently works as Staff Engineer in Schibsted developing systems using Kafka as a backbone.
Routing Billions of Events a Day: How we Do Routing in Schibsted
In this talk Carlos will present how his team in Schibsted have set up a streaming data platform using cloud technologies, Kafka and Kafka streams. Schibsted is a leader in media and online classifieds in the Nordic market, delivering news to most of the population of Scandinavia. The streaming platform aims to simplify data management, providing a single point to manage data streams independently of data types. It includes a data quality solution, a data registration solution and a routing solution. Kafka is used as the backbone of the system and the routing solution is built on top of it. Kafka streams is used downstream to enrich streams. The data quality solution allows consumers and providers to register data quality checks and share metrics about the data that is being sent and received. The data registration solution allows providers to register the datasets owned by them, so consumers can find them. At the same time consumers can request access to the datasets and owners can grant access to those datasets, providing visibility and simplifying the sharing process. A key component of the streaming platform is the routing solution implemented using Kafka. This allows consumers of data to request data and adapt it to their own needs by applying filters and transformations, giving great flexibility.