Designing Scalable and Extendable Data Pipeline for Call Of Duty Games
0 likes1,074 views
This document discusses scaling and extending the data pipeline for Call of Duty games. Some key points: 1) The data pipeline uses Apache Kafka topics partitioned for scalability, but too many topics creates operational overhead, so topic naming follows conventions expressing data types rather than producers/consumers. 2) A stream processing layer called "Refinery" enables measuring, validating, enriching, filtering and routing messages to make the pipeline flexible for new use cases. 3) A unified message envelope and Schema Registry supporting multiple formats allows easy integration of new games into the pipeline by standardizing the message format.
![// Application.java
props.put("value.deserializer", "com.example.CustomDeserializer");
// CustomDeserializer.java
public class CustomDeserializer implements Deserializer<???> {
@Override
public ??? deserialize(String topic, byte[] data) {
???
}
}
Custom deserialization](https://image.slidesharecdn.com/designing-scalable-and-extendable-data-pipeline-for-call-of-duty-games-180601190833/85/Designing-Scalable-and-Extendable-Data-Pipeline-for-Call-Of-Duty-Games-22-320.jpg)
Designing Scalable and Extendable Data Pipeline for Call Of Duty Games
- 1. Designing Scalable and Extendable Data Pipeline for Call of Duty Games Yaroslav Tkachenko Senior Data Engineer at Activision
- 4. 1+ PB Data lake size (AWS S3)
- 5. Number of topics in the biggest cluster (Apache Kafka) 600+
- 6. 10k+ Messages per second (Apache Kafka)
- 7. Scaling the data pipeline even further Volume Well-known industry techniques Games Using previous experience Use-cases Completely unpredictable Complexity
- 9. 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 Kafka topic Consumer or Producer Partition 1 Partition 2 Partition 3 Kafka topics are partitioned and replicated
- 10. We need to keep the number of topics and partitions low More topics means more operational burden. Number of partitions in a fixed cluster is not infinite. Autoscaling Kafka is impossible, scaling is hard.
- 11. Topic naming convention $env.$source.$title.$category-$version prod.glutton.1234.telemetry_match_event-v1 Unique game id “CoD WW2 on PSN”Producer
- 12. A proper solution has been invented decades ago. Think about databases.
- 13. Messaging system IS a form of a database Data topic = Database + Table. Data topic = Namespace + Data type.
- 15. Each approach has pros and cons • Topics that use metadata for their names are obviously easier to track and monitor (and even consume). • As a consumer, I can consume exactly what I want, instead of consuming a single large topic and extracting required values. • These dynamic fields can and will change. Producers (sources) and consumers will change. • Very efficient utilization of topics and partitions. • Finally, it’s impossible to enforce any constraints with a topic name. And you can always end up with dev data in prod topic and vice versa.
- 16. After removing necessary metadata from the topic names stream processing becomes mandatory.
- 17. Stream processing becomes mandatory Measuring → Validating → Enriching → Filtering & routing
- 18. Refinery
- 19. Having a single message schema for a topic is more than just a nice-to-have.
- 20. Number of supported message formats 8
- 21. Stream processor JSON Protobuf Custom Avro ? ? ? ?
- 22. // Application.java props.put("value.deserializer", "com.example.CustomDeserializer"); // CustomDeserializer.java public class CustomDeserializer implements Deserializer<???> { @Override public ??? deserialize(String topic, byte[] data) { ??? } } Custom deserialization
- 23. Message envelope anatomy ID, env, timestamp, source, game, ... Event Header / Metadata Body / Payload Message
- 24. Unified message envelope syntax = "proto2"; message MessageEnvelope { optional bytes message_id = 1; optional uint64 created_at = 2; optional uint64 ingested_at = 3; optional string source = 4; optional uint64 title_id = 5; optional string env = 6; optional UserInfo resource_owner = 7; optional SchemaInfo schema_info = 8; optional string message_name = 9; optional bytes message = 100; }
- 25. Schema Registry • API to manage message schemas • Single source of truth for all producers and consumers • It should be impossible to send a message to the pipeline without registering its schema in the Schema Registry! • Good Schema Registry supports immutability, versioning and basic validation • Activision uses custom Schema Registry implemented with Python and Cassandra
- 26. Summary: scaling and extending the data pipeline Games • By using unified message envelope and topic names adding a new game becomes almost effortless • “Operational” stream processing makes it possible • Still flexible enough: each game can use its own message payload format via Schema Registry Use-cases • Topic names express data types, not producers or consumers • Stream filtering & routing allows low-cost experiments • Data catalog built on top of Schema Registry promotes data discovery
- 27. Thanks! @sap1ens