Databus - LinkedIn's Change Data Capture Pipeline

Editor's Notes

• #4: Large scale storage systems, by nature, are distributed. Data is stored in multiple machines. Also, it is often the case that it is the same data (I mean, same content) that is stored in different places to support different access patterns, efficient retrieval, or for quick look up of derived data (as opposed to computing them during a look-up). In order to have such distributed systems work together to provide a service, two things are needed: CLICKData flow between these systems: When data is updated in one side on one system, it should be reflected in the other parts that store the same content. CLICKData consistency: The different parts of the system must converge at some point in time.So, we need a change capture system that supports such a distributed system.
• #5: Basically there are two ways changes can be captured. Applications can dual-write data into the database and the change streamWe capture the data change list from the commit logs that most databases have.CLICKDual writes appear really easy on the surface. But then when we start considering the transient failure scenarios, achieving consistency gets harder, sometimes impossible. You may need to get into 2P commits, etc. compromising on performance, or availability.CLICKOn the other hand, extracting changes from the database is almost like post-processing. Minimal or no performance penalty, application is unaware of the existence of a change capture system, and consistency is not an issue as long as you see the commit logs in entirety. No such thing as a free lunch, of course. Commit logs formats are proprietary, and extracting from them can be tough.We chose to take this approach of change extract.
• #6: Here are some of the use cases for Databus. In this picture, we have one source of truth for data – the primary database. Changes to the data are observed (or, consumed) by consumers, which may the turn around and update derived data serving systems. Or, data may be extracted into Hadoop (for example), to be re-loaded into some derived systems. At Linked in, we our primary database is in Oracle . For example, we have a database that holds member information. When rows in these databases are altered, the change events need to be propagated to a search index. The search index is used to serve queries from recruiters looking for appropriate candidates. Similarly there are other consumers of different databases, each having their own business logic to build derived data out of the primary database.Databus is used to capture these changes and provide them as change events to consumers.So, what would be the requirements of such a system?
• #7: … Let’s look at a brief biography of Databus
• #8: … Now for a close look at Databus
• #9: Use of ‘change’ in ‘change data’ more of a noun – rather than a verb – ‘data that has changed’ rather than alrer the data. – Industry std term.Change capture logic (CDC ) to extract changes – in a consistent way (preserving the consistency, ordering -> e.g. ways to extract order of commits, I D semantics)Publisher and Subscriber APIs that lets the CDC transform he extracted changes and publish those events with atomicity guarantees of the source... Applications preserved consistency when they applied the changes they received in a timely manner… and there were realities
• #10: Different type of applicationsSchemas evolving at sourceButSource cannot be burdened (a typical problem with V1 )Applications cannot be forced to move to latest version ( resulted in proliferation of different versions of change streams of same source)
• #11: External clock attached to source. Ordering defined by the source- e.g. commit ordering in Oracle -> increasing SCN – in mysqlbinlog increasing txns could be the scnsNo additional source of truth, no additional point of failureAbility to recreate event stream given SCN and sourceFor applications ; the ordering of events is same as that seen by the source – so eventually the source and apps will converge , SCN is used to track progress on the app, Apps can reason about consistency with source- as external clock SCN is used , SCN is logical – not tied to any particular change stream node, Apps need to be idempotent – as they can see a change more than once .Apps can reason about consistency – derived stores can reason amongst each other as they have SCN visibility – a concept that is useful to compare consistency across applicationsTimeline consistency: at least once guarantee – order of change events same as source db; no updates missed. SCN; all apps listening to the change stream see the same order of change events
• #12: Pul model - > as opposed to push – where producers keep track of their consumer progress- and call clients as long as they are available, pull model assumes the state required to servr a request lies with the consumer. Restartability is easier as state can be computed from source,SCN on any machine. This Is true at both change event stream and the consumer.Separation of concerns – between use cases of ‘online consumption’ – recent changes and ‘catchup/ bootstrap ‘ case – where older changes are required – different scalability properties IsolIsolate Sources and consumers – source can move , schemas can change, and of couse, producer speed and consumption speeds can vastly differWe are not just transport – we support meta data – such as schemas. We ensure that consumers have a good experience – while the change stream also becomes more managable – ultimately helping provisioning and consumer robustness.Also gives an option of adding more filtering options at the change stream.
• #13: Point: Change Capture is within the relay: each relay is self-sufficient, i.e. since eventBufferState = fn(source,SCN) – it has change capture logic to pull in the changes – if change capture were outside, then the change capture logic ,fan out has to take care of replication or write to a leader-follower relay cluster. EventBuffer wraps around if it runs out of memory.Point: Client Library: fetches changes from the databus stream (which is now Relay and Bootstrap Service) - . Point: Workload separation : cases – recent change, to older changes, snapshots – cannot rely solely on all changes fitting into memoryPoint: Bootstrap Consumer: special application that listens to changes and updates it’s log store (persistent change events) and snapshot store – (persistent copy of the database – stores change events in user-space)Point: Remember , client library automatically switches between appropriate service – relay or bootstrap depending on SCN requested by application - Point: Meta data: used by relay stream (schema awareness) Point: DB’s are saved from abusive scans from lagging consumers , (isolation) Counterpoints:Push-model -> addl state of consumers; or speed of consumption / speed of production have to match ; harder to maintain lossless guarantees;
• #14: Are all of these open-sourced? Oracle is.
• #15: Custom mysqlrepl /mysql slave instance– the Specifically custom storage engine of slave writes to tcp channel instead of disk – Slave state has an SCN (offset , log number) – that can be controlled – upto 3 days worth of rewindability (configurable)
• #16: Control Flow is depicted:Note the pull-model - .> at the CDC end, it’s easy to make data portable – the SCN ,Source is sufficient to re-create state in EventBuffer; easier restarts.; no state requires to be maintained on upstream system about it’s subscribers (as in case of push model).- Publish does not require persistence/durability guarantees – obtained from source of truth and the fact that change stream is a f(Source,SCN) .At end points –the CDC catpures changes from database and publishes to an event bufferAt the other end – applications subscribe to the change stream and receive callbacks when change data from the sources they have subscribed to become availablePoint: End points have API’s supporting transaction semantics (atomicity)Point: ‘Window’ or consistency window are points in the stream that are consistent with source at the specified SCN. Point: Consumers ‘see’ events one consistency window at a time –i.e. they are visible to the consumers after ‘end of window’ has been written.Questions: What if CDC was outside-> CDC can be a pull-model – but can they push-off box to event stream – Yes, but then event stream isn’t simple – cluster state (leader-follower) is shared amongst CDC and Relay. Also failure of CDC leads needs to be treated and monitored separately. Packaging CDC with event stream has operability advantages. Questions: is onRollback() triggered at same time rollback was shown in buffer? – No – this isn’t about one-to-one correspondence in time – but one in semantics - both have notion of ‘transactions’ – apps don’t see events not committed – end users- output of apps have the option of seeing the whole txn in entirety as well – very important for example in relay-chaining.
• #17: - Both Relay (Online changes) and Bootstrap (Older changes) together constitute the change event streamDo not share the same exact API – but semantically say the same thingGet events since a point in logical clock Both have ability to perform simple filters on the service side- Both have chunking/progress guarantees-HTTP based implementation-Efficient communication to clients
• #18: Database schemas to some neutral format for the databus events , we chose AvroTools are available to publish schema to ‘schema registry’ , schema generation from different source types.Schemas generated and stored in a place accessible by Change Stream – backward compatibility Tools are available to publish schema to ‘schema registry’ , schema generation from different source types.Ensures backward compatibility – relevant for bootstrap.Schemas available to consumers to deserialize.
• #19: Relay enacapsulates change capture logic – event buffer (remember the publish API) – implemented as a circular buffer - and the meta dataConstitutes the online – most frequently used part of the change stream – addreses 98% of requests on a typical day.
• #20: Relays talk to database directly – since they contain the change capture. This has horizontal scalability limits.
• #21: -Bootstrap Consumer : is a special application that consumes events from relays and writes to a persistence layer called ‘log store’ . -Another process applies changes to snapshot store – how – using the pk that was there in the publish API. Separate thread not shown here.-Seeding: bootstrapping the bootstrap
• #22: Databus Client Library: -Orchestrate consumption of change stream from bootstrap/relay-Uses a http fetch to get events from upstream / write to eventbuffer using efficient readEvents call-Currently uses polling mechanism to get events from upstream-Dispatcher uses iterator interface of EventBuffer to read the events and then call the user specified consumer implementations.-Client library by default persists SCN for lossless recovery.-Consumers need to be thread-safe, can take advantage of parallelism .… let’s look at a typical application
• #23: -Key: a single instance of client library can handle – multiple consumers subscribing to multiple change streams Different logic tuning required for bootstrap and online case - facilities providedSchema aware apps can force type conversion from one schema to another – as long as backward compatibility is preserved amongst change data. -Override of persisted SCN possible : cases where flush() is not guaranteed by the application (e.g. index - ) – so , apps store the SCN in the index; retrieve it on startup-Applications typically are distributed – so they have notion of some sort of partitions/partition awareness. It can be tempting to consume the entire event stream of an unpartitioned upstream store and then drop n-1/n th partition on the floor.It’s inefficient and expensive (for relay and consumer – latency wise as we shall see) – Instead…
• #24: - Here- client nodes refer to one instance of the client library – so that can be an application instance-Applications themselves are partition aware – write to partitioned indexes/storesNeed to distribute processing loadPartition function -> applied at source on the primary key - this is applied on the fly – and the source itself neednt’ be partitioned.- Partitions can be changed as more nodes are applied if the application accounts for ‘repartitioning’ . Checkpoints need to be reset, configuration needs to be changed.But this is hardly operation friendly..
• #25: The clients are partition aware- but the partition assignment is dynamicCluster awareness is introduced , client app clusters Operability advantages – ability to add/remove nodes with dynamic redistribution -Helix used to manage client clusters , and as SCN store ..….Now, Let’s look at some aspects of the current implementation.
• #26: … And on to some code – let’s take a look at the application
• #27: -Points to note: - how a source is specified. -How sources are specified-A consumer: - and a databus client uses subscription (register);
• #28: Key – show how payload is extracted. The API’s we have visited them earlier. … Now to dwell on performance
• #30: Setup:- Measure relay serving throughput and CPU utilization- Vary number of consumers and poll interval tpt_10 means throughput with a poll interval of 10ms, cpu_55 is cpu utilization with 55ms poll interval, etc.)- Consumers pulling at max speed (no additional processing)- Event size is 2.5 KBNo write traffic -- relay buffer pre-filledThe hypothesis was that we can support more consumers if the poll interval is long, and that is confirmed by the Observations:- Relay can easily saturate the network with minimal CPU utilization- Once network is saturated, CPU increases with number of consumers due to networking overhead (context switching)- Even with 200 consumers, CPU utilization is less than 10%- Higher poll intervals generally lead to less CPU utilization
• #32: Setup:- Measure read throughput of each consumer with update traffic on the relay- Vary the number of consumers and update rate- Consumers pulling at max speed (no additional processing)- Poll interval is 10 msEvent size is 2.5 KBObservations:- Drops mean consumer no longer being able to keep up- Reason is network saturation on the relay side; e.g. 2000 update/s * 20 consumers * 2.5KB = 100MBps < max network bandwidth < 200MBps = 2000 update/s * 40 consumers * 2.5KB
• #33: Setup:- Same as above but measure time in milliseconds for events to reach consumer- Added partitioning through server-side filtering to see what happens if network is not a bottleneckObservations:- Latency knees due to relay network saturation as before- Latency without SSF (Server side filtering) is around 10-20 ms (including an average 5ms overhead due to the poll interval)With SSF network is no longer a bottleneck; latency up to 15-45ms due to SSF computation overheadSo, the relay can scale to hundreds of consumers if they can tolerate a little bit of latency.
• #35: E2E latency has no meaning for bootstrap service, and they can easily saturate the network with multiple clients. So, we focused on comparing the serves out log store vs snapshot store.Setup:* Compare serving deltas vs serving all updates* Synthetic work load* Vary number of updates to existing keys vs new keys (i.e. inserts)Observations:* Catchup time is constant as it does not distinguish updates vs inserts* Break even point is around 1:1 updates vs insertsFor a small number of inserts, the benefit of snapshot is overwhelmingThe breakeven point seems to be when ½ of the changes are updates. We monitor the update rate in production and tune the bootstrap service.
• #36: Databusstream for Oracle :  things that scale with memberId and things that scale with connections (multiplicative , only inserts)  small sources – advertiser data (but consistency important)Applications:  search – multiple instances – large distributed deployment – low latency requirement – consistency  Bootstrap used in new ways – used to automatically provision new index nodes; new in memory in advertising data sets ; usef to fix legacy stores Espresso: source of truth 2013 and beyondPartitioned primary data store (transactional) based on Mysql store engine ; horizontally scalable ; change stream partitioned at source-of-truth rather than change-stream -> Change stream still requires trigger based ‘databusification’ in Oracle. Relay provisioning is still manual – in the sense – there is no self-serve mechanism of specify a source /automatic source discovery- that will triggerRelays being provisioned in the ‘cloud’ – depending on capacity estimates.Let’s look at some change capture implementations we have.. .
• #38: -
• #40: Overall: is external clock propagation a good idea overall? Is it necessary or a nice to have? - It becomes important in case of bootstrap ?Are checkpoints portable? If a mapping exists between SCN->CDC-GEN-UNIQ-NUM , or if an index exists at every layer-bootstrap and relay for SCN – then it can be handled as system levelImpl – and the client needn’t use SCN explicitly. SCN – external clock is a convenient way of storing logical state across instances of the change stream.
• #41: Point: Change Capture is within the relay: each relay is self-sufficient, i.e. since eventBufferState = fn(source,SCN) – it has change capture logic to pull in the changes – if change capture were outside, then the change capture logic ,fan out has to take care of replication or write to a leader-follower relay clusterPoint: Client Library: fetches changes from the databus stream (which is now Relay and Bootstrap Service) - . Point: Separation of use cases – recent change, to older changes, snapshots – cannot rely solely on all changes fitting into memoryPoint: Bootstrap Consumer: special application that listens to changes and updates it’s log store (persistent change events) and snapshot store – (persistent copy of the database – stores change events in user-space)Point: Remember , client library automatically switches between appropriate service – relay or bootstrap depending on SCN requested by applicationPoint: DB’s are saved from abusive scans from lagging consumers