Insta clustr seattle kafka meetup presentation bb
1 like•260 views
This document summarizes Instaclustr's lessons learned from building a managed Apache Kafka service. It provides an overview of Kafka and how it works, details Instaclustr's offering and development process, and discusses choices around hardware, security configuration, monitoring, backups and restores. Key topics covered include benchmarking storage types, enabling SSL, managing topics and users, and exposing metrics for monitoring brokers and topics.
Insta clustr seattle kafka meetup presentation bb
- 1. instaclustr.comTwitter @instaclustr [email protected] instaclustr.com Lessons Learned from Building an Apache Kafka Managed Service
- 2. instaclustr.com Introduction # Who am I? /bin/whoami â—Ź Ben Bromhead, CTO, Instaclustr # Who is Instaclustr? /bin/id -g -n â—Ź Experts in reliability at scale â—Ź Manage/Support 3k+ Cassandra, Spark and Elassandra nodes â—Ź Platform providers automated provisioning, monitoring and management â—Ź Available on AWS, GCP, Azure and IBM Cloud â—Ź Managed Apache Kafka released May 21st
- 3. instaclustr.com Agenda â—Ź What is Kafka â—Ź A quick intro to how it works â—Ź Context - our offering and development process â—Ź Hardware choice and benchmarking â—Ź Topic and user management â—Ź Broker security configuration â—Ź Monitoring â—Ź Backup and Restore
- 4. instaclustr.com What Is Apache Kafka? Key Characteristics ● Horizontal scalable, distributed system ● Performance ○ Low latency, high throughput ● Scalability ○ Linear broker scalability via partitioned topics ○ Linear consumer scalability via consumer groups ● Fault-tolerance ○ Data is replicated across multiple brokers ○ Automatic broker failover when primary replica goes offline ○ Automatic consumer failover when consumer in consumer group goes offline ● Apache Foundation Open Source ● Production Proven • Publish & Subscribe to streams of data (reliable message transport) • Transform and/or aggregate data streams using distributed processing applications (stream processing)
- 5. instaclustr.com Why use Apache Kafka? â—Ź Provide a buffering mechanism in front of a processing (ie deal with temporary incoming message rate greater than processing app can deal with) â—Ź A special case of buffering is to allow producers to publish messages with guaranteed delivery even if the consumers are down when the message is published â—Ź As an event store for events sourcing or Kappa architecture â—Ź Facilitate flexible, configurable architectures with many producers -> many consumers by separating the details who what is consuming messages for the apps that produce them (and vice-versa) â—Ź Perform stream analytics (with Kafka Streams)
- 6. instaclustr.com How does it work: producing records ● Each topic has a fixed number of partitions ● Records published to a topic by a producer are divided amongst the topic’s partitions ● Partitions are ordered, immutable lists ● Each new record is appended to the end of a partition ● Each partition is stored on a single leader broker, and may optionally be replicated to one or more follower brokers
- 7. instaclustr.com How does it work: consuming records â—Ź A consumer reads from one or more partitions â—Ź Consumer maintains an offset of the last record in the partition read â—Ź The consumer requests a micro-batch of records from Kafka. The broker uses the offset to provide the latest records to the consumer â—Ź Once the consumer has finished processing a record, it must commit the new offset â—Ź Because Kafka does not delete records immediately after they are read, consumers may reset the offset to a previous value to replay records
- 8. instaclustr.com How does it work: consumer groups ● Multiple consumers reading from a topic may be arranged into Consumer Groups ● A Consumer Group load-balances partitions amongst consumers ● If a consumer goes offline, the consumer group will automatically re-distribute it’s partitions amongst the remaining consumers
- 9. instaclustr.com How it works: Easier Abstractions â—Ź High-level API â—Ź Drop-in source (import) & sink (export) connectors exist for many popular technologies, including Amazon S3, Amazon Kinesis, Apache Cassandra, HDFS and JDBC Kafka Connect â—Ź Provides functionality to aggregate data, join multiple topics and perform complex transformations to live data as it arrives â—Ź The API abstracts away most of the difficult scalability, fault-tolerance and consistency problems associated with performing live aggregations on a distributed system Kafka Streams
- 10. instaclustr.com Instaclustr Managed Kafka - Key Features ● Available Now: ○ Open source Apache Kafka (Brokers) and Zookeeper automatically provisioned in AWS, GCP and Azure ○ Broker Monitoring ○ Instaclustr monitoring and provisioning API support ○ Private network clusters (AWS only) ○ Run in your cloud provider account or ours ● For GA (end June): ○ SOC2 compliant ○ User & credential management ○ More cluster config options ○ Topic Level and Synthetic transaction monitoring ○ Infrastructure config tuning ● Likely future release scope: ○ Topic Management UI ○ Cluster “copy” ○ Managed: ■Kafka Connect ■Schema Registry ■Mirror Maker ○ Dynamic scaling
- 11. instaclustr.com Instaclustr Managed Kafka - Development Process â—Ź First customer requests 2016 â—Ź Internal infrastructure deployment and usage of Kafka mid 2017 â—Ź Managed service platform development commenced November 2017 â—Ź Early access program with 4 customers commenced December 2017 â—Ź Public preview release 21 May 2018 â—Ź GA expected 25 June 2018
- 12. instaclustr.com Hardware Choice and Benchmarking - GP2 vs ST1 â—Ź AWS Benchmark - r4.large w 500GB disks â—Ź Avg 10% improved throughput with ST1 vs GP2 EBS â—Ź ST1 is 45% of the cost of GP2
- 13. instaclustr.com Hardware Choice and Benchmarking - GP2 vs ST1 â—Ź AWS Benchmark - r4.large w 500GB disks â—Ź Avg 10% improved throughput with ST1 vs GP2 EBS â—Ź ST1 is 45% of the cost of GP2
- 14. instaclustr.com Hardware Choice and Benchmarking - SSL vs non-SSL â—Ź AWS Benchmark - r4.large w 1500GB ST1 disks â—Ź 512 byte messages â—Ź ~30% decrease in throughput with Broker and Client SSL enabled
- 15. instaclustr.com Hardware Choice and Benchmarking - SSL vs non-SSL â—Ź AWS Benchmark - r4.large w 1500GB ST1 disks â—Ź 512 byte messages â—Ź ~30% decrease in throughput with Broker and Client SSL enabled
- 16. instaclustr.com Hardware Choice and Benchmarking - Number of Topics â—Ź Increasing topics small reduction performances â—Ź However, more topics = more partitions and significantly slows recovery time from node failure 10 Topic s 100 Topic s 1000 Topic s 5000 Topic s
- 17. instaclustr.com Hardware Choice and Benchmarking - Colocated Zookeeper â—Ź Often recommended to host zookeeper separately to Kafka. â—Ź However, recent changes have significantly reduced load on Zookeeper from Kafka. â—‹ Consumer offsets are no longer stored in Zookeeper. â—Ź Our benchmarking showed no measurable difference in performance, at least for smaller clusters. Consumer Rate - Colocated Consumer Rate - Separate 6 Broker Test with Node Restart
- 18. instaclustr.com Topic and User Configuration Management â—Ź Existing Kafka utilities for managing topic and user configuration required direct access to Zookeeper â—Ź However, Zookeeper does not have a robust external security model (TLS support, node to node auth, etc) â—Ź Providing Zookeeper access to customers introduces a whole class of very strange ways to break a cluster by corrupting Zookeeper â—Ź Solutions: â—‹ Developed command line tool to use Kafka API for topic configuration (https://github.com/instaclustr/ic-kafka-tools) â– may add to Instaclustr console later although we think maintaining topic config as a version controlled file in your repo is a better approach â—‹ Adding user management to Instaclustr console â– we do no want to keep cluster passwords in our central management system so this feature will require users to enter an existing Kafka credentials to be temporarily used by our system
- 19. instaclustr.com Broker Security Configuration â—Ź Using SCRAM (Salted Challenge Response Authentication Mechanism) for authentication â—‹ More secure â—‹ Allows easier rotation of credentials â—‹ Initial release for client->broker only with plain text for broker to broker â—‹ Decided to also use for broker->broker to allow rapid rotation of credentials as part of SOC2 security measures â—Ź TLS built on existing Cassandra infrastructure â—‹ New CA created per cluster â—‹ CA used to generate certificates for each node â—‹ CA pub cert available for clients to download for full validation of certificates â—Ź Access to managed clusters also follows same model as Cassandra â—‹ Public IPs and whitelisting in firewall (security group or equivalent) â—‹ Private IPs with VPC Peering (or equivalent in other cloud providers) â—‹ Private Network Clusters where nodes are not allocated public IPs and gateway box is used for admin access â—‹ Did not expose through firewall Zookeeper due to weak security model
- 20. instaclustr.com Monitoring â—Ź Metrics exposed via JMX allowing us to use our existing Cassandra monitoring â—‹ Custom agent -> RabbitMQ (planned to migrate to Kafka) -> Riemann -> Cassandra + Spark -> Console, APIs, Grafana â—Ź Exposing broker-level and per-topic metrics â—Ź Alerting? â—‹ The basics: service state, disk usage free space, server still exists â—‹ Kafka metrics: offline partitions, active controllers != 1, partition under replicated â—‹ Synthetic transactions: publish and consume message to controlled topic, measure success and latency
- 21. instaclustr.com Backup and Restore ● Internet wisdom = Kafka Backups is not a thing ○ Rely on replication within cluster or mirror maker replication to another cluster ● Hmm - we rarely use backups for Cassandra but there have been a few times we’ve been very glad to have them ○ Hardware failure is not an issue but corruption due to app bugs or user error can occur and be spread by replication ● Working on regular automated backup and restore of topic and security configuration ● Consider using Kafka Connect to write important message to offline backup
- 22. instaclustr.comTwitter @instaclustr [email protected] instaclustr.com Give a try! 14 Day Free Trial at instaclustr.com