Kafka Up And Running For Network Devops Set Your Network Data In Motion Eric Chou

Kafka Up And Running For Network Devops Set Your
Network Data In Motion Eric Chou download
https://ebookbell.com/product/kafka-up-and-running-for-network-
devops-set-your-network-data-in-motion-eric-chou-36653854
Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.
Kafka Up And Running For Network Devops Eric Chou
https://ebookbell.com/product/kafka-up-and-running-for-network-devops-
eric-chou-55476666
Apache Kafka Set Up Apache Kafka Clusters And Develop Custom Message
Producers And Consumers Using Practical Handson Examples Nishant Garg
https://ebookbell.com/product/apache-kafka-set-up-apache-kafka-
clusters-and-develop-custom-message-producers-and-consumers-using-
practical-handson-examples-nishant-garg-5476594
Turning Up The Heat Urban Political Ecology For A Climate Emergency
Maria Kaika Editor
https://ebookbell.com/product/turning-up-the-heat-urban-political-
ecology-for-a-climate-emergency-maria-kaika-editor-50975850
Kafka On The Shore Haruki Murakami
https://ebookbell.com/product/kafka-on-the-shore-haruki-
murakami-46476270

Kafka Streams In Action Second Edition Meap Version 8 2nd Bill Bejeck
https://ebookbell.com/product/kafka-streams-in-action-second-edition-
meap-version-8-2nd-bill-bejeck-46595524
Kafka And The Universal 1st Edition Arthur Cools Vivian Liska
https://ebookbell.com/product/kafka-and-the-universal-1st-edition-
arthur-cools-vivian-liska-46609216
Kafka And The Traveling Doll Jordi Sierra I Fabra
https://ebookbell.com/product/kafka-and-the-traveling-doll-jordi-
sierra-i-fabra-47220834
Kafkas Stereoscopes The Political Function Of A Literary Style Isak
Winkel Holm
https://ebookbell.com/product/kafkas-stereoscopes-the-political-
function-of-a-literary-style-isak-winkel-holm-50216298
Storm Bringer Volume 8 Kafka Asagiri Sango Harukawa
https://ebookbell.com/product/storm-bringer-volume-8-kafka-asagiri-
sango-harukawa-50511446

Kafka Up and Running for Network DevOps
Set Your Network Data in Motion
Eric Chou
This book is for sale at http://leanpub.com/network-devops-kafka-up-and-
running
This version was published on 2021-11-12
* * * * *
This is a Leanpub book. Leanpub empowers authors and publishers with the
Lean Publishing process. Lean Publishing is the act of publishing an in-
progress ebook using lightweight tools and many iterations to get reader
feedback, pivot until you have the right book and build traction once you do.
* * * * *
© 2021 Network Automation Nerds, LLC.
ISBN for EPUB version: 978-1-957046-01-3
ISBN for MOBI version: 978-1-957046-02-0

For my family, you are my ‘why’ for everything I do.
I would like to thank the open-source software community. My life would be
very different without the many dedicated, talented individuals in the open-
source community. Thank you all.

Table of Contents
Introduction
What is Kafka
Why do we need Kafka
Prerequisites for this book
Who this book is for
What this book covers
Download the example code files
Conventions used
Get in touch
Chapter 1. Kafka Introduction
History of Kafka
Kafka Use Cases
Disadvantages of Kafka
Kafka Concepts
Conclusion
Chapter 2. Kafka Installation and Testing
Network Lab Setup
Kafka Installation Overview
Install Java
Download Kafka
Configure Zookeeper
Configure Kafka
Start Zookper and Kafka manually
Test the Kafka operations
Configure System Services
Conclusion
Chapter 3. Kafka Concepts and Examples
Producers: Writing Messages
Consumers: Receiving Messages
Offsets in Action

Kafka Topic Administration
Replication
Conclusion
Chapter 4. Hosted Kafka Services
AWS Managed Kafka Service
Amazon MSK Costs
Launch Amazon MSK Cluster
Client Setup
Produce and Consume Data
Conclusion
Chapter 5. Cloud Provider Messaging Services
Amazon Kinesis
Amazon Kinesis Example
Azure Event Hub
Azure Event Hub Example
Google Cloud Pub/Sub
GCP Pub/Sub Python Example
Conclusion
Chapter 6. Network Operations with Kafka
Install Docker
Install Elasticsearch
Install Kibana
Network Data Feed
Network Data Pipeline
Network Log as a Service
Conclusion
Chapter 7. Other Kafka Considerations and Looking Ahead
Hardware Considerations
Kafka Broker and Topic Configurations
Schema Registry
Kafka Stream Processing
Cross-Cluster Data Mirroring
Additional Resources

Conclusion
Appendix A. Installing Lab Instance in Public Cloud

Introduction
Welcome to the world of data!
Unless you have been living under a rock for the last few years, you know
data processing, machine learning, and artificial intelligence are taking over
the world. Data exists everywhere around us. We can now check real-time
traffic information from online cameras before we even leave the house. We
can connect to our thermometers remotely to automatically adjust house
temperatures. Better yet, the thermometers can also be self-taught so that they
can adjust the temperatures all by themselves. Before our family weekend
movie nights, my kids love to leverage the WiFi-enabled lights to match the
lighting with our mood.
How do these cameras, lights, and thermometers able to take measurements
and generate data? It turns out the cost of small sensors and tiny computing
units have been coming down steadily since the early days and now can be
integrated into everyday items. However, the generated data by one or two
devices might not be sufficient enough to yield meaningful results. After all,
traffic information on one street might only benefit a tiny fraction of people
who travels on that street, but aggregated traffic information on all streets can
help everyone. Generally, it is by aggregating all disperse data sets across
hundreds of devices; we are able to derive useful information that helps us
with our daily lives. The data are constantly flowing between producers and
consumers of data.
Have you ever wondered how these data are being exchanged between data
producers and consumers? Does each of the devices provide an API
(Application Programming Interface) to be queried? Do each of them have
local databases that persist the data? What about data integrity, transmission
latency, or scalability?
There are many tools and projects that address these data streaming and
exchange issues. One of the most popular open-source tools widely used by
companies large and small alike is Apache Kafka.

What is Kafka
You might be thinking, “Don’t we already have lots of data storage systems?
Why do we need yet-another-storage-system?” You are right, and we do have
lots of storage solutions such as relational and non-relational databases, cache
systems, big data storage clusters, search solutions, and many more. But in
most of the data storage cases, the data is entered in once, stored in the
database, then retrieved later when needed. For example, when I visited my
dentist for the first time, they asked for my personal information, entered
them into a database so for my future visits, they could pull up my record.
This is very different than the traffic sensor data example that we discussed.
What sets Kafka apart is it was built from the ground up to treat data as
continuous flows of information that are constantly being produced,
enhanced, manipulated, and consumed. Instead of a focus on holding in
data like databases, key-value stores, search indexes, or caches, Kafka
architects itself as a system that allows data to be a continually evolving
stream of information.
According to the Apache Kafka project page:
Apache Kafka is an open-source distributed event streaming platform
used by thousands of companies for high-performance data pipelines,
streaming analytics, data integration, and mission-critical applications.
Companies known for a large amount of data, such as AirBnb, Datadog, Etsy,
and many others across different industries, use Kafka to build their data
pipeline. These data pipelines use a variety of services that both produce and
consume data in a continuous format.

Figure Intro. 1: Powered by Apache Kafka
(https://kafka.apache.org/powered-by)
Don’t worry if you have not heard of Kafka before or are not sure how, as
network DevOps engineers, this tool can help us. We will go a lot deeper into
Kafka in this book.

Why do we need Kafka
As a general overview, there are many uses cases for Kafka in network
engineers:
We can use Kafka to stream data, such as logs and NetFlow data, once
and be consumed by multiple receivers. Kafka takes care of the ordering
of messages, acknowledging receipt to producers, delivery confirmation
to consumers, and balancing the data between different recipients.
We can separate data into logical partitions called Topics in a single
Kafka cluster. This allows subscribers to only receive the data they are
interested in, so the log receiver will not need to receive flow data.
Kafka allows for an event-driven architecture, such as triggering events
based on different types of events. For example, a log receiver can page
an on-call engineer if it notices a BGP neighbor of a core device going
down.
Kafka allows us to build a centralized pipeline for network data
processing instead of having dispersed teams process bits and pieces of
data separately.
These are just some of the use cases of Kafka. By the end of this book, I am
sure we will be able to find much more creative use cases.

Prerequisites for this book
Basic knowledge of Linux command line is required to make the most out of
this book. We would use command-line tools such as using cd for changing
directories, ls for listing directories contents, and pwd to know where in the
directory tree you are currently operating from.
We will be using Python 3 as the programming language in this book. Python
is a popular language amongst network engineers with a large ecosystem of
tools and libraries. We will use Python to create Kafka producers, consumers
and interface with public cloud providers. However, I do not believe you
need to be an expert in Python 3 to understand the scripts in this book. If you
need a refresher on Python, a good place to go would be the official Python
Tutorial.

Who this book is for
This book is ideal for IT professionals and engineers who want to take
advantage of Kafka’s distributed, fault-tolerant streaming data platform. This
book can also be used by management to gain a general understanding of
Kafka and how it fits into the general IT infrastructure.

What this book covers
Chapter 1. Kafka Introduction, In this chapter, we will cover the general
concepts of Kafka. The core architecture, components, and tools. The idea
behind Kafka, how it was built, and how the components can help maintain
data streams at scale.
Chapter 2. Kafka Installation and Testing, In this chapter, we will install
Zookeeper and Kafka on a single Virtual Machine and configure both
components. We will also prepare our network lab to be used for future
examples. After installation, we will work on a few producer-consumer
examples using Kafka command-line tools.
Chapter 3. Kafka Concepts and Examples, In this chapter, we will provide
examples of Kafka usage for Producers and Consumers. The producers will
write messages to a Topic with consumers receiving the messages. We will
look at examples of offset, commit, and acknowledgment for data in the
topics.
Chapter 4. Hosted Kafka Services, When we want to move Kafka from our
lab setup into production, we can use the Kafka-hosting-as-a-service
provided by various cloud providers, such as Amazon AWS or Confluent
Cloud. In this chapter, we will provide a step-by-step guide to launch our
Kafka cluster using Amazon Managed Streaming for Apache Kafka.
Chapter 5. Cloud Providers Messaging Services, If we are not ready for a
managed Kafka cluster, the top public cloud providers, Amazon AWS,
Microsoft Azure, and Google Cloud, offer their adopted version of message
streaming service. The messaging services have various degrees of Kafka
compatibility. In this chapter, we will look at examples of AWS Kinesis,
Azure Event Hub, and Google Pub/Sub.
Chapter 6. Network Operations with Kafka, In this chapter, we will explore
examples of Kafa in network engineering. We will look at data feeds, data
enhancement, and Kafka Connect. The Kafka Connect reuses code provided

by the community. We will look at the File and Elasticsearch Kafka connect
plugins.
Chapter 7. Other Kafka Considerations and Looking Ahead, In this chapter,
we will discuss other Kafka considerations, such as hardware requirements,
Broker and Topic configuration, Schema registry, and many more. This
chapter will provide additional resources for readers to explore Kafka.

Download the example code files
The code examples used in this book can be downloaded from GitHub at
https://github.com/ericchou1/network-devops-kafka-up-and-running.

Conventions used
There are a number of text conventions used in this book to help organize the
flow. Information in bold and italic are used to indicate important or special
terms.
Code blocks are shown below:
1 print('hello world')
Command-line input or output will be shown as follows:
1 $ touch my_script.py
2
3 $ ls /
4 bin cdrom etc lib lib64 lost+found mnt proc run snap sw
5 apfile tmp var
6 boot dev home lib32 libx32 media opt root sbin srv sy
7 s usr
8
9 $ python
10 Python 3.8.10 (default, Jun 2 2021, 10:49:15)
11 [GCC 9.4.0] on linux
12 Type "help", "copyright", "credits" or "license" for more information.
13 >>> print('hello world')
14 hello world
15 >>> exit()
Warning, tips, and information will be specified in their own special block:
This is a tip section. It will include useful tips and tricks in relation to the topic discussed at
hand.
This is an information section. It will provide additional information to help you explore the
topic further.

Another Random Scribd Document
with Unrelated Content

containing a part of this work or any other work associated with
Project Gutenberg™.
1.E.5. Do not copy, display, perform, distribute or redistribute
this electronic work, or any part of this electronic work, without
prominently displaying the sentence set forth in paragraph 1.E.1
with active links or immediate access to the full terms of the
Project Gutenberg™ License.
1.E.6. You may convert to and distribute this work in any binary,
compressed, marked up, nonproprietary or proprietary form,
including any word processing or hypertext form. However, if
you provide access to or distribute copies of a Project
Gutenberg™ work in a format other than “Plain Vanilla ASCII” or
other format used in the official version posted on the official
Project Gutenberg™ website (www.gutenberg.org), you must,
at no additional cost, fee or expense to the user, provide a copy,
a means of exporting a copy, or a means of obtaining a copy
upon request, of the work in its original “Plain Vanilla ASCII” or
other form. Any alternate format must include the full Project
Gutenberg™ License as specified in paragraph 1.E.1.
1.E.7. Do not charge a fee for access to, viewing, displaying,
performing, copying or distributing any Project Gutenberg™
works unless you comply with paragraph 1.E.8 or 1.E.9.
1.E.8. You may charge a reasonable fee for copies of or
providing access to or distributing Project Gutenberg™
electronic works provided that:
• You pay a royalty fee of 20% of the gross profits you derive
from the use of Project Gutenberg™ works calculated using the
method you already use to calculate your applicable taxes. The
fee is owed to the owner of the Project Gutenberg™ trademark,
but he has agreed to donate royalties under this paragraph to
the Project Gutenberg Literary Archive Foundation. Royalty

payments must be paid within 60 days following each date on
which you prepare (or are legally required to prepare) your
periodic tax returns. Royalty payments should be clearly marked
as such and sent to the Project Gutenberg Literary Archive
Foundation at the address specified in Section 4, “Information
about donations to the Project Gutenberg Literary Archive
Foundation.”
• You provide a full refund of any money paid by a user who
notifies you in writing (or by e-mail) within 30 days of receipt
that s/he does not agree to the terms of the full Project
Gutenberg™ License. You must require such a user to return or
destroy all copies of the works possessed in a physical medium
and discontinue all use of and all access to other copies of
Project Gutenberg™ works.
• You provide, in accordance with paragraph 1.F.3, a full refund of
any money paid for a work or a replacement copy, if a defect in
the electronic work is discovered and reported to you within 90
days of receipt of the work.
• You comply with all other terms of this agreement for free
distribution of Project Gutenberg™ works.
1.E.9. If you wish to charge a fee or distribute a Project
Gutenberg™ electronic work or group of works on different
terms than are set forth in this agreement, you must obtain
permission in writing from the Project Gutenberg Literary
Archive Foundation, the manager of the Project Gutenberg™
trademark. Contact the Foundation as set forth in Section 3
below.
1.F.
1.F.1. Project Gutenberg volunteers and employees expend
considerable effort to identify, do copyright research on,
transcribe and proofread works not protected by U.S. copyright

law in creating the Project Gutenberg™ collection. Despite these
efforts, Project Gutenberg™ electronic works, and the medium
on which they may be stored, may contain “Defects,” such as,
but not limited to, incomplete, inaccurate or corrupt data,
transcription errors, a copyright or other intellectual property
infringement, a defective or damaged disk or other medium, a
computer virus, or computer codes that damage or cannot be
read by your equipment.
1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except
for the “Right of Replacement or Refund” described in
paragraph 1.F.3, the Project Gutenberg Literary Archive
Foundation, the owner of the Project Gutenberg™ trademark,
and any other party distributing a Project Gutenberg™ electronic
work under this agreement, disclaim all liability to you for
damages, costs and expenses, including legal fees. YOU AGREE
THAT YOU HAVE NO REMEDIES FOR NEGLIGENCE, STRICT
LIABILITY, BREACH OF WARRANTY OR BREACH OF CONTRACT
EXCEPT THOSE PROVIDED IN PARAGRAPH 1.F.3. YOU AGREE
THAT THE FOUNDATION, THE TRADEMARK OWNER, AND ANY
DISTRIBUTOR UNDER THIS AGREEMENT WILL NOT BE LIABLE
TO YOU FOR ACTUAL, DIRECT, INDIRECT, CONSEQUENTIAL,
PUNITIVE OR INCIDENTAL DAMAGES EVEN IF YOU GIVE
NOTICE OF THE POSSIBILITY OF SUCH DAMAGE.
1.F.3. LIMITED RIGHT OF REPLACEMENT OR REFUND - If you
discover a defect in this electronic work within 90 days of
receiving it, you can receive a refund of the money (if any) you
paid for it by sending a written explanation to the person you
received the work from. If you received the work on a physical
medium, you must return the medium with your written
explanation. The person or entity that provided you with the
defective work may elect to provide a replacement copy in lieu
of a refund. If you received the work electronically, the person
or entity providing it to you may choose to give you a second
opportunity to receive the work electronically in lieu of a refund.

If the second copy is also defective, you may demand a refund
in writing without further opportunities to fix the problem.
1.F.4. Except for the limited right of replacement or refund set
forth in paragraph 1.F.3, this work is provided to you ‘AS-IS’,
WITH NO OTHER WARRANTIES OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR ANY PURPOSE.
1.F.5. Some states do not allow disclaimers of certain implied
warranties or the exclusion or limitation of certain types of
damages. If any disclaimer or limitation set forth in this
agreement violates the law of the state applicable to this
agreement, the agreement shall be interpreted to make the
maximum disclaimer or limitation permitted by the applicable
state law. The invalidity or unenforceability of any provision of
this agreement shall not void the remaining provisions.
1.F.6. INDEMNITY - You agree to indemnify and hold the
Foundation, the trademark owner, any agent or employee of the
Foundation, anyone providing copies of Project Gutenberg™
electronic works in accordance with this agreement, and any
volunteers associated with the production, promotion and
distribution of Project Gutenberg™ electronic works, harmless
from all liability, costs and expenses, including legal fees, that
arise directly or indirectly from any of the following which you
do or cause to occur: (a) distribution of this or any Project
Gutenberg™ work, (b) alteration, modification, or additions or
deletions to any Project Gutenberg™ work, and (c) any Defect
you cause.
Section 2. Information about the Mission
of Project Gutenberg™

Project Gutenberg™ is synonymous with the free distribution of
electronic works in formats readable by the widest variety of
computers including obsolete, old, middle-aged and new
computers. It exists because of the efforts of hundreds of
volunteers and donations from people in all walks of life.
Volunteers and financial support to provide volunteers with the
assistance they need are critical to reaching Project
Gutenberg™’s goals and ensuring that the Project Gutenberg™
collection will remain freely available for generations to come. In
2001, the Project Gutenberg Literary Archive Foundation was
created to provide a secure and permanent future for Project
Gutenberg™ and future generations. To learn more about the
Project Gutenberg Literary Archive Foundation and how your
efforts and donations can help, see Sections 3 and 4 and the
Foundation information page at www.gutenberg.org.
Section 3. Information about the Project
Gutenberg Literary Archive Foundation
The Project Gutenberg Literary Archive Foundation is a non-
profit 501(c)(3) educational corporation organized under the
laws of the state of Mississippi and granted tax exempt status
by the Internal Revenue Service. The Foundation’s EIN or
federal tax identification number is 64-6221541. Contributions
to the Project Gutenberg Literary Archive Foundation are tax
deductible to the full extent permitted by U.S. federal laws and
your state’s laws.
The Foundation’s business office is located at 809 North 1500
West, Salt Lake City, UT 84116, (801) 596-1887. Email contact
links and up to date contact information can be found at the
Foundation’s website and official page at
www.gutenberg.org/contact

Section 4. Information about Donations to
the Project Gutenberg Literary Archive
Foundation
Project Gutenberg™ depends upon and cannot survive without
widespread public support and donations to carry out its mission
of increasing the number of public domain and licensed works
that can be freely distributed in machine-readable form
accessible by the widest array of equipment including outdated
equipment. Many small donations ($1 to $5,000) are particularly
important to maintaining tax exempt status with the IRS.
The Foundation is committed to complying with the laws
regulating charities and charitable donations in all 50 states of
the United States. Compliance requirements are not uniform
and it takes a considerable effort, much paperwork and many
fees to meet and keep up with these requirements. We do not
solicit donations in locations where we have not received written
confirmation of compliance. To SEND DONATIONS or determine
the status of compliance for any particular state visit
www.gutenberg.org/donate.
While we cannot and do not solicit contributions from states
where we have not met the solicitation requirements, we know
of no prohibition against accepting unsolicited donations from
donors in such states who approach us with offers to donate.
International donations are gratefully accepted, but we cannot
make any statements concerning tax treatment of donations
received from outside the United States. U.S. laws alone swamp
our small staff.
Please check the Project Gutenberg web pages for current
donation methods and addresses. Donations are accepted in a
number of other ways including checks, online payments and

credit card donations. To donate, please visit:
www.gutenberg.org/donate.
Section 5. General Information About
Project Gutenberg™ electronic works
Professor Michael S. Hart was the originator of the Project
Gutenberg™ concept of a library of electronic works that could
be freely shared with anyone. For forty years, he produced and
distributed Project Gutenberg™ eBooks with only a loose
network of volunteer support.
Project Gutenberg™ eBooks are often created from several
printed editions, all of which are confirmed as not protected by
copyright in the U.S. unless a copyright notice is included. Thus,
we do not necessarily keep eBooks in compliance with any
particular paper edition.
Most people start at our website which has the main PG search
facility: www.gutenberg.org.
This website includes information about Project Gutenberg™,
including how to make donations to the Project Gutenberg
Literary Archive Foundation, how to help produce our new
eBooks, and how to subscribe to our email newsletter to hear
about new eBooks.

Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.
More than just a book-buying platform, we strive to be a bridge
connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.
Join us on a journey of knowledge exploration, passion nurturing, and
personal growth every day!
ebookbell.com

Kafka Up And Running For Network Devops Set Your Network Data In Motion Eric Chou

More Related Content

Similar to Kafka Up And Running For Network Devops Set Your Network Data In Motion Eric Chou (20)

Recently uploaded (20)

Kafka Up And Running For Network Devops Set Your Network Data In Motion Eric Chou