Introduction to Amazon Kinesis Data Streams
0 likes92 views
This document provides an introduction and overview of Amazon Kinesis Data Streams. It begins with definitions of streaming data and examples. It then introduces Amazon Kinesis Data Streams as a scalable real-time data streaming service. It outlines the basic architecture involving producers, streams, shards, and consumers. It defines key concepts like records, partitions, and capacity modes. It also lists some basic operations on streams like creating, describing, putting records, and getting records. Finally, it concludes with references for further reading.
Introduction to Amazon Kinesis Data Streams
- 1. Presented By: Prateek Gupta Introduction to Amazon Kinesis Data Streams
- 2. Lack of etiquette and manners is a huge turn off. KnolX Etiquettes Punctuality Join the session 5 minutes prior to the session start time. We start on time and conclude on time! Feedback Make sure to submit a constructive feedback for all sessions as it is very helpful for the presenter. Silent Mode Keep your mobile devices in silent mode, feel free to move out of session in case you need to attend an urgent call. Avoid Disturbance Avoid unwanted chit chat during the session.
- 3. Our Agenda 02 Amazon Kinesis Data Streams 03 High-Level Architecture 04 Key Concepts and Terminology 05 Basic Operations 01 What is Streaming Data? 06 Demo
- 5. What is Streaming Data? Streaming data refers to the data that is generated continuously in real time by thousands of data sources and delivered to a system for processing. Key Points: ● Real-time ● Continuous flow ● Variety of sources ● Variety of formats ● Requires specialized processing Examples: ● Ecommerce purchases ● Game data ● Information from social networks ● Log data ● Stock prices ● GPS data ● IoT Sensor Data
- 6. Amazon Kinesis Data Streams Amazon Kinesis Data Streams is a real-time streaming data service by AWS. It makes it easy to collect and process real-time streaming data at high scale. Some key points to understand: ● Real-time data ● Highly Scalable ● Data sources ● Processing ● Cost-effective ● Easy to use
- 7. High-Level Architecture ● The producers continually push data to Kinesis Data Streams, and the consumers process the data in real time. ● Once the processing is done by the consumer, the result are stored using an AWS service such as Amazon DynamoDB, Amazon Redshift, or Amazon S3.
- 8. Key Concepts and Terminology ➢ Producer: It is an application that puts the data records into Amazon Kinesis Data Streams. ➢ Consumer: It is an application that retrieves the data records from Amazon Kinesis Data Streams and process them. ➢ Kinesis Data Stream: ○ A Kinesis data stream is a set of shards. ○ Each shard has a sequence of data records. ○ Each data record has a sequence number. ○ Data retains for 24 hours by default.
- 9. ➢ Shard: ○ A shard is a uniquely identified sequence of data records ○ A stream is composed of one or more shards, each of which provides a fixed unit of capacity. ○ Each shard can support up to 1000 PUT records per second(or 1MB/sec), and up to 1,000 GET records per second(or 2MB/sec) ○ The data capacity of a stream is a function of the number of shards. ○ If the data rate increases, increase the number of shards allocated to the stream. ➢ Data Record: ○ A data record is the unit of data stored in a Kinesis data stream. ○ Each data record is composed of a sequence number, a partition key, and a data blob(up to 1MB).
- 10. ➢ Sequence Number: ○ A sequence number is a unique identifier for each data record. ○ Allows to read data in the order and also to determine which records have been processed ➢ Partition Key: ○ A partition key is a meaningful identifier that is associated with each record. ○ It is used by the service to determine which shard to store the record in. ○ Specified by the data producer while putting data into a data stream ○ Records with the same partition key are stored together in the same shard. ➢ Retention Period: ○ Amount of time that data records are stored in an Amazon Kinesis Data Stream. ○ Default data retention period for a stream is 24 hours(configurable upto 365 days)
- 11. ➢ Capacity Mode: ○ The capacity mode determines how capacity is managed and the usage charges for a data stream. ○ Currently, in Kinesis Data Streams, we can choose between an on-demand mode and a provisioned mode for our data streams.
- 12. Basic Operations Amazon Kinesis Data Streams provides a number of operations that can be performed on a data stream. Here are some basic operations: ● create-stream ● describe-stream ● list-streams ● put-record ● get-shard-iterator ● get-records ● split-shard ● merge-shards ● delete-stream
- 13. Demo
- 14. References ● Kinesis Data Streams Official Documentation ● AWS Kinesis - Javatpoint
- 15. Thank You !