Build and Run Streaming Applications with Apache Flink and Amazon Kinesis Data Analytics for Java Applications - Steffen Hausmann, AWS
0 likes600 views
The document provides an overview of building and running streaming applications with Apache Flink and Amazon Kinesis Data Analytics for Java applications. It discusses architecture, scaling, monitoring, logging, data protection, and best practices for implementing these services. Additional resources for further learning are also noted.
Build and Run Streaming Applications with Apache Flink and Amazon Kinesis Data Analytics for Java Applications - Steffen Hausmann, AWS
- 1. © 2019, Amazon Web Services, Inc. or its Affiliates. Dr. Steffen Hausmann (@sthmmm) Specialist Solutions Architect Analytics, EMEA Amazon Web Services Build and run streaming applications with Apache Flink and Amazon Kinesis Data Analytics for Java Applications
- 2. © 2019, Amazon Web Services, Inc. or its Affiliates. Architecture for Streaming Analytics with Apache Flink Amazon Kinesis Data Streams Amazon Kinesis Data Analytics for Java Applications Amazon Elasticsearch Service Ingestion layer Processing layer Presentation layer
- 3. © 2019, Amazon Web Services, Inc. or its Affiliates.© 2019, Amazon Web Services, Inc. or its Affiliates. Let’s go build!
- 4. © 2019, Amazon Web Services, Inc. or its Affiliates.© 2019, Amazon Web Services, Inc. or its Affiliates. Kinesis Data Analytics under the hood and best practices
- 5. © 2019, Amazon Web Services, Inc. or its Affiliates. Amazon Kinesis Data Analytics under the hood Kinesis Data Analytics manages the underlying infrastructure • Operates Zookeeper for high availability • Configures checkpoints and savepoints • Automatic failover to healthy nodes Kinesis Processing Units (or KPUs) • Basic scaling unit • Determines parallelism of the application • Can be further adapted for I/O bound applications with ParallelismPerKPU
- 6. © 2019, Amazon Web Services, Inc. or its Affiliates. Scaling and Autoscaling A scaling operation causes downtime of the application • Scaling triggers a savepoint and • starts a new application with adapted parallelism Autoscaling • Scale up triggered in minutes after a constant spike in CPU usage • Scale down triggered a few hours after a drop in CPU usage • Disable autoscaling if you are not CPU bound
- 7. © 2019, Amazon Web Services, Inc. or its Affiliates. Monitoring and Metrics Application metrics are exposed through Amazon CloudWatch • IncomingRecords/ Bytes • Write/ ReadProvisionedThroughputExceeded • millisBehindLatest
- 8. © 2019, Amazon Web Services, Inc. or its Affiliates. Logging Application logs are exposed through CloudWatch Logs • Send custom messages with Log4J & SLF4J • Avoid extensive logging on the data plane path • Search and analyze logs with CloudWatch Logs Insights
- 9. © 2019, Amazon Web Services, Inc. or its Affiliates. Data protection and security Data Protection • You can encrypt data on the incoming Kinesis data stream • All data stored in running application storage is encrypted at rest • You can encrypt data in transit and at rest in Elasticsearch service Avoid baking credentials into your code • Use temporary credentials for integrating with AWS services • Use AWS Secrets Manager to retrieve and rotate passwords kinesisConsumerConfig.setProperty( AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO" );
- 10. © 2019, Amazon Web Services, Inc. or its Affiliates. Further Readings The AWS Big Data blog is a great resource to learn more about Apache Flink and stream processing on AWS in general. • https://aws.amazon.com/blogs/big-data/build-and-run-streaming-applications- with-apache-flink-and-amazon-kinesis-data-analytics-for-java-applications/ • https://aws.amazon.com/blogs/big-data/build-a-real-time-stream-processing- pipeline-with-apache-flink-on-aws/ • https://github.com/aws-samples/amazon-kinesis-analytics-taxi-consumer
- 11. © 2019, Amazon Web Services, Inc. or its Affiliates.© 2019, Amazon Web Services, Inc. or its Affiliates. Thank you!