Dataservices: Processing Big Data the Microservice Way
2 likes•544 views
The document discusses the implementation of data services and microservices for processing big data, emphasizing the transition from monolith to cloud-native architectures. It outlines the components and frameworks suitable for building scalable and efficient data services, with a focus on various messaging patterns, open source technologies, and cloud platforms. Additionally, it covers practical examples and Kubernetes definitions for managing microservice deployments.
Dataservices: Processing Big Data the Microservice Way
- 1. Mario-Leander Reimer [email protected] @LeanderReimer Dataservices Processing Big Data the Microservice Way New York, Feb 27, 2018
- 2. Mario-Leander Reimer Chief Technologist, QAware GmbH Contact Details Mail: [email protected] Twitter: @LeanderReimer Github: https://github.com/lreimer/data-services-javaee7 27.02.18 3 Developer && Architect 20+ years of experience #CloudNativeNerd Open Source Enthusiast
- 3. We want to go to the cloud … 4
- 4. 5 Device The System Traffic Data Historical Data Map Data Vehicle Data
- 5. 6 The system. The data center.
- 6. Enter Dataservices. { Big + Fast + Smart } Data Micro- services
- 7. BIG DATA All things distributed: Distributed Processing Distributed Databases 8 FAST DATA Low latency and high throughput: Stream processing Messaging Event-driven Data to information: Machine (deep) learning Advanced statistics Natural Language Processing SMART DATA
- 8. 9
- 9. 10 Components All Along the Software Lifecycle. DESIGN § Complexity unit § Data integrity unit § Coherent and cohesive features unit § Decoupled unit Design Components RUN § Release unit § Deployment unit § Runtime unit (crash, slow-down, access) § Scaling unit Ops Components n:1 NEW ! BUILD § Planning unit § Team assignment unit § Knowledge unit § Development unit § Integration unit Dev Components 1:1
- 10. 11 Dev Components Ops Components?:1 System Subsystems Components Services Good starting point Decomposition Trade-Offs Microservices Nanoservices Macroservices Monolith More flexible to scale Runtime isolation (crash, slow-down, …) Independent releases, deployments, teams Higher utilization possible - Distribution debt: Latency - Increasing infrastructure complexity - Increasing troubleshooting complexity - Increasing integration complexity
- 11. 12http://martinfowler.com/bliki/MonolithFirst.html We are here. We need to go here.
- 12. Decomposing the existing monolith was realistic. 13
- 13. 14 The basic idea: Input – Processing – Output. Data processing using a graph of microservices. I1 Sources P1 Pn Processors O1 Sinks Microservice (aka Dataservice) Message Queue
- 14. 15 Possible messaging patterns applied for reliable and flexible communication between dataservices. P1 C1Q1 Message Passing P1 C1 Q1 Cn Work Queue P1 C1Q1 CnQn Publish/Subscribe P1 C1 Q1 Q2 Remote Procedure Call
- 15. 16 The basic idea: Cloud-native platform for micro- and dataservices. CLUSTER OPERATING SYSTEM MICROSERVICE PLATFORM DATASERVICE PLATFORM DATASERVICES MICROSERVICES MESSAGING IMDG
- 16. 17 Some Open Source Dataservice Platforms. Standardized API with several open source implementations Microservices: JavaEE micro container Messaging: JMS, MQTT, Kafka, SQS Platforms: Docker, Kubernetes, OpenShift, DC/OS Stream processing tightly integrated with Kafka Microservices: main() Messaging: Kafka, Kafka Streams Platforms: any Kafka runs on Open source by Lightbend Microservices: Lagom, Play Messaging: akka Platforms: Conductr, ??? Open source project based on the Spring stack Microservices: Spring Boot, Spring Cloud Stream & Task Messaging: Kafka, RabbitMQ Platforms: PCF, Kuberntes, YARN, Mesos Java EE 7 / 8 Kafka Streams Lagom Framework Cloud Cloud Data Flow
- 17. Overview of Java EE 7 APIs suited for Dataservices. 18 CDI Extensions Web Fragments Bean Validation 1.1 CDI 1.1 Managed Beans 1.0 JCA 1.7 JPA 2.2JMS 2.0 JSP 2.3 EL 3.0 EJB 3.2 Batch 1.0 JSF 2.2 Interceptors 1.2 Mail 1.5 Common Annotations 1.3 JTA 1.2 JAX-WS 1.4 JAX-RS 2.0 Concurrency 1.0 JSON-P 1.0 WebSocket 1.1 JASPIC 1.1 JACC 1.5 Servlet 3.1 JCache 1.0
- 18. @MessageDriven(activationConfig = { @ActivationConfigProperty(propertyName = "serverURIs", propertyValue = "tcp://eclipse-mosquitto:1883"), @ActivationConfigProperty(propertyName = "cleanSession", propertyValue = "false"), @ActivationConfigProperty(propertyName = "automaticReconnect", propertyValue = "true"), @ActivationConfigProperty(propertyName = "filePersistence", propertyValue = "false"), @ActivationConfigProperty(propertyName = "connectionTimeout", propertyValue = "30"), @ActivationConfigProperty(propertyName = "maxInflight", propertyValue = "3"), @ActivationConfigProperty(propertyName = "keepAliveInterval", propertyValue = "5"), @ActivationConfigProperty(propertyName = "topicFilter", propertyValue = "de/qaware/oss/cloud/mqtt"), @ActivationConfigProperty(propertyName = "qos", propertyValue = "1") }) public class MqttSourceMDB implements MQTTListener { @OnMQTTMessage @TransactionAttribute(value = TransactionAttributeType.REQUIRED) @Transactional(Transactional.TxType.REQUIRED) public void onMQTTMessage(String topic, MqttMessage message) { JsonReader reader = Json.createReader(new ByteArrayInputStream(message.getPayload())); JsonObject jsonObject = reader.readObject(); // TODO do stuff with the JSON payload } } 19 Simple Message Driven Beans to receive messages. This also works for MQTT, Kafka, Amazon SQS, … For other JCA adapters visit https://github.com/payara/Cloud-Connectors
- 19. JsonObject currentWeather = Json.createObjectBuilder() .add("city", “London") .add("weather", “Drizzle") .build(); StringWriter payload = new StringWriter(); JsonWriter jsonWriter = Json.createWriter(payload); jsonWriter.writeObject(currentWeather); TextMessage msg = session.createTextMessage(payload.toString()); msg.setJMSType("CurrentWeather"); msg.setStringProperty("contentType", "application/vnd.weather.v1+json"); @ActivationConfigProperty(propertyName = "messageSelector", propertyValue = "(JMSType = 'CurrentWeather') AND (contentType = 'application/vnd.weather.v1+json‘)“) JsonReader reader = Json.createReader(new StringReader(body)); JsonObject jsonObject = reader.readObject(); 20 Use JSON-P to build your JsonObject and JsonArray instances. Use JSON-P to read JSON payloads. Use JSON-P to traverse and access JSON objects and arrays. Upcoming in Java EE 8: JSON Pointers and JSON Patch add even more flexibility. Use Mime-Type versioning for your JSON messages if required. Use JMS message selectors to filter on JMS type and content type. Alternatively use flexible binary protocols like ProtoBuf. Use JSON as payload format for loose coupling. Use JSON-P to implement tolerant reader pattern.
- 20. Cloud-ready runtimes suited for Dataservices. 21 … and many more.
- 21. Overview of the demo showcase. 22 JDBC Source Weather Processor Weather File Sink Weather DB Sink REST Source JAX-RS JMS MQTT Source JSON-P JMS Kafka Source JSON-P JMS CSV Source JBatch JMS JBatch JMS CSV In-Memory Datagrid Topic Queue Topic https://github.com/lreimer/data-services-javaee7 Location Processor JSON-P JMS JCache JSON-P JMS JCache CSV JMS JSON-P JPA JMS JSON-P JPA
- 22. Conceptual View on Kubernetes Building Blocks. 23
- 23. Most important Kubernetes concepts. 24 Services are an abstraction for a logical collection of pods. Pods are the smallest unit of compute in Kubernetes Deployments are an abstraction used to declare and update pods, RCs, … Replica Sets ensure that the desired number of pod replicas are running Labels are key/value pairs used to identify Kubernetes resources
- 24. apiVersion: extensions/v1beta1 kind: Deployment metadata: name: location-processor spec: replicas: 2 strategy: type: RollingUpdate template: metadata: labels: io.kompose.service: location-processor spec: containers: - name: location-processor image: lreimer/location-processor:1.0 ports: - containerPort: 8080 - containerPort: 5701 Example K8s Deployment Definition. 25
- 25. resources: # Define resources to help K8S scheduler # CPU is specified in units of cores # Memory is specified in units of bytes # required resources for a Pod to be started requests: memory: “196Mi" cpu: "250m" # the Pod will be restarted if limits are exceeded limits: memory: “512Mi" cpu: "500m" Resource Constraints Definition. 26
- 26. # container will receive requests if probe succeeds readinessProbe: httpGet: path: /api/application.wadl port: 8080 initialDelaySeconds: 30 timeoutSeconds: 5 # container will be killed if probe fails livenessProbe: httpGet: path: /admin/health port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 Liveness and Readiness Probes for Antifragility. 27
- 27. apiVersion: v1 kind: Service metadata: labels: io.kompose.service: location-processor name: location-processor spec: type: NodePort ports: - name: "http" port: 8080 targetPort: 8080 selector: io.kompose.service: location-processor Example K8s Service Definition. 28
- 28. Programmable MIDI Controller. Visualizes Deployments and Pods. Scales Deployments. Supports K8s, OpenShift, DC/OS. http://github.com/qaware/kubepad/ Java EE powered Dataservices on Kubernetes in Action. 29
- 29. Fork me on Github. https://github.com/lreimer/data-services-javaee7
- 30. Mario-Leander Reimer [email protected] @LeanderReimer xing.com/companies/qawaregmbh linkedin.com/company/qaware-gmbh slideshare.net/qaware twitter.com/qaware youtube.com/qawaregmbh github.com/qaware