Big Data Europe: Simplifying Development and Deployment of Big Data Applications
1 like1,365 views
The document presents the Big Data Europe project focused on simplifying the development and deployment of big data applications through a structured architecture and various components. It outlines the project's goals, participants, and detailed descriptions of BDI stacks and their lifecycles, including hardware and software layers and support components. Additionally, it provides technical insights into integration procedures and examples of BDI stack assembly, particularly focusing on the Halyard stack.
Big Data Europe: Simplifying Development and Deployment of Big Data Applications
- 1. Big Data Europe: Simplifying Development and Deployment of Big Data Applications Ivan Ermilov @ Meet-up v MSD IT: Bioinformatics
- 2. Outline 2 ● About BigDataEurope Project ● Architecture ● BDI Stacks and BDI Stacks Lifecycle ● Supporting BDI Components
- 3. BigDataEurope: BDE Project 3 ● Horizon2020 project ● 17 partners ● 7 pilots in various domains ● > 30 Big Data components ● > 250 stars on github
- 5. Support Layer Init Daemon GUIs Monitor App Layer Traffic Forecast Satellite Image Analysis Platform Layer Spark Flink Semantic Layer Ontario SANSA Semagrow Kafka Real-time Stream Monitoring ... ... Resource Management Layer (Swarm) Hardware Layer Premises Cloud (AWS, GCE, MS Azure, …) Data Layer Hadoop NOSQL Store CassandraElasticsearch ...RDF Store Platform Architecture 5
- 6. BDI Stack Lifecycle App templates Ready-made components, best practices BDI Stack Builder BDI Support Components, Instructions Swarm UI, docker-compose Pipeline/Logg ing Monitor 6
- 9. Adding Custom Apps FROM bde2020/spark-submit:2.1.0-hadoop2.7 ENV ENABLE_INIT_DAEMON=false ENV SPARK_APPLICATION_PYTHON_LOCATION= ENV SPARK_MASTER_NAME=sc6-spark-master ENV SPARK_APPLICATION_ARGS= ENV SPARK_MASTER_URL=spark://sc6-spark-master:7077 ENV SPARK_MASTER_PORT=7077 ... 9
- 11. Logging Monitor ● Network logging for HTTP ○ Capture network interface as PCAP ○ Convert to HAR (json) ○ Expand HAR ○ Dump into ELK stack 11
- 13. UI Integrator 13
- 14. Reverse Proxy/CSSWrapper ● Simple injection of custom CSS strabon: image: bde2020/strabon links: - csswrapper expose: - "8080" environment: VIRTUAL_HOST: "strabon.big-data-europe.aksw.org" VIRTUAL_PORT: "8080" CSS_SOURCE: "strabon" 14 https://www.big-data-europe.eu/using-reverse-proxy-inside-bde-platform-jwildernginx-setup-for-docke r-swarm/ https://github.com/big-data-europe/demo-integrator-ui
- 15. Swarm UI 15
- 16. Swarm UI 16
- 18. Example BDI Stack: Halyard ● Which BD components does Halyard use? ○ HDFS ○ YARN (for MapReduce jobs) ○ HBase ● Which interfaces are supported? ○ Shell scripts (bulkload, export etc) ○ RDF4J console ○ RDF4J REST Server + Workbench 18
- 19. Halyard: BDI Stack ● Hadoop ○ DFS ○ YARN ● HBase ● Zookeeper 19
- 20. Halyard: BDI Stack (yaml) 20 namenode: image: bde2020/hadoop-namenode:1.2.0-hadoop2.8-java8 container_name: namenode networks: - hbase volumes: - ./data/hadoop/namenode:/hadoop/dfs/name environment: - CLUSTER_NAME=test ports: - "50070:50070" env_file: - ./hadoop.env
- 21. Halyard: BDI Stack: Running Simply execute the command: docker-compose up -d 21
- 22. Adding Halyard to BDI Stack FROM bde2020/hadoop-base:1.2.0-hadoop2.8-java8 as hadoop-base FROM bde2020/hbase-base:1.0.0-hbase1.2.6 as hbase-base FROM openjdk:8 MAINTAINER Ivan Ermilov <[email protected]> ENV HADOOP_VERSION=2.8.0 COPY --from=hadoop-base /opt/hadoop-$HADOOP_VERSION /opt/hadoop-$HADOOP_VERSION RUN ln -s /opt/hadoop-$HADOOP_VERSION/etc/hadoop /etc/hadoop ENV PATH /opt/hadoop-$HADOOP_VERSION/bin:$PATH ENV HBASE_VERSION=1.2.6 COPY --from=hbase-base /opt/hbase-$HBASE_VERSION /opt/hbase-$HBASE_VERSION RUN ln -s /opt/hbase-$HBASE_VERSION/conf /etc/hbase ENV PATH /opt/hbase-$HBASE_VERSION/bin:$PATH ENV HALYARD_VERSION 1.2 … 22
- 23. Running Halyard SDK $ docker run -it --rm --network hbase --env-file ./hbase.env bde2020/halyard-sdk:1.0.0-halyard1.2 /bin/bash $ ./console 23
- 24. Running Halyard SDK 24 > create hbase > load http://danbri.org/foaf.rdf
- 25. Running Halyard SDK 25 > sparql select ?s ?p ?o {where ?s ?p ?o} . .
- 26. Halyard: BDI Stack (complete) 26
- 27. Thank you 27 Questions? Github: https://github.com/earthquakesan @AKSW: http://aksw.org/IvanErmilov.html Email: [email protected] Twitter: @earthquakesan LinkedIn: https://www.linkedin.com/in/iermilov/