AdminOps-Part-II-Installation.pdf of okera
- 1. AdminOps-Part-II-Installation August 24, 2020 1 Part II - Installation Learn how to install the Okera Active Data Access Platform (ODAP). 1.1 Table of Contents • Section 1.2 – Section 1.2.1 – Section 1.2.2 • Section 1.3 – Section 1.3.1 – Section ?? – Section 1.3.3 • Section 1.4 – Section 1.4.1 – Section 1.4.5 – Section 1.4.6 • Section 1.5 • Section 1.6 – Section 1.6.1 – Section 1.6.7 • Section 1.7 • Section 1.8 • Section 1.9 1.2 Deployment Considerations There are two choices to make before deploying ODAP: 1. Dependent Infrastructure This splits into the following sub-choices: • Infrastructure Provider: Amazon Web Services (AWS), Microsoft Azure, Google Cloud, On-Premises • Compute Service: AWS EC2 or EKS, AKS, GKS, OpenShift 1
- 2. • RDBMS Service: AWS RDS MySQL or Aurora, Azure Database, Cloud SQL, Managed RDBMS • Managed Storage: AWS S3, Azure ADLS, Cloud Storage 2. Cluster Types ODAS clusters can be used for different purposes, for example, as a shared metadata service only, or as an ephemeral data access cluster. This will affect how many nodes you need and the size of each node, as well as the configuration of the ODAS clusters themselves. 1.2.1 Cluster Types ODAS clusters can be used for different purposes, which may affect how they are configured. The following are possible cluster types: • Data Plane Cluster The default setup of an ODAS cluster, containing all services. • Metadata Service Cluster ODAS clusters can be configured to share services from other ODAS clusters. One of those use-cases is to have a shared ODAS cluster provide the metadata services, including the Schema Registry, and Policy and Audit Engine. • Data Access Cluster This is the counterpart to the previous type, that is, the services of another ODAS cluster are configured to get the metadata from the shared services, as opposed to using its own. In other words, this cluster is just offering data access service, but not any metadata services. 1.2.2 Cluster Types - Example Here two data access clusters are sharing a long-running metadata service cluster instance. 2
- 3. 1.3 Prerequisites Before installing the cluster, you need to further decide on the infrastructure, as well as ODAS specific settings. While discussed here as well, also consider reading the following documents: • Install Prerequisites for AWS and Azure • Service Ports • Release Notes • Gravity Requierments 1.3.1 General Prerequisites The following general prerequisites apply: • A 64-bit Linux machine. For example, for AWS we recommend Amazon Linux 2. • The Docker daemon (dockerd) should not be running. • You should not be running anything on port 53 (e.g. dnsmasq) • You should not be running local-dns, as it is not compatible with Kubernetes. • You should have at least 150 GiB of disk space available in an EBS volume, mounted at /. • While discussed below, the general machine type used should have – more than 8 cores (aka vCPUs), ideally starting with 16 cores and up – at least 32 GiB of memory, though eventually memory is pegged – more than 1 GigE network connectivity, ideally starting at 10 GigE and up 1.3.2 Networking Prerequisites For networking, you should configure a separate Security Group (or similar named, grouped net- work firewall rules). All outgoing connections should be allowed. For incoming connections, consider the following rules: Type Protocol Port Range Source Description SSH TCP 22 <admin cidr> SSH Custom TCP Rule TCP 8083 <vpc cidr> ODAS REST API/WebUI Custom TCP Rule TCP 12050 <vpc cidr> ODAS Planner API Custom TCP Rule TCP 12051 <admin cidr> ODAS Planner Diagnostics Custom TCP Rule TCP 13050 <vpc cidr> ODAS Worker API Custom TCP Rule TCP 13051 <admin cidr> ODAS Worker Diagnostics Custom TCP Rule TCP 14050 <vpc cidr> ODAS Presto API Custom TCP Rule TCP 32009 <admin cidr> ODAS Diagnostics All traffic All All <sg id> ODAS Internal Traffic 3
- 4. Notes: • IPv6 addresses are omitted for the sake of brevity, they follow the usual rules if configured • Replace <sg id> with your security group identifier, for instance, sg-0655cc29afad577d3 (for AWS) – This is rquired to allow all traffic between the machines in the same security group • Replace <admin cidr> with the CIDR address range where you will do administrative tasks to the cluster, such as SSH or viewing diagnostic UIs • Replace <vpc cidr> with the CIDR address range where other hosts are located requiring access the cluster, such as AWS EMR, Databricks, Tableau, and so on • By default, no processes are bound to ports 12051 and 13051 - but they are open to allow for diagnostics when needed. 1.3.3 IP Address Management Typically the IP address range of the host machines is part of the virtual machine service settings, for AWS this is part of the VPC. You need to ensure you have enough IP addresses available to run the servers you require (see next sub-section). On top of that, all containers in the cluster have their own IP address, which must not overlap with the host machine range. Also see Kubernetes Networking and Network Issues for details. 4
- 5. 1.3.4 IP Address Management (contd.) 1.4 Cluster Sizing Dependent on the use of Okera in your enterprise you may want to run one or more ODAS cluster. These clusters are sized based on their primary workload. For example, you may have a cluster for ad-hoc users, and another one for scheduled data pipelines. The following related topics are discussed next: * Per Node Memory Usage * Number of Nodes * Instance Types 1.4.1 Per Node Memory Usage The following are the general assumptions to be made about Okera’s access services: 5
- 6. • Available Memory Okera assumes that the machine and its resources is fully available for maximum use. In other words, there should be no other, non-Okera services running on the same machines. This is important especially for the ODAS Worker processes. • Managed Memory Allocation ODAS uses long-lived memory allocations with pooling, combined with efficient memory allocation for worker threads (TCMalloc). • Monitoring Resources Okera also assumes that monitoring is focused on other, more short-lived and bursty re- sources, such as I/O and CPU load. 1.4.2 Per Node Memory Usage (contd.) As mentioned, some ODAS processes need more memory than others. The following gives a brief overview: Process Memory Network Database Notes Web UI Low Low None Lightweight, uses REST Server for all functionality REST Server Medium Medium Medium For generic clients, streams data and metadata Catalog Low Low Medium Serves metadata, which might be larger (e.g. many partitions) Canary Low Low None Infrastructure services, not part of data or metadata paths ZooKeeper Low Low None Internal or- chestration service, holds cluster membership 6
- 7. Process Memory Network Database Notes Planner Medium/High Medium High Handles metadata, which may be extensive Worker High High None Handles all data I/O, caches data blocks 1.4.3 Per Node Memory Usage (contd.) More specifically, for the workers, the amount of memory needed for each node is roughly: (memory per task * number of concurrent tasks) + cache memory Assuming that the memory per task is usually in the low megabyte range, the overall memory needed should be in the low to mid gigabyte range, for example, 10 to 40 GB. See Memory Usage for more details. 1.4.4 Per Node Memory Usage (contd.) The diagram shows the execution of queries in stages (first plan, then execute) and how that affects the ODAS services: 7
- 8. 1.4.5 Number of Nodes The amount of servers in an ODAS cluster is dependent on the primary workload. For example, if you use a 100 node compute cluster, you will need about 10 ODAS nodes (of the same machine type, see next slide) for batch workloads. In case of specific low-latency requirements, for example doing ad-hoc queries with Presto, this ratio can drop to 4:1, that is, four compute nodes to 1 ODAS node. 1.4.6 Instance Types The instance types of the ODAS nodes is driven by the aforementioned ratio to the primary workload producer. It is vital though to not under-size the ODAS nodes to avoid running into memory issues early on. In practice it is common to have multiple environments, for example for development, as well as user-acceptance-testing (UAT) and production. The former usually need a smaller ODAS cluster than the latter ones. 8
- 9. 1.4.7 Instance Types (contd.) The following suggestions are for consideration only. Ensure that you are aware of dependent restrictions (such as network and storage throughput, or CPU throttling) based on the selected instance size. Proof of Concept/Development Cluster: Node Type Generic Instance Type AWS Instance Type Azure Worker 8+ cores, 16+ GiB RAM t3.2xlarge / m5.2xlarge / c5.4xlarge Standard_B8ms / Standard_D8_v3 / Standard_F16s_v2 Production Cluster: Node Type Generic Instance Type AWS Instance Type Azure Worker 16+ cores, 32+ GiB RAM m5.2xlarge / c5n.9xlarge Standard_D16_v3 / Standard_F32s_v2 See Cluster Sizing for more details. 1.5 Installation Types After deciding on the instance types and number of VMs needed, you can proceed to set up the Master machine. You have the choice of two types of installation: 1. Deployment on bare Linux machines 2. Deployment into an existing Kubernetes service Not all platforms and vendors are supported for these types, with the following table listing your choices: Platform Bare Linux Kubernetes Service Amazon Web Services Yes (EC2) Yes (EKS) Microsoft Azure No Yes (AKS) 9
- 10. 1.5.1 Installation Types (contd.) As mentioned in Part I - Introduction, both options end up running ODAS in a managed Kubernetes service. The difference is that option #1 is resulting into an autonomous Gravity-managed cluster, while option #2 is dependent on an existing, managed Kubernetes cluster. Once the cluster is installed either way, they are controlled from the Master node, which has the Okera tools installed to communicate with the cluster services. The (high-level) installation steps are: • Create a VM (or provide a bare-metal Linux machine) used as the Master node • Download the tarball and unpack it • Install (for option #1) or deploy (for option #2) ODAS • Modify the cluster configuration YAML file • Update the cluster with the new configuration Note: Optionally you can use the Okera provided install.sh script to handle some of these steps for you. 1.6 Master Node Installation After deciding on the instance types and number of VMs needed, you can proceed to set up the Master machine. The requirements differ depending on the installation type chosen: 1. Bare Linux In this case, the Master runs an entire Kubernetes Control Plane, as well as all Okera services. In other words, the machine needs to be at least as powerful as any other Worker node. 2. Existing Kubernetes Service Here the Master node is very lightweight as it only is used to execute the provided command- line tools. This means it can be a very small VM instance, or even your local personal computer. Each are discussed separately next. 1.6.1 Master Node Installation - Bare Linux Once you have provisioned the machine with a basic Linux set up (for example, an EC2 instance with Amazon Linux 2), log into the machine using SSH and run the following: $ # Easy install option $ curl https://okera-release-uswest.s3.amazonaws.com/2.0.3/gravity/install.sh | sh - This does the following: 10
- 11. • Creates a local directory that reflects the ODAS version (for example, okera-2.0.3) • Determines the closest region containing the installation files (tarball) • Downloads the single, autonomous installation tarball containing everything required • Prepares the installation and automation scripts in the current directory Note: The tarball is 6GB+ in size, make sure you have enough resources available 1.6.2 Master Node Installation - Bare Linux (contd.) Alternatively, you can either download the install.sh script and use a few extra command-line options to modify the default installation process: $ # Customize install script options $ curl -O https://okera-release-uswest.s3.amazonaws.com/2.0.3/gravity/install.sh $ sh install.sh --quickstart --token MySEkReTToKen#123 You have these options: Option Description --conf-file Allows you to hand in an existing cluster configuration file --quickstart Uses the provided configs/config-quickstart.yaml cluster configuration file --token Can be used to specify a custom token used for Gravity to manage clusters Note: If you do not specify a token, a random one is created automatically. 1.6.3 Master Node Installation - Bare Linux (contd.) Alternatively, you can completely bypass the installation script and perform the installation man- ually like so: $ # Custom setup $ mkdir okera-2.0.3 && cd okera-2.0.3 $ curl -O https://okera-release-uswest.s3.amazonaws.com/2.0.3/gravity/odas.tar $ tar -xvf odas.tar $ ./okctl prepare --config=configs/config-quickstart.yaml 1.6.4 Master Node Installation - Bare Linux (contd.) After the previous steps you should see the following files in your Okera installation directory: 11
- 12. 1.6.5 Master Node Installation - Bare Linux (contd.) Here a quick summary of what is available in the installation directory (omitting some of the system files): File/Directory Description README-odas.md A Markdown formatted file containing a lot of important and useful information about ODAS and its configuration. configs A directory with a few prebuilt cluster configuration files and templates. create.cmd Script that can be used to install ODAS, see next slide. gravity The Gravity command-line tool, can be used to manage the Gravity setup. join.asg.cmd Script containing the commands needed to automatically join a new node when provisioned. join.cmd Similar to before, but just containing the Gravity join commands. Requires for Gravity tooling to be installed already. okctl The Okera command-line tool to manage the ODAS cluster. 1.6.6 Master Node Installation - Bare Linux (contd.) Once the Master is prepared you can install the cluster itself, including Kubernetes and Okera services. Note: The --quickstart or --conf-file options of the install script already includes this steps and the cluster configuration as well. $ # Install services $ ./create.cmd This brings up a basic ODAS cluster, with default, minimal configuration. Next and final step is to configure the cluster, which is shown below in Section ??. 1.6.7 Master Node Installation - Existing Kubernetes Service If you already have an existing Kubernetes cluster, you can use your local machine or any VM that has kubectl, the Kubernetes client tool, installed. You also need to ensure that the configuration 12
- 13. of kubectl has been updated so that it can talk to the cluster. Test this by running the following command, also ensuring the listed nodes belong to the proper Kubernetes cluster: $ kubectl get nodes -owide NAME STATUS ROLES AGE VERSION INTERN ip-10-1-10-96.us-west-2.compute.internal Ready <none> 38m v1.12.10-eks-ffbd96 10.1.1 ip-10-1-2-109.us-west-2.compute.internal Ready <none> 44m v1.12.10-eks-ffbd96 10.1.2 1.6.8 Master Node Installation - Existing Kubernetes Service (contd.) Download the specific installation tarball for existing Kubernetes clusters like so: $ mkdir okera-2.0.3 && cd okera-2.0.3 $ curl -O https://okera-release-uswest.s3.amazonaws.com/2.0.3/gravity/okctl.tar $ tar -xvf okctl.tar This tarball is quite small and mostly contains the okctl command-line tool and the necessary Kubernetes YAML files for the Okera services. Then run the deploy command to apply the provided Okera YAML files to the cluster: $ ./okctl deploy --arch aws Note: okctl uses kubectl internally to communicate with the configured Kubernetes cluster 1.7 Minimal Cluster Configuration After the setup is complete the cluster is running in a minimal configuration. In this minimal mode, the cluster has no authentication enabled, no TLS network encryption set up, and lacks the ability to add new users. The cluster also uses an embedded PostgreSQL database system instance to store the metadata locally. You can use the following fixed credentials to log into the cluster’s WebUI (using http:// to skip TLS): Username Password Description root root The system administrator that access everything with this ODAS cluster. okera okera Provides a non-privileged user account that has no special access. 13
- 14. 1.8 LAB - Verify Cluster Setup Goal: Verify the cluster is available in its minimal configuration Time: 15 mins URL: http://training.demo.okera.rocks:8083 Tasks: • Access the WebUI and log in as an administrator – Username: root – Password: root • Explore the Datasets tab – Click on the okera_sample.sample table – Click on the “Preview” quick action button in the table details pane 1.9 Advanced Installation Topics The more advanced setup topics are discussed in Part III - Administration Continue with Part III - Administration 14