Easy Cloud Native Transformation using HashiCorp Nomad

Easy Cloud Native Transformation
with HashiCorp Nomad
Bram Vogelaar
@attachmentgenie

$ whoami
• Used to be a Molecular Biologist
• Then became a Dev
• Now an Ops
• Currently Cloud Engineer @ The Factory
• Amsterdam HUG organizer

Horizontal Scaling / Load Balancers

And than stuff got complicated….

The story starts with my personal website

Nomad
l Open Source tool for dynamic workload scheduling
l Batch, containerized, and non-containerized applications.
l Has native Consul and Vault integrations.
l Has token based access setup.
l Jobs written in (H)ashiCorp (C)onfiguration (L)anguage
https://www.nomadproject.io/

job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
network {
port "http" {
to = 80
}
}
task "nginx" {
driver = "docker"
config {
image = "${PRIVATE}.dkr.ecr.us-east-1.amazonaws.com/blog:19"
ports = ["http"]
Deploy the blog

1 == None
job "blog" {
type = "service"
group "hugo" {
count = 2

job "blog" {
type = "service"
group "hugo" {
count = 2
constraint {
operator = "distinct_hosts"
value = "true"
}
Force onto different hardware

job "blog" {
type = "service"
group "hugo" {
count = 2
Spread {
attribute = "${node.datacenter}"
}
Suggest onto different hardware

/etc/nomad.d/config.hcl
Client {
Enabled = true
Meta {
"rack" = "his"
}
}
Based on custom meta-data

job "blog" {
type = "service"
group "hugo" {
count = 2
Spread {
attribute = "${meta.rack}"
target "his" {
percent = 50
}
target "her" {
percent = 50
}
}
Based on custom meta-data

service {
name = ”blog"
provider = "nomad"
port = ”http"
}
Service Definition

template {
data = <<EOH
http {
server {
listen 80;
location / {
{{ range nomadService ”blog" }}
proxy_pass http://{{ .Address }}:{{ .Port }};
{{ end }}
}
}
}
EOH
destination = "local/api-servers"
Service Usage

Nomad Pack
• Levant
• Templating and packaging tool
• Easily deploy popular applications to Nomad
• Re-use common patterns across internal applications
• Find and share job specifications with the Nomad community
• Nightlies only right now!
https://github.com/hashicorp/nomad-pack-community-registry

Nomad Pack
• nomad-pack registry list
• nomad-pack run hello_world
• nomad-pack run hello_world --var message=hola
https://github.com/hashicorp/nomad-pack

https://github.com/attachmentgenie/vagrant-scheduler
Try it yourself

Consul
l Open-Source Service Discovery Tool
l Build-in KV store
l Service Mesh tool
https://www.consul.io/

service {
name = "blog"
port = "http"
check {
type = "tcp"
interval = "10s"
timeout = "2s"
}
}
Service Definition

Check {
type = "tcp"
interval = "10s"
timeout = "2s"
check_restart {
limit = 3
grace = "10s"
ignore_warnings = false
}
}
Stampeding herd

group "hugo" {
Restart {
interval = "10m"
attempts = 2
delay = "15s"
mode = "fail"
}
task "nginx" {
Restart failed jobs

group "hugo" {
Count = 2
reschedule {
delay = "30s"
delay_function = "constant" #constant, exponential, fibonacci
unlimited = true # or max_delay = “1h”
}
task "nginx" {
Reschedule a job

• https://github.com/ThomasObenaus/dummy-services
• https://github.com/Shopify/toxiproxy
Testing your assumptions

group "hugo" {
Count = 10
update {
max_parallel = 2
min_healthy_time = "30s"
healthy_deadline = "5m"
}
task "nginx" {
Updates

group "hugo" {
Count = 10
Update {
max_parallel = 1
canary = 10
auto_revert = true
auto_promote = false
}
task "nginx" {
Blue/Green Release

group "hugo" {
Count = 5
Update {
max_parallel = 1
canary = 1
auto_revert = true
auto_promote = true
}
task "nginx" {
Canary Release

service {
name = "blog"
tags = ["v2"]
}
$version++

group "hugo" {
Count = 5
Update {
max_parallel = 1
canary = 1
auto_revert = true
auto_promote = false
}
task "nginx" {
Canary Release++

kind = "service-router"
name = "blog"
routes = [
{
match {
http {
header = [
{
name = "group"
exact = "test"
}, ] } }
destination {
service = "blog"
service_subset = "v2"
} },]
Consul to the rescue

● Introduced in/with Nomad 0.11
● (Currently) independently release cycle
● Gaining new functionality every release
● Build in Functionality for horizontal and vertical scaling
● But extendable by your own (community) plugins
Nomad autoscaler

● Makes decisions based on a checks
● Checks are a combination of
• Data queried from an APM
• Defined STRATEGY
• Attempt to approach TARGET value
● Multiple Checks can be combined
• Answer with the most resources will win!
• ScaleOut and ScaleIn => ScaleOut
• ScaleOut and ScaleNone => ScaleOut
• ScaleIn and ScaleNone => ScaleNone
• ScaleOut(10) and ScaleOut(9) => ScaleOut(10)
• ScaleIn(3) and ScaleIn(4) => ScaleIn(4)
Auto-scaling TLDR

• job "autoscaler" {
type = "service"
group "autoscaler" {
count = 1
task "autoscaler" {
driver = "docker"
config {
image = "hashicorp/nomad-autoscaler:0.3.6"
command = "nomad-autoscaler"
args = [
"agent",
"-config",
"${NOMAD_TASK_DIR}/config.hcl",
"-http-bind-address",
"0.0.0.0",
]
Deploy the autoscaler

• /etc/nomad.d/config.hcl
• nomad {
address = "http://{{env "attr.unique.network.ip-address" }}:4646"
}
apm "prometheus" {
driver = "prometheus"
config = {
address = "http://prometheus.service.consul:9090"
}
}
strategy "target-value" {
driver = "target-value"
}
Config for the autoscaler

Metrics
https://prometheus.io/

• job "blog" {
type = "service"
group "hugo" {
count = 3
scaling {
enabled = true
min = 1
max = 20
policy {
cooldown = "20s"
check "avg_instance_sessions" {
source = "prometheus"
query = "scalar(avg(traefik_service_open_connections{service="blog@consulcatalog"}))"
target = 5
}
Enable autoscaling for the blog

Dashboards
https://grafana.com/oss/grafana/

agent: querying APM: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=target-value target=local-nomad
agent: calculating new count: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=target-value target=local-nomad
agent: next count outside limits: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=
agent: updated count to be within limits: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=target-value target=local-nomad from=3 to=1 min=1 max=10
agent: scaling target: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=target-value
Observe the autoscaler

hey -z 1m -c 30 http://127.0.0.1:8000
Apply load

Logs
https://grafana.com/oss/loki/

group "autoscaler" {
count = 1
task "autoscaler" {
driver = "docker"
config {
image = "hashicorp/nomad-autoscaler:0.3.6"
command = "nomad-autoscaler"
logging {
type = "loki"
config {
loki-url = 'http://loki.service.consul:3100/api/prom/push'
tag = "loki"
}
}
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
Direct to loki

task "promtail" {
driver = "docker"
lifecycle {
hook = "prestart"
sidecar = true
}
config {
image = "grafana/promtail:2.5.0"
args = [
"-config.file",
"local/promtail.yaml",
]
Promtail sidecar

• scrape_configs:
- job_name: system
entry_parser: raw
static_configs:
- targets:
- localhost
labels:
task: autoscaler
__path__: /alloc/logs/autoscaler*
pipeline_stages:
- match:
selector: '{task="autoscaler"}'
stages:
- regex:
expression: '.*policy_id=(?P<policy_id>[a-zA-Z0-9_-]+).*source=(?P<source>[a-zA-Z0-9_-
]+).*strategy=(?P<strategy>[a-zA-Z0-9_-]+).*target=(?P<target>[a-zA-Z0-9_-]+).*Group:(?P<group>[a-zA-Z0-
9]+).*Job:(?P<job>[a-zA-Z0-9_-]+).*Namespace:(?P<namespace>[a-zA-Z0-9_-]+)'
https://grafana.com/docs/loki/latest/clients/promtail/
Promtail sidecar

• apm "prometheus" {
driver = "prometheus"
config = {
address = "http://prometheus.service.consul:9090"
}
}
• target "aws-asg" {
driver = "aws-asg"
config = {
aws_region = "{{ $x := env "attr.platform.aws.placement.availability-zone" }}{{ $length := len $x |subtract 1 }}{{ slice $x 0 $length}}"
}
}
Grow into your platform

• scaling "cluster_policy" {
policy {
cooldown = "2m"
evaluation_interval = "1m"
check "cpu_allocated_percentage" {
source = "prometheus"
query =
"scalar(sum(nomad_client_allocated_cpu{node_class="hashistack"}*100/(nomad_client_unallocated_cpu{node_class="hashistack"}+nomad_client_allocated_cpu{n
ode_class="hashistack"}))/count(nomad_client_allocated_cpu{node_class="hashistack"}))"
target = 70
}
}
target "aws-asg" {
dry-run = "false"
aws_asg_name = "${client_asg_name}"
node_class = "hashistack"
node_drain_deadline = "5m”
Grow into your platform

agent.worker.check_handler: querying source: check=mem_allocated_percentage policy_id=bf68649a-d087-2e69-362e-bbe71b5544f7 source=prometheus strategy=target-value target=a
agent.worker.check_handler: calculating new count: check=mem_allocated_percentage policy_id=bf68649a-d087-2e69-362e-bbe71b5544f7 source=prometheus strategy=target-value ta
agent.worker.check_handler: scaling target: check=mem_allocated_percentage policy_id=bf68649a-d087-2e69-36
internal_plugin.aws-asg: successfully performed and verified scaling out: action=scale_out asg_name=hashistack-n
agent.worker.check_handler: successfully submitted scaling action to target: check=mem_allocated_percentage policy_id=bf68649a-d087-2e69-362e-bbe71b5544f7 source=prometheus
Observe the autoscaler again

https://github.com/hashicorp/nomad-
autoscaler/tree/master/demo/remote
Try it yourself

Moving it all to the cloud – QED

Contact
bram@attachmentgenie.com
@attachmentgenie
https://www.slideshare.net/attachmentgenie
https://hashiconf.com/europe/ <= Running Trusted Payloads With Nomad and Waypoint

Questions ?
The Floor is yours…

Easy Cloud Native Transformation using HashiCorp Nomad

More Related Content

What's hot (20)

Similar to Easy Cloud Native Transformation using HashiCorp Nomad (20)

More from Bram Vogelaar (20)

Recently uploaded (20)

Easy Cloud Native Transformation using HashiCorp Nomad