Terraform at Scale

Terraform at Scale
Hashiconf
Calvin French-Owen
Co-Founder of Segment
@calvinfo
September 7, 2016

How do we move
nimbly–while adding
people?

This talk
- Terraform at Segment
- What makes “good” Terraform
- What’s next

By the numbers
- 16 developers working with Terraform
- 94 microservices
- thousands of AWS resources

A year with Terraform
December 2012 – Launch day
April 2015 – Terraform first attempt (v1)
November 2015 – Terraform “redux” (v2)

Migrating to Terraform
April 2015

1. AWS accounts per environment

dev stage prod
old
prod
vpc peering

dev stage prod
old
prod
vpc peering
managed by Terraform

Separate accounts
- confidence to apply ‘at will’
- test the waters without screwing up the old account
- any sort of ‘global’ configs are okay

1. AWS accounts per environment
2. Docker and ECS

Terraform (our first attempt)
├── Makefile
├── README.md
└── environments
├── dev
├── production
└── stage

environments/stage
├── api.tf
├── bastion.tf
├── dns.tf
├── elasticache.tf
├── elbs.tf
├── iam.tf
├── outputs.tf
├── redis.tf
├── s3.tf
├── terraform.tfstate
├── terraform.tfvars
└── vpc.tf

resource "aws_ecs_task_definition" "app" {
family = "app"
container_definitions = <<EOF
[
{
"cpu": 1024,
"memory": 768,
"environment": [
{
"name": "NODE_ENV",
"value": "stage"
}
],
"image": "segment/app:1.54.14",
"name": "app",
"portMappings": [
{
"containerPort": 8000,
"hostPort": 8000
}
]
}
]
EOF
}

Life was better!Life was
better…

Life was better!Life was
better…
but not
good.

Terraform first attempt
├── Makefile
├── README.md
└── environments
├── ops
├── production
└── stage

family = "app"
[
{
"cpu": 1024,
"memory": 768,
"environment": [
{
"name": "NODE_ENV",
"value": "stage"
}
],
"name": "app",
"portMappings": [
{
"hostPort": 8000
}
]
}
]
EOF
}
<= stage

family = "app"
[
{
"cpu": 1024,
"memory": 768,
"environment": [
{
"name": "NODE_ENV",
"value": "stage"
}
],
"name": "app",
"portMappings": [
{
"hostPort": 8000
}
]
}
]
EOF
}
<= stage
family = "app"
[
{
"cpu": 1024,
"memory": 3072,
"environment": [
{
"name": "NODE_ENV",
"value": "production”,
}
],
"name": "app",
"portMappings": [
{
"hostPort": 3000
}
]
}
]
EOF
}
prod =>

$ terraform plan –target=aws_elb.feels_so_easy

$ terraform plan –target=aws_elb.oh_no_what_have_w

Terraform v1 Problems
1. massive shared state
2. locally stored state
3. drift between environments

Terraform v1 Problems
1. massive shared state: split states
2. locally stored state: remote state
3. drift between environments: modules

core
(vpc, networking, security groups, asgs)
auth api site db cdn
services

core
(vpc, networking, security groups, asgs)
auth api site db cdn
services
→readonly→

/**
* Remote state.
*/
resource "terraform_remote_state" "state" {
backend = "s3"
config {
bucket = "segment-ops"
key = "terraform/${var.environment}/terraform.tfstate"
}
}
data "template_file" ”test" {
template = "${file("${path.module}/init.tpl")}"
vars {
zone_id = "${terraform_remote_state.state.zone_id}"
}
}

/**
* Remote state.
*/
backend = "s3"
config {
}
}
vars {
}
}
read only!

/**
* Remote state.
*/
backend = "s3"
config {
}
}
vars {
}
}
read only!
reference

Modules enforce
configuration parity.

What makes good*
Terraform?
*for some definitions of good

1. Variables
2. Composition
3. State
4. Versioning

1. Variables
- anything a user might want to override should be a
variable
- use defaults liberally

1. Variables
resource "aws_instance" "bastion" {
ami = "${module.ami.ami_id}"
source_dest_check = false
instance_type = "${var.instance_type}"
subnet_id = "${var.subnet_id}"
key_name = "${var.key_name}"
vpc_security_group_ids = ["${split(",",var.security_groups)}"]
monitoring = true
tags {
Name = "bastion"
Environment = "${var.environment}"
}
}
configurable
configurable
configurable
configurable
configurable

1. Variables
source_dest_check = false
monitoring = true
tags {
Name = "bastion"
}
}
configurable
configurable
configurable
configurable
configurable
non-configurable
non-configurable
non-configurable

1. Variables
source_dest_check = ${var.source_dest_check}
monitoring = ${var.monitoring}
tags {
Name = "bastion"
}
}

2. Composition
- build modules as you need them
- it’s okay if not everything fits the abstraction

2. Composition – “full stack”
module “stack” {
source = “github.com/segmentio/stack”
name = “my-stack”
environment = “production”
}

2. Composition – inside stack
module "vpc" {
source = "./vpc”
…
}
module "security_groups" {
source = "./security-groups”
…
}
module "bastion" {
source = "./bastion”
…
}
module "dhcp" {
source = "./dhcp”
…
}

2. Composition – byo edition
module “cluster” {
source = “github.com/segmentio/stack//ecs-cluster”
environment = “prod”
name = “cdn”
vpc_id = “vpc-eff2eada”
image_id = “ami-204faaf3”
}

3. State management
- separate core from services
- states per service
- use atlas or s3
- use binary plans

4. Versioning
module “stack” {
source =
“github.com/segmentio/stack?ref=v1.x”
}

What’s next
- Applying in CI
- Atlas
- Data sources
- Terraform generation

Prior Art
Stack: github.com/segmentio/stack
Atlas Examples: github.com/hashicorp/atlas-examples

Terraform at Scale

More Related Content

What's hot (20)

Viewers also liked (19)

Similar to Terraform at Scale (20)

Recently uploaded (20)

Terraform at Scale