Patterns in Terraform 12+13: Data, Transformations and Resources

Lead DevOps Engineer - Salesforce
Robert Blumen
The 'Data,
Transformations,
Resources' Pattern in
Terraform

Outline
● Terraform anti-patterns
● New features in versions 12 and 13
● The "Data + Transforms + Resources" Pattern

What Terraform 11 Can Not Do
● Looping
● Conditional
● Nested loops
● Data structures

Data Types in Terraform
12 & 13

structured data types
● list
● map
● set
● tuple
● object

list(string)
list(tuple[string,number,bool])
map(string)
map(object({ id=string, cidr_block=string }))
composition of data types

variable group_members {
type = list(string)
description = "list of emails of the group members"
}
example

variable people {
type = list(object({ name=string, age=number }))
default = [
{
name = "John"
age = 32
}
]
}
type checking (continued)

~/Devel/devops-tools (rblumen) $ /opt/hashi/terraform-0.12.20
apply -var 'people=[ { name="Job", age=71 } ]'
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
~/Devel/devops-tools (rblumen) $ /opt/hashi/terraform-0.12.20
apply -var 'people=[ { name="Job", age=true } ]'
Error: Invalid value for input variable
The argument -var="people=..." does not contain a valid value for variable
"people": element 0: attribute "name": string required.

groups = [
{
"key" : "value"
},
{
"key-2": "value-2"
}
]
all_groups = merge(local.groups...)
###
{
"key": "value",
"key-2": "value"
}
python-style splatting

terraform 11 - indexed lists
resource providerX web_thing {
count = var.num_servers
name = element(var.server_names,count.index)
}
#####
web_thing[0]
web_thing[1]
web_thing[2]
web_thing[3]

locals {
server_names = [ "db", "front-end", "back-end" ]
}
terraform 12+ - maps
web_srv["db"]
web_srv["front-end"]
web_srv["back-end"]
resource providerY web_srv {
for_each = toset(var.server_names)
name = each.value
}

old - (count based):
aws_iam_policy.aws_admin_access[3]
new - (for_each):
aws_iam_policy.aws_admin_access["AmazonS3FullAccess"]
indexing

locals {
administrator_policies = [
"AmazonEC2FullAccess",
"AmazonS3FullAccess",
"AWSKeyManagementServicePowerUser",
"AWSLambdaFullAccess",
]
}
data aws_iam_policy aws_admin_access {
for_each = toset(local.administrator_policies)
arn = "arn:aws:iam::aws:policy/${each.value}"
}
for_each - iterating over a list

resource heroku_app_feature app_feature {
for_each = {
for f, e in {
spaces-tls-legacy : false,
spaces-tls-modern : true
spaces-strict-tls : false
} :
f => e
}
app = heroku_app.app.name
name = each.key
enabled = each.value
}
for_each - iterating over a map

data aws_iam_policy_document access {
for_each = toset(local.groups)
statement {
effect = "Allow"
actions = ["s3:PutObject", ]
resources = data.aws_s3_bucket.devel.arn
}
}
chaining iteration
resource aws_iam_policy access {
for_each = data.aws_iam_policy_document.access
name = "${each.key}Access"
policy = each.value.json
}
resource aws_iam_group_policy_attachment group_access {
for_each = aws_iam_policy.access
group = each.key
policy_arn = each.value.arn
}

● 12: for_each does not work with modules
● 13: for_each supports modules
terraform 12 versus 13

functions on sets
● setintersection
● setproduct
● setsubtract
● setunion

locals {
all_groups = toset([ "dev", "core", "admin", "ops", "chat" ])
some_groups = toset(["dev", "voice"])
other_groups = setsubtract(all_groups, some_groups) # in 12.21
}
resource aws_iam_group groups {
for_each = local.all_groups
name = each.value
}
resource aws_thing thing_one {
for_each = local.some_groups
group = aws_iam_group.groups[each.value]
}
resource aws_thing thing_two {
for_each = local.other_groups
group = aws_iam_group.groups[each.value]
}
implementing conditionals with sets

locals {
test_envs = [ "test1", "test2" ]
}
resource saas_thing_one test_only {
foreach = setintersection(toset([var.environment]), local.test_envs)
…
}
conditionals (2)

for expressions
users = [ "achman", "aziss", "bwong", "cshah", ]
# list
emails = [
for u in local.users : "${replace(u, "-", ".")}@pagerduty.com"
]
# map
emails_by_user = {
for u in local.users :
u => "${replace(u, "-", ".")}@pagerduty.com"
}

[for s in var.list :
upper(s)
if s != ""]
for expressions - conditionals

var user_email_assn {
type = map(string)
default = { ... }
}
local {
labels = [ for user, email in var.user_email_assn: "${user}:${email} ]
}
for expressions - map

{
for s in [ "abc", "def", "aef", "dps" ] :
substr(s, 0, 1) => s...
}
{
"a" = [
"abc",
"aef",
]
"d" = [
"def",
"dps",
]
}
for expressions - group by

resource backend_foo left_hand_thing {
for_each = ...
}
resource backend_bar right_hand_thing {
for_each = ..
}
resource backend_foo_bar left_right_assn {
for_each = ??
left_hand_thing = each.value.left
right_hand_thing = each.value.right
}
many-to-many

# groups have multiple members
# user may belong to more than one group
resource pagerduty_user { … }
resource pagerduty_team { … }
resource pagerduty_team_membership {
for_each = ??
user_id = ??
team_id = ??
}
example: PagerDuty

for g in groups {
create group
for u in groups.users {
create user
create create user_group_membership
}
}
conventional imperative language

locals {
on_call_teams = {
tier_1 = [ "abe", "jaxel", "beldon" ]
tier_2 = [ "abe", "janpax", "fanlo" ]
escalation = [ "adam", "shefty", "fanlo" ]
}
}
terraform solution

resource pagerduty_team teams {
for_each = keys(local.on_call_teams)
name = each.value
}
resource pagerduty_user users {
for_each = distinct(flatten(values(local.on_call_teams)))
name = each.value
email = "${each.value}@companyx.com"
}

locals {
team_user_pairs = [
for team, users in local.on_call_teams: [
for for user in users: {
user: user
team: team
}
]
]
team_users = {
for pair in flatten(local.team_user_pairs):
"${pair.team}-${pair.user}" => {
team_id: pagerduty_team[pair.team].id,
user_id: pagerduty_user[pair.user].id
}
}
}
resource pagerduty_team_membership membership {
for_each = local.team_users
team_id = each.value.team_id
user_id = each.value.user_id
}

locals {
on_call_teams = {
tier_1 = [ "abe", "jaxel", "beldon" ]
tier_2 = [ "abe", "janpax", "fanlo" ]
escalation = [ "adam", "shefty", "fanlo" ]
}
}
###
resource saas_team_membership membership {
for_each = ??
user = each.key
groups = each.value
}
one:many inverting

user_group_pairs = flatten([
for group, members in local.group_memberships : [
for user in members : {
user = user
group = group
}
]
])
groups_by_user = {
for pair in local.user_group_pairs :
pair.user => pair.group...
}
}

resource saas_team_membership membership {
for_each = local.groups_by_user
user = each.key
groups = each.value
}

The
"Data
+ Transformations
+ Resources"
Pattern

objectives
● understandable
● easy to change
● economical (DRY)

● data
○ structure according to your domain
○ use data structures
○ normalize/minimize duplication
○ optimize for frequent changes
● transform
○ use for, if and set functions
○ inputs: your data
○ outputs: what your provider's resources need
● resources
○ create each resource once
○ chain resources
pattern for code organization

Patterns in Terraform 12+13: Data, Transformations and Resources

More Related Content

What's hot (19)

Similar to Patterns in Terraform 12+13: Data, Transformations and Resources (20)

Recently uploaded (20)

Patterns in Terraform 12+13: Data, Transformations and Resources