Immutable infrastructure - Beyond stateless
1 like•118 views
The document discusses immutable infrastructure as a paradigm where components are replaced rather than changed, emphasizing the need for automation and stateless processes. It highlights the advantages of consistent environments and easier rollbacks, while addressing challenges like high upfront investments and time-consuming cluster updates. Additionally, it presents deployment strategies and tools for managing stateful services in cloud environments.
![def _ip_to_replace(self):
cassandra_dc = EC2Snitch.get_datacenter()
cassandra_rack = EC2Snitch.get_rack()
...
dead_nodes = cluster.find_dead_nodes(cassandra_dc, cassandra_rack)
...
ips_to_replace = [
node.address
for node in dead_nodes
if not EC2Checks(self.region).is_alive(node.address)
]
if len(ips_to_replace) == 0:
return None
return ips_to_replace[0]](https://image.slidesharecdn.com/immutableinfrastructure-beyondstateless-180622141940/85/Immutable-infrastructure-Beyond-stateless-35-320.jpg)
![class Node(object):
def __init__(self, instance_id, session):
self.instance_id = instance_id
def is_healthy(self):
return self._instance.state['Name'] != "running":
def terminate(self):
....](https://image.slidesharecdn.com/immutableinfrastructure-beyondstateless-180622141940/85/Immutable-infrastructure-Beyond-stateless-43-320.jpg)
![class Cluster(object):
def __init__(self, name, session):
self.name = name
def wait_until_healthy(self):
while True:
if self.asg.capacity == len(self.healthy_nodes) and self.is_healthy():
break
time.sleep(10)
def healthy_nodes(self):
return [n for n in self.nodes if n.is_healthy()]
def is_healthy(self):
return True](https://image.slidesharecdn.com/immutableinfrastructure-beyondstateless-180622141940/85/Immutable-infrastructure-Beyond-stateless-44-320.jpg)
![class KafkaCluster(Cluster):
node_type = KafkaNode
def is_healthy(self):
health = [self._get_kafka_cluster_status(node.public_dns_name, port=8000)
for node in self.nodes]
return all(x == "green" for x in health)
@staticmethod
def _get_kafka_cluster_status(server_address, port, timeout):
response = requests.get(
"http://{}:{}/cluster".format(server_address, port)
)
return response.json()["status"]](https://image.slidesharecdn.com/immutableinfrastructure-beyondstateless-180622141940/85/Immutable-infrastructure-Beyond-stateless-48-320.jpg)
Immutable infrastructure - Beyond stateless
- 3. JORGEDIASJORGEDIAS Father , coder , polyglot , human . Infrastructure engineer by Schibsted.
- 4. AGENDAAGENDA Introduction to immutable infrastructure Implementing it in practice Application for stateful services
- 6. IMMUTABLEIMMUTABLE adjective | im·mu·ta·ble | (ˌ)i(m)-ˈmyü-tə-bəl not capable of or susceptible to change
- 7. PARADIGMPARADIGM Components are replaced rather than changed Build new servers on every deploy and discard the old ones
- 9. Mutable Infrastructure Unique Servers Heterogeneous fleet Configuration Drift Hard to reproduce environment Costly failure events Mostly static
- 11. New tools are needed Automation is a necessity Architecture changes for apps
- 13. 12FACTORAPP12FACTORAPP General guidelines for building SAAS applications Great ideas for working on the cloud
- 14. CONFIGCONFIG Config is everything that is likely to change between deploys
- 15. Separate your config from your code Store config in the environment Use environment variables
- 16. PROCESSESPROCESSES Processes are stateless and share-nothing
- 17. No persistent data between requests Filesystem can be used as a brief cache
- 19. Processes shut down gracefully on SIGTERM Processes should also be robust against sudden death
- 20. BUILD Code + Dependencies CODE RELEASE CONFIG RUN Execute Processes APP Build + Configuration
- 24. DEPLOYMENTSTRATEGIESDEPLOYMENTSTRATEGIES Rolling terminate Blue/green deployments Canary releases
- 26. CONTEXTCONTEXT Storage team offers managed databases to internal teams Small team of 5 engineers and 1 manager 10s of clusters in production 100s of servers
- 29. Treat the system like you have no state Any node can die and the system must be OK Embrace distributed systems “Controlled chaos“™ through Continuous deployment
- 30. ISN'TTHATDANGEROUS?ISN'TTHATDANGEROUS? Yours Truly “ If we can’t kill a node in a controlled matter then when it inevitably fails we’ll be in real trouble. ”
- 32. NODELIFECYCLENODELIFECYCLE Load configuration Generate configuration files Gathers information about the cluster from cloud If it’s the first node bootstraps the cluster Joins an existing cluster as a new node or replacement Respond to healthcheck to indicate success
- 33. CASSANDRA-NEOPTOLEMUSCASSANDRA-NEOPTOLEMUS In house solution alternative to Priam from Netflix 4000 loc (3500 python)
- 34. def main(): block_devices = storage.get_block_devices_by_priority() storage.create_raid(block_devices) ... seeds_manager = SeedsManager(cluster_name=cluster_name) seed_ips = seeds_manager.list_seeds_private_ips() cassandra_configs = CassandraConfigs( cluster_name=cluster_name, seed_ips=seed_ips, .... ) cassandra_configs.generate_cassandra_yaml() cassandra_configs.generate_cassandra_env_extra() ....
- 35. def _ip_to_replace(self): cassandra_dc = EC2Snitch.get_datacenter() cassandra_rack = EC2Snitch.get_rack() ... dead_nodes = cluster.find_dead_nodes(cassandra_dc, cassandra_rack) ... ips_to_replace = [ node.address for node in dead_nodes if not EC2Checks(self.region).is_alive(node.address) ] if len(ips_to_replace) == 0: return None return ips_to_replace[0]
- 36. ZOOKEEPERCLUSTERZOOKEEPERCLUSTER function main() { genZookeeperConfig "${clustername}" > /etc/zookeeper/zoo.cfg .... enis=$(getAvailableEnis "zookeeper-${clustername}") attachEni "${enis}" .... local myid="$(getEniTag ${eni} ZookeeperId)" echo "${myid}" > /var/lib/zookeeper/myid local myrole="$(getZookeeperRole "${clustername}" "${myid}" )" .... }
- 39. ROLLINGUPDATESROLLINGUPDATES Small framework extensible for many cluster types Currently supports Cassandra, Kafka and Zookeeper
- 40. Cluster and nodes Healthchecks at both levels Implements updating algorithms
- 41. TERMINATINGNODESTERMINATINGNODES Send signal to node to start shutdown Wait for cleanup and node termination Wait for new node to be spawn Wait for the cluster recovery and proceed with next node
- 42. def rolling_terminate(cluster): nodes = cluster.nodes for node in cluster.nodes_to_upgrade(): cluster.wait_until_healthy() node.terminate() cluster.wait_until_healthy()
- 43. class Node(object): def __init__(self, instance_id, session): self.instance_id = instance_id def is_healthy(self): return self._instance.state['Name'] != "running": def terminate(self): ....
- 44. class Cluster(object): def __init__(self, name, session): self.name = name def wait_until_healthy(self): while True: if self.asg.capacity == len(self.healthy_nodes) and self.is_healthy(): break time.sleep(10) def healthy_nodes(self): return [n for n in self.nodes if n.is_healthy()] def is_healthy(self): return True
- 45. class CassandraNode(Node): def terminate(self): self._disablebinary() self._wait_for_hint_delivery() self._drain_node() super().terminate() def is_healthy(self): return super().is_healthy() and self._check_node_disk_usage() and self._check_node_hints() and self._check_node_compactions() and self._check_service_is_up()
- 46. class CassandraCluster(Cluster): node_type = CassandraNode def is_healthy(self): ssm_command = "/opt/cassandra/bin/nodetool status | grep -c ^'UN '" health = SSM.run(self.nodes, ssm_command) return all(x == self.asg.capacity and x == len(self.healthy_nodes) for x in health)
- 47. class KafkaNode(Node): def is_healthy(self): return super().is_healthy() and self._kafka_server_is_up() def _kafka_server_is_up(self): try: response = requests.get( "http://{}:{}/".format(self.public_dns_name, 8000) ) return response.status_code == 200 except RequestException: return False
- 48. class KafkaCluster(Cluster): node_type = KafkaNode def is_healthy(self): health = [self._get_kafka_cluster_status(node.public_dns_name, port=8000) for node in self.nodes] return all(x == "green" for x in health) @staticmethod def _get_kafka_cluster_status(server_address, port, timeout): response = requests.get( "http://{}:{}/cluster".format(server_address, port) ) return response.json()["status"]
- 49. OURTOOLSOURTOOLS
- 50. EMBRACEYOURCLOUDPROVIDEREMBRACEYOURCLOUDPROVIDER We use many AWS services
- 52. PREPARINGFORDISASTERPREPARINGFORDISASTER If you loose customers data you’re in BIG trouble
- 53. Developed a tool soon to be opensourced for Cassandra Backup
- 54. SUMMARYSUMMARY
- 55. ADVANTAGESADVANTAGES Known state and less deployment failures Testing infrastructure in isolation is possible Consistent environments between dev/pre/pro Easy to rollback
- 56. TRADEOFFSTRADEOFFS High up-front investment Rolling update of a whole cluster takes time
- 57. TAKEAWAYSTAKEAWAYS Immutable infrastructure is a means to an end not a goal. It’s not about the tools it’s about the ideas. You can implement it gradually. It may not be a good fit for you.