SlideShare a Scribd company logo

Building for operations - Reinhardt Quelle

•
D
Devopsdays

The document discusses building applications for operability. It describes the different perspectives of operations and development teams. Operations teams view servers as part of clusters and focus on system-level configuration and components. Development teams focus mainly on application configuration. The document emphasizes that applications need to be built to allow for sharing of data and configurations across server clusters to support high availability and scalability. It provides examples of how server infrastructure and applications are structured, such as front-end servers routing requests to application servers.

TechnologyEducation
1 of 75
Downloaded 29 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
Building Applications for Operability Reinhardt Quelle Operations Architect rquelle@cisco.com rquelle@* V16e © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) 1
scr © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Great stuff, but not that’s not what my team does… © 2011 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) 3
We host and manage SaaS Applications Cisco Social Jabber Millions of Meetings for 10s of Millions of Users totaling Billions of Minutes each month 7x24x365 WebEx © 2011-2012 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) 4
DevOps Damon Edwards (DTO) John Willis (now at Dell) suggest “CAMS” as useful mnemonic. © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
DevOps > Tools •Automation © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
DevOps > Tools •Automation •Metrics © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
DevOps > Tools •Automation •Metrics •Sharing © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
DevOps > Tools •Culture •Automation •Metrics •Sharing © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
DevOps > Tools •Culture / Collaboration •Automation •Metrics •Sharing © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Collaboration Requires: • shared goals • communication • empathy • © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Yay, We’re going to do DevOps! http://www.realclearworld.com/blog/2009/01/ © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
When you start, its more like this: http://www.news.com.au/world/are-barack-obama-and-vladamir-putin-holding-highlevel-talks-or-are-they-on-an-awkward-date/storyfndir2ev-1226665803294 © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
At a more personal level: remember that first date? http://www.hercampus.com/love/dating-hooking/9-first-date-disasters-how-deal-them © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Where do we begin? © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Perspective helps (a lot) http://www.navigonlp.com/blog/stop-look-look-again/boat-land-perspective-2/ © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
I come from an Operations background © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Green = Good © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Illustrating the Ops perspective to Engineering http://cdn.twentytwowords.com/wp-content/uploads/Rhinos-Perspective-e1369734532511-634x447.jpg © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
A quick illustration: A real, albeit deceased, product (RIP) © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Developer’s View Compute Node resolv.conf ntp.conf smb.conf ... App © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle (rquelle@cisco.com)
Operations’ View Compute Node /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... App © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle (rquelle@cisco.com)
Servers run in "clusters" Compute Node Compute Node Compute Node inetd config inetd config config sshd config rsyslog config LVM /etc/snmp/* config LVM ... ... ntp.conf ... DR:BD ... © 2013 Cisco and/or its affiliates. All rights reserved. config ... App start/stop orchestration config ntp.conf .... sshd config rsyslog config ... ntp.conf ... .... DR:BD ... ntp.conf App config config ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... App config B Reinhardt Quelle (rquelle@cisco.com) sshd rsyslog smb.conf config A start/stop orchestration /etc/sysconfig/* /etc/logrotate/* LVM /etc/snmp/* ... config B config resolv.conf smb.conf config A /etc/passwd inetd /etc/* resolv.conf DR:BD /etc/sysconfig/* /etc/logrotate/* LVM /etc/snmp/* ... config B start/stop orchestration config LVM ... App sshd smb.conf config A /etc/passwd inetd /etc/logrotate/* resolv.conf smb.conf /etc/sysconfig/* /etc/* .... .... resolv.conf DR:BD config rsyslog /etc/snmp/* Compute Node /etc/passwd inetd /etc/* /etc/* rsyslog /etc/sysconfig/* /etc/logrotate/* /etc/logrotate/* sshd Compute Node /etc/passwd /etc/passwd /etc/sysconfig/* start/stop orchestration ... config A App config B start/stop orchestration config A config B
Which means they may have to share data: Compute Node Compute Node Compute Node inetd sshd /etc/sysconfig/* config config inetd config sshd config LVM rsyslog ... ... ntp.conf ... DR:BD ... config ntp.conf /etc/sysconfig/* /etc/snmp/* config A App start/stop orchestration inetd sshd /etc/sysconfig/* config config rsyslog config ... ... ntp.conf rsyslog config ... .... DR:BD ntp.conf ... App ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... config A App config B start/stop orchestration config smb.conf ... config B rsyslog /etc/logrotate/* resolv.conf smb.conf config A config LVM /etc/snmp/* resolv.conf DR:BD sshd /etc/sysconfig/* config /etc/* LVM .... /etc/passwd inetd /etc/logrotate/* /etc/* ... config B /etc/passwd /etc/logrotate/* smb.conf ... start/stop orchestration sshd resolv.conf smb.conf App config LVM .... .... resolv.conf DR:BD /etc/snmp/* config LVM config inetd /etc/* /etc/* rsyslog /etc/sysconfig/* Compute Node /etc/passwd /etc/logrotate/* /etc/logrotate/* /etc/snmp/* Compute Node /etc/passwd /etc/passwd ... config A App config B start/stop orchestration config A config B start/stop orchestration spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle (rquelle@cisco.com) LUN
Which means they may have to share data: Compute Node Compute Node Compute Node inetd sshd /etc/sysconfig/* config config inetd config sshd config LVM rsyslog ... ... ntp.conf ... DR:BD ... config ntp.conf /etc/sysconfig/* /etc/snmp/* config A App start/stop orchestration inetd sshd /etc/sysconfig/* config config rsyslog config ... ... ntp.conf rsyslog config ... .... DR:BD ntp.conf ... App ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... config A App config B start/stop orchestration config smb.conf ... config B rsyslog /etc/logrotate/* resolv.conf smb.conf config A config LVM /etc/snmp/* resolv.conf DR:BD sshd /etc/sysconfig/* config /etc/* LVM .... /etc/passwd inetd /etc/logrotate/* /etc/* ... config B /etc/passwd /etc/logrotate/* smb.conf ... start/stop orchestration sshd resolv.conf smb.conf App config LVM .... .... resolv.conf DR:BD /etc/snmp/* config LVM config inetd /etc/* /etc/* rsyslog /etc/sysconfig/* Compute Node /etc/passwd /etc/logrotate/* /etc/logrotate/* /etc/snmp/* Compute Node /etc/passwd /etc/passwd ... config A App config B start/stop orchestration config A config B start/stop orchestration spare SAN LUN LUN LUN LUN Note: there are *much* better ways to share data! © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
We have several front-end servers Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd /etc/logrotate/* sshd config rsyslog config LVM ... sshd config rsyslog config LVM ... config A Service X config B Postpath Unified Node (PUN) /etc/passwd /etc/sysconfig/* rsyslog config LVM ... .... sshd config rsyslog config ... ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config sshd config rsyslog config ... ntp.conf ... sshd ntp.conf ... rsyslog config ... /etc/snmp/* .... config rsyslog /etc/sysconfig/* config sshd config ... /etc/logrotate/* /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... ... resolv.conf DR:BD ntp.conf ... smb.conf ... ... config A PPSD config B cluster manager /etc/passwd inetd /etc/* smb.conf PPSD config A config B LVM config resolv.conf DR:BD Service Y /etc/logrotate/* /etc/* .... config B Postpath Unified Node (PUN) /etc/sysconfig/* config LVM /etc/snmp/* config A /etc/passwd inetd /etc/logrotate/* resolv.conf DR:BD Service X config B /etc/* .... smb.conf /etc/passwd inetd LVM /etc/snmp/* resolv.conf ... ntp.conf ... Service Y config B /etc/logrotate/* /etc/* DR:BD .... resolv.conf config A Postpath Unified Node (PUN) /etc/sysconfig/* config LVM /etc/snmp/* config A /etc/passwd inetd /etc/logrotate/* config ... ... Postpath Unified Node (PUN) sshd DR:BD smb.conf Service Y config ... /etc/snmp/* /etc/* ntp.conf ... /etc/sysconfig/* config resolv.conf ntp.conf smb.conf inetd config LVM .... resolv.conf config B sshd rsyslog /etc/snmp/* /etc/* .... config A config /etc/logrotate/* ... DR:BD /etc/snmp/* /etc/* Service X inetd /etc/logrotate/* ... DR:BD /etc/passwd /etc/sysconfig/* config config A PPSD config B cluster manager config A PPSD config B cluster manager smb.conf ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle (rquelle@cisco.com) LUN
...which also have to be redundant. Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd sshd /etc/passwd /etc/sysconfig/* config config inetd sshd config /etc/logrotate/* config config resolv.conf /etc/sysconfig/* ... inetd ... sshd ntp.conf config smb.conf rsyslog Service X Service Y DR:BD config B config resolv.conf ... inetd ... sshd ntp.conf config smb.conf /etc/* rsyslog config B ... ntp.conf Service X config A /etc/snmp/* config B Postpath Unified Node (PUN) Service Y sshd config rsyslog config LVM ... Service X config B sshd config rsyslog config LVM /etc/sysconfig/* config ntp.conf ... ntp.conf ... sshd config /etc/snmp/* config A config B /etc/* config ... Service Y DR:BD .... ... config A resolv.conf config B ... ntp.conf smb.conf .... DR:BD ntp.conf ... rsyslog ... config B config rsyslog ntp.conf ... config ... config B ... /etc/logrotate/* .... /etc/snmp/* /etc/* .... resolv.conf ntp.conf ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... config A PPSD config B cluster manager config LVM /etc/snmp/* ... PPSD config rsyslog /etc/sysconfig/* config sshd /etc/* DR:BD ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle (rquelle@cisco.com) LUN config A config B /etc/passwd inetd /etc/logrotate/* smb.conf config A cluster manager sshd resolv.conf DR:BD Service Y config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/* .... config A /etc/passwd inetd LVM /etc/snmp/* ... PPSD ... Service X config B Postpath Unified Node (PUN) config smb.conf ... Service Y config A /etc/logrotate/* resolv.conf config A cluster manager config B /etc/* smb.conf PPSD config A /etc/sysconfig/* config LVM /etc/snmp/* resolv.conf DR:BD Service X /etc/passwd inetd /etc/logrotate/* /etc/* .... /etc/logrotate/* smb.conf rsyslog config B ... Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* /etc/snmp/* /etc/passwd /etc/sysconfig/* ... Postpath Unified Node (PUN) /etc/sysconfig/* config .... config resolv.conf smb.conf config A /etc/passwd inetd sshd ntp.conf config LVM ... Service X inetd ... .... resolv.conf smb.conf config A ... /etc/* ... config A Service Y DR:BD config B LVM /etc/logrotate/* config ... LVM resolv.conf config DR:BD /etc/sysconfig/* LVM .... /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd DR:BD /etc/snmp/* ... config A /etc/sysconfig/* config /etc/logrotate/* .... /etc/logrotate/* config ... LVM config A config /etc/passwd LVM config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... DR:BD inetd /etc/logrotate/* /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config
And that means load balancers to direct traffic Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd /etc/logrotate/* sshd config sshd /etc/snmp/* config config resolv.conf /etc/sysconfig/* LVM ... inetd ... sshd ntp.conf config smb.conf config A Service Y DR:BD config B sshd /etc/snmp/* config config resolv.conf ... inetd ... sshd ntp.conf config smb.conf /etc/* rsyslog resolv.conf config B ... ntp.conf Service X config A /etc/snmp/* config B Load Balancer Postpath Unified Node (PUN) Service Y config A Service X config B config rsyslog config LVM ... /etc/sysconfig/* config sshd config rsyslog config ... resolv.conf ntp.conf ntp.conf ... config B Service X .... sshd config rsyslog ntp.conf ... Service X config B .... resolv.conf config B ... ntp.conf ... sshd rsyslog ntp.conf ... config ... config rsyslog /etc/sysconfig/* config sshd /etc/snmp/* config ... /etc/logrotate/* /etc/snmp/* /etc/* .... /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... resolv.conf DR:BD ntp.conf ... smb.conf ... ... config A PPSD config B config A PPSD config B cluster manager config A PPSD config B cluster manager smb.conf ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle (rquelle@cisco.com) LUN config A config B /etc/passwd inetd LVM config resolv.conf DR:BD config B /etc/logrotate/* /etc/* .... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config LVM /etc/snmp/* config A /etc/passwd inetd ... cluster manager Service Y DR:BD ... config A Postpath Unified Node (PUN) config resolv.conf DR:BD Service Y smb.conf PPSD config A config B /etc/* ... config A smb.conf /etc/logrotate/* /etc/* resolv.conf DR:BD config A /etc/sysconfig/* config LVM /etc/snmp/* /etc/* .... /etc/snmp/* config ... LVM /etc/passwd inetd /etc/logrotate/* LVM /etc/snmp/* /etc/logrotate/* smb.conf config B ... Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd /etc/passwd /etc/sysconfig/* ... Postpath Unified Node (PUN) /etc/sysconfig/* config .... smb.conf /etc/passwd inetd sshd ntp.conf config config resolv.conf rsyslog ... config A inetd ... .... smb.conf Service X ... /etc/* ... config A Service Y DR:BD config B LVM /etc/logrotate/* config ... LVM ... config A /etc/snmp/* config DR:BD /etc/sysconfig/* LVM .... config Front-End Server /etc/* /etc/passwd DR:BD /etc/snmp/* /etc/sysconfig/* rsyslog .... /etc/logrotate/* config ... LVM Service X config /etc/passwd DR:BD config /etc/logrotate/* Front-End Server /etc/* rsyslog .... rsyslog inetd /etc/logrotate/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config
There are multiple clusters Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config sshd config inetd config config LVM config A Service X /etc/logrotate/* sshd ntp.conf config ... config A Service X Postpath Unified Node (PUN) config A Postpath Unified Node (PUN) Postpath Unified Node (PUN) sshd config rsyslog /etc/* .... resolv.conf config B ... ntp.conf Postpath Unified Node (PUN) config ntp.conf PPSD config B DR:BD cluster manager DR:BD ... /etc/snmp/* /etc/* .... resolv.conf ntp.conf ... ntp.conf smb.conf ... smb.conf ... config A PPSD ... .... smb.conf config B /etc/logrotate/* config config LVM /etc/* resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config sshd rsyslog /etc/snmp/* .... config A /etc/passwd inetd ... /etc/* smb.conf ... config config LVM config B Postpath Unified Node (PUN) /etc/sysconfig/* config sshd rsyslog /etc/snmp/* resolv.conf ntp.conf ... DR:BD smb.conf config B /etc/logrotate/* ... .... config A Service Y /etc/passwd inetd config LVM /etc/* resolv.conf config A ... config A Service X config B /etc/sysconfig/* config sshd rsyslog /etc/snmp/* .... cluster manager /etc/snmp/* ... config A Service Y DR:BD config B config A /etc/logrotate/* ... /etc/* resolv.conf PPSD inetd config LVM /etc/snmp/* ... /etc/logrotate/* config ... smb.conf /etc/passwd /etc/sysconfig/* config /etc/logrotate/* ... DR:BD config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/logrotate/* config config LVM /etc/passwd /etc/sysconfig/* smb.conf resolv.conf config B ... config A Service X config B /etc/sysconfig/* config sshd .... config resolv.conf LVM ... /etc/passwd inetd rsyslog sshd ntp.conf config smb.conf Service Y config B ... rsyslog ... config A DR:BD .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/* config ... LVM resolv.conf ... /etc/snmp/* smb.conf rsyslog config B ... /etc/snmp/* config /etc/logrotate/* smb.conf Service X config Front-End Server /etc/* LVM /etc/sysconfig/* .... ... config A Service Y DR:BD config B /etc/sysconfig/* config sshd rsyslog config resolv.conf inetd /etc/passwd /etc/* config ... LVM ... DR:BD /etc/snmp/* smb.conf rsyslog LVM .... /etc/logrotate/* sshd ntp.conf config ... DR:BD /etc/sysconfig/* config /etc/passwd config resolv.conf inetd /etc/snmp/* Front-End Server /etc/* rsyslog .... ... inetd /etc/logrotate/* /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config sshd /etc/logrotate/* PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager spare SAN LUN LUN LUN Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd config config DR:BD rsyslog ... .... inetd DR:BD ... /etc/snmp/* Service Y DR:BD ntp.conf config A Service X resolv.conf ntp.conf config A Postpath Unified Node (PUN) config A Service Y config inetd Postpath Unified Node (PUN) config config inetd .... ntp.conf Postpath Unified Node (PUN) config config config ... inetd /etc/sysconfig/* config sshd config rsyslog config /etc/logrotate/* /etc/logrotate/* /etc/snmp/* /etc/snmp/* /etc/* /etc/* .... ... LVM ... .... ... DR:BD LVM resolv.conf ntp.conf /etc/passwd /etc/sysconfig/* config sshd .... LVM DR:BD resolv.conf config B Postpath Unified Node (PUN) /etc/* ... ... inetd rsyslog /etc/snmp/* .... LVM DR:BD config A Service Y config B /etc/passwd config /etc/* .... ntp.conf config A Service X /etc/logrotate/* /etc/snmp/* /etc/* resolv.conf ... DR:BD ... resolv.conf ntp.conf resolv.conf ntp.conf smb.conf smb.conf smb.conf ... ... ntp.conf smb.conf ... PPSD /etc/* resolv.conf ... config A config B /etc/sysconfig/* config sshd rsyslog /etc/logrotate/* /etc/snmp/* ... /etc/snmp/* ... config A config B ... smb.conf /etc/passwd /etc/sysconfig/* config sshd rsyslog /etc/logrotate/* ... Service Y config B /etc/passwd config LVM config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config sshd DR:BD /etc/logrotate/* config ... Service Y DR:BD config B ... /etc/passwd inetd rsyslog config A Service X smb.conf ... config B /etc/passwd /etc/sysconfig/* smb.conf LVM ... config A smb.conf Service X .... config resolv.conf sshd ntp.conf config rsyslog .... config B ... Service Y DR:BD config B ... inetd /etc/* config ... LVM resolv.conf DR:BD /etc/snmp/* smb.conf ... config A ... /etc/logrotate/* sshd ntp.conf config rsyslog .... config B ... /etc/snmp/* config LVM /etc/passwd /etc/sysconfig/* /etc/* config ... LVM config A /etc/logrotate/* config rsyslog .... config resolv.conf inetd /etc/logrotate/* sshd ntp.conf config config B ... /etc/sysconfig/* smb.conf Service X LVM Front-End Server /etc/* config /etc/passwd config resolv.conf rsyslog /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* config ... inetd /etc/logrotate/* sshd /etc/snmp/* Front-End Server /etc/* LVM /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog Load Balancer LUN ... config A PPSD config B cluster manager config A config A PPSD config B cluster manager PPSD config B cluster manager smb.conf ... config A config A PPSD config B cluster manager config B cluster manager spare SAN LUN LUN LUN Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd config config LVM ... DR:BD ... inetd rsyslog config resolv.conf smb.conf Service X Service Y DR:BD config B config LVM ... DR:BD /etc/snmp/* ... ... config A config B ... inetd config resolv.conf sshd ntp.conf config smb.conf rsyslog LVM Service X ntp.conf ... config A Service Y DR:BD config B config A Postpath Unified Node (PUN) config B ... config config Service Y config A Service X config B /etc/snmp/* /etc/sysconfig/* config sshd config config /etc/snmp/* Service X ntp.conf config ... DR:BD ... /etc/snmp/* config B /etc/* config ... Service Y DR:BD .... ... config A resolv.conf config B ... ntp.conf ... config A Service X config B /etc/snmp/* config config /etc/snmp/* /etc/sysconfig/* config sshd config config /etc/logrotate/* /etc/snmp/* /etc/* /etc/* .... LVM ... DR:BD ... .... LVM ... DR:BD ... resolv.conf ntp.conf smb.conf smb.conf ... ... resolv.conf ntp.conf smb.conf ... ... config A PPSD config B config A PPSD config B cluster manager config A PPSD config B cluster manager smb.conf ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN config A config B /etc/passwd inetd rsyslog /etc/logrotate/* .... resolv.conf ntp.conf Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/* ... ... config A config B /etc/passwd inetd rsyslog /etc/logrotate/* LVM DR:BD resolv.conf ntp.conf Service Y Postpath Unified Node (PUN) config .... LVM cluster manager config A smb.conf smb.conf PPSD /etc/logrotate/* smb.conf LVM resolv.conf /etc/passwd /etc/sysconfig/* config sshd /etc/* .... resolv.conf ntp.conf inetd rsyslog /etc/logrotate/* /etc/* ... ... config A config B Postpath Unified Node (PUN) /etc/passwd inetd rsyslog /etc/logrotate/* LVM /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd DR:BD .... inetd rsyslog .... smb.conf /etc/passwd inetd rsyslog ... /etc/* ... config B DR:BD /etc/snmp/* smb.conf Service X ... /etc/logrotate/* config ... .... config A /etc/snmp/* config LVM /etc/passwd /etc/sysconfig/* /etc/* resolv.conf /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/logrotate/* config ... LVM config /etc/logrotate/* sshd /etc/snmp/* /etc/passwd /etc/sysconfig/* sshd ntp.conf config config A config Front-End Server /etc/* .... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd LUN LUN Reinhardt Quelle (rquelle@cisco.com)
... and geo-redundant data centers. GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* config A sshd ntp.conf config rsyslog .... resolv.conf config B ... config A Service X resolv.conf config B ... config B Postpath Unified Node (PUN) config B Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) config config Postpath Unified Node (PUN) config config LVM config LVM config config config B Postpath Unified Node (PUN) inetd config config config LVM config LVM /etc/logrotate/* config LUN LUN Webmail config LVM ADM inetd smb.conf ... ... config A config /etc/sysconfig/* inetd sshd ntp.conf config Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config /etc/snmp/* resolv.conf config A Service X Service X config B inetd Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) config LVM config inetd config Service Y DR:BD config B resolv.conf config A Service X config B inetd Service X config B inetd config /etc/* .... ... config A resolv.conf config B ... ntp.conf smb.conf LVM config inetd inetd /etc/sysconfig/* config /etc/logrotate/* config /etc/logrotate/* sshd /etc/snmp/* /etc/* config /etc/snmp/* /etc/* rsyslog .... LVM ... config /etc/* rsyslog .... LVM ... config .... ... resolv.conf ntp.conf ... DR:BD resolv.conf ntp.conf ... smb.conf DR:BD ntp.conf ... smb.conf ... smb.conf ... config A PPSD /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* config DR:BD ... config A config B cluster manager config B Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf PPSD inetd /etc/logrotate/* config LVM resolv.conf ... ... config A Service Y config B /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... DR:BD smb.conf config A config B ... config A Service X config B Postpath Unified Node (PUN) /etc/* rsyslog .... resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/snmp/* /etc/* ... ... PPSD cluster manager Service Y config B /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD ... config A Service X ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config rsyslog .... config A /etc/passwd /etc/sysconfig/* config sshd /etc/* ntp.conf /etc/passwd /etc/logrotate/* smb.conf LVM resolv.conf config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) smb.conf config B inetd rsyslog .... smb.conf config A Service Y /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf .... /etc/snmp/* config ... LVM ... config A config LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... /etc/passwd config A PPSD cluster manager sshd ntp.conf config /etc/* config B ... config A Service X ... ... DR:BD ... inetd ... resolv.conf ntp.conf smb.conf /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* ... ... smb.conf /etc/snmp/* config LVM resolv.conf config A config B cluster manager /etc/snmp/* config ... LVM config A Service X /etc/logrotate/* config rsyslog .... ... ... PPSD Load Balancer /etc/passwd /etc/sysconfig/* config sshd rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config sshd /etc/snmp/* /etc/* LVM DR:BD ... config B sshd ntp.conf config Postpath Unified Node (PUN) /etc/logrotate/* config rsyslog .... ntp.conf smb.conf config A PPSD cluster manager inetd rsyslog inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/passwd /etc/sysconfig/* config sshd /etc/* ... ... DR:BD ... config A config B cluster manager config B Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf PPSD inetd /etc/snmp/* config LVM resolv.conf ... ... config A Service Y config B /etc/passwd /etc/logrotate/* config rsyslog .... ... DR:BD smb.conf config A config B config A Service X config B Postpath Unified Node (PUN) /etc/* rsyslog .... resolv.conf ntp.conf ntp.conf ... config A /etc/sysconfig/* config sshd /etc/snmp/* /etc/* ... ... inetd /etc/logrotate/* sshd /etc/snmp/* config PPSD resolv.conf config B ... smb.conf /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config LVM DR:BD Service Y config B /etc/passwd /etc/sysconfig/* config sshd rsyslog config A Service X config B /etc/passwd cluster manager ... config A Service Y DR:BD config B .... ... config A Service Y config A Service X ntp.conf /etc/* smb.conf ... config A /etc/snmp/* config ... LVM resolv.conf config B ... smb.conf Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... ... config A Service Y DR:BD config B ntp.conf sshd ntp.conf config /etc/* config ... LVM ... config A config B ... inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... ADM /etc/passwd inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config A config B rsyslog .... /etc/snmp/* smb.conf Postpath Unified Node (PUN) PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager spare spare SAN SAN LUN LUN LUN Webmail LUN RPC Front-End Server /etc/sysconfig/* smb.conf rsyslog Service Y DR:BD inetd sshd ntp.conf config ... DR:BD /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A inetd DR:BD ... /etc/snmp/* smb.conf ntp.conf Service X config A config B ... Service Y DR:BD /etc/sysconfig/* sshd ntp.conf config ... DR:BD smb.conf rsyslog resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) config resolv.conf ntp.conf smb.conf smb.conf ... .... Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle (rquelle@cisco.com) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf config A config B config A config B /etc/passwd /etc/* rsyslog LVM ... PPSD cluster manager Service X /etc/logrotate/* resolv.conf DR:BD ... config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf config A config B Service Y /etc/passwd /etc/logrotate/* resolv.conf ... config A config B Postpath Unified Node (PUN) /etc/passwd LVM ... config A config B ... /etc/* config PPSD cluster manager Service X smb.conf ... /etc/logrotate/* /etc/snmp/* /etc/logrotate/* smb.conf LVM ... config A ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... smb.conf /etc/passwd inetd /etc/snmp/* inetd /etc/* ... config B ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... config B ... /etc/sysconfig/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD /etc/snmp/* config Front-End Server /etc/* rsyslog .... inetd smb.conf Service X config /etc/logrotate/* sshd /etc/snmp/* config /etc/logrotate/* config ... LVM ... config A config B ... inetd /etc/logrotate/* config Front-End Server /etc/* LVM /etc/passwd config resolv.conf .... /etc/passwd /etc/sysconfig/* config sshd rsyslog .... inetd /etc/* config ... LVM resolv.conf Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/logrotate/* sshd ntp.conf config rsyslog .... ... config A config Front-End Server /etc/* /etc/sysconfig/* LVM /etc/passwd config resolv.conf /etc/* ADM Front-End Server /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM /etc/passwd config resolv.conf LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... LVM config A config B ... /etc/logrotate/* LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail Front-End Server /etc/passwd inetd /etc/logrotate/* config config LVM LUN ADM Front-End Server /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. config B LUN Front-End Server inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... LVM config A PPSD config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM DR:BD ntp.conf ... DR:BD smb.conf PPSD cluster manager LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog rsyslog .... ... resolv.conf ntp.conf ... ... config B cluster manager spare LUN RPC Front-End Server Service X /etc/snmp/* /etc/* rsyslog .... ... DR:BD smb.conf config A PPSD sshd /etc/snmp/* config resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B inetd /etc/logrotate/* sshd /etc/* rsyslog .... resolv.conf PPSD cluster manager config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/snmp/* /etc/* ... ... ... config A config B ... Postpath Unified Node (PUN) SAN LUN sshd ntp.conf Service Y config B spare inetd .... resolv.conf smb.conf /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD smb.conf SAN Load Balancer /etc/* ... config A config B ... config A Service X config B /etc/sysconfig/* config /etc/logrotate/* sshd resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/sysconfig/* config rsyslog .... ... ... PPSD cluster manager config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/snmp/* config LVM DR:BD ... config B /etc/logrotate/* smb.conf resolv.conf ... /etc/* rsyslog .... ntp.conf smb.conf config A PPSD cluster manager /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM config B ... config A Service X /etc/logrotate/* config /etc/* ... ... DR:BD ... .... inetd rsyslog .... ... config A smb.conf Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A config B cluster manager config B /etc/snmp/* ... ... /etc/* config ... Service Y DR:BD config B config A /etc/sysconfig/* config sshd /etc/snmp/* config LVM resolv.conf PPSD config B cluster manager inetd /etc/logrotate/* config rsyslog .... ... ... ... config A PPSD sshd /etc/snmp/* /etc/* LVM DR:BD smb.conf ... config A config B cluster manager config A Service X ntp.conf /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd ntp.conf ... DR:BD smb.conf PPSD rsyslog .... ... resolv.conf ntp.conf ... DR:BD ... resolv.conf Service Y Postpath Unified Node (PUN) /etc/passwd inetd /etc/snmp/* ... config B config config LVM DR:BD /etc/logrotate/* smb.conf rsyslog LVM config B ... config A Service X Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config B /etc/logrotate/* sshd ntp.conf config smb.conf config A Service Y /etc/passwd inetd /etc/snmp/* /etc/* rsyslog .... resolv.conf config A config B config B /etc/logrotate/* sshd /etc/snmp/* /etc/* ... ... PPSD Service Y DR:BD config B /etc/sysconfig/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd .... ... config A rsyslog .... config resolv.conf ... ... /etc/* config ... ... /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD inetd config A Service X ntp.conf config A Service X /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog cluster manager resolv.conf config B ... /etc/snmp/* smb.conf rsyslog sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X /etc/passwd inetd Service Y DR:BD config B sshd ntp.conf config LVM ... config A Service Y config A Service X ntp.conf inetd .... ... config A inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... smb.conf ... config A /etc/snmp/* /etc/passwd /etc/sysconfig/* config sshd rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog LVM smb.conf Service X sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf /etc/* config ... Service Y DR:BD config B inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf LVM Service X config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog
Which means data must be replicated - somehow. GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* rsyslog Service X rsyslog config A Service X sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X inetd sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X /etc/logrotate/* sshd ntp.conf config rsyslog config B ... config A Service X Service Y DR:BD config B ntp.conf config A config B ... config config A config A Service X config B Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd /etc/snmp/* config config rsyslog /etc/logrotate/* rsyslog DR:BD Postpath Unified Node (PUN) config config /etc/logrotate/* config LUN LUN Webmail config .... LVM ... resolv.conf resolv.conf ntp.conf ... ADM smb.conf ... PPSD ... config A inetd config /etc/sysconfig/* inetd sshd ntp.conf config smb.conf config A Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config smb.conf rsyslog LVM resolv.conf config B ... config A Service X resolv.conf config B Postpath Unified Node (PUN) Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd config config LVM config LVM Postpath Unified Node (PUN) config DR:BD config LVM config A Service X config resolv.conf Postpath Unified Node (PUN) inetd .... resolv.conf ntp.conf smb.conf config config config config LVM /etc/logrotate/* config /etc/snmp/* /etc/* rsyslog .... LVM ... DR:BD config .... ... resolv.conf ntp.conf ... smb.conf DR:BD ntp.conf ... smb.conf ... smb.conf ... config A PPSD sshd /etc/snmp/* config resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* LVM resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config rsyslog .... ... resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... ... config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* LVM DR:BD smb.conf config A config B ... Postpath Unified Node (PUN) Service Y config B PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager spare spare SAN SAN LUN LUN LUN Webmail LUN RPC Front-End Server /etc/sysconfig/* smb.conf rsyslog Service Y DR:BD inetd sshd ntp.conf config ... DR:BD /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A inetd DR:BD ... /etc/snmp/* smb.conf ntp.conf Service X config A config B ... Service Y DR:BD /etc/sysconfig/* sshd ntp.conf config ... DR:BD smb.conf rsyslog resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) config resolv.conf ntp.conf smb.conf smb.conf ... .... Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle (rquelle@cisco.com) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf config A config B config A config B /etc/passwd /etc/* rsyslog LVM ... PPSD cluster manager Service X /etc/logrotate/* resolv.conf DR:BD ... config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf config A config B Service Y /etc/passwd /etc/logrotate/* resolv.conf ... config A config B Postpath Unified Node (PUN) /etc/passwd LVM ... config A config B ... /etc/* config PPSD cluster manager Service X smb.conf ... /etc/logrotate/* /etc/snmp/* /etc/logrotate/* smb.conf LVM ... config A ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... smb.conf /etc/passwd inetd /etc/snmp/* inetd /etc/* ... config B ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... config B ... /etc/sysconfig/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD /etc/snmp/* config Front-End Server /etc/* rsyslog .... inetd smb.conf Service X config /etc/logrotate/* sshd /etc/snmp/* config /etc/logrotate/* config ... LVM ... config A config B ... inetd /etc/logrotate/* config Front-End Server /etc/* LVM /etc/passwd config resolv.conf .... /etc/passwd /etc/sysconfig/* config sshd rsyslog .... inetd /etc/* config ... LVM resolv.conf Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/logrotate/* sshd ntp.conf config rsyslog .... ... config A config Front-End Server /etc/* /etc/sysconfig/* LVM /etc/passwd config resolv.conf /etc/* ADM Front-End Server /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM /etc/passwd config resolv.conf LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... LVM config A config B ... /etc/logrotate/* LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail Front-End Server /etc/passwd inetd /etc/logrotate/* config config LVM LUN ADM Front-End Server /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. /etc/* ... config A config B ... config A Service X config B /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/sysconfig/* config /etc/* ... ... PPSD cluster manager config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/snmp/* config LVM DR:BD ... config A Service X config B /etc/logrotate/* sshd rsyslog .... ntp.conf /etc/passwd /etc/logrotate/* smb.conf LVM config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) smb.conf config A .... inetd rsyslog .... smb.conf config A Service Y /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf /etc/sysconfig/* config config B cluster manager resolv.conf /etc/passwd inetd resolv.conf ... PPSD config B config LVM /etc/snmp/* config ... LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... config B ... config A Service X /etc/snmp/* config ... sshd ntp.conf config ... ... DR:BD smb.conf config A config B cluster manager inetd smb.conf /etc/* rsyslog .... ntp.conf ... PPSD Service Y DR:BD config B /etc/logrotate/* sshd /etc/snmp/* config ... DR:BD ... config B config A Service X /etc/logrotate/* /etc/sysconfig/* config /etc/passwd config resolv.conf ... ... /etc/* ... config A Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A PPSD cluster manager Load Balancer /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config /etc/* LVM resolv.conf ... config A config B cluster manager sshd ntp.conf config LVM inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... ... /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... ... DR:BD smb.conf PPSD sshd /etc/snmp/* config LVM ntp.conf ... ... config B Postpath Unified Node (PUN) /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B config A Service X config B /etc/logrotate/* sshd /etc/snmp/* config resolv.conf PPSD ... config A /etc/sysconfig/* config /etc/* rsyslog .... ... ... DR:BD inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* rsyslog cluster manager ntp.conf smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd inetd resolv.conf config B ... ... config A Service Y Service Y DR:BD config B .... ... config A smb.conf ... Service X config A Service X ntp.conf /etc/* config ... LVM config B ... /etc/snmp/* config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog smb.conf config A sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... .... ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* LUN RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config B rsyslog .... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... LVM config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM DR:BD config A PPSD config B cluster manager LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog rsyslog ntp.conf ... DR:BD smb.conf ... config B cluster manager spare LUN RPC Front-End Server Service X /etc/snmp/* /etc/* rsyslog ... DR:BD smb.conf config A PPSD sshd /etc/snmp/* config .... LVM ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config LVM resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... DR:BD ... config A /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* config LVM ntp.conf smb.conf config B cluster manager inetd config B Postpath Unified Node (PUN) /etc/passwd /etc/sysconfig/* config /etc/* rsyslog .... resolv.conf config A PPSD config B cluster manager ... SAN LUN sshd ntp.conf Service Y config B spare inetd .... resolv.conf config A Service X config B /etc/passwd inetd /etc/logrotate/* config /etc/* ... ... config A PPSD Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD ... config A config B cluster manager SAN Load Balancer /etc/* ... config A config B ... smb.conf config A Service Y config B /etc/passwd inetd /etc/logrotate/* rsyslog .... smb.conf ... PPSD config /etc/* ntp.conf ... DR:BD smb.conf ... config B cluster manager sshd /etc/snmp/* ... resolv.conf ntp.conf ... smb.conf config A PPSD config LVM ... resolv.conf ntp.conf ... DR:BD ... config rsyslog .... LVM ... resolv.conf ntp.conf smb.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config .... LVM resolv.conf config A config B cluster manager sshd /etc/snmp/* /etc/* config .... ... ... PPSD config /etc/* config LVM DR:BD ... sshd /etc/snmp/* /etc/* rsyslog .... ntp.conf smb.conf inetd config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* /etc/* resolv.conf config A config B inetd config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd ... ... PPSD /etc/snmp/* config ... Service Y DR:BD config B ... config LVM DR:BD Service X smb.conf ... /etc/logrotate/* sshd rsyslog /etc/logrotate/* smb.conf config A ntp.conf smb.conf ... Postpath Unified Node (PUN) cluster manager /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM smb.conf /etc/sysconfig/* config inetd rsyslog ... /etc/passwd inetd .... ... ... .... resolv.conf smb.conf Service Y Postpath Unified Node (PUN) /etc/snmp/* config /etc/* ... config A ... config B config DR:BD /etc/snmp/* config ... LVM resolv.conf LVM /etc/logrotate/* smb.conf smb.conf Service X /etc/sysconfig/* config sshd Front-End Server /etc/* inetd .... ... config A Service Y DR:BD config B ntp.conf /etc/passwd inetd rsyslog /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf Front-End Server /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf ADM /etc/passwd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf config B ... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B sshd ntp.conf config /etc/* config ... LVM inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf Front-End Server /etc/* config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... config A inetd /etc/logrotate/* sshd /etc/snmp/* config Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog
But wait! There’s more! GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* rsyslog Service X rsyslog config A Service X sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X inetd sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X /etc/logrotate/* sshd ntp.conf config rsyslog config B ... config A Service X Service Y DR:BD config B ntp.conf Security Appliances config A config B ... config config A config A Service X config B Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd /etc/snmp/* config config rsyslog /etc/logrotate/* rsyslog DR:BD Postpath Unified Node (PUN) config config /etc/logrotate/* config LUN LUN Webmail config .... LVM ... resolv.conf resolv.conf ntp.conf ... ADM smb.conf ... PPSD ... config A inetd config /etc/sysconfig/* inetd sshd ntp.conf config smb.conf config A Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config smb.conf rsyslog LVM resolv.conf config B ... config A Service X resolv.conf config B Postpath Unified Node (PUN) Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd config config LVM config LVM Postpath Unified Node (PUN) config DR:BD config LVM config A Service X config resolv.conf Postpath Unified Node (PUN) inetd config config config config LVM /etc/logrotate/* config /etc/snmp/* /etc/* rsyslog .... LVM ... DR:BD config .... ... resolv.conf ntp.conf ... smb.conf DR:BD ntp.conf ... smb.conf ... smb.conf ... config A PPSD sshd /etc/snmp/* config resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* LVM resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config rsyslog .... ... resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... ... config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* LVM DR:BD smb.conf config A config B ... Postpath Unified Node (PUN) PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager SAN Identity Servers LUN LUN LUN Webmail LUN RPC Front-End Server /etc/sysconfig/* smb.conf rsyslog Service Y DR:BD inetd sshd ntp.conf config ... DR:BD /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A inetd DR:BD ... /etc/snmp/* smb.conf ntp.conf Service X config A config B ... Service Y DR:BD /etc/sysconfig/* sshd ntp.conf config ... DR:BD smb.conf rsyslog resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) config resolv.conf ntp.conf smb.conf smb.conf ... .... Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle (rquelle@cisco.com) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf config A config B config A config B /etc/passwd /etc/* rsyslog LVM ... PPSD cluster manager Service X /etc/logrotate/* resolv.conf DR:BD ... config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf config A config B Service Y /etc/passwd /etc/logrotate/* resolv.conf ... config A config B Postpath Unified Node (PUN) /etc/passwd LVM ... config A config B ... /etc/* config PPSD cluster manager Service X smb.conf ... /etc/logrotate/* /etc/snmp/* /etc/logrotate/* smb.conf LVM ... config A ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... smb.conf /etc/passwd inetd /etc/snmp/* inetd /etc/* ... config B ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... config B ... /etc/sysconfig/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD /etc/snmp/* config Front-End Server /etc/* rsyslog .... inetd smb.conf Service X config /etc/logrotate/* sshd /etc/snmp/* config /etc/logrotate/* config ... LVM ... config A config B ... inetd /etc/logrotate/* config Front-End Server /etc/* LVM /etc/passwd config resolv.conf .... /etc/passwd /etc/sysconfig/* config sshd rsyslog .... inetd /etc/* config ... LVM resolv.conf Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/logrotate/* sshd ntp.conf config rsyslog .... ... config A config Front-End Server /etc/* /etc/sysconfig/* LVM /etc/passwd config resolv.conf /etc/* ADM Front-End Server /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM /etc/passwd config resolv.conf LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... LVM config A config B ... /etc/logrotate/* LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail Front-End Server /etc/passwd inetd /etc/logrotate/* config config LVM LUN ADM Front-End Server /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. ntp.conf smb.conf spare SAN LVM .... resolv.conf Service Y config B spare DR:BD /etc/* ... config A config B ... config A Service X config B /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/sysconfig/* config /etc/* ... ... PPSD cluster manager config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/snmp/* config LVM DR:BD ... config A Service X config B /etc/logrotate/* sshd rsyslog .... ntp.conf /etc/passwd /etc/logrotate/* smb.conf LVM config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) smb.conf config A .... inetd rsyslog .... smb.conf config A Service Y /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf /etc/sysconfig/* config config B cluster manager resolv.conf /etc/passwd inetd resolv.conf ... PPSD config B config LVM /etc/snmp/* config ... LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... config B ... config A Service X /etc/snmp/* config ... sshd ntp.conf config ... ... DR:BD smb.conf config A config B cluster manager inetd smb.conf /etc/* rsyslog .... ntp.conf ... PPSD Service Y DR:BD config B /etc/logrotate/* sshd /etc/snmp/* config ... DR:BD ... config B config A Service X /etc/logrotate/* /etc/sysconfig/* config /etc/passwd config resolv.conf ... ... /etc/* ... config A Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A PPSD cluster manager Load Balancer /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config /etc/* LVM resolv.conf ... config A config B cluster manager sshd ntp.conf config LVM inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... ... /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... ... DR:BD smb.conf PPSD sshd /etc/snmp/* config LVM ntp.conf ... ... config B Postpath Unified Node (PUN) /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B config A Service X config B /etc/logrotate/* sshd /etc/snmp/* config resolv.conf PPSD ... config A /etc/sysconfig/* config /etc/* rsyslog .... ... ... DR:BD inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* rsyslog cluster manager ntp.conf smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd inetd resolv.conf config B ... ... config A Service Y Service Y DR:BD config B .... ... config A smb.conf ... Service X config A Service X ntp.conf /etc/* config ... LVM config B ... /etc/snmp/* config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog smb.conf config A sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... .... ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* LUN RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config B rsyslog .... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... rsyslog config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Billing Mgt Servers config A PPSD config B cluster manager LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog Service X ntp.conf ... DR:BD smb.conf ... config B cluster manager spare LUN RPC Front-End Server sshd /etc/snmp/* /etc/* rsyslog ... DR:BD smb.conf config A PPSD sshd /etc/snmp/* config .... LVM ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config LVM resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... DR:BD ... config A /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* config LVM ntp.conf smb.conf config B cluster manager inetd config B Postpath Unified Node (PUN) /etc/passwd /etc/sysconfig/* config /etc/* rsyslog .... resolv.conf config A PPSD config B cluster manager ... SAN LUN inetd ntp.conf Service Y config B spare Load Balancer .... resolv.conf config A Service X config B /etc/passwd inetd /etc/logrotate/* config /etc/* ... ... config A PPSD Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD ... config A config B cluster manager SAN Monitoring Servers /etc/* ... config A config B ... smb.conf config A Service Y config B /etc/passwd inetd /etc/logrotate/* rsyslog .... smb.conf ... PPSD config /etc/* ntp.conf ... DR:BD smb.conf ... config B cluster manager sshd /etc/snmp/* ... resolv.conf ntp.conf ... smb.conf config A PPSD config LVM ... resolv.conf ntp.conf ... DR:BD ... config rsyslog .... LVM ... resolv.conf ntp.conf smb.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config .... LVM resolv.conf config A config B cluster manager sshd /etc/snmp/* /etc/* config .... ... ... PPSD config /etc/* config LVM DR:BD ... sshd /etc/snmp/* /etc/* rsyslog .... ntp.conf smb.conf inetd config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* /etc/* resolv.conf config A config B inetd config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd ... ... PPSD /etc/snmp/* config ... Service Y DR:BD config B ... config LVM DR:BD Service X smb.conf ... /etc/logrotate/* sshd rsyslog /etc/logrotate/* smb.conf config A ntp.conf smb.conf ... Postpath Unified Node (PUN) cluster manager /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM smb.conf /etc/sysconfig/* config inetd rsyslog ... /etc/passwd inetd .... ... ... .... resolv.conf smb.conf Service Y Postpath Unified Node (PUN) /etc/snmp/* config /etc/* ... config A ... config B config DR:BD /etc/snmp/* config ... LVM resolv.conf LVM /etc/logrotate/* smb.conf smb.conf Service X /etc/sysconfig/* config sshd Front-End Server /etc/* inetd .... ... config A Service Y DR:BD config B ntp.conf /etc/passwd inetd rsyslog /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf Front-End Server /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf ADM /etc/passwd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf config B ... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B sshd ntp.conf config /etc/* config ... LVM inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf Front-End Server /etc/* config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... config A inetd /etc/logrotate/* sshd /etc/snmp/* config Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog
But wait! There’s more! GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* rsyslog Service X rsyslog config A Service X sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X inetd sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X /etc/logrotate/* sshd ntp.conf config rsyslog config B ... config A Service X Service Y DR:BD config B ntp.conf Security Appliances config A config B ... config A Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd config rsyslog sshd config rsyslog /etc/snmp/* smb.conf smb.conf ... smb.conf ... config A PPSD config B PPSD config B smb.conf PPSD config B cluster manager config /etc/* .... LVM ... DR:BD ... resolv.conf resolv.conf ntp.conf LUN LUN smb.conf ... smb.conf ... PPSD config B cluster manager ... config A ADM inetd config /etc/sysconfig/* inetd sshd ntp.conf config smb.conf config A Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config smb.conf rsyslog LVM resolv.conf config B ... config A Service X resolv.conf config B config config LVM config DR:BD config LVM config A Service X resolv.conf config inetd config LVM config config /etc/logrotate/* sshd /etc/snmp/* config config LVM .... ... resolv.conf ntp.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A config A PPSD config B cluster manager config B cluster manager SAN Identity Servers LUN LUN LUN Webmail LUN RPC Front-End Server Front-End Server Front-End Server ... /etc/sysconfig/* smb.conf LVM config A Service Y DR:BD config B ... ... /etc/snmp/* /etc/sysconfig/* smb.conf rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A ... inetd Front-End Server /etc/snmp/* resolv.conf ntp.conf Service X config A config B Service Y DR:BD Front-End Server inetd ... /etc/sysconfig/* smb.conf resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... ... /etc/snmp/* /etc/sysconfig/* smb.conf rsyslog LVM resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf smb.conf ... ... config Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config .... /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle (rquelle@cisco.com) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... config A config B Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager config A config B /etc/passwd /etc/* rsyslog LVM ... ... Service X /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B config A config B Postpath Unified Node (PUN) /etc/* rsyslog resolv.conf ... Service Y /etc/passwd /etc/logrotate/* LVM ... config A config B Postpath Unified Node (PUN) /etc/passwd /etc/* config PPSD cluster manager config A config B ... /etc/logrotate/* /etc/snmp/* Service X smb.conf ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/logrotate/* smb.conf LVM ... config A smb.conf /etc/passwd inetd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... ... config B /etc/snmp/* inetd /etc/* smb.conf Service X ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... ... config A config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config /etc/* config B ... /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... inetd /etc/logrotate/* config ... LVM ... config A config B ... config /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd rsyslog .... /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... /etc/* config ... LVM ... config A config config LVM /etc/logrotate/* smb.conf rsyslog .... /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* /etc/sysconfig/* DR:BD /etc/passwd config resolv.conf sshd ntp.conf config /etc/* ADM /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config /etc/* Identity Servers LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... inetd /etc/logrotate/* config ... LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail /etc/passwd inetd /etc/logrotate/* config config LVM DR:BD LUN ADM /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. config resolv.conf ... ... config B cluster manager spare SAN LVM /etc/snmp/* /etc/* rsyslog .... ... DR:BD smb.conf config A PPSD /etc/sysconfig/* config /etc/* LVM ntp.conf ... DR:BD ... config A config B cluster manager config B /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... resolv.conf ntp.conf smb.conf PPSD sshd /etc/snmp/* config LVM resolv.conf ... ... config A Service Y config B Postpath Unified Node (PUN) /etc/* rsyslog .... ... DR:BD smb.conf config A config B Monitoring Servers ntp.conf config A Service X /etc/sysconfig/* config spare DR:BD .... resolv.conf /etc/passwd inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* rsyslog .... resolv.conf ntp.conf /etc/* ... config A config B ... ... config A config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* sshd /etc/* ... ... PPSD cluster manager /etc/snmp/* config ... Service Y DR:BD config B smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B Postpath Unified Node (PUN) /etc/passwd /etc/snmp/* config LVM DR:BD ... config A Service X ntp.conf ... config A Service X config B inetd /etc/logrotate/* sshd rsyslog .... ntp.conf smb.conf config A config B /etc/passwd /etc/logrotate/* smb.conf LVM config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config smb.conf config A Service Y .... inetd rsyslog .... ... config B /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf Postpath Unified Node (PUN) resolv.conf ... PPSD cluster manager resolv.conf /etc/sysconfig/* config config LVM /etc/snmp/* config ... LVM DR:BD /etc/logrotate/* smb.conf rsyslog /etc/passwd /etc/snmp/* config ... sshd ntp.conf config .... config B ... config A Service X inetd ... DR:BD smb.conf config A config B cluster manager inetd smb.conf /etc/* rsyslog .... ntp.conf ... PPSD Service Y DR:BD config B /etc/logrotate/* sshd /etc/snmp/* config ... DR:BD ... config B cluster manager config A Service X /etc/logrotate/* /etc/sysconfig/* config /etc/passwd config resolv.conf ... ... /etc/* ... config A Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A PPSD config B Load Balancer /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config /etc/* LVM resolv.conf ... config A config B cluster manager sshd ntp.conf config LVM inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... ... /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... ... DR:BD smb.conf PPSD sshd /etc/snmp/* config LVM ntp.conf ... ... config A Service Y config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A Service X /etc/passwd inetd /etc/logrotate/* config /etc/* rsyslog .... resolv.conf config A config B ... config A config B Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/snmp/* /etc/* ... ... PPSD inetd /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD ntp.conf smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd rsyslog config A Service X config B Postpath Unified Node (PUN) cluster manager resolv.conf config B ... ... config A Service Y /etc/passwd /etc/sysconfig/* config Service Y DR:BD config B .... ... config A smb.conf ... config A Service X inetd config A Service X ntp.conf /etc/* config ... LVM config B ... /etc/snmp/* config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog smb.conf Postpath Unified Node (PUN) sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... .... ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* LUN RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config B rsyslog .... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... rsyslog config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog Billing Mgt Servers config A PPSD config B cluster manager spare LUN RPC Front-End Server Service X ntp.conf SAN Webmail sshd /etc/snmp/* .... ... ... config A PPSD config config /etc/* LVM DR:BD smb.conf config A config B cluster manager sshd rsyslog /etc/snmp/* .... ntp.conf Security Appliances /etc/sysconfig/* config /etc/logrotate/* config /etc/* resolv.conf ... config A PPSD config B inetd /etc/logrotate/* sshd rsyslog /etc/snmp/* ... ... smb.conf ... config A PPSD cluster manager config config LVM DR:BD ntp.conf config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd rsyslog /etc/snmp/* ... ... resolv.conf ntp.conf ... config A config B cluster manager Postpath Unified Node (PUN) .... LVM DR:BD resolv.conf ... /etc/sysconfig/* config /etc/* ... ... smb.conf ... config A PPSD cluster manager config config .... LVM DR:BD ntp.conf smb.conf ... config A config B cluster manager LUN inetd ntp.conf Service Y config B spare Load Balancer .... resolv.conf config A Service X config B /etc/passwd inetd /etc/logrotate/* sshd rsyslog /etc/snmp/* /etc/* ... ... resolv.conf ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config .... LVM DR:BD resolv.conf SAN Monitoring Servers /etc/* ... config A config B ... smb.conf config A Service Y config B /etc/passwd inetd config /etc/* ... ... sshd rsyslog /etc/snmp/* .... LVM DR:BD ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config config /etc/* ... ... resolv.conf ntp.conf sshd rsyslog /etc/snmp/* .... LVM DR:BD resolv.conf cluster manager config config /etc/* ... ... sshd rsyslog /etc/snmp/* .... LVM DR:BD /etc/logrotate/* config /etc/* .... ntp.conf inetd config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config /etc/* resolv.conf inetd config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd /etc/snmp/* PPSD config A Service X config B ... ... /etc/snmp/* config ... Service Y DR:BD config B ... Postpath Unified Node (PUN) config Service X smb.conf ... config LVM DR:BD /etc/logrotate/* smb.conf config A ntp.conf smb.conf ... /etc/logrotate/* sshd rsyslog /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM smb.conf /etc/sysconfig/* config inetd rsyslog ... /etc/passwd inetd .... ... ... .... resolv.conf smb.conf Service Y Postpath Unified Node (PUN) /etc/snmp/* config /etc/* ... config A ... config B config DR:BD /etc/snmp/* config ... LVM resolv.conf LVM /etc/logrotate/* smb.conf smb.conf Service X /etc/sysconfig/* config sshd Front-End Server /etc/* inetd .... ... config A Service Y DR:BD config B ntp.conf /etc/passwd inetd rsyslog /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf Front-End Server /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf ADM /etc/passwd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf config B ... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B sshd ntp.conf config /etc/* config ... LVM inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf Front-End Server /etc/* config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... config A inetd /etc/logrotate/* sshd /etc/snmp/* config Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog Billing Mgt Servers
“Turtles all the way down” Even individual systems are not like your developer’s workstation: physical hostname service hostname hostname eth0 physical /etc/passwd inetd config sshd config NIC /etc/sysconfig/* /etc/logrotate/* rsyslog LVM eth0 inetd /etc/snmp/* / /etc/* config config ... sshd config rsyslog config LVM hd ntp.conf ... NIC /etc/sysconfig/* ... /etc/snmp/* / /etc/* .... eth1 mgt eth2 iSCSI, NFS resolv.conf smb.conf DR:BD ntp.conf ... ... smb.conf ... Service Y eth0:1 cluster /etc/logrotate/* .... resolv.conf DR:BD /etc/passwd / config A config B Service Y config A config B /var.. SAN inst1 © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) inst2
Deployment Environment is Complex • Many nodes in cluster = many “points” to monitor, including systems and services • Dependencies on external services (DNS, Active Directory, SAN) • Dependencies within application stack • Sequencing of operations critical © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
So, lets just use PaaS! $ cf deploy appname Container App A © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Uh,Yeah. HTTPS HTTP Internet LB/ SSL CF Router UAM REST Server Ubuntu 12.04 Ubuntu 12.04 NATS HM Cloud Controller Ubuntu 12.04 C/C: NATS/ HTTP HTTP/S Data/Msg Bus Container Container App A App B Container Container App A' DEA App B' DEA 1 DEA 1 Ubuntu DEA 1 12.04 Ubuntu 12.04 Ubuntu 12.04 © 2013 Cisco and/or its affiliates. All rights reserved. VHOST A VHOST B RabbitMQ RabbitMQ CentOS 6.4 CentOS 6.4 Reinhardt Quelle (rquelle@cisco.com) Keysp. A Keysp. B Cassandra Cassandra CentOS 6.4 Cassandra CentOS 6.4 Cassandra CentOS 6.4 CentOS 6.4
I’m not going to discuss application architecture in detail • Designing for Always-on Services: • We don’t ever stop serving customers (intentionally, anyway) • Avoiding the Fallacies of Distributed Computing: Emphasized by load and geography • Embracing Failure: At scale, something is ALWAYS breaking • Designing for the real world Circuit breakers, exponential fallback, dynamic configuration • The need for loosely coupled services: Multi-service synchronized updates are extraordinarily costly Read Web Operations (Alspaw, et al), Release It! (Nygaard) © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Running Bits in Production 1 Where do we want the bits to run? Deployment Targets 2 How do they get there? Packaging/Distribution 3 How are they supposed to behave? Configuration 4 How do we start and stop them? Command/Control (aka Orchestration) 5 Are we sure they are running (or ready to run)? Health Checks and Verification 6 When will we run into a wall? Metrics 7 Ow! A wall! Alerts 8 How did we end up at that wall? Logs © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Note: “Monitoring” is multifaceted Where do we want the bits to run? Deployment Targets 2 How do they get there? Packaging/Distribution 3 How are they supposed to behave? Configuration 4 How do we start and stop them? Command/Control (aka Orchestration) 5 Are we sure they are running (or ready to run)? Health Checks and Verification 6 When will we run into a wall? Metrics 7 Was that a wall we just hit? Alerts 8 What just happened there? Logs © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) Monitoring 1
1 Where does a package run? Service ≠ Server • Multiple servers may run same service • A single server may run multiple services even several of the same kind (esp. 32bit) © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
One Service, Multiple Hosts LB Front-End Server Front-End Server /etc/passwd inetd config /etc/sysconfig/* /etc/passwd inetd config /etc/logrotate/* sshd config rsyslog config LVM ... /etc/logrotate/* sshd config rsyslog config LVM /etc/snmp/* ... /etc/* .... ... ntp.conf /etc/snmp/* /etc/* resolv.conf DR:BD /etc/sysconfig/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... Service X © 2013 Cisco and/or its affiliates. All rights reserved. config A config B smb.conf ... Service Y config A config B Reinhardt Quelle (rquelle@cisco.com) Service X config A config B Service Y config A config B
Multiple Services, One Host Application Compute Node /etc/passwd inetd config sshd config rsyslog config LVM /etc/sysconfig/* ... /etc/logrotate/* /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... App Instance © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle (rquelle@cisco.com) App Instance config A' config B'
Deployment Target Choices Where bits run: Shared system May depend on: Virtual Machine Balancing Resources Dedicated server Failover Strategies Security or Isolation Changes over time “Refactoring your Infrastructure” © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
2 Packaging/Distribution • How is software Bundled, Transported and Unpacked/Installed • Goal is to get a specific version of bits extracted onto a system in a predictable way with its dependencies © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Distribution Choices • Recommended: • versioned and signed OS package (RPM, Deb) from controlled repository • puppet “code” is in package, too - puppetfile is source • Less desirable: • direct version control repository (svn, git) checkout • scp tarball • Note there are different “units of deployment” • Images, Packages, Applications © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Configuration 3 • How is this instance supposed to behave? • How is this server supposed to be configured? • May be multiple instances on a given server • Configuration is often changed over time • Consider “static” and “dynamic” configuration © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Standard OS Platform Postpath Unified Node (PUN) /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
+ Application Postpath Unified Node (PUN) /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... PPSD © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle (rquelle@cisco.com)
Fail Postpath Unified Node (PUN) /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... PPSD © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle (rquelle@cisco.com) uh, oh...
Application Configuration • Unique per application instance • Does not overwrite system/environment • Application switches: apache2 -f nginx -c postfix -c Select Config apachectl configtest nginx -t postfix check Verify Config Reload Changed Config Dump Config © 2013 Cisco and/or its affiliates. All rights reserved. apachectl graceful nginx reload postfix reload postconf Reinhardt Quelle (rquelle@cisco.com)
Various classes of configuration • Static (ish) • Environments: Dev, QA, Staging, Production • Data Centers: SJC, DFW, NRT, AMS • Clusters: ISJ1, AC • Use Puppet/Chef/cfengine • Dynamic • Cluster membership • Service hosts: DB, Logs, Metrics, Identity • Use Zookeeper/Noah/Eureka/DNS - MX, GSS/Anycast © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Common Practice: Configuration Management Compute Node configuration daemon Machine Spec config sshd config rsyslog ... DR:BD ... /etc/sysconfig/* config LVM Cluster Spec -* - TOSCA - CloudFormation - Heat /etc/passwd inetd /etc/logrotate/* /etc/snmp/* /etc/* + Environmental Data - Puppet Hiera - Chef Databags .... resolv.conf configuration master ntp.conf smb.conf ... - Puppet Modules - Chef Cookbooks App config A config B Compute Node configuration daemon /etc/passwd inetd config sshd config rsyslog config LVM ... DR:BD ... /etc/sysconfig/* /etc/logrotate/* /etc/snmp/* /etc/* .... resolv.conf ntp.conf smb.conf ... App © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) config A config B
Need a new Dev/QA test environment? Compute Node /etc/passwd configuration daemon inetd /etc/sysconfig/* config /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf -* - TOSCA - CloudFormation - Heat Machine Spec DR:BD mstr A Cluster Spec smb.conf ... App + Environmental Data ntp.conf ... config A config B Compute Node /etc/passwd configuration daemon - Puppet Hiera - Chef Databags inetd /etc/sysconfig/* config /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* - Puppet Modules - Chef Cookbooks .... resolv.conf DR:BD ntp.conf ... smb.conf ... App config A config B Environmental Data Compute Node - Puppet Hiera - Chef Databags configuration daemon /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD mstr B ... ntp.conf smb.conf ... App config A config B Compute Node configuration daemon /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) App config A config B
Make applications configurationfriendly apache2 -f /path/to/config nginx -c /path/to/config postfix -c /path/to/config Select Config apachectl configtest nginx -t postfix check Verify Config Reload Changed Config Dump Config © 2013 Cisco and/or its affiliates. All rights reserved. apachectl graceful nginx reload postfix reload postconf Reinhardt Quelle (rquelle@cisco.com)
Make configuration cooperative and configuration management friendly: logrotate include /etc/logrotate.d monit include /etc/monit.d sudo #include /etc/sudoers.d Sensu © 2013 Cisco and/or its affiliates. All rights reserved. “Deep Merge” of JSON config Reinhardt Quelle (rquelle@cisco.com)
Command and Control (aka “orchestration”) 4 • Starting and stopping services usually requires specific sequence • By definition, cluster has multiple systems on which processes must be controlled • • • Sequencing must be enforced, repeatable Dependencies must be managed Ad-hoc processes may be required on demand © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Many tools available • Fabric • Capistrano • MCollective • RunDeck • And many, many more © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
So now we’re running... Sure about that? © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Are We Alive? 5 • • • • • Health Checks provide specific status Often variable depth: • shallow/frequent, deep/less frequent Many Consumers • Cluster managers, load balancers, process monitors Personal favorite - “Pre-Flight checks” Call on demand © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Now you can run, but... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Where are we headed? 6 • • • Metrics help us understand trends and predict future... • Operations team can deal with server, environment or give us historical context for some event. Collected by central tools for every device in environment Monitoring OS ? Network Equipment App © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Application Metrics • Metrics used in development for profiling often continue to be interesting in production • Highly specific to a specific service: number of open connections quota usage message duration message size distribution AD Lookup response time, uptime message distribution across users/ folders cache hit/miss rate MAPI operation count © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Example Application Metrics Apache $ apache2ctl status memcached $ echo -ne "statsrn" | nc -i1 localhost 11211 ... STAT curr_connections 2 STAT total_connections 71072406 STAT connection_structures 11 STAT cmd_get 58419832 STAT cmd_set 4127175 STAT get_hits 53793350 STAT get_misses 4626482 ... squid [calh@photocache3 ~] $ /usr/sbin/squidclient -p 80 cache_object://localhost/5min ... client_http.all_median_svc_time = 0.000911 seconds client_http.miss_median_svc_time = 0.042766 seconds client_http.nm_median_svc_time = 0.000000 seconds client_http.nh_median_svc_time = 0.024508 seconds client_http.hit_median_svc_time = 0.000911 seconds ... mysql mysql> SHOW SLAVE STATUS G ... Connect_retry: 60 Master_Log_File: db1-bin.439 Read_Master_Log_Pos: 374103193 Relay_Log_File: dbslave-relay.125 Relay_Log_Pos: 854427197 Replicate_ignore_db: Last_errno: 0 Last_error: Skip_counter: 0 Exec_master_log_pos: 587702364 Relay_log_space: 854431912 ... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Again, many tools available • JMX • NewRelic • statsd (extremely light-weight and easy to use) • snmp (highly standard, but high barrier to use and configuration for application layer) • collectd + graphite © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
7 Alerts • Actionable • May be derived from other sources, or raised directly. • Have to be routed to be useful - thank you, Sean! • The application teams will get these! © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
8 What have you been up to? • • Logging key for debugging, tracing, compliance, etc. Logs should be collected off-box for multiple constituents, including: Operations, Support, InfoSec/Compliance, Engineering • Maximum utility when levels manageable at runtime without application restart • Logstash, Splunk... no excuses © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Logs vs. Stats • Logs can be fall-back facility for monitoring of health and metrics But this is less than desirable - Shifts development to ops, usually late in the game - Can be brittle Usually less efficient © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Logging in Production • Clustered, parallel processes require host and process IDs • Activity flows across services - how do we correlate? • • Audit logs important - who did what when? • Passwords must never be logged Keep privacy in mind - if logs have private data, we can’t easily take them out of production. © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
“Wow, that’s a lot we have to build!” © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Or not. © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
If applications provide standard management interfaces... Logging Metrics App Health Process Mgt Config © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
...they can “plug in” to existing infrastructure Log Host OS Logging Logging Metrics Health Process Mgt Stats Host Metrics Logging Alert Manager Metrics App Health Health Process Mgt Cluster/Orchestration Manager CM Config Config Configuration Manager Config Config VCS Config Engineering © 2013 Cisco and/or its affiliates. All rights reserved. Operations Reinhardt Quelle (rquelle@cisco.com)
Operation’s Asks • Make Operability an integral part of product requirements and designs • Engage your operations team as soon as possible to let us help you leverage our platform and services • Talk with us about HA architectures: we don’t have a magic wand! © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
Thank you! www.cisco.com/careers © 2010 Cisco and/or its affiliates. All rights reserved. 75

More Related Content

PDF
How we built Packet's bare metal cloud platform
Packet
 
PPTX
Devs are from Mars, Ops are from Venus, Maish Saidel-Keesing, Cisco
DevOpsDays Tel Aviv
 
PPTX
Devs are from Mars, Ops are from Venus
Maish Saidel-Keesing
 
PPTX
The Data Center and Hadoop
DataWorks Summit
 
PDF
Cisco Connect Toronto 2017 - Introducing the Network Intuitive
Cisco Canada
 
PPTX
Cloud Computing and the Promise of Everything as a Service
Lew Tucker
 
PDF
Complete Download DevOps for networking boost your organization's growth by i...
raiyaalaiaya
 
PDF
Deploying Applications in Today’s Network Infrastructure
Cisco Canada
 
How we built Packet's bare metal cloud platform
Packet
 
Devs are from Mars, Ops are from Venus, Maish Saidel-Keesing, Cisco
DevOpsDays Tel Aviv
 
Devs are from Mars, Ops are from Venus
Maish Saidel-Keesing
 
The Data Center and Hadoop
DataWorks Summit
 
Cisco Connect Toronto 2017 - Introducing the Network Intuitive
Cisco Canada
 
Cloud Computing and the Promise of Everything as a Service
Lew Tucker
 
Complete Download DevOps for networking boost your organization's growth by i...
raiyaalaiaya
 
Deploying Applications in Today’s Network Infrastructure
Cisco Canada
 

Similar to Building for operations - Reinhardt Quelle (20)

PDF
Cisco Managed Private Cloud in Your Data Center: Public cloud experience on ...
Cisco DevNet
 
PDF
PDF DevOps for networking boost your organization's growth by incorporating n...
keshnadjunev
 
PDF
Why Automate the Network?
Hank Preston
 
PDF
Cisco Connect Ottawa 2018 dev net
Cisco Canada
 
PDF
Devops For Networking Steven Armstrong Armstrong Steven
bosomsuaa
 
PDF
DEVNET-1149 Leveraging Rapid Development with PaaS on Cisco Cloud
Cisco DevNet
 
PDF
Synergy 2015 Session Slides: SYN408 XenDesktop 7.6 Architecture - Dealing Wit...
Citrix
 
PDF
Presentation data center design overview
xKinAnx
 
PPTX
Devops for Netops
Karthi Sadasivan
 
PPTX
OpenStack Enabling DevOps
Cisco DevNet
 
PDF
Cisco Connect Halifax 2018 Simple IT
Cisco Canada
 
PDF
Cisco Connect Toronto 2018 consuming public and private clouds
Cisco Canada
 
PDF
CCNP Service Provider Training and Certification
rituraj9b
 
PDF
Ccie And Ccde Written Exam Evolving Technologies Version 11 Study Guide Nicho...
nkosijakirne
 
PDF
finCODE US 2018 - En route to CI/CD nirvana
Rémi Vichery
 
PPTX
DevOps in Network Engineering ( Network Automation using DevNet)
akintunero101
 
PPTX
DEVNET-1125 Partner Case Study - “Project Hybrid Engineer”
Cisco DevNet
 
PPTX
What the business thinks about
DevOps4Networks
 
PPTX
0413-OSS-CKN.pptx
JaamacAbdikhayr
 
PPTX
vBrownBag OpenStack Networking Talk
mestery
 
Cisco Managed Private Cloud in Your Data Center: Public cloud experience on ...
Cisco DevNet
 
PDF DevOps for networking boost your organization's growth by incorporating n...
keshnadjunev
 
Why Automate the Network?
Hank Preston
 
Cisco Connect Ottawa 2018 dev net
Cisco Canada
 
Devops For Networking Steven Armstrong Armstrong Steven
bosomsuaa
 
DEVNET-1149 Leveraging Rapid Development with PaaS on Cisco Cloud
Cisco DevNet
 
Synergy 2015 Session Slides: SYN408 XenDesktop 7.6 Architecture - Dealing Wit...
Citrix
 
Presentation data center design overview
xKinAnx
 
Devops for Netops
Karthi Sadasivan
 
OpenStack Enabling DevOps
Cisco DevNet
 
Cisco Connect Halifax 2018 Simple IT
Cisco Canada
 
Cisco Connect Toronto 2018 consuming public and private clouds
Cisco Canada
 
CCNP Service Provider Training and Certification
rituraj9b
 
Ccie And Ccde Written Exam Evolving Technologies Version 11 Study Guide Nicho...
nkosijakirne
 
finCODE US 2018 - En route to CI/CD nirvana
Rémi Vichery
 
DevOps in Network Engineering ( Network Automation using DevNet)
akintunero101
 
DEVNET-1125 Partner Case Study - “Project Hybrid Engineer”
Cisco DevNet
 
What the business thinks about
DevOps4Networks
 
0413-OSS-CKN.pptx
JaamacAbdikhayr
 
vBrownBag OpenStack Networking Talk
mestery
 
Ad

More from Devopsdays (20)

PDF
Dev opsdays scriptcode
Devopsdays
 
PDF
Zero to hero - Geoff Webb
Devopsdays
 
PDF
Letting go gavin - Mc Donald
Devopsdays
 
PDF
Dw tpain - Gordon Klok
Devopsdays
 
PDF
Dev ops finishes what agile started - Manfred Moser
Devopsdays
 
PDF
Game of thrones - Jonathan Thorpe
Devopsdays
 
PDF
Heka - Rob Miller
Devopsdays
 
PDF
Gaming dev ops - Eduardo Saito
Devopsdays
 
PDF
From the classroom to the cloud a journey with node.js - Christopher Hogue
Devopsdays
 
PDF
Dev ops at mobify - Kyle Young
Devopsdays
 
PDF
Your business needs devops, so don’t follow - Brian johnson
Devopsdays
 
PDF
Test kitchen 1.0 - Fletcher Nichol
Devopsdays
 
PDF
Living system or build factory - Chris Maxwell
Devopsdays
 
PDF
From vagrant to production - Mark Eijsermans
Devopsdays
 
PDF
Dev ops lessons learned - Michael Collins
Devopsdays
 
PDF
Taking devops to the Next Level - Max Martin
Devopsdays
 
PDF
Sensu intro - Sean Porter
Devopsdays
 
PDF
Ops for everyone - John Britton
Devopsdays
 
PDF
Effective monitoring with statsd - Alexis lê-quôc
Devopsdays
 
PDF
Being healthy dev and ops in cookpad - Issei Naruta
Devopsdays
 
Dev opsdays scriptcode
Devopsdays
 
Zero to hero - Geoff Webb
Devopsdays
 
Letting go gavin - Mc Donald
Devopsdays
 
Dw tpain - Gordon Klok
Devopsdays
 
Dev ops finishes what agile started - Manfred Moser
Devopsdays
 
Game of thrones - Jonathan Thorpe
Devopsdays
 
Heka - Rob Miller
Devopsdays
 
Gaming dev ops - Eduardo Saito
Devopsdays
 
From the classroom to the cloud a journey with node.js - Christopher Hogue
Devopsdays
 
Dev ops at mobify - Kyle Young
Devopsdays
 
Your business needs devops, so don’t follow - Brian johnson
Devopsdays
 
Test kitchen 1.0 - Fletcher Nichol
Devopsdays
 
Living system or build factory - Chris Maxwell
Devopsdays
 
From vagrant to production - Mark Eijsermans
Devopsdays
 
Dev ops lessons learned - Michael Collins
Devopsdays
 
Taking devops to the Next Level - Max Martin
Devopsdays
 
Sensu intro - Sean Porter
Devopsdays
 
Ops for everyone - John Britton
Devopsdays
 
Effective monitoring with statsd - Alexis lê-quôc
Devopsdays
 
Being healthy dev and ops in cookpad - Issei Naruta
Devopsdays
 
Ad

Recently uploaded (20)

PDF
GDG Cloud Munich - Intro - Luiz Carneiro - #BuildWithAI - July - Abdel.pdf
Luiz Carneiro
 
PPTX
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
PDF
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
PDF
Peak of Data & AI Encore - Real-Time Insights & Scalable Editing with ArcGIS
Safe Software
 
PDF
Tea4chat - another LLM Project by Kerem Atam
a0m0rajab1
 
PPTX
Dev Dives: Automate, test, and deploy in one place—with Unified Developer Exp...
AndreeaTom
 
PPTX
Simple and concise overview about Quantum computing..pptx
mughal641
 
PPTX
Agile Chennai 18-19 July 2025 Ideathon | AI Powered Microfinance Literacy Gui...
AgileNetwork
 
PDF
How ETL Control Logic Keeps Your Pipelines Safe and Reliable.pdf
Stryv Solutions Pvt. Ltd.
 
PDF
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
PPTX
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
PDF
Get More from Fiori Automation - What’s New, What Works, and What’s Next.pdf
Precisely
 
PDF
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
 
PDF
Oracle AI Vector Search- Getting Started and what's new in 2025- AIOUG Yatra ...
Sandesh Rao
 
PPTX
The Future of AI & Machine Learning.pptx
pritsen4700
 
PDF
Software Development Methodologies in 2025
KodekX
 
PDF
AI Unleashed - Shaping the Future -Starting Today - AIOUG Yatra 2025 - For Co...
Sandesh Rao
 
PDF
Presentation about Hardware and Software in Computer
snehamodhawadiya
 
PDF
NewMind AI Weekly Chronicles - July'25 - Week IV
NewMind AI
 
PDF
MASTERDECK GRAPHSUMMIT SYDNEY (Public).pdf
Neo4j
 
GDG Cloud Munich - Intro - Luiz Carneiro - #BuildWithAI - July - Abdel.pdf
Luiz Carneiro
 
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
Peak of Data & AI Encore - Real-Time Insights & Scalable Editing with ArcGIS
Safe Software
 
Tea4chat - another LLM Project by Kerem Atam
a0m0rajab1
 
Dev Dives: Automate, test, and deploy in one place—with Unified Developer Exp...
AndreeaTom
 
Simple and concise overview about Quantum computing..pptx
mughal641
 
Agile Chennai 18-19 July 2025 Ideathon | AI Powered Microfinance Literacy Gui...
AgileNetwork
 
How ETL Control Logic Keeps Your Pipelines Safe and Reliable.pdf
Stryv Solutions Pvt. Ltd.
 
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
Get More from Fiori Automation - What’s New, What Works, and What’s Next.pdf
Precisely
 
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
 
Oracle AI Vector Search- Getting Started and what's new in 2025- AIOUG Yatra ...
Sandesh Rao
 
The Future of AI & Machine Learning.pptx
pritsen4700
 
Software Development Methodologies in 2025
KodekX
 
AI Unleashed - Shaping the Future -Starting Today - AIOUG Yatra 2025 - For Co...
Sandesh Rao
 
Presentation about Hardware and Software in Computer
snehamodhawadiya
 
NewMind AI Weekly Chronicles - July'25 - Week IV
NewMind AI
 
MASTERDECK GRAPHSUMMIT SYDNEY (Public).pdf
Neo4j
 

Building for operations - Reinhardt Quelle

  • 1. Building Applications for Operability Reinhardt Quelle Operations Architect [email protected] rquelle@* V16e © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected]) 1
  • 2. scr © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 3. Great stuff, but not that’s not what my team does… © 2011 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected]) 3
  • 4. We host and manage SaaS Applications Cisco Social Jabber Millions of Meetings for 10s of Millions of Users totaling Billions of Minutes each month 7x24x365 WebEx © 2011-2012 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected]) 4
  • 5. DevOps Damon Edwards (DTO) John Willis (now at Dell) suggest “CAMS” as useful mnemonic. © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 6. DevOps > Tools •Automation © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 7. DevOps > Tools •Automation •Metrics © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 8. DevOps > Tools •Automation •Metrics •Sharing © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 9. DevOps > Tools •Culture •Automation •Metrics •Sharing © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 10. DevOps > Tools •Culture / Collaboration •Automation •Metrics •Sharing © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 11. Collaboration Requires: • shared goals • communication • empathy • © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 12. Yay, We’re going to do DevOps! http://www.realclearworld.com/blog/2009/01/ © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 13. When you start, its more like this: http://www.news.com.au/world/are-barack-obama-and-vladamir-putin-holding-highlevel-talks-or-are-they-on-an-awkward-date/storyfndir2ev-1226665803294 © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 14. At a more personal level: remember that first date? http://www.hercampus.com/love/dating-hooking/9-first-date-disasters-how-deal-them © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 15. Where do we begin? © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 16. Perspective helps (a lot) http://www.navigonlp.com/blog/stop-look-look-again/boat-land-perspective-2/ © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 17. I come from an Operations background © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 18. Green = Good © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 19. Illustrating the Ops perspective to Engineering http://cdn.twentytwowords.com/wp-content/uploads/Rhinos-Perspective-e1369734532511-634x447.jpg © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 20. A quick illustration: A real, albeit deceased, product (RIP) © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 21. Developer’s View Compute Node resolv.conf ntp.conf smb.conf ... App © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle ([email protected])
  • 23. Servers run in "clusters" Compute Node Compute Node Compute Node inetd config inetd config config sshd config rsyslog config LVM /etc/snmp/* config LVM ... ... ntp.conf ... DR:BD ... © 2013 Cisco and/or its affiliates. All rights reserved. config ... App start/stop orchestration config ntp.conf .... sshd config rsyslog config ... ntp.conf ... .... DR:BD ... ntp.conf App config config ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... App config B Reinhardt Quelle ([email protected]) sshd rsyslog smb.conf config A start/stop orchestration /etc/sysconfig/* /etc/logrotate/* LVM /etc/snmp/* ... config B config resolv.conf smb.conf config A /etc/passwd inetd /etc/* resolv.conf DR:BD /etc/sysconfig/* /etc/logrotate/* LVM /etc/snmp/* ... config B start/stop orchestration config LVM ... App sshd smb.conf config A /etc/passwd inetd /etc/logrotate/* resolv.conf smb.conf /etc/sysconfig/* /etc/* .... .... resolv.conf DR:BD config rsyslog /etc/snmp/* Compute Node /etc/passwd inetd /etc/* /etc/* rsyslog /etc/sysconfig/* /etc/logrotate/* /etc/logrotate/* sshd Compute Node /etc/passwd /etc/passwd /etc/sysconfig/* start/stop orchestration ... config A App config B start/stop orchestration config A config B
  • 24. Which means they may have to share data: Compute Node Compute Node Compute Node inetd sshd /etc/sysconfig/* config config inetd config sshd config LVM rsyslog ... ... ntp.conf ... DR:BD ... config ntp.conf /etc/sysconfig/* /etc/snmp/* config A App start/stop orchestration inetd sshd /etc/sysconfig/* config config rsyslog config ... ... ntp.conf rsyslog config ... .... DR:BD ntp.conf ... App ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... config A App config B start/stop orchestration config smb.conf ... config B rsyslog /etc/logrotate/* resolv.conf smb.conf config A config LVM /etc/snmp/* resolv.conf DR:BD sshd /etc/sysconfig/* config /etc/* LVM .... /etc/passwd inetd /etc/logrotate/* /etc/* ... config B /etc/passwd /etc/logrotate/* smb.conf ... start/stop orchestration sshd resolv.conf smb.conf App config LVM .... .... resolv.conf DR:BD /etc/snmp/* config LVM config inetd /etc/* /etc/* rsyslog /etc/sysconfig/* Compute Node /etc/passwd /etc/logrotate/* /etc/logrotate/* /etc/snmp/* Compute Node /etc/passwd /etc/passwd ... config A App config B start/stop orchestration config A config B start/stop orchestration spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle ([email protected]) LUN
  • 25. Which means they may have to share data: Compute Node Compute Node Compute Node inetd sshd /etc/sysconfig/* config config inetd config sshd config LVM rsyslog ... ... ntp.conf ... DR:BD ... config ntp.conf /etc/sysconfig/* /etc/snmp/* config A App start/stop orchestration inetd sshd /etc/sysconfig/* config config rsyslog config ... ... ntp.conf rsyslog config ... .... DR:BD ntp.conf ... App ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... config A App config B start/stop orchestration config smb.conf ... config B rsyslog /etc/logrotate/* resolv.conf smb.conf config A config LVM /etc/snmp/* resolv.conf DR:BD sshd /etc/sysconfig/* config /etc/* LVM .... /etc/passwd inetd /etc/logrotate/* /etc/* ... config B /etc/passwd /etc/logrotate/* smb.conf ... start/stop orchestration sshd resolv.conf smb.conf App config LVM .... .... resolv.conf DR:BD /etc/snmp/* config LVM config inetd /etc/* /etc/* rsyslog /etc/sysconfig/* Compute Node /etc/passwd /etc/logrotate/* /etc/logrotate/* /etc/snmp/* Compute Node /etc/passwd /etc/passwd ... config A App config B start/stop orchestration config A config B start/stop orchestration spare SAN LUN LUN LUN LUN Note: there are *much* better ways to share data! © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 26. We have several front-end servers Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd /etc/logrotate/* sshd config rsyslog config LVM ... sshd config rsyslog config LVM ... config A Service X config B Postpath Unified Node (PUN) /etc/passwd /etc/sysconfig/* rsyslog config LVM ... .... sshd config rsyslog config ... ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config sshd config rsyslog config ... ntp.conf ... sshd ntp.conf ... rsyslog config ... /etc/snmp/* .... config rsyslog /etc/sysconfig/* config sshd config ... /etc/logrotate/* /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... ... resolv.conf DR:BD ntp.conf ... smb.conf ... ... config A PPSD config B cluster manager /etc/passwd inetd /etc/* smb.conf PPSD config A config B LVM config resolv.conf DR:BD Service Y /etc/logrotate/* /etc/* .... config B Postpath Unified Node (PUN) /etc/sysconfig/* config LVM /etc/snmp/* config A /etc/passwd inetd /etc/logrotate/* resolv.conf DR:BD Service X config B /etc/* .... smb.conf /etc/passwd inetd LVM /etc/snmp/* resolv.conf ... ntp.conf ... Service Y config B /etc/logrotate/* /etc/* DR:BD .... resolv.conf config A Postpath Unified Node (PUN) /etc/sysconfig/* config LVM /etc/snmp/* config A /etc/passwd inetd /etc/logrotate/* config ... ... Postpath Unified Node (PUN) sshd DR:BD smb.conf Service Y config ... /etc/snmp/* /etc/* ntp.conf ... /etc/sysconfig/* config resolv.conf ntp.conf smb.conf inetd config LVM .... resolv.conf config B sshd rsyslog /etc/snmp/* /etc/* .... config A config /etc/logrotate/* ... DR:BD /etc/snmp/* /etc/* Service X inetd /etc/logrotate/* ... DR:BD /etc/passwd /etc/sysconfig/* config config A PPSD config B cluster manager config A PPSD config B cluster manager smb.conf ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle ([email protected]) LUN
  • 27. ...which also have to be redundant. Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd sshd /etc/passwd /etc/sysconfig/* config config inetd sshd config /etc/logrotate/* config config resolv.conf /etc/sysconfig/* ... inetd ... sshd ntp.conf config smb.conf rsyslog Service X Service Y DR:BD config B config resolv.conf ... inetd ... sshd ntp.conf config smb.conf /etc/* rsyslog config B ... ntp.conf Service X config A /etc/snmp/* config B Postpath Unified Node (PUN) Service Y sshd config rsyslog config LVM ... Service X config B sshd config rsyslog config LVM /etc/sysconfig/* config ntp.conf ... ntp.conf ... sshd config /etc/snmp/* config A config B /etc/* config ... Service Y DR:BD .... ... config A resolv.conf config B ... ntp.conf smb.conf .... DR:BD ntp.conf ... rsyslog ... config B config rsyslog ntp.conf ... config ... config B ... /etc/logrotate/* .... /etc/snmp/* /etc/* .... resolv.conf ntp.conf ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... config A PPSD config B cluster manager config LVM /etc/snmp/* ... PPSD config rsyslog /etc/sysconfig/* config sshd /etc/* DR:BD ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle ([email protected]) LUN config A config B /etc/passwd inetd /etc/logrotate/* smb.conf config A cluster manager sshd resolv.conf DR:BD Service Y config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/* .... config A /etc/passwd inetd LVM /etc/snmp/* ... PPSD ... Service X config B Postpath Unified Node (PUN) config smb.conf ... Service Y config A /etc/logrotate/* resolv.conf config A cluster manager config B /etc/* smb.conf PPSD config A /etc/sysconfig/* config LVM /etc/snmp/* resolv.conf DR:BD Service X /etc/passwd inetd /etc/logrotate/* /etc/* .... /etc/logrotate/* smb.conf rsyslog config B ... Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* /etc/snmp/* /etc/passwd /etc/sysconfig/* ... Postpath Unified Node (PUN) /etc/sysconfig/* config .... config resolv.conf smb.conf config A /etc/passwd inetd sshd ntp.conf config LVM ... Service X inetd ... .... resolv.conf smb.conf config A ... /etc/* ... config A Service Y DR:BD config B LVM /etc/logrotate/* config ... LVM resolv.conf config DR:BD /etc/sysconfig/* LVM .... /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd DR:BD /etc/snmp/* ... config A /etc/sysconfig/* config /etc/logrotate/* .... /etc/logrotate/* config ... LVM config A config /etc/passwd LVM config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... DR:BD inetd /etc/logrotate/* /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config
  • 28. And that means load balancers to direct traffic Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd /etc/logrotate/* sshd config sshd /etc/snmp/* config config resolv.conf /etc/sysconfig/* LVM ... inetd ... sshd ntp.conf config smb.conf config A Service Y DR:BD config B sshd /etc/snmp/* config config resolv.conf ... inetd ... sshd ntp.conf config smb.conf /etc/* rsyslog resolv.conf config B ... ntp.conf Service X config A /etc/snmp/* config B Load Balancer Postpath Unified Node (PUN) Service Y config A Service X config B config rsyslog config LVM ... /etc/sysconfig/* config sshd config rsyslog config ... resolv.conf ntp.conf ntp.conf ... config B Service X .... sshd config rsyslog ntp.conf ... Service X config B .... resolv.conf config B ... ntp.conf ... sshd rsyslog ntp.conf ... config ... config rsyslog /etc/sysconfig/* config sshd /etc/snmp/* config ... /etc/logrotate/* /etc/snmp/* /etc/* .... /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... resolv.conf DR:BD ntp.conf ... smb.conf ... ... config A PPSD config B config A PPSD config B cluster manager config A PPSD config B cluster manager smb.conf ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle ([email protected]) LUN config A config B /etc/passwd inetd LVM config resolv.conf DR:BD config B /etc/logrotate/* /etc/* .... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config LVM /etc/snmp/* config A /etc/passwd inetd ... cluster manager Service Y DR:BD ... config A Postpath Unified Node (PUN) config resolv.conf DR:BD Service Y smb.conf PPSD config A config B /etc/* ... config A smb.conf /etc/logrotate/* /etc/* resolv.conf DR:BD config A /etc/sysconfig/* config LVM /etc/snmp/* /etc/* .... /etc/snmp/* config ... LVM /etc/passwd inetd /etc/logrotate/* LVM /etc/snmp/* /etc/logrotate/* smb.conf config B ... Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd /etc/passwd /etc/sysconfig/* ... Postpath Unified Node (PUN) /etc/sysconfig/* config .... smb.conf /etc/passwd inetd sshd ntp.conf config config resolv.conf rsyslog ... config A inetd ... .... smb.conf Service X ... /etc/* ... config A Service Y DR:BD config B LVM /etc/logrotate/* config ... LVM ... config A /etc/snmp/* config DR:BD /etc/sysconfig/* LVM .... config Front-End Server /etc/* /etc/passwd DR:BD /etc/snmp/* /etc/sysconfig/* rsyslog .... /etc/logrotate/* config ... LVM Service X config /etc/passwd DR:BD config /etc/logrotate/* Front-End Server /etc/* rsyslog .... rsyslog inetd /etc/logrotate/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config
  • 29. There are multiple clusters Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config sshd config inetd config config LVM config A Service X /etc/logrotate/* sshd ntp.conf config ... config A Service X Postpath Unified Node (PUN) config A Postpath Unified Node (PUN) Postpath Unified Node (PUN) sshd config rsyslog /etc/* .... resolv.conf config B ... ntp.conf Postpath Unified Node (PUN) config ntp.conf PPSD config B DR:BD cluster manager DR:BD ... /etc/snmp/* /etc/* .... resolv.conf ntp.conf ... ntp.conf smb.conf ... smb.conf ... config A PPSD ... .... smb.conf config B /etc/logrotate/* config config LVM /etc/* resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config sshd rsyslog /etc/snmp/* .... config A /etc/passwd inetd ... /etc/* smb.conf ... config config LVM config B Postpath Unified Node (PUN) /etc/sysconfig/* config sshd rsyslog /etc/snmp/* resolv.conf ntp.conf ... DR:BD smb.conf config B /etc/logrotate/* ... .... config A Service Y /etc/passwd inetd config LVM /etc/* resolv.conf config A ... config A Service X config B /etc/sysconfig/* config sshd rsyslog /etc/snmp/* .... cluster manager /etc/snmp/* ... config A Service Y DR:BD config B config A /etc/logrotate/* ... /etc/* resolv.conf PPSD inetd config LVM /etc/snmp/* ... /etc/logrotate/* config ... smb.conf /etc/passwd /etc/sysconfig/* config /etc/logrotate/* ... DR:BD config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/logrotate/* config config LVM /etc/passwd /etc/sysconfig/* smb.conf resolv.conf config B ... config A Service X config B /etc/sysconfig/* config sshd .... config resolv.conf LVM ... /etc/passwd inetd rsyslog sshd ntp.conf config smb.conf Service Y config B ... rsyslog ... config A DR:BD .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/* config ... LVM resolv.conf ... /etc/snmp/* smb.conf rsyslog config B ... /etc/snmp/* config /etc/logrotate/* smb.conf Service X config Front-End Server /etc/* LVM /etc/sysconfig/* .... ... config A Service Y DR:BD config B /etc/sysconfig/* config sshd rsyslog config resolv.conf inetd /etc/passwd /etc/* config ... LVM ... DR:BD /etc/snmp/* smb.conf rsyslog LVM .... /etc/logrotate/* sshd ntp.conf config ... DR:BD /etc/sysconfig/* config /etc/passwd config resolv.conf inetd /etc/snmp/* Front-End Server /etc/* rsyslog .... ... inetd /etc/logrotate/* /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config sshd /etc/logrotate/* PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager spare SAN LUN LUN LUN Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd config config DR:BD rsyslog ... .... inetd DR:BD ... /etc/snmp/* Service Y DR:BD ntp.conf config A Service X resolv.conf ntp.conf config A Postpath Unified Node (PUN) config A Service Y config inetd Postpath Unified Node (PUN) config config inetd .... ntp.conf Postpath Unified Node (PUN) config config config ... inetd /etc/sysconfig/* config sshd config rsyslog config /etc/logrotate/* /etc/logrotate/* /etc/snmp/* /etc/snmp/* /etc/* /etc/* .... ... LVM ... .... ... DR:BD LVM resolv.conf ntp.conf /etc/passwd /etc/sysconfig/* config sshd .... LVM DR:BD resolv.conf config B Postpath Unified Node (PUN) /etc/* ... ... inetd rsyslog /etc/snmp/* .... LVM DR:BD config A Service Y config B /etc/passwd config /etc/* .... ntp.conf config A Service X /etc/logrotate/* /etc/snmp/* /etc/* resolv.conf ... DR:BD ... resolv.conf ntp.conf resolv.conf ntp.conf smb.conf smb.conf smb.conf ... ... ntp.conf smb.conf ... PPSD /etc/* resolv.conf ... config A config B /etc/sysconfig/* config sshd rsyslog /etc/logrotate/* /etc/snmp/* ... /etc/snmp/* ... config A config B ... smb.conf /etc/passwd /etc/sysconfig/* config sshd rsyslog /etc/logrotate/* ... Service Y config B /etc/passwd config LVM config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config sshd DR:BD /etc/logrotate/* config ... Service Y DR:BD config B ... /etc/passwd inetd rsyslog config A Service X smb.conf ... config B /etc/passwd /etc/sysconfig/* smb.conf LVM ... config A smb.conf Service X .... config resolv.conf sshd ntp.conf config rsyslog .... config B ... Service Y DR:BD config B ... inetd /etc/* config ... LVM resolv.conf DR:BD /etc/snmp/* smb.conf ... config A ... /etc/logrotate/* sshd ntp.conf config rsyslog .... config B ... /etc/snmp/* config LVM /etc/passwd /etc/sysconfig/* /etc/* config ... LVM config A /etc/logrotate/* config rsyslog .... config resolv.conf inetd /etc/logrotate/* sshd ntp.conf config config B ... /etc/sysconfig/* smb.conf Service X LVM Front-End Server /etc/* config /etc/passwd config resolv.conf rsyslog /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* config ... inetd /etc/logrotate/* sshd /etc/snmp/* Front-End Server /etc/* LVM /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog Load Balancer LUN ... config A PPSD config B cluster manager config A config A PPSD config B cluster manager PPSD config B cluster manager smb.conf ... config A config A PPSD config B cluster manager config B cluster manager spare SAN LUN LUN LUN Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd config config LVM ... DR:BD ... inetd rsyslog config resolv.conf smb.conf Service X Service Y DR:BD config B config LVM ... DR:BD /etc/snmp/* ... ... config A config B ... inetd config resolv.conf sshd ntp.conf config smb.conf rsyslog LVM Service X ntp.conf ... config A Service Y DR:BD config B config A Postpath Unified Node (PUN) config B ... config config Service Y config A Service X config B /etc/snmp/* /etc/sysconfig/* config sshd config config /etc/snmp/* Service X ntp.conf config ... DR:BD ... /etc/snmp/* config B /etc/* config ... Service Y DR:BD .... ... config A resolv.conf config B ... ntp.conf ... config A Service X config B /etc/snmp/* config config /etc/snmp/* /etc/sysconfig/* config sshd config config /etc/logrotate/* /etc/snmp/* /etc/* /etc/* .... LVM ... DR:BD ... .... LVM ... DR:BD ... resolv.conf ntp.conf smb.conf smb.conf ... ... resolv.conf ntp.conf smb.conf ... ... config A PPSD config B config A PPSD config B cluster manager config A PPSD config B cluster manager smb.conf ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN config A config B /etc/passwd inetd rsyslog /etc/logrotate/* .... resolv.conf ntp.conf Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/* ... ... config A config B /etc/passwd inetd rsyslog /etc/logrotate/* LVM DR:BD resolv.conf ntp.conf Service Y Postpath Unified Node (PUN) config .... LVM cluster manager config A smb.conf smb.conf PPSD /etc/logrotate/* smb.conf LVM resolv.conf /etc/passwd /etc/sysconfig/* config sshd /etc/* .... resolv.conf ntp.conf inetd rsyslog /etc/logrotate/* /etc/* ... ... config A config B Postpath Unified Node (PUN) /etc/passwd inetd rsyslog /etc/logrotate/* LVM /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd DR:BD .... inetd rsyslog .... smb.conf /etc/passwd inetd rsyslog ... /etc/* ... config B DR:BD /etc/snmp/* smb.conf Service X ... /etc/logrotate/* config ... .... config A /etc/snmp/* config LVM /etc/passwd /etc/sysconfig/* /etc/* resolv.conf /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/logrotate/* config ... LVM config /etc/logrotate/* sshd /etc/snmp/* /etc/passwd /etc/sysconfig/* sshd ntp.conf config config A config Front-End Server /etc/* .... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd LUN LUN Reinhardt Quelle ([email protected])
  • 30. ... and geo-redundant data centers. GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* config A sshd ntp.conf config rsyslog .... resolv.conf config B ... config A Service X resolv.conf config B ... config B Postpath Unified Node (PUN) config B Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) config config Postpath Unified Node (PUN) config config LVM config LVM config config config B Postpath Unified Node (PUN) inetd config config config LVM config LVM /etc/logrotate/* config LUN LUN Webmail config LVM ADM inetd smb.conf ... ... config A config /etc/sysconfig/* inetd sshd ntp.conf config Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config /etc/snmp/* resolv.conf config A Service X Service X config B inetd Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) config LVM config inetd config Service Y DR:BD config B resolv.conf config A Service X config B inetd Service X config B inetd config /etc/* .... ... config A resolv.conf config B ... ntp.conf smb.conf LVM config inetd inetd /etc/sysconfig/* config /etc/logrotate/* config /etc/logrotate/* sshd /etc/snmp/* /etc/* config /etc/snmp/* /etc/* rsyslog .... LVM ... config /etc/* rsyslog .... LVM ... config .... ... resolv.conf ntp.conf ... DR:BD resolv.conf ntp.conf ... smb.conf DR:BD ntp.conf ... smb.conf ... smb.conf ... config A PPSD /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* config DR:BD ... config A config B cluster manager config B Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf PPSD inetd /etc/logrotate/* config LVM resolv.conf ... ... config A Service Y config B /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... DR:BD smb.conf config A config B ... config A Service X config B Postpath Unified Node (PUN) /etc/* rsyslog .... resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/snmp/* /etc/* ... ... PPSD cluster manager Service Y config B /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD ... config A Service X ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config rsyslog .... config A /etc/passwd /etc/sysconfig/* config sshd /etc/* ntp.conf /etc/passwd /etc/logrotate/* smb.conf LVM resolv.conf config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) smb.conf config B inetd rsyslog .... smb.conf config A Service Y /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf .... /etc/snmp/* config ... LVM ... config A config LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... /etc/passwd config A PPSD cluster manager sshd ntp.conf config /etc/* config B ... config A Service X ... ... DR:BD ... inetd ... resolv.conf ntp.conf smb.conf /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* ... ... smb.conf /etc/snmp/* config LVM resolv.conf config A config B cluster manager /etc/snmp/* config ... LVM config A Service X /etc/logrotate/* config rsyslog .... ... ... PPSD Load Balancer /etc/passwd /etc/sysconfig/* config sshd rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config sshd /etc/snmp/* /etc/* LVM DR:BD ... config B sshd ntp.conf config Postpath Unified Node (PUN) /etc/logrotate/* config rsyslog .... ntp.conf smb.conf config A PPSD cluster manager inetd rsyslog inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/passwd /etc/sysconfig/* config sshd /etc/* ... ... DR:BD ... config A config B cluster manager config B Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf PPSD inetd /etc/snmp/* config LVM resolv.conf ... ... config A Service Y config B /etc/passwd /etc/logrotate/* config rsyslog .... ... DR:BD smb.conf config A config B config A Service X config B Postpath Unified Node (PUN) /etc/* rsyslog .... resolv.conf ntp.conf ntp.conf ... config A /etc/sysconfig/* config sshd /etc/snmp/* /etc/* ... ... inetd /etc/logrotate/* sshd /etc/snmp/* config PPSD resolv.conf config B ... smb.conf /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config LVM DR:BD Service Y config B /etc/passwd /etc/sysconfig/* config sshd rsyslog config A Service X config B /etc/passwd cluster manager ... config A Service Y DR:BD config B .... ... config A Service Y config A Service X ntp.conf /etc/* smb.conf ... config A /etc/snmp/* config ... LVM resolv.conf config B ... smb.conf Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... ... config A Service Y DR:BD config B ntp.conf sshd ntp.conf config /etc/* config ... LVM ... config A config B ... inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... ADM /etc/passwd inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config A config B rsyslog .... /etc/snmp/* smb.conf Postpath Unified Node (PUN) PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager spare spare SAN SAN LUN LUN LUN Webmail LUN RPC Front-End Server /etc/sysconfig/* smb.conf rsyslog Service Y DR:BD inetd sshd ntp.conf config ... DR:BD /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A inetd DR:BD ... /etc/snmp/* smb.conf ntp.conf Service X config A config B ... Service Y DR:BD /etc/sysconfig/* sshd ntp.conf config ... DR:BD smb.conf rsyslog resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) config resolv.conf ntp.conf smb.conf smb.conf ... .... Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle ([email protected]) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf config A config B config A config B /etc/passwd /etc/* rsyslog LVM ... PPSD cluster manager Service X /etc/logrotate/* resolv.conf DR:BD ... config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf config A config B Service Y /etc/passwd /etc/logrotate/* resolv.conf ... config A config B Postpath Unified Node (PUN) /etc/passwd LVM ... config A config B ... /etc/* config PPSD cluster manager Service X smb.conf ... /etc/logrotate/* /etc/snmp/* /etc/logrotate/* smb.conf LVM ... config A ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... smb.conf /etc/passwd inetd /etc/snmp/* inetd /etc/* ... config B ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... config B ... /etc/sysconfig/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD /etc/snmp/* config Front-End Server /etc/* rsyslog .... inetd smb.conf Service X config /etc/logrotate/* sshd /etc/snmp/* config /etc/logrotate/* config ... LVM ... config A config B ... inetd /etc/logrotate/* config Front-End Server /etc/* LVM /etc/passwd config resolv.conf .... /etc/passwd /etc/sysconfig/* config sshd rsyslog .... inetd /etc/* config ... LVM resolv.conf Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/logrotate/* sshd ntp.conf config rsyslog .... ... config A config Front-End Server /etc/* /etc/sysconfig/* LVM /etc/passwd config resolv.conf /etc/* ADM Front-End Server /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM /etc/passwd config resolv.conf LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... LVM config A config B ... /etc/logrotate/* LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail Front-End Server /etc/passwd inetd /etc/logrotate/* config config LVM LUN ADM Front-End Server /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. config B LUN Front-End Server inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... LVM config A PPSD config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM DR:BD ntp.conf ... DR:BD smb.conf PPSD cluster manager LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog rsyslog .... ... resolv.conf ntp.conf ... ... config B cluster manager spare LUN RPC Front-End Server Service X /etc/snmp/* /etc/* rsyslog .... ... DR:BD smb.conf config A PPSD sshd /etc/snmp/* config resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B inetd /etc/logrotate/* sshd /etc/* rsyslog .... resolv.conf PPSD cluster manager config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/snmp/* /etc/* ... ... ... config A config B ... Postpath Unified Node (PUN) SAN LUN sshd ntp.conf Service Y config B spare inetd .... resolv.conf smb.conf /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD smb.conf SAN Load Balancer /etc/* ... config A config B ... config A Service X config B /etc/sysconfig/* config /etc/logrotate/* sshd resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/sysconfig/* config rsyslog .... ... ... PPSD cluster manager config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/snmp/* config LVM DR:BD ... config B /etc/logrotate/* smb.conf resolv.conf ... /etc/* rsyslog .... ntp.conf smb.conf config A PPSD cluster manager /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM config B ... config A Service X /etc/logrotate/* config /etc/* ... ... DR:BD ... .... inetd rsyslog .... ... config A smb.conf Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A config B cluster manager config B /etc/snmp/* ... ... /etc/* config ... Service Y DR:BD config B config A /etc/sysconfig/* config sshd /etc/snmp/* config LVM resolv.conf PPSD config B cluster manager inetd /etc/logrotate/* config rsyslog .... ... ... ... config A PPSD sshd /etc/snmp/* /etc/* LVM DR:BD smb.conf ... config A config B cluster manager config A Service X ntp.conf /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd ntp.conf ... DR:BD smb.conf PPSD rsyslog .... ... resolv.conf ntp.conf ... DR:BD ... resolv.conf Service Y Postpath Unified Node (PUN) /etc/passwd inetd /etc/snmp/* ... config B config config LVM DR:BD /etc/logrotate/* smb.conf rsyslog LVM config B ... config A Service X Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config B /etc/logrotate/* sshd ntp.conf config smb.conf config A Service Y /etc/passwd inetd /etc/snmp/* /etc/* rsyslog .... resolv.conf config A config B config B /etc/logrotate/* sshd /etc/snmp/* /etc/* ... ... PPSD Service Y DR:BD config B /etc/sysconfig/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd .... ... config A rsyslog .... config resolv.conf ... ... /etc/* config ... ... /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD inetd config A Service X ntp.conf config A Service X /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog cluster manager resolv.conf config B ... /etc/snmp/* smb.conf rsyslog sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X /etc/passwd inetd Service Y DR:BD config B sshd ntp.conf config LVM ... config A Service Y config A Service X ntp.conf inetd .... ... config A inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... smb.conf ... config A /etc/snmp/* /etc/passwd /etc/sysconfig/* config sshd rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog LVM smb.conf Service X sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf /etc/* config ... Service Y DR:BD config B inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf LVM Service X config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog
  • 31. Which means data must be replicated - somehow. GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* rsyslog Service X rsyslog config A Service X sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X inetd sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X /etc/logrotate/* sshd ntp.conf config rsyslog config B ... config A Service X Service Y DR:BD config B ntp.conf config A config B ... config config A config A Service X config B Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd /etc/snmp/* config config rsyslog /etc/logrotate/* rsyslog DR:BD Postpath Unified Node (PUN) config config /etc/logrotate/* config LUN LUN Webmail config .... LVM ... resolv.conf resolv.conf ntp.conf ... ADM smb.conf ... PPSD ... config A inetd config /etc/sysconfig/* inetd sshd ntp.conf config smb.conf config A Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config smb.conf rsyslog LVM resolv.conf config B ... config A Service X resolv.conf config B Postpath Unified Node (PUN) Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd config config LVM config LVM Postpath Unified Node (PUN) config DR:BD config LVM config A Service X config resolv.conf Postpath Unified Node (PUN) inetd .... resolv.conf ntp.conf smb.conf config config config config LVM /etc/logrotate/* config /etc/snmp/* /etc/* rsyslog .... LVM ... DR:BD config .... ... resolv.conf ntp.conf ... smb.conf DR:BD ntp.conf ... smb.conf ... smb.conf ... config A PPSD sshd /etc/snmp/* config resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* LVM resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config rsyslog .... ... resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... ... config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* LVM DR:BD smb.conf config A config B ... Postpath Unified Node (PUN) Service Y config B PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager spare spare SAN SAN LUN LUN LUN Webmail LUN RPC Front-End Server /etc/sysconfig/* smb.conf rsyslog Service Y DR:BD inetd sshd ntp.conf config ... DR:BD /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A inetd DR:BD ... /etc/snmp/* smb.conf ntp.conf Service X config A config B ... Service Y DR:BD /etc/sysconfig/* sshd ntp.conf config ... DR:BD smb.conf rsyslog resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) config resolv.conf ntp.conf smb.conf smb.conf ... .... Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle ([email protected]) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf config A config B config A config B /etc/passwd /etc/* rsyslog LVM ... PPSD cluster manager Service X /etc/logrotate/* resolv.conf DR:BD ... config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf config A config B Service Y /etc/passwd /etc/logrotate/* resolv.conf ... config A config B Postpath Unified Node (PUN) /etc/passwd LVM ... config A config B ... /etc/* config PPSD cluster manager Service X smb.conf ... /etc/logrotate/* /etc/snmp/* /etc/logrotate/* smb.conf LVM ... config A ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... smb.conf /etc/passwd inetd /etc/snmp/* inetd /etc/* ... config B ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... config B ... /etc/sysconfig/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD /etc/snmp/* config Front-End Server /etc/* rsyslog .... inetd smb.conf Service X config /etc/logrotate/* sshd /etc/snmp/* config /etc/logrotate/* config ... LVM ... config A config B ... inetd /etc/logrotate/* config Front-End Server /etc/* LVM /etc/passwd config resolv.conf .... /etc/passwd /etc/sysconfig/* config sshd rsyslog .... inetd /etc/* config ... LVM resolv.conf Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/logrotate/* sshd ntp.conf config rsyslog .... ... config A config Front-End Server /etc/* /etc/sysconfig/* LVM /etc/passwd config resolv.conf /etc/* ADM Front-End Server /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM /etc/passwd config resolv.conf LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... LVM config A config B ... /etc/logrotate/* LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail Front-End Server /etc/passwd inetd /etc/logrotate/* config config LVM LUN ADM Front-End Server /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. /etc/* ... config A config B ... config A Service X config B /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/sysconfig/* config /etc/* ... ... PPSD cluster manager config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/snmp/* config LVM DR:BD ... config A Service X config B /etc/logrotate/* sshd rsyslog .... ntp.conf /etc/passwd /etc/logrotate/* smb.conf LVM config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) smb.conf config A .... inetd rsyslog .... smb.conf config A Service Y /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf /etc/sysconfig/* config config B cluster manager resolv.conf /etc/passwd inetd resolv.conf ... PPSD config B config LVM /etc/snmp/* config ... LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... config B ... config A Service X /etc/snmp/* config ... sshd ntp.conf config ... ... DR:BD smb.conf config A config B cluster manager inetd smb.conf /etc/* rsyslog .... ntp.conf ... PPSD Service Y DR:BD config B /etc/logrotate/* sshd /etc/snmp/* config ... DR:BD ... config B config A Service X /etc/logrotate/* /etc/sysconfig/* config /etc/passwd config resolv.conf ... ... /etc/* ... config A Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A PPSD cluster manager Load Balancer /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config /etc/* LVM resolv.conf ... config A config B cluster manager sshd ntp.conf config LVM inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... ... /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... ... DR:BD smb.conf PPSD sshd /etc/snmp/* config LVM ntp.conf ... ... config B Postpath Unified Node (PUN) /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B config A Service X config B /etc/logrotate/* sshd /etc/snmp/* config resolv.conf PPSD ... config A /etc/sysconfig/* config /etc/* rsyslog .... ... ... DR:BD inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* rsyslog cluster manager ntp.conf smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd inetd resolv.conf config B ... ... config A Service Y Service Y DR:BD config B .... ... config A smb.conf ... Service X config A Service X ntp.conf /etc/* config ... LVM config B ... /etc/snmp/* config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog smb.conf config A sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... .... ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* LUN RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config B rsyslog .... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... LVM config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM DR:BD config A PPSD config B cluster manager LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog rsyslog ntp.conf ... DR:BD smb.conf ... config B cluster manager spare LUN RPC Front-End Server Service X /etc/snmp/* /etc/* rsyslog ... DR:BD smb.conf config A PPSD sshd /etc/snmp/* config .... LVM ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config LVM resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... DR:BD ... config A /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* config LVM ntp.conf smb.conf config B cluster manager inetd config B Postpath Unified Node (PUN) /etc/passwd /etc/sysconfig/* config /etc/* rsyslog .... resolv.conf config A PPSD config B cluster manager ... SAN LUN sshd ntp.conf Service Y config B spare inetd .... resolv.conf config A Service X config B /etc/passwd inetd /etc/logrotate/* config /etc/* ... ... config A PPSD Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD ... config A config B cluster manager SAN Load Balancer /etc/* ... config A config B ... smb.conf config A Service Y config B /etc/passwd inetd /etc/logrotate/* rsyslog .... smb.conf ... PPSD config /etc/* ntp.conf ... DR:BD smb.conf ... config B cluster manager sshd /etc/snmp/* ... resolv.conf ntp.conf ... smb.conf config A PPSD config LVM ... resolv.conf ntp.conf ... DR:BD ... config rsyslog .... LVM ... resolv.conf ntp.conf smb.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config .... LVM resolv.conf config A config B cluster manager sshd /etc/snmp/* /etc/* config .... ... ... PPSD config /etc/* config LVM DR:BD ... sshd /etc/snmp/* /etc/* rsyslog .... ntp.conf smb.conf inetd config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* /etc/* resolv.conf config A config B inetd config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd ... ... PPSD /etc/snmp/* config ... Service Y DR:BD config B ... config LVM DR:BD Service X smb.conf ... /etc/logrotate/* sshd rsyslog /etc/logrotate/* smb.conf config A ntp.conf smb.conf ... Postpath Unified Node (PUN) cluster manager /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM smb.conf /etc/sysconfig/* config inetd rsyslog ... /etc/passwd inetd .... ... ... .... resolv.conf smb.conf Service Y Postpath Unified Node (PUN) /etc/snmp/* config /etc/* ... config A ... config B config DR:BD /etc/snmp/* config ... LVM resolv.conf LVM /etc/logrotate/* smb.conf smb.conf Service X /etc/sysconfig/* config sshd Front-End Server /etc/* inetd .... ... config A Service Y DR:BD config B ntp.conf /etc/passwd inetd rsyslog /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf Front-End Server /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf ADM /etc/passwd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf config B ... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B sshd ntp.conf config /etc/* config ... LVM inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf Front-End Server /etc/* config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... config A inetd /etc/logrotate/* sshd /etc/snmp/* config Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog
  • 32. But wait! There’s more! GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* rsyslog Service X rsyslog config A Service X sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X inetd sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X /etc/logrotate/* sshd ntp.conf config rsyslog config B ... config A Service X Service Y DR:BD config B ntp.conf Security Appliances config A config B ... config config A config A Service X config B Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd /etc/snmp/* config config rsyslog /etc/logrotate/* rsyslog DR:BD Postpath Unified Node (PUN) config config /etc/logrotate/* config LUN LUN Webmail config .... LVM ... resolv.conf resolv.conf ntp.conf ... ADM smb.conf ... PPSD ... config A inetd config /etc/sysconfig/* inetd sshd ntp.conf config smb.conf config A Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config smb.conf rsyslog LVM resolv.conf config B ... config A Service X resolv.conf config B Postpath Unified Node (PUN) Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd config config LVM config LVM Postpath Unified Node (PUN) config DR:BD config LVM config A Service X config resolv.conf Postpath Unified Node (PUN) inetd config config config config LVM /etc/logrotate/* config /etc/snmp/* /etc/* rsyslog .... LVM ... DR:BD config .... ... resolv.conf ntp.conf ... smb.conf DR:BD ntp.conf ... smb.conf ... smb.conf ... config A PPSD sshd /etc/snmp/* config resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* LVM resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config rsyslog .... ... resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... ... config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* LVM DR:BD smb.conf config A config B ... Postpath Unified Node (PUN) PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager SAN Identity Servers LUN LUN LUN Webmail LUN RPC Front-End Server /etc/sysconfig/* smb.conf rsyslog Service Y DR:BD inetd sshd ntp.conf config ... DR:BD /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A inetd DR:BD ... /etc/snmp/* smb.conf ntp.conf Service X config A config B ... Service Y DR:BD /etc/sysconfig/* sshd ntp.conf config ... DR:BD smb.conf rsyslog resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) config resolv.conf ntp.conf smb.conf smb.conf ... .... Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle ([email protected]) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf config A config B config A config B /etc/passwd /etc/* rsyslog LVM ... PPSD cluster manager Service X /etc/logrotate/* resolv.conf DR:BD ... config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf config A config B Service Y /etc/passwd /etc/logrotate/* resolv.conf ... config A config B Postpath Unified Node (PUN) /etc/passwd LVM ... config A config B ... /etc/* config PPSD cluster manager Service X smb.conf ... /etc/logrotate/* /etc/snmp/* /etc/logrotate/* smb.conf LVM ... config A ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... smb.conf /etc/passwd inetd /etc/snmp/* inetd /etc/* ... config B ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... config B ... /etc/sysconfig/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD /etc/snmp/* config Front-End Server /etc/* rsyslog .... inetd smb.conf Service X config /etc/logrotate/* sshd /etc/snmp/* config /etc/logrotate/* config ... LVM ... config A config B ... inetd /etc/logrotate/* config Front-End Server /etc/* LVM /etc/passwd config resolv.conf .... /etc/passwd /etc/sysconfig/* config sshd rsyslog .... inetd /etc/* config ... LVM resolv.conf Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/logrotate/* sshd ntp.conf config rsyslog .... ... config A config Front-End Server /etc/* /etc/sysconfig/* LVM /etc/passwd config resolv.conf /etc/* ADM Front-End Server /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM /etc/passwd config resolv.conf LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... LVM config A config B ... /etc/logrotate/* LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail Front-End Server /etc/passwd inetd /etc/logrotate/* config config LVM LUN ADM Front-End Server /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. ntp.conf smb.conf spare SAN LVM .... resolv.conf Service Y config B spare DR:BD /etc/* ... config A config B ... config A Service X config B /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/sysconfig/* config /etc/* ... ... PPSD cluster manager config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/snmp/* config LVM DR:BD ... config A Service X config B /etc/logrotate/* sshd rsyslog .... ntp.conf /etc/passwd /etc/logrotate/* smb.conf LVM config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) smb.conf config A .... inetd rsyslog .... smb.conf config A Service Y /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf /etc/sysconfig/* config config B cluster manager resolv.conf /etc/passwd inetd resolv.conf ... PPSD config B config LVM /etc/snmp/* config ... LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... config B ... config A Service X /etc/snmp/* config ... sshd ntp.conf config ... ... DR:BD smb.conf config A config B cluster manager inetd smb.conf /etc/* rsyslog .... ntp.conf ... PPSD Service Y DR:BD config B /etc/logrotate/* sshd /etc/snmp/* config ... DR:BD ... config B config A Service X /etc/logrotate/* /etc/sysconfig/* config /etc/passwd config resolv.conf ... ... /etc/* ... config A Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A PPSD cluster manager Load Balancer /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config /etc/* LVM resolv.conf ... config A config B cluster manager sshd ntp.conf config LVM inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... ... /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... ... DR:BD smb.conf PPSD sshd /etc/snmp/* config LVM ntp.conf ... ... config B Postpath Unified Node (PUN) /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B config A Service X config B /etc/logrotate/* sshd /etc/snmp/* config resolv.conf PPSD ... config A /etc/sysconfig/* config /etc/* rsyslog .... ... ... DR:BD inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* rsyslog cluster manager ntp.conf smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd inetd resolv.conf config B ... ... config A Service Y Service Y DR:BD config B .... ... config A smb.conf ... Service X config A Service X ntp.conf /etc/* config ... LVM config B ... /etc/snmp/* config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog smb.conf config A sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... .... ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* LUN RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config B rsyslog .... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... rsyslog config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Billing Mgt Servers config A PPSD config B cluster manager LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog Service X ntp.conf ... DR:BD smb.conf ... config B cluster manager spare LUN RPC Front-End Server sshd /etc/snmp/* /etc/* rsyslog ... DR:BD smb.conf config A PPSD sshd /etc/snmp/* config .... LVM ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config LVM resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... DR:BD ... config A /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* config LVM ntp.conf smb.conf config B cluster manager inetd config B Postpath Unified Node (PUN) /etc/passwd /etc/sysconfig/* config /etc/* rsyslog .... resolv.conf config A PPSD config B cluster manager ... SAN LUN inetd ntp.conf Service Y config B spare Load Balancer .... resolv.conf config A Service X config B /etc/passwd inetd /etc/logrotate/* config /etc/* ... ... config A PPSD Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD ... config A config B cluster manager SAN Monitoring Servers /etc/* ... config A config B ... smb.conf config A Service Y config B /etc/passwd inetd /etc/logrotate/* rsyslog .... smb.conf ... PPSD config /etc/* ntp.conf ... DR:BD smb.conf ... config B cluster manager sshd /etc/snmp/* ... resolv.conf ntp.conf ... smb.conf config A PPSD config LVM ... resolv.conf ntp.conf ... DR:BD ... config rsyslog .... LVM ... resolv.conf ntp.conf smb.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config .... LVM resolv.conf config A config B cluster manager sshd /etc/snmp/* /etc/* config .... ... ... PPSD config /etc/* config LVM DR:BD ... sshd /etc/snmp/* /etc/* rsyslog .... ntp.conf smb.conf inetd config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* /etc/* resolv.conf config A config B inetd config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd ... ... PPSD /etc/snmp/* config ... Service Y DR:BD config B ... config LVM DR:BD Service X smb.conf ... /etc/logrotate/* sshd rsyslog /etc/logrotate/* smb.conf config A ntp.conf smb.conf ... Postpath Unified Node (PUN) cluster manager /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM smb.conf /etc/sysconfig/* config inetd rsyslog ... /etc/passwd inetd .... ... ... .... resolv.conf smb.conf Service Y Postpath Unified Node (PUN) /etc/snmp/* config /etc/* ... config A ... config B config DR:BD /etc/snmp/* config ... LVM resolv.conf LVM /etc/logrotate/* smb.conf smb.conf Service X /etc/sysconfig/* config sshd Front-End Server /etc/* inetd .... ... config A Service Y DR:BD config B ntp.conf /etc/passwd inetd rsyslog /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf Front-End Server /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf ADM /etc/passwd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf config B ... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B sshd ntp.conf config /etc/* config ... LVM inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf Front-End Server /etc/* config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... config A inetd /etc/logrotate/* sshd /etc/snmp/* config Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog
  • 33. But wait! There’s more! GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* rsyslog Service X rsyslog config A Service X sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X inetd sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X /etc/logrotate/* sshd ntp.conf config rsyslog config B ... config A Service X Service Y DR:BD config B ntp.conf Security Appliances config A config B ... config A Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd config rsyslog sshd config rsyslog /etc/snmp/* smb.conf smb.conf ... smb.conf ... config A PPSD config B PPSD config B smb.conf PPSD config B cluster manager config /etc/* .... LVM ... DR:BD ... resolv.conf resolv.conf ntp.conf LUN LUN smb.conf ... smb.conf ... PPSD config B cluster manager ... config A ADM inetd config /etc/sysconfig/* inetd sshd ntp.conf config smb.conf config A Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config smb.conf rsyslog LVM resolv.conf config B ... config A Service X resolv.conf config B config config LVM config DR:BD config LVM config A Service X resolv.conf config inetd config LVM config config /etc/logrotate/* sshd /etc/snmp/* config config LVM .... ... resolv.conf ntp.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A config A PPSD config B cluster manager config B cluster manager SAN Identity Servers LUN LUN LUN Webmail LUN RPC Front-End Server Front-End Server Front-End Server ... /etc/sysconfig/* smb.conf LVM config A Service Y DR:BD config B ... ... /etc/snmp/* /etc/sysconfig/* smb.conf rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A ... inetd Front-End Server /etc/snmp/* resolv.conf ntp.conf Service X config A config B Service Y DR:BD Front-End Server inetd ... /etc/sysconfig/* smb.conf resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... ... /etc/snmp/* /etc/sysconfig/* smb.conf rsyslog LVM resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf smb.conf ... ... config Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config .... /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle ([email protected]) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... config A config B Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager config A config B /etc/passwd /etc/* rsyslog LVM ... ... Service X /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B config A config B Postpath Unified Node (PUN) /etc/* rsyslog resolv.conf ... Service Y /etc/passwd /etc/logrotate/* LVM ... config A config B Postpath Unified Node (PUN) /etc/passwd /etc/* config PPSD cluster manager config A config B ... /etc/logrotate/* /etc/snmp/* Service X smb.conf ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/logrotate/* smb.conf LVM ... config A smb.conf /etc/passwd inetd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... ... config B /etc/snmp/* inetd /etc/* smb.conf Service X ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... ... config A config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config /etc/* config B ... /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... inetd /etc/logrotate/* config ... LVM ... config A config B ... config /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd rsyslog .... /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... /etc/* config ... LVM ... config A config config LVM /etc/logrotate/* smb.conf rsyslog .... /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* /etc/sysconfig/* DR:BD /etc/passwd config resolv.conf sshd ntp.conf config /etc/* ADM /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config /etc/* Identity Servers LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... inetd /etc/logrotate/* config ... LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail /etc/passwd inetd /etc/logrotate/* config config LVM DR:BD LUN ADM /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. config resolv.conf ... ... config B cluster manager spare SAN LVM /etc/snmp/* /etc/* rsyslog .... ... DR:BD smb.conf config A PPSD /etc/sysconfig/* config /etc/* LVM ntp.conf ... DR:BD ... config A config B cluster manager config B /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... resolv.conf ntp.conf smb.conf PPSD sshd /etc/snmp/* config LVM resolv.conf ... ... config A Service Y config B Postpath Unified Node (PUN) /etc/* rsyslog .... ... DR:BD smb.conf config A config B Monitoring Servers ntp.conf config A Service X /etc/sysconfig/* config spare DR:BD .... resolv.conf /etc/passwd inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* rsyslog .... resolv.conf ntp.conf /etc/* ... config A config B ... ... config A config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* sshd /etc/* ... ... PPSD cluster manager /etc/snmp/* config ... Service Y DR:BD config B smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B Postpath Unified Node (PUN) /etc/passwd /etc/snmp/* config LVM DR:BD ... config A Service X ntp.conf ... config A Service X config B inetd /etc/logrotate/* sshd rsyslog .... ntp.conf smb.conf config A config B /etc/passwd /etc/logrotate/* smb.conf LVM config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config smb.conf config A Service Y .... inetd rsyslog .... ... config B /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf Postpath Unified Node (PUN) resolv.conf ... PPSD cluster manager resolv.conf /etc/sysconfig/* config config LVM /etc/snmp/* config ... LVM DR:BD /etc/logrotate/* smb.conf rsyslog /etc/passwd /etc/snmp/* config ... sshd ntp.conf config .... config B ... config A Service X inetd ... DR:BD smb.conf config A config B cluster manager inetd smb.conf /etc/* rsyslog .... ntp.conf ... PPSD Service Y DR:BD config B /etc/logrotate/* sshd /etc/snmp/* config ... DR:BD ... config B cluster manager config A Service X /etc/logrotate/* /etc/sysconfig/* config /etc/passwd config resolv.conf ... ... /etc/* ... config A Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A PPSD config B Load Balancer /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config /etc/* LVM resolv.conf ... config A config B cluster manager sshd ntp.conf config LVM inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... ... /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... ... DR:BD smb.conf PPSD sshd /etc/snmp/* config LVM ntp.conf ... ... config A Service Y config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A Service X /etc/passwd inetd /etc/logrotate/* config /etc/* rsyslog .... resolv.conf config A config B ... config A config B Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/snmp/* /etc/* ... ... PPSD inetd /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD ntp.conf smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd rsyslog config A Service X config B Postpath Unified Node (PUN) cluster manager resolv.conf config B ... ... config A Service Y /etc/passwd /etc/sysconfig/* config Service Y DR:BD config B .... ... config A smb.conf ... config A Service X inetd config A Service X ntp.conf /etc/* config ... LVM config B ... /etc/snmp/* config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog smb.conf Postpath Unified Node (PUN) sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... .... ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* LUN RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config B rsyslog .... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... rsyslog config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog Billing Mgt Servers config A PPSD config B cluster manager spare LUN RPC Front-End Server Service X ntp.conf SAN Webmail sshd /etc/snmp/* .... ... ... config A PPSD config config /etc/* LVM DR:BD smb.conf config A config B cluster manager sshd rsyslog /etc/snmp/* .... ntp.conf Security Appliances /etc/sysconfig/* config /etc/logrotate/* config /etc/* resolv.conf ... config A PPSD config B inetd /etc/logrotate/* sshd rsyslog /etc/snmp/* ... ... smb.conf ... config A PPSD cluster manager config config LVM DR:BD ntp.conf config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd rsyslog /etc/snmp/* ... ... resolv.conf ntp.conf ... config A config B cluster manager Postpath Unified Node (PUN) .... LVM DR:BD resolv.conf ... /etc/sysconfig/* config /etc/* ... ... smb.conf ... config A PPSD cluster manager config config .... LVM DR:BD ntp.conf smb.conf ... config A config B cluster manager LUN inetd ntp.conf Service Y config B spare Load Balancer .... resolv.conf config A Service X config B /etc/passwd inetd /etc/logrotate/* sshd rsyslog /etc/snmp/* /etc/* ... ... resolv.conf ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config .... LVM DR:BD resolv.conf SAN Monitoring Servers /etc/* ... config A config B ... smb.conf config A Service Y config B /etc/passwd inetd config /etc/* ... ... sshd rsyslog /etc/snmp/* .... LVM DR:BD ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config config /etc/* ... ... resolv.conf ntp.conf sshd rsyslog /etc/snmp/* .... LVM DR:BD resolv.conf cluster manager config config /etc/* ... ... sshd rsyslog /etc/snmp/* .... LVM DR:BD /etc/logrotate/* config /etc/* .... ntp.conf inetd config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config /etc/* resolv.conf inetd config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd /etc/snmp/* PPSD config A Service X config B ... ... /etc/snmp/* config ... Service Y DR:BD config B ... Postpath Unified Node (PUN) config Service X smb.conf ... config LVM DR:BD /etc/logrotate/* smb.conf config A ntp.conf smb.conf ... /etc/logrotate/* sshd rsyslog /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM smb.conf /etc/sysconfig/* config inetd rsyslog ... /etc/passwd inetd .... ... ... .... resolv.conf smb.conf Service Y Postpath Unified Node (PUN) /etc/snmp/* config /etc/* ... config A ... config B config DR:BD /etc/snmp/* config ... LVM resolv.conf LVM /etc/logrotate/* smb.conf smb.conf Service X /etc/sysconfig/* config sshd Front-End Server /etc/* inetd .... ... config A Service Y DR:BD config B ntp.conf /etc/passwd inetd rsyslog /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf Front-End Server /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf ADM /etc/passwd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf config B ... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B sshd ntp.conf config /etc/* config ... LVM inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf Front-End Server /etc/* config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... config A inetd /etc/logrotate/* sshd /etc/snmp/* config Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog Billing Mgt Servers
  • 34. “Turtles all the way down” Even individual systems are not like your developer’s workstation: physical hostname service hostname hostname eth0 physical /etc/passwd inetd config sshd config NIC /etc/sysconfig/* /etc/logrotate/* rsyslog LVM eth0 inetd /etc/snmp/* / /etc/* config config ... sshd config rsyslog config LVM hd ntp.conf ... NIC /etc/sysconfig/* ... /etc/snmp/* / /etc/* .... eth1 mgt eth2 iSCSI, NFS resolv.conf smb.conf DR:BD ntp.conf ... ... smb.conf ... Service Y eth0:1 cluster /etc/logrotate/* .... resolv.conf DR:BD /etc/passwd / config A config B Service Y config A config B /var.. SAN inst1 © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected]) inst2
  • 35. Deployment Environment is Complex • Many nodes in cluster = many “points” to monitor, including systems and services • Dependencies on external services (DNS, Active Directory, SAN) • Dependencies within application stack • Sequencing of operations critical © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 36. So, lets just use PaaS! $ cf deploy appname Container App A © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 37. Uh,Yeah. HTTPS HTTP Internet LB/ SSL CF Router UAM REST Server Ubuntu 12.04 Ubuntu 12.04 NATS HM Cloud Controller Ubuntu 12.04 C/C: NATS/ HTTP HTTP/S Data/Msg Bus Container Container App A App B Container Container App A' DEA App B' DEA 1 DEA 1 Ubuntu DEA 1 12.04 Ubuntu 12.04 Ubuntu 12.04 © 2013 Cisco and/or its affiliates. All rights reserved. VHOST A VHOST B RabbitMQ RabbitMQ CentOS 6.4 CentOS 6.4 Reinhardt Quelle ([email protected]) Keysp. A Keysp. B Cassandra Cassandra CentOS 6.4 Cassandra CentOS 6.4 Cassandra CentOS 6.4 CentOS 6.4
  • 38. I’m not going to discuss application architecture in detail • Designing for Always-on Services: • We don’t ever stop serving customers (intentionally, anyway) • Avoiding the Fallacies of Distributed Computing: Emphasized by load and geography • Embracing Failure: At scale, something is ALWAYS breaking • Designing for the real world Circuit breakers, exponential fallback, dynamic configuration • The need for loosely coupled services: Multi-service synchronized updates are extraordinarily costly Read Web Operations (Alspaw, et al), Release It! (Nygaard) © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 39. Running Bits in Production 1 Where do we want the bits to run? Deployment Targets 2 How do they get there? Packaging/Distribution 3 How are they supposed to behave? Configuration 4 How do we start and stop them? Command/Control (aka Orchestration) 5 Are we sure they are running (or ready to run)? Health Checks and Verification 6 When will we run into a wall? Metrics 7 Ow! A wall! Alerts 8 How did we end up at that wall? Logs © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 40. Note: “Monitoring” is multifaceted Where do we want the bits to run? Deployment Targets 2 How do they get there? Packaging/Distribution 3 How are they supposed to behave? Configuration 4 How do we start and stop them? Command/Control (aka Orchestration) 5 Are we sure they are running (or ready to run)? Health Checks and Verification 6 When will we run into a wall? Metrics 7 Was that a wall we just hit? Alerts 8 What just happened there? Logs © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected]) Monitoring 1
  • 41. 1 Where does a package run? Service ≠ Server • Multiple servers may run same service • A single server may run multiple services even several of the same kind (esp. 32bit) © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 42. One Service, Multiple Hosts LB Front-End Server Front-End Server /etc/passwd inetd config /etc/sysconfig/* /etc/passwd inetd config /etc/logrotate/* sshd config rsyslog config LVM ... /etc/logrotate/* sshd config rsyslog config LVM /etc/snmp/* ... /etc/* .... ... ntp.conf /etc/snmp/* /etc/* resolv.conf DR:BD /etc/sysconfig/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... Service X © 2013 Cisco and/or its affiliates. All rights reserved. config A config B smb.conf ... Service Y config A config B Reinhardt Quelle ([email protected]) Service X config A config B Service Y config A config B
  • 43. Multiple Services, One Host Application Compute Node /etc/passwd inetd config sshd config rsyslog config LVM /etc/sysconfig/* ... /etc/logrotate/* /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... App Instance © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle ([email protected]) App Instance config A' config B'
  • 44. Deployment Target Choices Where bits run: Shared system May depend on: Virtual Machine Balancing Resources Dedicated server Failover Strategies Security or Isolation Changes over time “Refactoring your Infrastructure” © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 45. 2 Packaging/Distribution • How is software Bundled, Transported and Unpacked/Installed • Goal is to get a specific version of bits extracted onto a system in a predictable way with its dependencies © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 46. Distribution Choices • Recommended: • versioned and signed OS package (RPM, Deb) from controlled repository • puppet “code” is in package, too - puppetfile is source • Less desirable: • direct version control repository (svn, git) checkout • scp tarball • Note there are different “units of deployment” • Images, Packages, Applications © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 47. Configuration 3 • How is this instance supposed to behave? • How is this server supposed to be configured? • May be multiple instances on a given server • Configuration is often changed over time • Consider “static” and “dynamic” configuration © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 48. Standard OS Platform Postpath Unified Node (PUN) /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 49. + Application Postpath Unified Node (PUN) /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... PPSD © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle ([email protected])
  • 50. Fail Postpath Unified Node (PUN) /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... PPSD © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle ([email protected]) uh, oh...
  • 51. Application Configuration • Unique per application instance • Does not overwrite system/environment • Application switches: apache2 -f nginx -c postfix -c Select Config apachectl configtest nginx -t postfix check Verify Config Reload Changed Config Dump Config © 2013 Cisco and/or its affiliates. All rights reserved. apachectl graceful nginx reload postfix reload postconf Reinhardt Quelle ([email protected])
  • 52. Various classes of configuration • Static (ish) • Environments: Dev, QA, Staging, Production • Data Centers: SJC, DFW, NRT, AMS • Clusters: ISJ1, AC • Use Puppet/Chef/cfengine • Dynamic • Cluster membership • Service hosts: DB, Logs, Metrics, Identity • Use Zookeeper/Noah/Eureka/DNS - MX, GSS/Anycast © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 53. Common Practice: Configuration Management Compute Node configuration daemon Machine Spec config sshd config rsyslog ... DR:BD ... /etc/sysconfig/* config LVM Cluster Spec -* - TOSCA - CloudFormation - Heat /etc/passwd inetd /etc/logrotate/* /etc/snmp/* /etc/* + Environmental Data - Puppet Hiera - Chef Databags .... resolv.conf configuration master ntp.conf smb.conf ... - Puppet Modules - Chef Cookbooks App config A config B Compute Node configuration daemon /etc/passwd inetd config sshd config rsyslog config LVM ... DR:BD ... /etc/sysconfig/* /etc/logrotate/* /etc/snmp/* /etc/* .... resolv.conf ntp.conf smb.conf ... App © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected]) config A config B
  • 54. Need a new Dev/QA test environment? Compute Node /etc/passwd configuration daemon inetd /etc/sysconfig/* config /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf -* - TOSCA - CloudFormation - Heat Machine Spec DR:BD mstr A Cluster Spec smb.conf ... App + Environmental Data ntp.conf ... config A config B Compute Node /etc/passwd configuration daemon - Puppet Hiera - Chef Databags inetd /etc/sysconfig/* config /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* - Puppet Modules - Chef Cookbooks .... resolv.conf DR:BD ntp.conf ... smb.conf ... App config A config B Environmental Data Compute Node - Puppet Hiera - Chef Databags configuration daemon /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD mstr B ... ntp.conf smb.conf ... App config A config B Compute Node configuration daemon /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected]) App config A config B
  • 55. Make applications configurationfriendly apache2 -f /path/to/config nginx -c /path/to/config postfix -c /path/to/config Select Config apachectl configtest nginx -t postfix check Verify Config Reload Changed Config Dump Config © 2013 Cisco and/or its affiliates. All rights reserved. apachectl graceful nginx reload postfix reload postconf Reinhardt Quelle ([email protected])
  • 56. Make configuration cooperative and configuration management friendly: logrotate include /etc/logrotate.d monit include /etc/monit.d sudo #include /etc/sudoers.d Sensu © 2013 Cisco and/or its affiliates. All rights reserved. “Deep Merge” of JSON config Reinhardt Quelle ([email protected])
  • 57. Command and Control (aka “orchestration”) 4 • Starting and stopping services usually requires specific sequence • By definition, cluster has multiple systems on which processes must be controlled • • • Sequencing must be enforced, repeatable Dependencies must be managed Ad-hoc processes may be required on demand © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 58. Many tools available • Fabric • Capistrano • MCollective • RunDeck • And many, many more © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 59. So now we’re running... Sure about that? © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 60. Are We Alive? 5 • • • • • Health Checks provide specific status Often variable depth: • shallow/frequent, deep/less frequent Many Consumers • Cluster managers, load balancers, process monitors Personal favorite - “Pre-Flight checks” Call on demand © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 61. Now you can run, but... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 62. Where are we headed? 6 • • • Metrics help us understand trends and predict future... • Operations team can deal with server, environment or give us historical context for some event. Collected by central tools for every device in environment Monitoring OS ? Network Equipment App © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 63. Application Metrics • Metrics used in development for profiling often continue to be interesting in production • Highly specific to a specific service: number of open connections quota usage message duration message size distribution AD Lookup response time, uptime message distribution across users/ folders cache hit/miss rate MAPI operation count © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 64. Example Application Metrics Apache $ apache2ctl status memcached $ echo -ne "statsrn" | nc -i1 localhost 11211 ... STAT curr_connections 2 STAT total_connections 71072406 STAT connection_structures 11 STAT cmd_get 58419832 STAT cmd_set 4127175 STAT get_hits 53793350 STAT get_misses 4626482 ... squid [calh@photocache3 ~] $ /usr/sbin/squidclient -p 80 cache_object://localhost/5min ... client_http.all_median_svc_time = 0.000911 seconds client_http.miss_median_svc_time = 0.042766 seconds client_http.nm_median_svc_time = 0.000000 seconds client_http.nh_median_svc_time = 0.024508 seconds client_http.hit_median_svc_time = 0.000911 seconds ... mysql mysql> SHOW SLAVE STATUS G ... Connect_retry: 60 Master_Log_File: db1-bin.439 Read_Master_Log_Pos: 374103193 Relay_Log_File: dbslave-relay.125 Relay_Log_Pos: 854427197 Replicate_ignore_db: Last_errno: 0 Last_error: Skip_counter: 0 Exec_master_log_pos: 587702364 Relay_log_space: 854431912 ... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 65. Again, many tools available • JMX • NewRelic • statsd (extremely light-weight and easy to use) • snmp (highly standard, but high barrier to use and configuration for application layer) • collectd + graphite © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 66. 7 Alerts • Actionable • May be derived from other sources, or raised directly. • Have to be routed to be useful - thank you, Sean! • The application teams will get these! © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 67. 8 What have you been up to? • • Logging key for debugging, tracing, compliance, etc. Logs should be collected off-box for multiple constituents, including: Operations, Support, InfoSec/Compliance, Engineering • Maximum utility when levels manageable at runtime without application restart • Logstash, Splunk... no excuses © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 68. Logs vs. Stats • Logs can be fall-back facility for monitoring of health and metrics But this is less than desirable - Shifts development to ops, usually late in the game - Can be brittle Usually less efficient © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 69. Logging in Production • Clustered, parallel processes require host and process IDs • Activity flows across services - how do we correlate? • • Audit logs important - who did what when? • Passwords must never be logged Keep privacy in mind - if logs have private data, we can’t easily take them out of production. © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 70. “Wow, that’s a lot we have to build!” © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 71. Or not. © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 72. If applications provide standard management interfaces... Logging Metrics App Health Process Mgt Config © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 73. ...they can “plug in” to existing infrastructure Log Host OS Logging Logging Metrics Health Process Mgt Stats Host Metrics Logging Alert Manager Metrics App Health Health Process Mgt Cluster/Orchestration Manager CM Config Config Configuration Manager Config Config VCS Config Engineering © 2013 Cisco and/or its affiliates. All rights reserved. Operations Reinhardt Quelle ([email protected])
  • 74. Operation’s Asks • Make Operability an integral part of product requirements and designs • Engage your operations team as soon as possible to let us help you leverage our platform and services • Talk with us about HA architectures: we don’t have a magic wand! © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
  • 75. Thank you! www.cisco.com/careers © 2010 Cisco and/or its affiliates. All rights reserved. 75