How eStruxture Data Centers is Using ECE to Rapidly Scale Their Business
1 like•2,391 views
The document outlines Estrutura Data Centers' growth, highlighting key acquisitions and expansions to enhance their service capabilities, including the use of Elastic Cloud Enterprise (ECE) for monitoring and managing data center infrastructure. It discusses the importance of a data-driven approach to improve operational efficiency and customer service while addressing common industry challenges. The presentation also emphasizes the versatility and scalability of ECE to enable rapid deployment and management of resources across multiple locations.
![SNMP WALKS VIA CONF FILE
Do what work!
input{
snmp {
interval => 10
get => [
"1.3.6.1.2.1.1.5.0"
]
walk => [
"1.3.6.1.4.1.476.1.42.3.9.20.1.20.1.2.1" ,
"1.3.6.1.4.1.476.1.42.3.9.20.1.10.1.2.1"
]
hosts => [
{host => "udp:ip_address/port" community => “community_name" version => “snmp_version" } ,
{host => "udp:ip_address/port" community => "community_name " version => "snmp_version " } ,
{host => "udp:ip_address/port" community => "community_name " version => "snmp_version " }
]
}
}
filter {
de_dot {}
}
output {
kafka {
codec => json
topic_id => “topic_name"
bootstrap_servers => “kafka_ip:customized_kafka_port"
client_id => “client_name"
}
}](https://image.slidesharecdn.com/elasticontorontocustomertalkestruxture-190923183542/85/How-eStruxture-Data-Centers-is-Using-ECE-to-Rapidly-Scale-Their-Business-18-320.jpg)
How eStruxture Data Centers is Using ECE to Rapidly Scale Their Business
- 1. Eric Charpentier | Enterprise Architect eStruxture Data Centers September 18th 2019 How eStruxture Data Centers is using ECE to Rapidly Scale Our Business
- 2. WHO IS ESTRUXTURE? AND WHY DOES IT MATTER? !2
- 3. !3 eStruxture | TIMELINE OF FUNDING AND ACQUISITIONS Acquisition of Montreal- based Netelligent Hosting Services and launch of eStruxture with initial funding led by Canderel February 2017 Second round of funding by CDPQ eStruxture announces expansion of MTL-1 facility doubling cabinet space and increasing power capacity to 5MW June 2017 Acquisition of Montreal Gazette printing facility for 2nd data center expansion in Montreal April 2017 November 2017 MTL-1 MTL-2 Third round of funding by Fengate November 2017
- 4. !4 Acquisition of Kolotek from Gaz Metro Expansion to Vancouver via the acquisition of Backbone Datavault facility April 2018 Expansion of VAN-1 and acquisition of second Vancouver area data center (Burnaby) February 2018 January 2019 MTL-3 VAN-2 VAN-1 New credit facility with Scotiabank, NBC and IQ June 2019 Expansion in Calgary through the acquisition of the Shaw data center August 2019 CAL-1
- 5. eStruxture | CURRENT DATA CENTERS !5 High-Density Power in The Heart of Downtown Montréal ▪ 5MW ▪ 25,000 square feet ▪ 20+ carriers present in the facility ▪ High power density available (30kW/cabinet available standard) Capacity and Flexibility ▪ 30MW ▪ 187,000 square feet ▪ Large, expandable facility ▪ High power density available (30kW/cabinet available standard) ▪ Dedicated office and storage space Tier III Uptime Certified Facility ▪ 1.2MW ▪ 20,000 square feet ▪ High power density available (15kW/cabinet available standard) ▪ Easy to access, away from downtown traffic Innovative and Strategically Located Facility ▪ 2.4MW ▪ 10,000 square feet ▪ High power density available (up to 30kW/cabinet) ▪ All major carriers on site ▪ Mount Pleasant location – a hotbed for media studios MTL-1 MTL-2 MTL-3 Scalable, Tier III Uptime Certified Facility ▪ 6MW ▪ 65,000 square feet ▪ High power density available (30kW/cabinet available standard) ▪ Multiple customer amenities on site VAN-1 VAN-2 CAL-1 State-of-the-art, Expandable Facility ▪ 10MW ▪ 54,000 square feet ▪ High power density available (up to 30kW/cabinet) ▪ Located in Burnaby – proximal to stable, plentiful power and dense fiber optic infrastructure
- 7. !7 BACKGROUND KEN NAME CUSTOMER DEMOGRAPHICS GOALS HOBBIES & INTERESTS COMMON OBJECTIONS BIGGEST FEARS ▪ Experienced manager with a dedicated 7X24 L1/L2 support team ▪ Ken Oliver ▪ Father, husband great colleague ▪ Colocation services ▪ Network ▪ Managed services ▪ Cloud services ▪ Too many systems ▪ Alarm pollution ▪ Too many manual interventions ▪ Support his team ▪ Provide the best possible service to his customers ▪ Meet the highest levels of industry compliance ▪ Overworking his employees ▪ Missing customer commitments ▪ Not working on the right things ▪ Play with his kids ▪ Star Wars ▪ Make sure that everyone has a fun day
- 8. !8 WELCOME TO THE REALM OF DATA-DRIVEN DATA CENTERS Imagine a world where you, as a support manager, can rest easy at night, knowing that your infrastructure is monitored, analyzed, and troubleshooted automatically so that your team can focus on serving our customers.
- 9. 9 MACHINES DO THE BORING, REPETITIVE TASKS AND HUMANS SOLVE PROBLEMS
- 10. !10 Eliminate the duplication of costly tools that serve the same purpose Reduce manual interventions during support/installation Eliminate the eyes on glass operations model Speed up time to integration upon acquisition/construction of a site Avoid the old mindset of command & control approach to NOC management Eliminate the exceptions to manage for each sites (80/20 rule) WHAT PROBLEMS ARE WE TRYING TO SOLVE
- 11. !11 TRADITIONAL METHODOLOGY ▪ Technology based approach ▪ Cooling ▪ Electrical ▪ Mechanical ▪ Network ▪ Hardware ▪ Application ▪ Vendor based tools ▪ Vendor locked toolsets ▪ DCIM (partial monitoring) ▪ Multi-screen approach ▪ Workload intensive Connect to tools and databases to aggregate your data which increases the risks of data loss, connectivity problems and latency.
- 12. !12 A NEW APPROACH Based on philosophy, scalability and innovation Monitor Potential Bad Things1 Alert before they happen Monitor Actual Bad Things2 Alert when they do happen, which is, unfortunately, inevitable Monitor Good Things3 Alert when they stop happening 4 Steps of MonitoringOur Core Tenets Identify as many problems as possible. Good monitoring doesn't just tell you when your site is completely down. Identify problems as early as possible. The sooner you know about an issue, the better chance you'll have to address it before it affects users. Longer lead times are your friend. Generate as few false alarms as possible. False alarms (aka false positives) can lead to "alert fatigue“ Tune and Continuously Improve4 Iterate!
- 13. !13 A NEW APPROACH Based on philosophy, scalability and innovation Why Elastic? 1.Multiple sources of data under one roof 2.Ingest data from app logs, device logs, infra logs 3.It indexes it with its own tool, correlates, infers data and can aggregate it 4.Tool to understand the data and answer questions we have 5.Can build visuals of the data 6.Xpacs for extra features NextGen monitoring vs today 1. Automated monitoring vs Manual monitoring 2. What are the goals for monitoring vs alarm storms 3.What resources will we monitor vs A sea of resources 4.How often will we monitor the resources vs Constant monitoring 5.Who should be notified when something goes wrong vs Notify everyone all the time
- 14. 14 I HAVE HEARD THIS PROMISE BEFORE… Image courtesy of StateTechMagazine.com
- 15. Why did we choose ECE One word: Versatility High availability Ability to build our architecture based on availability zones to eliminate the risk of downtime Hardware customization Ability to choose the hardware appropriate to the business condition Separation of roles for scalability Ability to re-engineer the architecture to scale. If we require more resources, we can get additional nodes or modify the architecture to separate allocators, directors and proxies (depending on need) Speed Ability to rapidly create new, customized deployments on demand without having to provision new nodes of VMs Elastic Cloud Enterprise
- 16. !16 ITERATE – PHASE 1 Watcher Dashboards Logstash pipeline ?
- 17. MODBUS PROTOCOLS Sometimes, hybrid functions work best input{ exec { command => "/opt/rh/rh-python36/root/usr/bin/python3.6 /etc/logstash/scripts/script_name.py ip_address 168 2 f7 --device=1" interval => 10 add_field => { host => “host_name"} add_field => { IP_Address => “ip_address"} add_field => { CLLI_Code => “device_name"} add_field => { Country => “Country"} add_field => { Location => “City"} add_field => { Floor => “Floor"} add_field => { Room => “Room"} add_field => { DeviceType => “Equipment_type"} add_field => { DeviceNumber => “Equipment_number"} type => "OutputFrequency" } output { kafka { codec => json topic_id => “topic_name" bootstrap_servers => “kafka_ip:customized_kafka_port" client_id => “client_name" } }
- 18. SNMP WALKS VIA CONF FILE Do what work! input{ snmp { interval => 10 get => [ "1.3.6.1.2.1.1.5.0" ] walk => [ "1.3.6.1.4.1.476.1.42.3.9.20.1.20.1.2.1" , "1.3.6.1.4.1.476.1.42.3.9.20.1.10.1.2.1" ] hosts => [ {host => "udp:ip_address/port" community => “community_name" version => “snmp_version" } , {host => "udp:ip_address/port" community => "community_name " version => "snmp_version " } , {host => "udp:ip_address/port" community => "community_name " version => "snmp_version " } ] } } filter { de_dot {} } output { kafka { codec => json topic_id => “topic_name" bootstrap_servers => “kafka_ip:customized_kafka_port" client_id => “client_name" } }
- 19. !19 ITERATE, ITERATE – PHASE 2 Watcher Dashboards Logstash pipeline Multiple Logstash Modbus SNMP SNMP poller IT Life cycle management Users & Roles
- 20. Confidential and Proprietary !20 THE CASE FOR ENRICHING DATA 9/16/2019 Consumption Self-serve CONSUMPTION ▪ Enable consumption- based billing to all sites ▪ Enable consumption- based billing to all services ▪ Eliminates days of manual labor in favor of processing in seconds CORRELATION ▪ All systems and/or equipment logs into one data repository ▪ Ability to relate simultaneous alarms to one root cause ▪ Ability to relate all impacted customers to events in seconds SELF-SERVE ▪ Customer portals ▪ Maintenance notifications ▪ Ad hoc requests ▪ Ability to drill-down the data chain in seconds
- 21. MAXIMIZE THE USE OF YOUR INPUT PLUGINS JDBC Input Plugin input { jdbc { jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar" jdbc_driver_class => "com.mysql.jdbc.Driver" jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb" jdbc_user => "mysql" parameters => { "favorite_artist" => "Beethoven" } schedule => "* * * * *" statement => "SELECT * from songs where artist = :favorite_artist" } }
- 22. Watcher Dashboards !22 ITERATE, ITERATE, ITERATE Logstash pipeline Multiple Logstash Modbus SNMP SNMP poller IT winlogbeat metricbeat Life cycle management APM Machine Learning Canvas Users & Roles Logstash pipeline Client portal
- 25. Over power alarms Per month before we started using Watcher RESULTS AT A GLANCE Additional panels 1344 breakers more monitored than before Per week Ability to set a range and interval criteria compared to the devices which are point-in-time 7000+ 16 600
- 26. THE NEED TO MODIFY HISTORICAL DATA How we learned to go with the data flow GET /test_topic/_search { "query": { "match": { “device_label": “CAMTL200EL08UPS1001" } } } } POST test_topic/_update_by_query { "query": { "match": { “device_label": "CAMTL200EL08UPS1001" } }, "script": { "source": "ctx._source.device_label='CAMTL200EL08UPS2001'" } }
- 27. !27 • Make your work visible from current status to the roadmap • Use a JSON friendly editor (really!) • Version control everything • Create a production environment and a development environment completely isolated from one another • Define your storage policy before you start • Bring all your logstash and kafka logs into separate indices to analyze on a regular basis while you develop • Don’t assume that your environment will be static (dynamic mapping?!?) but also basic components like device names LESSONS LEARNED
- 28. !28 And while we’re not quite at the stage where Ken can rest easy at night, we are now allowing him to take the occasional nap… but not too many…
- 29. Thank you! Presented by Eric Charpentier Enterprise Architect eStruxture Data Centers [email protected] https://www.linkedin.com/in/eric-charpentier/