Sybase Global Infrastructure
Download as PPTX, PDF0 likes170 views
This document provides an overview of Sybase's global IT infrastructure, including: - 5 data centers located across the US, UK and Hong Kong supporting over 3100 physical servers and 3500 VMs - 1.5 petabytes of storage from various vendors and 120TB of backups per week - Metrics on help desk tickets, messaging/mobility users, Access Anywhere/VDI instances - Organization of infrastructure teams into Networks, Client Infrastructure, Data Centers, Applications groups - Use of operations teams in India for lower-cost support
Sybase Global Infrastructure
- 1. Global IT Infrastructure Services Sybase Infrastructure RobertMobley May2012
- 2. 2 Infrastructure Overview Data Centers: 5 (Dublin CA, Englewood CO, Concord MA, Maidenhead UK, Hong Kong) Physical Servers: 3100 (Dell, IBM, Sun and HP) – Over 3500 VMs (server class and desktop VMs) – 15 Virtual Centers and 330+ ESX servers (primarily blades) – VMware, IBM LPAR, SUN LDOM and Containers Storage: 1.5 Petabytes (EMC, NetApp, Dell/Compellent, Nexsan, HDS) Backups: 120 TB per week (Symantec NetBackup, Data Domain) 32% 26% 24% 5% 3% 9% 1% Solaris Linux Windows AIX HPUX ESX Others 46% 30% 24% Physical Servers Server VMs Desktop VMs
- 3. 3 Infrastructure Overview Client workstation and virtual desktops: 6000 desktops: primary OS XP, 80% Dell laptops, TCO $1200 / System / year Access Anywhere (shared virtual desktop) enabled users: 2452, TCO $800 / instance / year Virtual Desktop Machines deployed (VDI) 1952 , TCO $1200 / System / year 22 FTE to support physical and virtual desktops 9 FTE on Global Resolver team for second level support for Client Infrastructure Help Desk: Avg. 6000 support tickets handled per month placed by call, web or email request 41% ticket first call resolution rate Average time per call: 5 mins 8 FTE on the Help Desk for 24/7 support Messaging, Mobility and Cellular: 6000+ Lotus Notes email boxes, 3000 distribution lists, 19.6M messages / month, 30 TB of email data 2200 Sybase mobile devices – combination of iOS, Android, Windows Mobile and Blackberry. BYOD reduced hardware costs by $11k USD / month Cellular: 900K minutes, 250K SMS messages, 1.9 TB of data services, $300K USD/ month globally Afaria as a service providing mobile management and SUP apps to 30k+ devices 4.5 FTE to support messaging, mobility and cellular
- 4. 4 Infrastructure Overview • Data Network: Verizon MPLS (2.1 Gbps combined capacity) Internet: 6 Points of entry/exit (300 Mbps combined capacity) VPN: 5 gateways (Average of 477 sessions at any time) • Video Conferencing: 56 Polycom room-based units (18K Minutes / 530 connections per month typical) 350 Cisco desktop units (88K Minutes per month typical) 2 Cisco Telepresence units (408 Minutes per month typical) • Telephony: Global implementation Cisco IP telephony (2.1M minutes per month typical, 5000 phones) Call routing between all offices using Sybase WAN EMEA and North America based Call Center (34 agents)
- 5. 5 Infrastructure Overview • Databases: ASE, DB2, Microsoft SQL Server, and ASA (200 databases, 4 TB data) • Application and Web Servers: EAServer, Sun Web servers, Load Balancers (Approx. 250 services) • SAP System Landscapes: 150 Servers, 50 Instances Business Solutions: 2 CRMs (5 tier), 1 BW (5 tier), 2 PI (5 tier) Support Platform: 2 SolMan Education: IDES Technical Components: TREX, Portal, Web Dispatcher, NetWeaver Gateway, NetWeaver Mobile, NWDI
- 6. 6 Organization Structure Our organization is comprised of four primary functional groups: Networks Client Infrastructure Data Centers Application Infrastructure •Each functional group has an Architecture and Engineering function (Build) and an Implementation and Operation function (Run). The focus is on migrating the Run functions to low cost geographies, freeing up individuals to transition into the Build function. •Members of the Engineering teams tend to be the most experienced and highly skilled members of the team. They are responsible for strategic planning, roadmap development, architecture, design of new systems, and project management for larger projects. •Operations is responsible for Break/Fix, Move/Add/Change/Delete, most administrative functions, implementation of new systems as members of a team, and some limited project management of basic projects.
- 7. 7 Low Cost Support Model • All of the Infrastructure teams have an Operations component in Pune, India. For Applications Infrastructure (GAIT), Data Center, and Client Infrastructure, these teams handle 7x24 support on a global basis. Network Operations is staffed by 3 contractors in Dublin, CA who handle 5x24 support. Network Implementation staff cover the other 2 days. • All Operations staff are responsible for working cases regardless of location. We capture metrics on the number of out of geography ticket are resolved by individual to determine our success. • All managers up to and including me are on call 7x24x365 to handle escalations.
- 8. 8 Infrastructure Services Network Client Infrastructure • WAN • LAN • Internet • VPN (IPsec and SSL) • One-time password authentication • Remote access services (iPass) • Wireless • IP telephony systems (Cisco) • Voicemail • Audio conferencing (internal and external) • Web conferencing (internal and external) • Video conferencing (internal and external) • Perimeter security (firewalls, IDS, etc) • Security incident response • Risk assessment • Windows AD architecture • Desktop applications • Lotus Notes • Lotus SameTime (Chat) • AccessAnywhere Virtual Desktop • ELT support • MS Office • Windows O/S • Anti-virus • Patching (Servers and workstations) • Microsoft licensing and compliance • Help desk • Password synchronization • Mobile messaging and Cellular devices /plans • Desktop backup and encryption Data Center Applications Infrastructure (GAIT) • Physical and virtual servers • Server OS (Unix, Linux, Windows) • SAN / storage infrastructure • High Availability design, support • Backup/Restore • Enterprise Monitoring • Power and cooling management • Data Center/Lab support • Disaster Recovery services • Virtual Infrastructure for VDI/AccessAnywhere • Engineering Development support • Cloud Services (PAAS, IAAS) • R&D (Engineering and IT) • Our First Adopter Program • ASE, DB2, MS SQL Server builds and administration • SAP Solution System Landscape design, implementation, operation and maintenance • SAP ASE OS/DB Migration • SAP NetWeaver Mobile Infrastructure/Integration • SAP Solution Manager, NWDI • Web and application server builds and administration • SSL certificate management • Load balancers • FTP services • Web Single Sign-on / Federated sign-on • LDAP (where it is independent of AD)
- 9. 9 Organization Philosophy We must keep our services running to meet our SLAs This requires an operation effort as the base of our organization. Everything else builds on this. Implementation is project based, and thus should be done in a different level from operations Implementation of already engineered solutions requires an intermediate degree of autonomy and experience Engineering new solutions requires a higher degree of experience and knowledge. Architecture requires an ability to think beyond what is currently available and as such requires the highest degree of experience and knowledge.
- 10. 10 Organization Philosophy • Outsource or out-task the work that is commodity • Actively manage our vendors, especially outsourcers. • Work together • Have a global approach to everything we do.
- 11. 11 Engagement with Global Infrastructure There are three ways for work to enter our organization • Problem Tickets = Deliver to SLA • Projects with resource plans = Deliver to project timeline • Management assignment = Deliver to expectation Work can be escalated through management to change the delivery Do not accept incident work without tickets. If one does not exist, you must create it. Do not accept project work without a project plan. Insist on a scope and project plan that includes how many hours of what skill set are needed at what time. If it doesn’t exist, negotiate with the PM. Be careful in taking or giving management assignment. Remember that something has to give.
- 12. 12 Service Line Management Each Part of the organization offers lines of service that are individually identifiable These are broken out on ESA (time tracking) so that the cost of supporting the service is visible We have been working toward tracking the costs separately so that we can demonstrate the cost of service Service owners are asked to think as if they were in business for themselves: Is the price point competitive in the market? Should I continue to offer this service? What changes can I make to improve the position in the market? Is the service level competitive in the market? Am I measuring the right things to manage my service?
- 13. 13 Service Line Management - Client Infra
- 14. 14 Service Line Management - Data Center
- 15. 15 Service Line Management - Apps Infra
- 16. 16 Service Line Management - Network
- 17. 17 Metrics Data Center # of physical servers # of virtual servers % of capacity – Power, AC, Space for each data center SLA performance # tickets Amount of backup / storage % capacity of backup, storage, virtual infra Netcom Availability Congestion Calls VPN connections SLA performance # tickets Client Infrastructure # of physical desktops # of AA users # of mail accounts Amount of Spam filtered # of Viruses detected / cleaned SLA performance # tickets Application Infrastructure SLA performance Availability Exceptions Capacity Business Usage # tickets Proactive vs. reactive maintenance
- 18. 18 Global Desktop Service Ticket Metrics and SLA Performance Provider(s): (Multiple Items) Tickets Column Labels Priority Achieved Breached Grand Total 1 - Critical 265 22 287 RESP 113 12 125 RSLV 152 10 162 2 - High 1120 279 1399 RESP 505 147 652 RSLV 615 132 747 3 - Moderate 18912 4480 23392 RESP 8849 2804 11653 RSLV 10063 1676 11739 4 - Low 508 122 630 RESP 192 100 292 RSLV 316 22 338 Grand Total 20805 4903 25708 Green Queue Flow Tickets In Tickets Out Change Queue Notes / Trend Current MTD 223 354 (131) 626 Apr-12 1143 965 178 End of month Mar-12 788 829 (41) End of month Feb-12 867 929 (62) End of month Jan-12 960 954 6 End of month Dec-11 809 712 97 End of month Nov-11 834 712 122 End of month Oct-11 853 821 32 End of month Sep-11 934 917 17 End of month Aug-11 937 956 (19) End of month Jul-11 965 899 66 End of month Jun-11 941 1033 (92) End of month May-2011 1039 1051 (12) End of month Total May'11 thru April '12 11070 10778 Totals do not include May '12 MTD Global Desktop SLA Compliance by Priority for (05-07-2011 - 05-08-2012) Messaging Queue Ticket Flow Overall Queue Status and Health : Notes/Coments on Metrics (this week) Global Desktop SLA Summary Page RESP RSLV RESP RSLV RESP RSLV RESP RSLV 1 - Critical 2 - High 3 - Moderate 4 - Low Breached 12 10 147 132 2804 1676 100 22 Achieved 113 152 505 615 8849 10063 192 316 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% %AchievedtoTotal Global Desktop SLA Compliance by Priority
- 19. 19 Messaging and Mobility Ticket Metrics and SLA Performance Provider(s): (Multiple Items) Tickets Column Labels Priority Achieved Breached Grand Total 1 - Critical 252 16 268 RESP 118 7 125 RSLV 134 9 143 2 - High 652 127 779 RESP 290 52 342 RSLV 362 75 437 3 - Moderate 8151 1197 9348 RESP 3908 447 4355 RSLV 4243 750 4993 4 - Low 39 11 50 RESP 14 5 19 RSLV 25 6 31 Grand Total 9094 1351 10445 Green Queue Flow Tickets In Tickets Out Change Queue Notes / Trend Current MTD 93 75 18 141 Apr-12 319 334 (15) End of month Mar-12 256 312 (56) End of month Feb-12 340 348 (8) End of month Jan-12 404 409 (5) End of month Dec-11 313 275 38 End of month Nov-11 369 383 (14) End of month Oct-11 357 322 35 End of month Sep-11 254 310 (56) End of month Aug-11 344 341 3 End of month Jul-11 341 337 4 End of month Jun-11 292 286 6 End of month May-2011 291 428 (137) End of month Totals 3880 4085 Totals exclude current month (May '12) Messaging SLA Compliance by Priority for (05-08-2011 - 05-08-2012) Messaging Queue Ticket Flow Overall Queue Status and Health : Notes/Coments on Metrics (this week) Messaging SLA Summary Page RESP RSLV RESP RSLV RESP RSLV RESP RSLV 1 - Critical 2 - High 3 - Moderate 4 - Low Breached 7 9 52 75 447 750 5 6 Achieved 118 134 290 362 3908 4243 14 25 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% %AchievedtoTotal Messaging SLA Compliance by Priority
- 20. 20 Access Anywhere / Desktop Virtualization Ticket Metrics and SLA Performance Provider(s): (Multiple Items) Tickets Column Labels Priority Achieved Breached Grand Total 1 - Critical 33 7 40 RESP 15 4 19 RSLV 18 3 21 2 - High 333 75 408 RESP 155 36 191 RSLV 178 39 217 3 - Moderate 4651 630 5281 RESP 2335 302 2637 RSLV 2316 328 2644 4 - Low 14 4 18 RESP 5 2 7 RSLV 9 2 11 Grand Total 5031 716 5747 Green Queue Flow Tickets In Tickets Out Change Queue Notes / Trend Apr-12 333 267 66 Mar-12 188 184 4 End of month Feb-12 155 161 (6) End of month Jan-12 218 203 15 End of month Dec-11 106 91 15 End of month Nov-11 146 164 (18) End of month Oct-11 161 173 (12) End of month Sep-11 157 144 13 End of month Aug-11 180 192 (12) End of month Jul-11 144 134 10 End of month Jun-11 135 124 11 End of month May-2011 171 168 3 End of month Totals 2094 2005 Access Anywhere SLA Compliance by Priority for (05-08-2011 - 05-08-2012) Messaging Queue Ticket Flow Overall Queue Status and Health : Notes/Coments on Metrics (this week) Access Anywhere SLA Summary Page RESP RSLV RESP RSLV RESP RSLV RESP RSLV 1 - Critical 2 - High 3 - Moderate 4 - Low Breached 4 3 36 39 302 328 2 2 Achieved 15 18 155 178 2335 2316 5 9 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% %AchievedtoTotal Access Anywhere SLA Compliance by Priority
- 21. 21 SLA Metrics – Global Netcom
- 22. 22 SLA Metrics – Application Infrastructure
- 23. 23 SLA Metrics – Global Server
- 24. 24 Infrastructure Costs - excluding Telecommunications - 5,000,000 10,000,000 15,000,000 20,000,000 25,000,000 30,000,000 35,000,000 40,000,000 2007 2008 2009 2010 2011 Depreciation & Amort Premises Marketing Outside Services Other Operating Expense Travel & Entertainment Resource Pool Transfer Employee Development Other Headcount Variable Recruiting & Relocation Occupancy Allocation Temporary Headcount Benefits Salaries & Bonuses Between 2007 and 2011, Infrastructure reduced our OPEX budget by 20%
- 25. 25 Cost Control - 2,000,000 4,000,000 6,000,000 8,000,000 10,000,000 12,000,000 14,000,000 16,000,000 2007 2008 2009 2010 2011 Headcount Direct Headcount Variable Nonheadcount Variable Fixed Expenses Capitalized Software In 2010 telecommunications costs were moved to IT
- 26. 26 Cost Control 0 500,000 1,000,000 1,500,000 2,000,000 2,500,000 - 5,000,000 10,000,000 15,000,000 20,000,000 25,000,000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Telecom Costs (USD) Bandwidth (KBPS)
- 27. 27 Metrics – How I think about them A single number by itself indicates very little. A trend tells a much better story Metrics allow us to ask questions aimed at getting what we want # of tickets is only interesting to me in context As a part of a trend Compared across a like group Compared against ESA coding Compared against project load SLA compliance is interesting in absolute and in context As a part of a trend Across specific customer groups Compared by Incident priority
- 28. 28 Prioritization • Service – Provide outstanding service to our internal customers. Meet or exceed our SLAs. Take on projects that reduce the probability of outages or the impact that a failure can have on the severity of an outage. • Cost – reduce the cost of the services that we provide. Decrease or keep steady the run rate costs. Take on projects that provide positive NPV. • First Adopter – Provide effective internal support to Engineering and Marketing in the design and delivery of our products. Be the first and best customer of identified products. • Enable the business – Provide upgrades and features that allow our business organizations to be more effective in doing their jobs. Take on projects that tie directly to corporate objectives.
- 29. 29 Focal Goals • SCFE • Actively participate in the process of global standardization. As appropriate, participate in development and implementation of standards, policies, procedures, and practices. Comply with standards, policies, procedures, and practices. • Complete assigned projects on time, within budget, meeting stakeholder expectations. • Resolve assigned problem tickets within the established SLA. • Assist other parts of the Global Infrastructure team as requested • Manage your workload through providing feedback to management when you are nearing your capacity, or when you have time available to take on other work.
- 30. 30 Infrastructure Manager roles Operations •SLA compliance •Queue management •Escalation management •Vendor operations relationships management . •Staff management •Staff and peer communication management •System monitoring •Staff training plans: existing technologies . •Cost control •Customer relations •Maintenance contracts •Standards compliance •Transition from engineering Engineering •Architecture •Design •Vision and roadmap •Vendor relationships management: “forward looking” Staff management •Staff and peer communication management •Develop management tools •Staff training plans: new and future technologies •Cost forecasting •Customer requirement forecasting •Standards development •Transition to operations
- 31. 31 Everyone’s responsibilities • Be a problem solver, not just a problem identifier. • Support IT and Infrastructure initiatives • Use permissions and authority appropriately • Work globally • Provide good hand-offs – you would expect the same. • Ask for help if you need it, give help if you are asked • Provide honest feedback