IT Performance Problems
Download as PPTX, PDF0 likes92 views
Bill Alderson is an expert in responding to high-profile IT problems at government and commercial organizations. He has helped restore communications at the Pentagon after 9/11 and optimized networks and applications for the Joint Chiefs of Staff and other military organizations. In his presentation, Alderson will demonstrate the tools and techniques used to perform deep packet analysis and pinpoint the root causes of complex IT issues in order to optimize systems and avoid unnecessary upgrades.
IT Performance Problems
- 2. Session Date & Time • Date: Wednesday, June 29, 2016 Time: 1100-1145 Location: Tuscany Ballroom
- 3. • Bill Alderson responds to Information Technology high visibility, high stakes technical problems. Network outage, slowness, slow applications or disasters affecting government and commercial Information Technology Enterprise environments. ABC News told the story of how Bill and his team helped restore communications at the Pentagon immediately following 911. Bill assisted with six deployments to Iraq and Afghanistan requested by Army G2, Joint Chiefs and US Central Command diagnosing Biometrics and others critical systems. One of his missions is to help executives and technologists see both technical and leadership root causes that can be obviated through common sense best practices.
- 4. Bill Alderson Infographic Bio
- 5. • Deep packet analysis remains essential for definitive irrefutable diagnosis and optimization of complex systems. Bill demonstrates the tools, techniques and methods used to annotate complex technology findings so that technologists, managers, executives and vendors can agree on root cause. Once the problem is identified and agreed upon the true pinpoint mitigation can begin. The days of shotgun style "forklift wholesale upgrades" on everything have passed. We must optimize existing assets allowing them to perform well. Bill has proven ability to optimize large scale networks and applications from experience in analyzing the Pentagon immediately following 911, analysis of Biometrics applications across Iraq and Afghanistan, numerous optimizations of Joint Chiefs of Staff and OSD network analysis. Experience from analysis of the largest 100 commercial enterprise networks such as Stock Exchanges, Financial, Insurance and Healthcare institutions will be demonstrated with annotated examples for CIO, Executives and top level technologists.
- 6. IT Critical Problem Resolution Technology and Psychology [email protected]
- 7. “Swiss Army Knife” Portfolio of Tools Select Well. Avoid Spending Only on “Suites” All-in-one-tools Although easier to “buy” don’t solve many problems. They leave you “broke and broken” with a gold plated toolset.
- 13. Down - Intermittent - Slow
- 14. Technical vs. Leadership Root Cause
- 15. The Needle
- 16. The Environment
- 17. Packet Traces
- 18. Store Every Packet? Who’s can and is going to analyze them and when?
- 19. Finding The Stack With The Problem
- 21. Measured at the Server Fast TCP connect time. Fast Ack from F5 does not show true client response time which is why Apalytics provided Internet Monitoring. 1.4 second Get response is very slow which is why detailed platform and application analysis was performed. The 2nd & 3rd Gets were fast at 1 millisecond proving some commands are
- 22. CF Longest Requests 1,958,266ms = ~32 minutes from one request 391,692ms = ~7 minutes
- 23. Page Analysis from the Internet DNS does not play a role in slowness. Connection time varies and at time approaches 200 milliseconds which can be at the platform, internet, network, load balancer or firewalls. Connection delay analysis will require multiple capture points to definitively pinpoint and should be considered when multi-point capture test points can be configured at the Security Tap devices. But that is not material for improvement of this application at this timeFirst byte time is the most concerning issue in the infrastructure. Last byte time is also a concern as it appears that platform TCP/IP stack services are slow to move data out onto the wire after the first byte has started. It may also be that platform improvements may improve both response times and output speed. Page load time is a composite of all elements of the page that must come together to provide the user with the visual page and the main context of the query. This too is concerning, but it is caused by the slowness of the individual components of the page as they add serially to the response time which are represented in the main concerns. An example of the total page would be small visual images and data making up the user interface view (i.e., logos) that are not part of a computational or lookup, but rather a static image that should be served rapidly by the server.
- 24. Network Intrinsic Application Analysis
- 27. Application Monitoring Design Phase
- 28. Multi-tier Macro vs. Micro Event Process Net-Ser-Tr-Sw-Q Security Auth User Click Client Network WebSvr Network AppSrv Network SQLSvr Network AppSvr Network Mainframe Network AppSvr Network WebSvr Network Client User Display UpdateMacro Response Time Micro Response Time
- 29. HTTP Post from client Web1 Middlewa re 155ms HTTP / SQL Multi-tier 1
- 30. Back to client With HTTP SQL Calls complete Query and returns Rows to Web1 SQL Calls finish .497 SQL Call start -.231 SQL Resp Time =.266 Web1 Middleware 12ms HTTP / SQL Multi-tier 2
- 31. Logon A is 72 milliseconds… Logon B is 420 milliseconds! Oracle Logon Slow
- 32. Micro-Analysis Phase Web App I/F #1&2 SQL TransLogger MF#1 MF#2 Time Breakdown
- 33. TCP Satellite Retrans 3.5 Seconds
- 36. Citrix Session Abort Signature “Chernobyl Packet” The packet that evidenced a problem on a Citrix server. This pattern was used as a signature on the Infinistream Sniffers to find these problems until they were remediated. Prior to this users were stuck in this cycle for hours.
- 37. Citrix User Filer Access Error Details
- 38. Blind vs. Pinpoint Upgrades Blind Upgrade = Shotgun Approach = Forklift Upgrade
- 39. Root Cause Optimization Definitive Root Cause Analysis Pinpoint Cause Measure ROI Potential Pinpoint Purchases Validate & Prove ROI Award Innovation Optimization Root Cause Analysis
- 40. IT Critical Problem Resolution Technology and Psychology [email protected]