![Logļ¬les - 4
[#|2013-04-16T16:04:44.319+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean|
_ThreadID=14;_ThreadName=pool-1-thread-9;|Starting to initialize the Top Summary Stats Data Store timer|#]
[#|2013-04-16T16:04:44.335+0200|INFO|sun-appserver2.1|com.appdynamics.TOP.SUMMARY.STATS.WRITE|
_ThreadID=14;_ThreadName=pool-1-thread-9;|START TIME for timer service(TopSummaryStatsWriterTimerTaskBean) will be: Tue
Apr 16 16:05:00 CEST 2013|#]
[#|2013-04-16T16:04:44.338+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean|
_ThreadID=14;_ThreadName=pool-1-thread-9;|Successfully initialized the Top Summary Stats Data Store timer|#]
[#|2013-04-16T16:04:44.338+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean|
_ThreadID=14;_ThreadName=pool-1-thread-9;|Starting to initialize the Top Summary Stats Data Purger timer|#]
[#|2013-04-16T16:04:44.369+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean|
_ThreadID=14;_ThreadName=pool-1-thread-9;|Successfully initialized the Top Summary Stats Data Purger timer|#]
[#|2013-04-16T16:04:44.369+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean|
_ThreadID=14;_ThreadName=pool-1-thread-9;|Starting to initialize the Top Summary Stats Detail String cache timer|#]
[#|2013-04-16T16:04:44.376+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean|
_ThreadID=14;_ThreadName=pool-1-thread-9;|Successfully initialized the Top Summary Stats Detail String cache timer|#]
[#|2013-04-16T16:04:44.376+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean|
_ThreadID=14;_ThreadName=pool-1-thread-9;|Starting to initialize the Top Summary Stats rollup timer|#]](https://image.slidesharecdn.com/nochmehrschweineundschlangen-neuetippszumperformancetroubleshooting1x1-140130134406-phpapp02/85/Application-Performance-Troubleshooting-1x1-Part-2-Noch-mehr-Schweine-und-Schlangen-23-320.jpg)
Application Performance Troubleshooting 1x1 - Part 2 - Noch mehr Schweine und Schlangen
- 1. Noch mehr Schweine und Schlangen Neue Tipps zum Performance Troubleshooting Rainer Schuppe AppDynamics GmbH
- 2. Rainer Customer Support System Support / Ops Consultant / Dev Solution Architect Sales Engineer +Rainer Schuppe
- 3. Reprise: Why care about performance Where to start? What to do? Who to blame? Tooling Usecases - Symptoms & Diagnostics
- 6. Complexity increases Release 1.1 Release 1.2 Release 1.23 Tomcat Release 1.5 .NET Amazon EC2 Windows Azure CLOUD Release 2.4 Release 2.5 Release 2.6 Release 3.0 Login Search Flight View Flight Status Make Reservation Tomcat Mule, Tibco, AG Tomcat ESB VMWare WEB 2.0 Memcached Weblogic Release 1.4 Release 1.5 Release 1.6 Release 2.0 Browser Logic AJAX Web Frameworks Oracle Coherence Hadoop Cassandra MongoDB SOA .NET MQ AGILE Release 3.4 Release 3.5 Release 3.6 Release 4.0 SQL Server Release 4.4 Release 4.5 Release 4.6 Release 5.0 JBoss Release 1.4 Release 1.5 Release 1.6 Release 2.0 ATG, Vignette, Sharepoint BIG DATA
- 7. Generic Troubleshooting Process Alert / Detection Rootcause Detection Triage Diagnosis Data / Information Solution Finding Move on with life Fix
- 8. Triage ā¢ Determine who needs to ļ¬x it ā¢ Starts with overview and comparison to ānormalā performance ā¢ First level task (Operators) ā¢ First indication of problem type ā¢ Works best with transactional data
- 9. 50 ms .NET 10 ms Amazon EC2 60 ms Windows Azure Release 1.1 Release 1.2 Release 1.23 Tomcat Release 1.5 45,3 ms CLOUD 50 ms Release 2.4 Release 2.5 Release 2.6 Release 3.0 Login Search Flight View Flight Status Make Reservation Tomcat 145 Mule, Tibco, AG ms 145 ms ESB 145 ms 145 ms 10 ms WEBms 100 2.0 Memcached 250 ms Weblogic Release 1.4 Release 1.5 Release 1.6 Release 2.0 Browser Logic AJAX Web Frameworks 300.NET ms 300 ms 310 ms AGILE Release 3.4 Release 3.5 Release 3.6 Release 4.0 SQL Server 150 ms Tomcat 160 VMWare ms 145 ms Oracle Release 4.4 Release 4.5 Release 4.6 Release 5.0 Coherence SOA 1 MQ ms 15 ms 250 ms JBoss Release 1.4 Release 1.5 Release 1.6 Release 2.0 ATG, Vignette, Sharepoint Hadoop Cassandra MongoDB BIG DATA
- 10. Release 1.1 Release 1.2 Release 1.23 Tomcat Release 1.5 Pr .NET ob lem Amazon EC2 Windows Azure CLOUD Release 2.4 Release 2.5 Release 2.6 Release 3.0 Login Search Flight View Flight Status Make Reservation Tomcat Mule, Tibco, AG Tomcat ESB VMWare WEB 2.0 Memcached Weblogic Release 1.4 Release 1.5 Release 1.6 Release 2.0 Browser Logic AJAX Web Frameworks Oracle Coherence Hadoop Cassandra MongoDB SOA .NET MQ AGILE Release 3.4 Release 3.5 Release 3.6 Release 4.0 SQL Server Release 4.4 Release 4.5 Release 4.6 Release 5.0 JBoss Release 1.4 Release 1.5 Release 1.6 Release 2.0 ATG, Vignette, Sharepoint BIG DATA
- 11. Key: = bad = not bad
- 12. Diagnose ā¢ Determine the root of the problem ā¢ Uses ļ¬rst level information to narrow scope ā¢ Needs specialists ā¢ Lots of data / information needed in real time and historical ā¢ Usually needs iterations ā¢ More than 1 tool used in the process
- 13. Rootcause detection ā¢ Conļ¬rm the rootcause after you diagnosed it ā¢ Document it ā¢ Recreate it in test if possible ā¢ Needs the same data as diagnostics
- 14. Solution ļ¬nding ā¢ Find a solution for the problem ā¢ Architect a workaround or a ļ¬x ā¢ Again needs the diagnostic data ā¢ Run some test runs with different options check them in realtime ā¢ Conļ¬rm the idea for the ļ¬x ā¢ May be a different team then the trouble shooters
- 15. How to get the data? ā¢ Intuition ā¢ Experience ā¢ Tools ā¢ Logļ¬les ā¢ Communication
- 16. Tooling Ā© val-j - sxc.hu
- 17. 3 Key Things Impact Performance & Availability Concurrency Data Volume Resource
- 18. Why do things crash and slow down? Development Concurrency Data Volume Resource QA/Test Concurrency Data Volume Resource Production Concurrency Data Volume Resource
- 19. Technologies Logging ARM Bytecode Instrumentation / Aspects Sampling JMX (Java Management Extensions) PMI (IBM WebSphere speciļ¬c) Dev Test Prod
- 20. Logļ¬les Pros: Dev Test Prod ā¢ Anything can be logged ā¢ Easy to implement (if you have the sourcecode) Cons: ā¢ Only what the developer thinks is needed ā¢ I/O heavy ā¢ No chance for change if you donāt own the source code ā¢ Lots of ļ¬les - no TX context usually ā¢ How to correlate in distributed environment?
- 21. Logļ¬les - 2 Logging can be the source of problems itself e.g. Log4Net ā¢ Synchronous local ļ¬le system access ā¢ The more you log the longer it takes ā¢ Can only be diagnosed with another tool Dev Test Prod
- 22. Bringing down production with Logging Microsoft.Win32.Win32Native:SetFilePointerWin32
- 23. Logļ¬les - 4 [#|2013-04-16T16:04:44.319+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean| _ThreadID=14;_ThreadName=pool-1-thread-9;|Starting to initialize the Top Summary Stats Data Store timer|#] [#|2013-04-16T16:04:44.335+0200|INFO|sun-appserver2.1|com.appdynamics.TOP.SUMMARY.STATS.WRITE| _ThreadID=14;_ThreadName=pool-1-thread-9;|START TIME for timer service(TopSummaryStatsWriterTimerTaskBean) will be: Tue Apr 16 16:05:00 CEST 2013|#] [#|2013-04-16T16:04:44.338+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean| _ThreadID=14;_ThreadName=pool-1-thread-9;|Successfully initialized the Top Summary Stats Data Store timer|#] [#|2013-04-16T16:04:44.338+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean| _ThreadID=14;_ThreadName=pool-1-thread-9;|Starting to initialize the Top Summary Stats Data Purger timer|#] [#|2013-04-16T16:04:44.369+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean| _ThreadID=14;_ThreadName=pool-1-thread-9;|Successfully initialized the Top Summary Stats Data Purger timer|#] [#|2013-04-16T16:04:44.369+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean| _ThreadID=14;_ThreadName=pool-1-thread-9;|Starting to initialize the Top Summary Stats Detail String cache timer|#] [#|2013-04-16T16:04:44.376+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean| _ThreadID=14;_ThreadName=pool-1-thread-9;|Successfully initialized the Top Summary Stats Detail String cache timer|#] [#|2013-04-16T16:04:44.376+0200|INFO|sun-appserver2.1|com.singularity.ee.controller.beans.ControllerManagerBean| _ThreadID=14;_ThreadName=pool-1-thread-9;|Starting to initialize the Top Summary Stats rollup timer|#]
- 24. Proļ¬ler Pros: ā¢ No conļ¬g needed ā¢ Lots of data - lots of detail Cons: ā¢ Lots of data - not suitable for production ā¢ Needs experience ā¢ No transactional concept / context Dev Test
- 25. Proļ¬ler
- 26. JMX (and similar) Pros: ā¢ ā¢ ā¢ Built into most application servers JConsole is part of the JDK Easy to implement MBeans Cons: ā¢ ā¢ ā¢ ā¢ No transaction context Not available for 3rd party No historical data Usually one JVM only Dev Test Prod
- 28. APM tools (free) Pros: ā¢ They are free ā¢ Transaction context (most of them) ā¢ Quick setup (the commercial ones) Dev Test Prod Cons: ā¢ Usually functionally constrained (commercial) ā¢ Hard to conļ¬gure (open source) ā¢ Usually no history
- 29. Dev APM tools (commercial) Test Pros: ā¢ Transactions, Historical data ā¢ Distributed monitoring ā¢ Deep dive diagnostics ā¢ Production ļ¬t Cons: ā¢ Costly ā¢ Choose the right one Prod
- 30. Diagnosis There are just 2 sorts of issues
- 31. Ā© NLTeddy - sxc.hu
- 32. Ā© ross666 - sxc.hu
- 33. 50 shades of slow (appx.) ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ Constantly slow (Turtle) Slowly, but constantly slower Exponentially slower Suddenly slower Sporadically slow Spontaneous crash
- 34. The wonderful world of errors ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ Sudden outage Always erroneous Sporadically Errormessages Silent death / Bleed to death Increasing errorrates Wrong / meaningless error messages
- 35. Diagnosis ā Rough Flow Look at symptoms Eliminate deļ¬nite non-causes Prioritize the suspicions Conļ¬rm suspicion / Eliminate suspicion ā¢ Compare with ānormalā ā¢ Gather more information ā¢ Deļ¬ne root cause and conļ¬rm it ā¢ Redo from Start ā¢ ā¢ ā¢ ā¢
- 36. Possible Causes (in no particular order) ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ Bad Coding Too much load Backend not reachable / slow Conļ¬icting resources Memory Leak Resource Leak Network / Hardware Problem
- 38. Symptoms ā¢ ā¢ ā¢ User complaints - Slow performance Exceptions appeared in logļ¬les Alerts for ops triggered
- 39. Average Response Time (ms) 11.000 Connection Timeout 8.250 5.500 2.750 0 time 10:01 10:03 10:05 10:07 10:09 10:11 10:13 10:15 10:17 10:19 10:21 10:23 10:25 10:27 10:29
- 40. Connection Pool vs. Errors 15,00 org.hibernate.util.JDBCExceptionReporter : Cannot get a connection, pool error Timeout waiting for idle object 11,25 7,50 3,75 0 10:00 10:02 10:04 10:06 10:08 10:10 10:12 10:14 10:16 10:18 10:20 10:22 10:24 10:26 10:28 10:30
- 41. 1st Diagnosis ā¢ ā¢ ā¢ ā¢ ā¢ OK - We do have a problem Database connection pool depleted Waiting times stacking 10 minutes until errors appear in logs But WHY?
- 42. Open Questions ā¢ ā¢ ā¢ ā¢ ā¢ Which database? Which DB Pool? Transaction speciļ¬c? Problem on DB? What is the load?
- 43. How to ļ¬nd data ā¢ ā¢ ā¢ ā¢ Check log for DB connection info Ask architect which TX are using this pool Use JMX to check pool metrics Check load info (if available)
- 46. Who else is using the DB? Ask your architect!
- 47. Who else is using the DB? Ask your tool!
- 48. What did we ļ¬nd out? ā¢ ā¢ There were other TXās using the DB ā¢ This TX had a speciļ¬c DB connection pool It was just a single transaction with the DB problem OK - Now letās check the load
- 49. Load and DB connections Trx/ min Avg RT Pool Limit Pool Usage Trx Stalls Why the sudden load increase?
- 50. Root Cause ā¢ ā¢ Loadbalancer was not working correctly ā¢ Many different pools made this conļ¬g necessary DB connection pool size was not appropriate for this load
- 51. The missing link Release 1.1 Release 1.2 Release 1.23 Tomcat Release 1.5 .NET Amazon EC2 Windows Azure CLOUD Release 2.4 Release 2.5 Release 2.6 Release 3.0 Login Browser(s) Purchase Search Flight Search Flight Flight Flight View Status Status Login Make Reservation Native Mobile App MOBILE Tomcat Mule, Tibco, AG Tomcat ESB VMWare WEB 2.0 Memcached Weblogic Network Release 1.4 Release 1.5 Release 1.6 Release 2.0 Browser Logic AJAX Web Frameworks Oracle Coherence Hadoop Cassandra MongoDB SOA .NET MQ AGILE Release 3.4 Release 3.5 Release 3.6 Release 4.0 SQL Server Release 4.4 Release 4.5 Release 4.6 Release 5.0 JBoss Release 1.4 Release 1.5 Release 1.6 Release 2.0 ATG, Vignette, Sharepoint BIG DATA
- 54. A quick look at memory and resources
- 55. Linear Memory Leak Symptoms: ā¢ ā¢ ā¢ ā¢ OOM (Out of memory error) Slow over time with spikes Sawtooth with upward trend ā¢ Causes ā¢ ā¢ Objects added to linear structures without being removed (e.g., linked lists) Other API misuse (addListener() without corresponding removeListener(), etc.)
- 56. Linear Memory Leak Aggregate detection: ā¢ ā¢ ā¢ linear growth in heap utilization GC time growth Speciļ¬c detection: ā¢ ā¢ ā¢ ā¢ Figure out object types being leaked Verbose GC Find related APIs and search code for misuse
- 57. Linear Memory Leak Challenges ā¢ ā¢ ā¢ References - many small objects are referenced in one collection Death by 1000 cuts (Papierschnitte) Speciļ¬c detection: ā¢ ā¢ ā¢ ā¢ Figure out object types being leaked Verbose GC Find related APIs and search code for misuse
- 58. Speciļ¬c detection ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ Heap Dump Comparison Needs at least 2 dumps Stops the JVM Can take several minutes each Creates tons of data Finds the object, not the code responsible for the leak Proļ¬ler High overhead - not for production Lots of data APM Solution ā¢ ā¢ ā¢ Collection based algorithm ā ļ¬nds only collection leaks Instance counting Trade off between low overhead and usefulness of data
- 63. Exponential Memory Leak Causes: ā¢ Objects added to most data structures without being removed (e.g., vectors, hashtables) ā¢ Other API misuse (as Linear Leak) ā¢ Aggregate detection: ā¢ exponential growth in heap ā¢ Speciļ¬c detection: ā¢ Same as Linear Leak ā¢
- 64. Resource Leak Causes: ā¢ API misuse of Java objects with resourcestyle lifecycle (create->use->destroy) ā¢ Aggregate detection: ā¢ Slow over time ā¢ Growth in heap (if youāre lucky) ā¢ Speciļ¬c detection: ā¢ Audit code for API misuses ā¢ Object instance tracking ā¢
- 65. Resource conļ¬ict (block / wait)
- 66. Resource conļ¬ict / blocking ā¢ ā¢ ā¢ Causes: ā¢ Overcautious data integrity strategy ā¢ Synchronising is always good Aggregate detection: ā¢ Stalled threads ā¢ High thread usage - low CPU usage Speciļ¬c detection: ā¢ Thread dumps as needed ā¢ Stack traces / graphs ā¢ CPU block / wait timing measurement
- 67. Fragen ?
- 68. Bad Coding: Inļ¬nite Loop Causes: ā¢ Inļ¬nite loop in code ā¢ Aggregate detection: ā¢ Stalled threads ā¢ Permanently high usage of CPU / threads ā¢ Speciļ¬c detection: ā¢ Thread dumps as needed ā¢ Stack traces / graphs ā¢
- 69. Bad Coding: CPU-Bound Component Causes: ā¢ Idiot with a āLearn Java in 24 Hoursā book ā¢ Aggregate Detection: ā¢ Response time measurement ā¢ Aggregate CPU utilization ā¢ Speciļ¬c Detection: ā¢ Detailed CPU utilization ā¢ Typical Cure: ā¢ Cache of data or of performed calculations ā¢
- 70. Layer-itis Causes: ā¢ ā¢ ā¢ Poorly implemented data bridge layer, or simply too many of them DB -> XML -> XSLT -> More XML -> āCustom Data Management Layerā -> Consumer Aggregate Detection: ā¢ ā¢ Response time measurements Speciļ¬c Detection: ā¢ ā¢ ā¢ Call graphs - Call trace (stack trace not enough) Ask for a design or architecture document
- 71. O/R Mapper misuse Causes: ā¢ ā¢ ā¢ ā¢ Hibernate ļ¬xes everything Massive SQL statements (length and amount) Wrong data strategy Aggregate Detection: ā¢ ā¢ ā¢ Response time measurements DB time measurements Speciļ¬c Detection: ā¢ ā¢ Call stacks / snapshots
- 72. Caching issues
- 73. The Unending Retry Causes: ā¢ Continual attempts to call backend + unavailable backend ā¢ Aggregate Detection / Speciļ¬c Detection: ā¢ Response time measurement ā¢ Backend detection - measurement (time & # of calls) ā¢ Stalled TX count ā¢ Exceptions ā¢ Busy thread count ā¢
- 74. donāt forget about thrown exceptions
- 75. Threading: Deadlock / Livelock Causes: ā¢ Fundamental error in threading / lock acquisition strategy ā¢ Aggregate Detection: ā¢ Stalled threads / permanently high concurrent usage ā¢ Speciļ¬c Detection: ā¢ Deadlock detection in JVM ā¢ Thread dumps ā¢ Busy thread count ā¢
- 76. Threading: Deadlock Found one Java-level deadlock: ============================= "Thread-2": Ā Ā waiting to lock monitor 102054308 (object 7f3113800, a java.lang.Object), Ā Ā which is held by "Thread-1" "Thread-1": Ā Ā waiting to lock monitor 1020348b8 (object 7f3113810, a java.lang.Object), Ā Ā which is held by "Thread-2" Ā Java stack information for the threads listed above: =================================================== "Thread-2": Ā Ā Ā Ā at DeadlockTest$2.run(DeadlockTest.java:42) Ā Ā Ā Ā - waiting to lock <7f3113800> (a java.lang.Object) Ā Ā Ā Ā - locked <7f3113810> (a java.lang.Object) Ā Ā Ā Ā at java.lang.Thread.run(Thread.java:680) "Thread-1": Ā Ā Ā Ā at DeadlockTest$1.run(DeadlockTest.java:26) Ā Ā Ā Ā - waiting to lock <7f3113810> (a java.lang.Object) Ā Ā Ā Ā - locked <7f3113800> (a java.lang.Object) Ā Ā Ā Ā at java.lang.Thread.run(Thread.java:680) Ā
- 77. Threading: Chokepoint Causes: ā¢ Many threads bottlenecked waiting for one lock ā¢ Aggregate Detection: ā¢ Stalled threads / high concurrent usage ā¢ Exponential slowness ā¢ Low CPU usage ā¢ Speciļ¬c Detection: ā¢ Request response time monitoring ā¢ CPU block / wait timing ā¢
- 79. Internal Resource Bottleneck ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ Causes: Overusage of internal resource (threads, database connections, etc.) Underallocation of same Aggregate Detection: Stalled threads / high concurrent usage Call rate and average response time of internal resource Speciļ¬c Detection: Also compare with methods from Resource Leak, External Bottleneck, and Overusage of External System
- 80. External Bottleneck Causes: ā¢ ā¢ ā¢ External system (database, authentication server) is slow Compare with Overusage of external system Aggregate Detection: ā¢ ā¢ ā¢ Response time on backend calls Exceptions Speciļ¬c Detection: ā¢ ā¢ ā¢ Callgraphs Speciļ¬c monitoring on those backends
- 81. Commit happy
- 82. Production Ground to a halt for 2 hours And again the next day Trx/ min Avg RT Pool Limit Pool Usage Trx Stalls
- 83. Overusage of External System Causes: ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ ā¢ Poor design or tuning of interaction with backend system (e.g., join between two million-row tables for each user logon) O/R mapper misconļ¬guration Aggregate Detection: Response time measurement Speciļ¬c Detection: Timing on backend systems Also need tools for those backend systems
- 85. query too much data
- 86. Applications will become more Complex and Change Faster Distributed Monolithic AGILE Release 1.1 Release 1.2 Release 1.23 Release 1.5 .NET Service WebLogic Service WEB 2.0 3rd Party Web Service ESB/MQ Browser(s) Purchase Search Flight Flight Status Login Native Mobile App Oracle CDN Network Apache Sybase NOSQL JBoss Service SOA MySQL Cassandra PHP Service Memcached MOBILE DB2 SQL Server Tomcat Service JBoss Service VMWare Private Amazon EC2 Public CLOUD PostgreSQL Hadoop BIG DATA Copyright Ā© 2013 AppDynamics. All rights reserved. 86
- 87. ā¢ ā¢ One interesting problem occurs when the size of transactions with backend systems needs to be tuned Can be intertwined with / exacerbated by Layer-itis and Overusage of External System Many small requests System constantly wastes resources dispatching / unmarshalling many xactions and results āDeath by a thousand cutsā āJust Rightā One HUGE request System periodically slows to a crawl as many resources get thrown at large chunk of work āPig in a Pythonā