SlideShare a Scribd company logo
Java Performance: 
Biggest Mistakes 
- More on http://blog.dynatrace.com 
- Dynatrace Free Trial: http://bit.ly/dttrial 
Andreas Grabner - @grabnerandi 
1 @Dynatrace
And this is where I am from 
2 @Dynatrace
3 @Dynatrace
4 @Dynatrace
Can you tell my age by looking at my first 
computer? 
5 @Dynatrace
That’s why I ended up talking about performance 
6 @Dynatrace
Nobody wants this … 
7 @Dynatrace
8 @Dynatrace
Nor this … 
9 @Dynatrace
10 @Dynatrace
As it leads to this … 
11 @Dynatrace
The “War Room” 
Facebook – December 2012 
12 @Dynatrace
And potentially to this … 
13 @Dynatrace
14 @Dynatrace
And this … 
15 @Dynatrace
16 @Dynatrace
And that’s why Business 
doesn’t like it either … 
17 @Dynatrace
YES we know this 
80%Dev Time in Bug Fixing 
$60B Defect Costs 
BUT 
~80% of problems 
caused by ~20% patterns 
18 @Dynatrace
6 Situations on 
WHY this happened, 
HOW to avoid it 
19 @Dynatrace
20 @Dynatrace
“Blindly” (Re)use 
Existing 
Components 
21 @Dynatrace
Requirement: We need a report 
22 @Dynatrace
Using Hibernate results in 4k+ SQL Statements to 
display 3 items! 
Hibernate Executes 
4k+ Statements 
Individual Execution 
VERY FAST 
But Total SUM 
takes 6s 
23 @Dynatrace
24 @Dynatrace
Not every 
Architect makes 
good decisions 
25 @Dynatrace
Project: Online Room Reservation System 
• Symptoms 
• HTML takes between 60 and 120s to render 
• High GC Time 
• Assumptions 
• Bad GC Tuning 
• Probably bad Database Performance as rendering was simple 
26 @Dynatrace
Developers built own monitoring 
void roomreservationReport(int roomid) 
{ 
long startTime = System.currentTimeMillis(); 
Object data = loadDataForRoom(roomid); 
long dataLoadTime = System.currentTimeMillis() - startTime; 
generateReport(data, roomid); 
27 @Dynatrace 
} 
Result: 
Avg. Data Load Time: 45s! 
DB Tool says: 
Avg. SQL Query: <1ms!
#1: Loading too much data 
24889! Calls to the 
Database API! 
High CPU and High 
Memory Usage to keep all 
data in Memory 
28 @Dynatrace
#2: On individual connections 12444! 
individual 
connections 
Classical N+1 
Query Problem 
Individual SQL 
really <1ms 
29 @Dynatrace
#3: Putting all data in temp Hashtable 
Lots of time 
spent in 
Hashtable.get 
Called from their 
Entity Objects 
30 @Dynatrace
Lesson Learned 
• Custom Measuring 
• Was impacted by Garbage Collection 
• Just measured overall time but not # SQL Executions 
void roomreservationReport(int roomid) 
{ 
long startTime = System.currentTimeMillis(); 
Object data = loadDataForRoom(roomid); 
long dataLoadTime = System.currentTimeMillis() - startTime; 
generateReport(data, roomid); 
} 
• Learn SQL and don’t use Hashtables as Workaround 
31 @Dynatrace
32 @Dynatrace
Implementation 
Flaws 
33 @Dynatrace
Business Impact requires Action! 
34 @Dynatrace
Solution: Cache to the RESCUE!! 
35 @Dynatrace
Implementation and Rollout 
Implemented 
InMemory Cache 
Worked well in 
Load Testing 
36 @Dynatrace
Result: Out of Memory Crashes!! 
Still crashes 
Fixed Version Deployed Problem fixed! 
37 @Dynatrace
38 @Dynatrace
Disconnected 
Teams 
39 @Dynatrace
“Teamwork” between Dev and Ops 
SEV1 Problem in Production 
Need access to log files 
Where are they? Can’t get them 
Need to increase log level 
Can’t do! Can’t change config 
files in prod! 
40 @Dynatrace
Solution: Implement a Custom “On Demand” 
Remote Logger 
41 @Dynatrace
Implementation and Rollout 
Implemented 
Custom Logger 
Worked well in 
Load Testing 
42 @Dynatrace
What happened? 
~ 1Mio Lock Exceptions 
in 30 mins 
43 @Dynatrace
Root Cause: A special WebSphere Setting! 
Log Service provides a 
synchronized log file across 
Log Service provides a synchronized 
ALL JVMs 
log file across ALL JVMs 
44 @Dynatrace
45 @Dynatrace
“Deployment” 
Gone Bad! 
46 @Dynatrace
Test Environment 
Production Environment 
8x slower 
3x more SQL 
47 @Dynatrace
Test Environment Production Environment 
Hibernate, 
Hibernate, 
Classloading, 
XML – The Key 
Classloading, XML 
– The Key Hotspots 
I/O for Web 
Requests Hotspots 
doesn’t 
even show up! 
That’s Normal: 
Having I/O for Web 
Request as main 
contributor 
48 @Dynatrace
Top Contributor 
Class.getInterfaces 
Called from Hibernates 
FieldInterceptionHelper 
These calls all originate 
form thousands of calls to 
find item by code 
49 @Dynatrace
Top Methods related to XML 
Processing 
Classloading is triggered through 
CustomMonnkey and the Xalan 
CustomMonkey and the Xalan 
Parser 
50 @Dynatrace
51 @Dynatrace
#Push 
without a Plan 
52 @Dynatrace
Mobile Landing Page of Super Bowl Ad 
434 Resources in total on that page: 
230 JPEGs, 75 PNGs, 50 GIFs, … 
Total size of ~ 
20MB 
53 @Dynatrace
Fifa.com during Worldcup 
http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/ 
54 @Dynatrace
Waiting for 
Bug Reports 
55 @Dynatrace
Look behind the scenes 
56 @Dynatrace
57 @Dynatrace
Performance Clinic: Bring Your Own App 
When: Thursday, December 18th 
Lunch Session: NOON – 3PM 
After Work Session: 4PM – 7PM 
Where: Dynatrace Office 
404 Wyman Street, Waltham 
Register: http://bit.ly/onlineperfclinic 
58 @Dynatrace
Quick Demo 
59 @Dynatrace
30 Days Dynatrace Free Trial 
• Get it here: http://bit.ly/dttrial 
• YouTube Tutorials: http://bit.ly/dttutorials 
• Live Q&A Sessions: http://bit.ly/onlineperfclinic 
• Contact me: agrabner@dynatrace.com - @grabnerandi 
•Special Offer: 
• SaaS VIP Program: http://bit.ly/dtaasvip 
60 @Dynatrace
Andreas Grabner 
Dynatrace Developer Advocate 
@grabnerandi 
http://blog.dynatrace.com 
61 @Dynatrace

More Related Content

What's hot (20)

PPTX
Docker/DevOps Meetup: Metrics-Driven Continuous Performance and Scalabilty
Andreas Grabner
 
PPTX
From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam
Andreas Grabner
 
PPTX
Application Quality Gates in Continuous Delivery: Deliver Better Software Fas...
Andreas Grabner
 
PPTX
(R)evolutionize APM
Andreas Grabner
 
PPTX
Mobile User Experience: Auto Drive through Performance Metrics
Andreas Grabner
 
PPTX
Top Java Performance Problems and Metrics To Check in Your Pipeline
Andreas Grabner
 
PPTX
Sydney Continuous Delivery Meetup May 2014
Andreas Grabner
 
PPTX
OOP 2016 - Building Software That Eats The World
Andreas Grabner
 
PPTX
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
Andreas Grabner
 
PPTX
JavaOne 2015: Top Performance Patterns Deep Dive
Andreas Grabner
 
PDF
Nginx performance monitoring with Dynatrace
Harald Zeitlhofer
 
PPTX
Boston DevOps Days 2016: Implementing Metrics Driven DevOps - Why and How
Andreas Grabner
 
PPTX
How to explain DevOps to your mom
Andreas Grabner
 
PPTX
DevOps Pipelines and Metrics Driven Feedback Loops
Andreas Grabner
 
PPTX
Metrics Driven DevOps - Automate Scalability and Performance Into your Pipeline
Andreas Grabner
 
PPTX
Release Readiness Validation with Keptn for Austrian Online Banking Software
Andreas Grabner
 
PPTX
JavaOne - Performance Focused DevOps to Improve Cont Delivery
Andreas Grabner
 
PPTX
StarWest 2013 Performance is not an afterthought – make it a part of your Agi...
Andreas Grabner
 
PPTX
Keptn - Automated Operations & Continuous Delivery for k8s
Andreas Grabner
 
PDF
Continuously Integrating Distributed Code at Netflix
Atlassian
 
Docker/DevOps Meetup: Metrics-Driven Continuous Performance and Scalabilty
Andreas Grabner
 
From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam
Andreas Grabner
 
Application Quality Gates in Continuous Delivery: Deliver Better Software Fas...
Andreas Grabner
 
(R)evolutionize APM
Andreas Grabner
 
Mobile User Experience: Auto Drive through Performance Metrics
Andreas Grabner
 
Top Java Performance Problems and Metrics To Check in Your Pipeline
Andreas Grabner
 
Sydney Continuous Delivery Meetup May 2014
Andreas Grabner
 
OOP 2016 - Building Software That Eats The World
Andreas Grabner
 
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
Andreas Grabner
 
JavaOne 2015: Top Performance Patterns Deep Dive
Andreas Grabner
 
Nginx performance monitoring with Dynatrace
Harald Zeitlhofer
 
Boston DevOps Days 2016: Implementing Metrics Driven DevOps - Why and How
Andreas Grabner
 
How to explain DevOps to your mom
Andreas Grabner
 
DevOps Pipelines and Metrics Driven Feedback Loops
Andreas Grabner
 
Metrics Driven DevOps - Automate Scalability and Performance Into your Pipeline
Andreas Grabner
 
Release Readiness Validation with Keptn for Austrian Online Banking Software
Andreas Grabner
 
JavaOne - Performance Focused DevOps to Improve Cont Delivery
Andreas Grabner
 
StarWest 2013 Performance is not an afterthought – make it a part of your Agi...
Andreas Grabner
 
Keptn - Automated Operations & Continuous Delivery for k8s
Andreas Grabner
 
Continuously Integrating Distributed Code at Netflix
Atlassian
 

Viewers also liked (20)

PDF
Continuous Delivery in a content centric world
Jeroen Reijn
 
PDF
Andy carvell-sound cloud
App Promotion Summit Conference
 
PPT
Los Coches Del Papa
Jazmín Lobo
 
PDF
Creating High Performance Big Data Applications with the Java Persistence API
DATAVERSITY
 
PPT
Optimizing Java Performance
Konstantin Pavlov
 
PPTX
Java performance
Sachin Shukla
 
PDF
High Performance With Java
malduarte
 
PDF
Performance van Java 8 en verder - Jeroen Borgers
NLJUG
 
PDF
High Performance Web Design
Koji Ishimoto
 
PDF
Java Performance
SSA KPI
 
PDF
Java Performance & Profiling
Isuru Perera
 
PDF
WSO2 Identity Server
Prabath Siriwardena
 
PPTX
Microservices for performance - GOTO Chicago 2016
Peter Lawrey
 
PDF
Composer the right way
Rafael Dohms
 
PPTX
Una estrategia para el desarrollo del senderismo en Venezuela
Fundación Programa Andes Tropicales
 
PPT
Java Performance, Threading and Concurrent Data Structures
Hitendra Kumar
 
PPTX
High performance java ee with j cache and cdi
Payara
 
PDF
SSO with the WSO2 Identity Server
WSO2
 
PPTX
Using dynaTrace to optimise application performance
Richard Bishop
 
PPT
Practical Steps For Building High Performance Teams
Elijah Ezendu
 
Continuous Delivery in a content centric world
Jeroen Reijn
 
Andy carvell-sound cloud
App Promotion Summit Conference
 
Los Coches Del Papa
Jazmín Lobo
 
Creating High Performance Big Data Applications with the Java Persistence API
DATAVERSITY
 
Optimizing Java Performance
Konstantin Pavlov
 
Java performance
Sachin Shukla
 
High Performance With Java
malduarte
 
Performance van Java 8 en verder - Jeroen Borgers
NLJUG
 
High Performance Web Design
Koji Ishimoto
 
Java Performance
SSA KPI
 
Java Performance & Profiling
Isuru Perera
 
WSO2 Identity Server
Prabath Siriwardena
 
Microservices for performance - GOTO Chicago 2016
Peter Lawrey
 
Composer the right way
Rafael Dohms
 
Una estrategia para el desarrollo del senderismo en Venezuela
Fundación Programa Andes Tropicales
 
Java Performance, Threading and Concurrent Data Structures
Hitendra Kumar
 
High performance java ee with j cache and cdi
Payara
 
SSO with the WSO2 Identity Server
WSO2
 
Using dynaTrace to optimise application performance
Richard Bishop
 
Practical Steps For Building High Performance Teams
Elijah Ezendu
 
Ad

Similar to Java Performance Mistakes (20)

PPTX
Industry Keynote at Large Scale Testing Workshop 2015
Wolfgang Gottesheim
 
PDF
JUG Poznan - 2017.01.31
Omnilogy
 
PDF
Performance Metrics for your Delivery Pipeline - Wolfgang Gottesheim
JAXLondon2014
 
PDF
Become a Performance Diagnostics Hero
TechWell
 
PPTX
Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing ...
Andreas Grabner
 
PPTX
STP 2014 - Lets Learn from the Top Performance Mistakes in 2013
Andreas Grabner
 
PPTX
Performance Metrics for your Build Pipeline - presented at Vienna WebPerf Oct...
Andreas Grabner
 
PDF
Performance tuning the Spring Pet Clinic sample application
Julien Dubois
 
PPTX
Optimizing Application Performance - 2022.pptx
JasonTuran2
 
PDF
Performance? That's what version 2 is for!
Eduard Tudenhoefner
 
PDF
Tools to effectively monitor your mobile and web apps
Jerry Tan
 
PDF
Sensible scaling
Rowan Merewood
 
PPT
W-JAX Performance Workshop - Database Performance
Alois Reitbauer
 
PDF
JCON World 2023 - Cache, but Cache Wisely.pdf
DevenPhillips
 
PDF
Top-10-Java-Performance-Problems.pdf
KiranChinnagangannag
 
PPT
Low Hanging Fruits In J EE Performance
Alois Reitbauer
 
PPTX
Profiling and Tuning a Web Application - The Dirty Details
Achievers Tech
 
PDF
Application Performance Troubleshooting 1x1 - Von Schweinen, Schlangen und Pa...
rschuppe
 
PDF
Programming for Performance
Cris Holdorph
 
PPSX
Aws meetup Singapore 14 July
Jerry Tan
 
Industry Keynote at Large Scale Testing Workshop 2015
Wolfgang Gottesheim
 
JUG Poznan - 2017.01.31
Omnilogy
 
Performance Metrics for your Delivery Pipeline - Wolfgang Gottesheim
JAXLondon2014
 
Become a Performance Diagnostics Hero
TechWell
 
Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing ...
Andreas Grabner
 
STP 2014 - Lets Learn from the Top Performance Mistakes in 2013
Andreas Grabner
 
Performance Metrics for your Build Pipeline - presented at Vienna WebPerf Oct...
Andreas Grabner
 
Performance tuning the Spring Pet Clinic sample application
Julien Dubois
 
Optimizing Application Performance - 2022.pptx
JasonTuran2
 
Performance? That's what version 2 is for!
Eduard Tudenhoefner
 
Tools to effectively monitor your mobile and web apps
Jerry Tan
 
Sensible scaling
Rowan Merewood
 
W-JAX Performance Workshop - Database Performance
Alois Reitbauer
 
JCON World 2023 - Cache, but Cache Wisely.pdf
DevenPhillips
 
Top-10-Java-Performance-Problems.pdf
KiranChinnagangannag
 
Low Hanging Fruits In J EE Performance
Alois Reitbauer
 
Profiling and Tuning a Web Application - The Dirty Details
Achievers Tech
 
Application Performance Troubleshooting 1x1 - Von Schweinen, Schlangen und Pa...
rschuppe
 
Programming for Performance
Cris Holdorph
 
Aws meetup Singapore 14 July
Jerry Tan
 
Ad

More from Andreas Grabner (15)

PPTX
KCD Munich - Cloud Native Platform Dilemma - Turning it into an Opportunity
Andreas Grabner
 
PPTX
OpenTelemetry For GitOps: Tracing Deployments from Git Commit to Production
Andreas Grabner
 
PPTX
Don't Deploy Into the Dark: DORA Metrics for your K8s GitOps Deployments
Andreas Grabner
 
PPTX
Observability and Orchestration of your GitOps Deployments with Keptn
Andreas Grabner
 
PPTX
Adding Security to your SLO-based Release Validation with Keptn
Andreas Grabner
 
PPTX
A Guide to Event-Driven SRE-inspired DevOps
Andreas Grabner
 
PPTX
Jenkins Online Meetup - Automated SLI based Build Validation with Keptn
Andreas Grabner
 
PPTX
Continuous Delivery and Automated Operations on k8s with keptn
Andreas Grabner
 
PPTX
Shipping Code like a keptn: Continuous Delivery & Automated Operations on k8s
Andreas Grabner
 
PPTX
Top Performance Problems in Distributed Architectures
Andreas Grabner
 
PPTX
Applying AI to Performance Engineering: Shift-Left, Shift-Right, Self-Healing
Andreas Grabner
 
PPTX
Monitoring as a Self-Service in Atlassian DevOps Toolchain
Andreas Grabner
 
PPTX
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
Andreas Grabner
 
PPTX
AWS Summit - Trends in Advanced Monitoring for AWS environments
Andreas Grabner
 
PPTX
DevOps Transformation at Dynatrace and with Dynatrace
Andreas Grabner
 
KCD Munich - Cloud Native Platform Dilemma - Turning it into an Opportunity
Andreas Grabner
 
OpenTelemetry For GitOps: Tracing Deployments from Git Commit to Production
Andreas Grabner
 
Don't Deploy Into the Dark: DORA Metrics for your K8s GitOps Deployments
Andreas Grabner
 
Observability and Orchestration of your GitOps Deployments with Keptn
Andreas Grabner
 
Adding Security to your SLO-based Release Validation with Keptn
Andreas Grabner
 
A Guide to Event-Driven SRE-inspired DevOps
Andreas Grabner
 
Jenkins Online Meetup - Automated SLI based Build Validation with Keptn
Andreas Grabner
 
Continuous Delivery and Automated Operations on k8s with keptn
Andreas Grabner
 
Shipping Code like a keptn: Continuous Delivery & Automated Operations on k8s
Andreas Grabner
 
Top Performance Problems in Distributed Architectures
Andreas Grabner
 
Applying AI to Performance Engineering: Shift-Left, Shift-Right, Self-Healing
Andreas Grabner
 
Monitoring as a Self-Service in Atlassian DevOps Toolchain
Andreas Grabner
 
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
Andreas Grabner
 
AWS Summit - Trends in Advanced Monitoring for AWS environments
Andreas Grabner
 
DevOps Transformation at Dynatrace and with Dynatrace
Andreas Grabner
 

Recently uploaded (20)

PDF
Reasons for the succes of MENARD PRESSUREMETER.pdf
majdiamz
 
PPTX
GitOps_Without_K8s_Training_detailed git repository
DanialHabibi2
 
PPTX
265587293-NFPA 101 Life safety code-PPT-1.pptx
chandermwason
 
PDF
Water Industry Process Automation & Control Monthly July 2025
Water Industry Process Automation & Control
 
PPTX
What is Shot Peening | Shot Peening is a Surface Treatment Process
Vibra Finish
 
PDF
Electrical Engineer operation Supervisor
ssaruntatapower143
 
PDF
Biomechanics of Gait: Engineering Solutions for Rehabilitation (www.kiu.ac.ug)
publication11
 
PPTX
The Role of Information Technology in Environmental Protectio....pptx
nallamillisriram
 
PDF
Viol_Alessandro_Presentazione_prelaurea.pdf
dsecqyvhbowrzxshhf
 
PPT
Carmon_Remote Sensing GIS by Mahesh kumar
DhananjayM6
 
PDF
MAD Unit - 1 Introduction of Android IT Department
JappanMavani
 
PPTX
Worm gear strength and wear calculation as per standard VB Bhandari Databook.
shahveer210504
 
DOCX
CS-802 (A) BDH Lab manual IPS Academy Indore
thegodhimself05
 
PPT
PPT2_Metal formingMECHANICALENGINEEIRNG .ppt
Praveen Kumar
 
PPTX
Mechanical Design of shell and tube heat exchangers as per ASME Sec VIII Divi...
shahveer210504
 
PDF
Basic_Concepts_in_Clinical_Biochemistry_2018كيمياء_عملي.pdf
AdelLoin
 
PPTX
MATLAB : Introduction , Features , Display Windows, Syntax, Operators, Graph...
Amity University, Patna
 
PPTX
Product Development & DevelopmentLecture02.pptx
zeeshanwazir2
 
PPTX
Damage of stability of a ship and how its change .pptx
ehamadulhaque
 
PPTX
Heart Bleed Bug - A case study (Course: Cryptography and Network Security)
Adri Jovin
 
Reasons for the succes of MENARD PRESSUREMETER.pdf
majdiamz
 
GitOps_Without_K8s_Training_detailed git repository
DanialHabibi2
 
265587293-NFPA 101 Life safety code-PPT-1.pptx
chandermwason
 
Water Industry Process Automation & Control Monthly July 2025
Water Industry Process Automation & Control
 
What is Shot Peening | Shot Peening is a Surface Treatment Process
Vibra Finish
 
Electrical Engineer operation Supervisor
ssaruntatapower143
 
Biomechanics of Gait: Engineering Solutions for Rehabilitation (www.kiu.ac.ug)
publication11
 
The Role of Information Technology in Environmental Protectio....pptx
nallamillisriram
 
Viol_Alessandro_Presentazione_prelaurea.pdf
dsecqyvhbowrzxshhf
 
Carmon_Remote Sensing GIS by Mahesh kumar
DhananjayM6
 
MAD Unit - 1 Introduction of Android IT Department
JappanMavani
 
Worm gear strength and wear calculation as per standard VB Bhandari Databook.
shahveer210504
 
CS-802 (A) BDH Lab manual IPS Academy Indore
thegodhimself05
 
PPT2_Metal formingMECHANICALENGINEEIRNG .ppt
Praveen Kumar
 
Mechanical Design of shell and tube heat exchangers as per ASME Sec VIII Divi...
shahveer210504
 
Basic_Concepts_in_Clinical_Biochemistry_2018كيمياء_عملي.pdf
AdelLoin
 
MATLAB : Introduction , Features , Display Windows, Syntax, Operators, Graph...
Amity University, Patna
 
Product Development & DevelopmentLecture02.pptx
zeeshanwazir2
 
Damage of stability of a ship and how its change .pptx
ehamadulhaque
 
Heart Bleed Bug - A case study (Course: Cryptography and Network Security)
Adri Jovin
 

Java Performance Mistakes

  • 1. Java Performance: Biggest Mistakes - More on http://blog.dynatrace.com - Dynatrace Free Trial: http://bit.ly/dttrial Andreas Grabner - @grabnerandi 1 @Dynatrace
  • 2. And this is where I am from 2 @Dynatrace
  • 5. Can you tell my age by looking at my first computer? 5 @Dynatrace
  • 6. That’s why I ended up talking about performance 6 @Dynatrace
  • 7. Nobody wants this … 7 @Dynatrace
  • 9. Nor this … 9 @Dynatrace
  • 11. As it leads to this … 11 @Dynatrace
  • 12. The “War Room” Facebook – December 2012 12 @Dynatrace
  • 13. And potentially to this … 13 @Dynatrace
  • 15. And this … 15 @Dynatrace
  • 17. And that’s why Business doesn’t like it either … 17 @Dynatrace
  • 18. YES we know this 80%Dev Time in Bug Fixing $60B Defect Costs BUT ~80% of problems caused by ~20% patterns 18 @Dynatrace
  • 19. 6 Situations on WHY this happened, HOW to avoid it 19 @Dynatrace
  • 21. “Blindly” (Re)use Existing Components 21 @Dynatrace
  • 22. Requirement: We need a report 22 @Dynatrace
  • 23. Using Hibernate results in 4k+ SQL Statements to display 3 items! Hibernate Executes 4k+ Statements Individual Execution VERY FAST But Total SUM takes 6s 23 @Dynatrace
  • 25. Not every Architect makes good decisions 25 @Dynatrace
  • 26. Project: Online Room Reservation System • Symptoms • HTML takes between 60 and 120s to render • High GC Time • Assumptions • Bad GC Tuning • Probably bad Database Performance as rendering was simple 26 @Dynatrace
  • 27. Developers built own monitoring void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); 27 @Dynatrace } Result: Avg. Data Load Time: 45s! DB Tool says: Avg. SQL Query: <1ms!
  • 28. #1: Loading too much data 24889! Calls to the Database API! High CPU and High Memory Usage to keep all data in Memory 28 @Dynatrace
  • 29. #2: On individual connections 12444! individual connections Classical N+1 Query Problem Individual SQL really <1ms 29 @Dynatrace
  • 30. #3: Putting all data in temp Hashtable Lots of time spent in Hashtable.get Called from their Entity Objects 30 @Dynatrace
  • 31. Lesson Learned • Custom Measuring • Was impacted by Garbage Collection • Just measured overall time but not # SQL Executions void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); } • Learn SQL and don’t use Hashtables as Workaround 31 @Dynatrace
  • 34. Business Impact requires Action! 34 @Dynatrace
  • 35. Solution: Cache to the RESCUE!! 35 @Dynatrace
  • 36. Implementation and Rollout Implemented InMemory Cache Worked well in Load Testing 36 @Dynatrace
  • 37. Result: Out of Memory Crashes!! Still crashes Fixed Version Deployed Problem fixed! 37 @Dynatrace
  • 39. Disconnected Teams 39 @Dynatrace
  • 40. “Teamwork” between Dev and Ops SEV1 Problem in Production Need access to log files Where are they? Can’t get them Need to increase log level Can’t do! Can’t change config files in prod! 40 @Dynatrace
  • 41. Solution: Implement a Custom “On Demand” Remote Logger 41 @Dynatrace
  • 42. Implementation and Rollout Implemented Custom Logger Worked well in Load Testing 42 @Dynatrace
  • 43. What happened? ~ 1Mio Lock Exceptions in 30 mins 43 @Dynatrace
  • 44. Root Cause: A special WebSphere Setting! Log Service provides a synchronized log file across Log Service provides a synchronized ALL JVMs log file across ALL JVMs 44 @Dynatrace
  • 46. “Deployment” Gone Bad! 46 @Dynatrace
  • 47. Test Environment Production Environment 8x slower 3x more SQL 47 @Dynatrace
  • 48. Test Environment Production Environment Hibernate, Hibernate, Classloading, XML – The Key Classloading, XML – The Key Hotspots I/O for Web Requests Hotspots doesn’t even show up! That’s Normal: Having I/O for Web Request as main contributor 48 @Dynatrace
  • 49. Top Contributor Class.getInterfaces Called from Hibernates FieldInterceptionHelper These calls all originate form thousands of calls to find item by code 49 @Dynatrace
  • 50. Top Methods related to XML Processing Classloading is triggered through CustomMonnkey and the Xalan CustomMonkey and the Xalan Parser 50 @Dynatrace
  • 52. #Push without a Plan 52 @Dynatrace
  • 53. Mobile Landing Page of Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB 53 @Dynatrace
  • 54. Fifa.com during Worldcup http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/ 54 @Dynatrace
  • 55. Waiting for Bug Reports 55 @Dynatrace
  • 56. Look behind the scenes 56 @Dynatrace
  • 58. Performance Clinic: Bring Your Own App When: Thursday, December 18th Lunch Session: NOON – 3PM After Work Session: 4PM – 7PM Where: Dynatrace Office 404 Wyman Street, Waltham Register: http://bit.ly/onlineperfclinic 58 @Dynatrace
  • 59. Quick Demo 59 @Dynatrace
  • 60. 30 Days Dynatrace Free Trial • Get it here: http://bit.ly/dttrial • YouTube Tutorials: http://bit.ly/dttutorials • Live Q&A Sessions: http://bit.ly/onlineperfclinic • Contact me: [email protected] - @grabnerandi •Special Offer: • SaaS VIP Program: http://bit.ly/dtaasvip 60 @Dynatrace
  • 61. Andreas Grabner Dynatrace Developer Advocate @grabnerandi http://blog.dynatrace.com 61 @Dynatrace

Editor's Notes

  • #2: More detailed stories can also be found on our blog: http://blog.dynatrace.com All examples have been found using Dynatrace Free Trial – http://bit.ly/dttrial
  • #3: Just a bit about myself: That’s where I am from: Austria!
  • #4: Not the country with the kangaroos!
  • #6: That was my first Amiga 500 which I got when I was 9 years old!
  • #7: And that’s my professional background
  • #8: But now lets get started …
  • #19: Nobody likes to spend time in bugfixing
  • #56: Stop waiting for problems to happen
  • #57: Start looking behind the scenes