Make Java Profilers Lie Less

Make Java Profilers
Lie Less
Jaroslav Bachorík, Oracle
@yardus
Jaroslav Bachorík, 22nd
October 2015

Safe Harbor Statement
Jaroslav Bachorík Prague, 22nd
October 2015
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not
a commitment to deliver any material, code, or functionality, and should not be
relied upon in making purchasing decisions. The development, release, and timing
of any features or functionality described for Oracle’s products remains at the sole
discretion of Oracle.
Copyright ©2015, Oracle and/or its affiliates. All rights reserved.

About Me
October 2015
●
Present
●
Member Of JVM Serviceability Group
●
BTrace [1] Maintainer
●
Past
●
VisualVM [2] Contributor
●
NetBeans Profiler [3] Engineer

Agenda
October 2015
1) Let's Define 'Java Profiler'
2) Types Of Java Profilers
3) Under The Hood Of Java Profiler
4) Show Me The Magic!
5) There Must Be A Better Way
6) Conclusion
7) Anything I Forgot To Mention? (Q&A)

1) Let's Define
Java Profiler
October 2015

Profiling (Definition)
October 2015
Profiling ("program profiling", "software profiling") is a form of dynamic program
analysis that measures, for example
●
the space (memory) or time complexity of a program
●
the usage of particular instructions
●
or the frequency and duration of function calls.
Most commonly, profiling information serves to aid program optimization.
-Wikipedia

Java Runtime
October 2015
●
Also „Java Virtual Machine” (JVM)
●
Intermediate Execution Form – JVM Bytecode
●
Programming Language Agnostic
●
Dynamic In Nature
●
Interpreted vs. Compiled
●
Just In Time (JIT) Compilation
●
Dynamic Optimizations
●
Native Representation Of Bytecode Changes In Time

Profiling Java (JVM, In Fact)
October 2015
●
CPU
●
Utilization
●
Frequency And Duration Of Method Calls
●
Memory
●
Space Complexity
●
Garbage Collection (GC) Issues

2) Types Of
Java Profilers
October 2015

Java Profilers
October 2015
●
Sampling Profilers
●
Coarse Grained Data
●
Low Overhead (Usually)
●
Stack, Heap Histo
●
Instrumenting (Or Tracing) Profilers
●
Fine Grained Data
●
Injecting Bytecode
●
Significant Overhead (Usually)
●
Class Retransform (JVMTI, java.lang.instrument)

3) Under The Hood
Of Java Profiler
October 2015

Sampling Profilers
October 2015
●
Collecting Thread Stack Samples
●
Thread.getAllStackTraces()
"AWT-XAWT" #19 daemon prio=6 os_prio=0 tid=0x00007f08a80ef800 nid=0x647b runnable
[0x00007f08b67e5000]
java.lang.Thread.State: RUNNABLE
at sun.awt.X11.XToolkit.waitForEvents(Native Method)
at sun.awt.X11.XToolkit.run(XToolkit.java:568)
at sun.awt.X11.XToolkit.run(XToolkit.java:532)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None

Sampling Profilers
October 2015
●
Reconstruct Application Profile From Samples
●
Approximation
●
Errors Due To
●
Sampling Granularity
●
Sampling Bias

Sampling Granularity
October 2015
●
Grabbing Stacktrace With Certain Frequency
●
Might Miss Methods Shorter Than Sampling Period
●
Shortening Sampling Period Is Limited
●
Timer Resolution
●
Overhead
●
fn(Number Of Threads, Stack Depth)
●
Safepoints

Sampling Bias
October 2015
●
Preferring Certain Methods
●
Regular Execution
●
Execution Frequency ~ Sampling Frequency)
●
3 Out Of 4 Samples Ended In The Same Method
●
Other Methods Are Neglected
Timeline
Sampler

Sampling Bias
October 2015
●
Minimized By Irregular Sampling Period
●
Offsetting Each Sampling Interval
●
RND(-N, N)
●
Maximize Random Interval Randomness
●
Normal Distribution w Big Variance
●
Safepoints Are Problematic

Safepoint
October 2015
●
Signaling Known State Of JVM
●
Inserted By Interpreter
●
After Each Bytecode
●
Arbitrarily Placed By JIT
●
On Backedge Of
Non-Counted Loops
●
On Method Exit
●
On JNI Call Exit
●
May Be Omitted In Hot Code

Sampling And Safepoints
October 2015
Thread A Thread B Thread C
Code Safepoint
Hot Code
Safepoint Reached
Thread.getAllStackTraces()
Invisible Code

Instrumenting Profilers
October 2015
●
ByteCode Instrumentation (BCI)
●
Injecting Profiler Specific Bytecode Sequences
●
JVMTI Redefine/RetransformClasses
●
Additional Runtime Information
●
Method Parameters Values
●
Local Variables
●
Added Bytecode Will Affect JIT Compilers Decisions

0: sipush 3
3: invokestatic #96 // 'rootMethodEntry:(C)V'
6: nop
7: nop
8: iload_0
9: iload_1
20: iadd
11: sipush 3
14: invokestatic #90 // 'methodExit:(C)V'
17: nop
18: nop
19: ireturn
20: astore_1
21: sipush 3
24: invokestatic #90 // 'methodExit:(C)V'
27: aload_1
28: athrow
BCI Example
October 2015
0: iload_0
1: iload_1
2: iadd
3: ireturn
int add(int a, int b)
Profile

BCI & JIT
October 2015
●
JIT Decisions Affected By Injected Bytecode
●
Inlining
●
Inlined Method Size
●
Caller Size
●
Escape Analysis
●
TLAB Allocation
●
Stack Allocation

Inlining
October 2015
int x(int a, int b)
return a + c;
int c = a - b;
Inline!
int y(int a, int b)
return a - b;
int x(int a, int b)
return a + y(b, a);
●
Maximum Inline Size = 3
●
Method „y”
●
Size = 1
●
Will Be Inlined
●
Method „x”
●
Size = 1
●
After Inlining Size = 2

Inlining w BCI
October 2015
int y(int a, int b)
Profiler.entry();
return a - b;
Profiler.exit();
int x(int a, int b)
return a + y(b, a);
int x(int a, int b)
return a + c;
int c = a - b;
Profiler.entry();
Profiler.exit();
Inline?
●
●
Maximum Caller Size Violated

Inlining w BCI
October 2015
●
●
Maximum Callee Size Violated
int y(int a, int b)
Profiler.entry();
return a - b;
Profiler.exit();
int x(int a, int b)
return a + y(b, a);
Inline?

Escape Analysis w BCI
October 2015
Original Code Instrumented Code
Stack
●
Non-escaping instance allocated
on stack
●
No GC involved
Heap
●
Profiler.inspect(i) makes 'i' to escape
●
Instance is allocated on heap
●
GC is involved
void dummy()
MyInstance i = new Instance();
i.doOperation();
void dummy()
MyInstance i = new Instance();
i.doOperation();
Profiler.inspect(i);

Unintended Consequences
October 2015
●
JIT
●
Optimization Decisions
●
ReOpt/DeOpt
●
Code Cache
●
Application Execution
●
Timing
●
Locking Order
●
Additional Heap Allocations

Profiler Whopsies
October 2015
●
Sampling Profilers
●
Unexpected Overhead Due To Safepoints
●
Unexpected Bias Due To Safepoints
●
Instrumenting Profilers
●
BCI Affects JIT Optimization Decisions
●
BCI Affects Memory Management
●
BCI Affects Lock & Synchronization

4) Show Me
The Magic! (Demo)
October 2015

Profiling Results (BCI)
October 2015
Inlining Duration Total % Duration Total %
Default 0.44 6.81 6.46 8.2 660 1.24
3 0.44 6.98 6.3 9.5 684 1.39
40 0.44 6.03 7.3 7.4 653 1.13
200 0.44 9.58 4.59 15 680 2.21
1000 0.44 10.5 4.19 24 725 3.31
Run Method Profile Method

5) There Must Be
A Better Way
October 2015

AsyncGetCallTrace
October 2015
●
Internal API for Oracle Studio Performance Analyzer
●
Used by Java Flight Recorder (Oracle Commercial Feature)
●
Unsupported API
●
Despite That Used by 3rd
Parties
●
Honest Profiler [4]

AsyncGetCallTrace
October 2015
●
Interrupt Running Thread And Call Handler
●
Only Async-Safe Code In Handler
●
See 'man 7 signal' For The List
●
Does Not Depend On Safepoints
●
Shows What Is Actually Being Scheduled
●
Does Not Guarantee Complete Data
●
Missing Frames
●
Incomplete Frames
●
No Inlined Methods

Java Flight Recorder
October 2015
●
Oracle Commercial Feature
●
Free To Use In Development
●
-XX:+UnlockCommercialFeatures
●
-XX:+FlightRecorder
●
Use Java Mission Control To Inspect Recording
●
Available in JDK 8+ as 'jmc'
●
Use 'jcmd' To Initiate Recording
●
jcmd <PID> JFR.start duration=60s filename=myreco.jfr

October 2015
●
Very Low Overhead
●
Based On Events
●
Many Events Built In JVM
●
Reducing Need for BCI
●
Application Specific Events (WLS, DMS, 3rd
party)
●
Using AsyncGetCallTrace Internally
●
Avoiding Safepoints Pitfalls
●
All Limitations Of AsyncGetCallTrace

System Profiler
October 2015
●
-XX:+PreserveFramePointer
●
JDK 8u60, 9
●
Full Application Stack Available
●
Only Linux
●
perf
●
Need To Map Symbols
●
perf-map-agent [5]
●
Still Few Wrinkles To Iron
●
Symbols Changing In Time
CPU Flame Graphs [6]

Microbenchmarks
October 2015
●
Measure Performance In Controlled Environment
●
Stabilize JVM (JIT etc.) Before Measuring
●
Particulary Effective During Development
●
Optimize Performance Single Method At A Time
●
Guards Against Performance Regressions
●
Will Not Help Uncovering The Problems

Microbenchmarks
October 2015
●
Hard To Do Properly
●
Require Intimate Knowledge of JVM
●
Java Microbenchmark Harness (JMH) [7]
●
Abstract Away Gory Details
●
Integrated Profilers (stack, perf, perfasm, gc, …)
●
OpenJDK Project

JMH Profilers
October 2015
●
JMH Can Profile Benchmarked Methods
●
stack, gc, perf, perfnorm, perfasmperfasm
●
'perfasm' Profiler
●
Using 'perf' To Collect Profile
●
Shows Annotated Assembly Code
●
Java Instruction
●
Source Line Number
●
Inlining
●
Requires 'hsdis' [8]

BCI-Less JVMTI Profiler
October 2015
●
JVMTI Method Entry/Exit Event
●
Does Not Affect JIT
●
Does Not Need Stack Traces
●
Once Enabled ALL Methods Are Tracked
●
Huge Overhead

Improved BCI-Less Profiler
October 2015
●
Half-Baked Idea So Far
●
JVMTI Events With Reduced Overhead
●
Provide 'Slots' By Interpreter/Compiler
●
NOPs At Method Entry/Exit
●
Enable Profiling Per Method
●
Replace Slot By Handler Hook
●
Implementation At Interpreter/Compiler Level

6) Conclusion
October 2015

Sampling And Instrumentation
October 2015
●
Incomplete Or Skewed Data
●
Profiling Affects Application Execution
●
„Low Hanging Fruits” Almost Not Affected
●
Error Increasing With Detail Level
●
„Heisenberg Uncertainity Principle”
●
Strive For Repeatable Measurements
●
Know Your „O”

Non Traditionals
October 2015
●
AsyncGetCallTrace
●
Not Supported
●
Possibly Missing Frames
●
●
Commercial Product
●
→ AsyncGetCallTrace
●
System Profiler
●
Only Linux
●
Still Missing Pieces
●
Microbenchmarks
●
Won't Help To Identify Problems

Improving Situation
October 2015
●
AsyncGetCallTrace
●
Promote To Suported API
●
-XX:+PreserveFramePointer Synergy
●
Allow Unwinding Inlined Frames
●
Native Probe API
●
Use Interpreter/Compiler 'Slots' For Easy Instrumentation
●
Very Complicated Implementation

7) Did I Forget
Anything? (Q&A)
October 2015

Resources
October 2015
●
[1] https://github.com/jbachorik/btrace
●
[2] http://visualvm.java.net/
●
[3] https://profiler.netbeans.org/
●
[4] https://github.com/RichardWarburton/honest-profiler
●
[5] https://github.com/jrudolph/perf-map-agent
●
[6] http://techblog.netflix.com/2015/07/java-in-flames.html
●
[7] http://openjdk.java.net/projects/code-tools/jmh
●
[8] https://github.com/AdoptOpenJDK/jitwatch/wiki/Building-hsdis

THANK YOU!
October 2015
jaroslav.bachorik@oracle.com, @yardus

Make Java Profilers Lie Less

More Related Content

Viewers also liked(15)

Similar to Make Java Profilers Lie Less(20)

Recently uploaded(20)

Make Java Profilers Lie Less