Advanced Production Debugging
19 likes135,678 views
The document discusses advanced production debugging techniques, emphasizing the importance of logging at scale and state extraction using tools like btrace and java agents. It outlines best practices for logging, including transactional IDs and thread naming for improved clarity in multi-threaded environments. Additionally, it addresses challenges in modern production debugging such as state isolation and data distribution, providing insights into practical implementations for effective debugging.
![Transactional IDs
• Modern logging is done over a multi–threads / processes.
• Generate a UUID at every thread entry point into your app – the transaction ID.
• Append the ID into each log entry.
• Try to maintain it across machines – critical for debugging Reactive and microservice
apps.
[20-07 07:32:51][BRT -1473 -S4247] ERROR - Unable to retrieve data for
Job J141531. {CodeAnalysisUtil TID: Uu7XoelHfCTUUlvol6d2a9pU}
[SQS-prod_taskforce1_BRT-Executor-1-thread-2]](https://image.slidesharecdn.com/tal-weiss-scaladays-150611142831-lva1-app6892/85/Advanced-Production-Debugging-6-320.jpg)
![Thread Names (2)
For example:
Thread.currentThread().setName(
Context + TID + Params + current Time, ...);
Before:
“pool-1-thread-1″ #17 prio=5 os_prio=31
tid=0x00007f9d620c9800 nid=0x6d03 in Object.wait()
[0x000000013ebcc000
After:
”Queue Processing Thread, MessageID: AB5CAD, type:
AnalyzeGraph, queue: ACTIVE_PROD, Transaction_ID: 5678956,
Start Time: 10/8/2014 18:34″ #17 prio=5 os_prio=31
tid=0x00007f9d620c9800 nid=0x6d03 in Object.wait()
[0x000000013ebcc000]](https://image.slidesharecdn.com/tal-weiss-scaladays-150611142831-lva1-app6892/85/Advanced-Production-Debugging-9-320.jpg)
![”MsgID: AB5CAD, type: Analyze, queue: ACTIVE_PROD, TID:
5678956, TS: 11/8/20014 18:34 "
#17 prio=5 os_prio=31 tid=0x00007f9d620c9800 nid=0x6d03
in Object.wait() [0x000000013ebcc000]](https://image.slidesharecdn.com/tal-weiss-scaladays-150611142831-lva1-app6892/85/Advanced-Production-Debugging-18-320.jpg)
![Usage
• No JVM restart needed. Works remotely.
• btrace [-I <include-path>] [-p <port>] [-cp <classpath>] <pid> <btrace-script> [<args>]
• Example: Btrace 9550 myScript.java
• Available at: kenai.com/projects/btrace](https://image.slidesharecdn.com/tal-weiss-scaladays-150611142831-lva1-app6892/85/Advanced-Production-Debugging-22-320.jpg)
Advanced Production Debugging
- 2. About Me Co-founder – Takipi, JVM Production Debugging. Director, AutoCAD Web & Mobile. Software Architect at IAI Aerospace. Coding for the past 16 years - C++, Delphi, .NET, Java. Focus on real-time, scalable systems. Blogs at blog.takipi.com
- 3. Overview Dev-stage debugging is forward-tracing. Production debugging is focused on backtracing. Modern production debugging poses two challenges: • State isolation. • Data distribution.
- 4. Agenda 1. Logging at scale. 2. Preemptive jstacks 1. Extracting state with Btrace 1. Extracting state with custom Java agents.
- 5. Best Logging Practices 1. Code context. 2. Time + duration. 3. Transactional data (for async & distributed debugging). A primary new consumer is a log analyzer. Context trumps content.
- 6. Transactional IDs • Modern logging is done over a multi–threads / processes. • Generate a UUID at every thread entry point into your app – the transaction ID. • Append the ID into each log entry. • Try to maintain it across machines – critical for debugging Reactive and microservice apps. [20-07 07:32:51][BRT -1473 -S4247] ERROR - Unable to retrieve data for Job J141531. {CodeAnalysisUtil TID: Uu7XoelHfCTUUlvol6d2a9pU} [SQS-prod_taskforce1_BRT-Executor-1-thread-2]
- 7. 1. Don’t catch exceptions within loops and log them (implicit and explicit). For long running loops this will flood the log, impede performance and bring a server down. void readData { while (hasNext()) { try { readData(); } catch {Exception e) { logger.errror(“error reading “ X + “ from “ Y, e); } } 2. Do not log Object.toString(), especially collections. Can create an implicit loop. If needed – make sure length is limited. Logging Performance
- 8. Thread Names • Thread name is a mutable property. • Can be set to hold transaction specific state. • Some frameworks (e.g. EJB) don’t like that. • Can be super helpful when debugging in tandem with jstack.
- 9. Thread Names (2) For example: Thread.currentThread().setName( Context + TID + Params + current Time, ...); Before: “pool-1-thread-1″ #17 prio=5 os_prio=31 tid=0x00007f9d620c9800 nid=0x6d03 in Object.wait() [0x000000013ebcc000 After: ”Queue Processing Thread, MessageID: AB5CAD, type: AnalyzeGraph, queue: ACTIVE_PROD, Transaction_ID: 5678956, Start Time: 10/8/2014 18:34″ #17 prio=5 os_prio=31 tid=0x00007f9d620c9800 nid=0x6d03 in Object.wait() [0x000000013ebcc000]
- 11. Modern Stacks - Java 8
- 12. Modern Stacks - Scala
- 15. Preemptive jstack • A production debugging foundation. • Presents two issues – – Activated only in retrospect. – No state: does not provide any variable state. • Let’s see how we can overcome these with preemptive jstacks.
- 18. ”MsgID: AB5CAD, type: Analyze, queue: ACTIVE_PROD, TID: 5678956, TS: 11/8/20014 18:34 " #17 prio=5 os_prio=31 tid=0x00007f9d620c9800 nid=0x6d03 in Object.wait() [0x000000013ebcc000]
- 19. Jstack Triggers • A queue exceeds capacity. • Throughput exceeds or drops below a threshold. • CPU usage passes a threshold. • Locking failures / Deadlock. Integrate as a first class citizen with your logging infrastructure.
- 21. BTrace • An advanced open-source tool for extracting state from a live JVM. • Uses a Java agent and a meta-scripting language to capture state. • Pros: Lets you probe variable state without modifying / restarting the JVM. • Cons: read-only querying using a custom syntax and libraries.
- 22. Usage • No JVM restart needed. Works remotely. • btrace [-I <include-path>] [-p <port>] [-cp <classpath>] <pid> <btrace-script> [<args>] • Example: Btrace 9550 myScript.java • Available at: kenai.com/projects/btrace
- 23. BTrace - Restrictions • Can not create new objects. • Can not create new arrays. • Can not throw exceptions. • Can not catch exceptions. • Can not make arbitrary instance or static method calls - only the public static methods of com.sun.btrace.BTraceUtils class may be called from a BTrace program. • Can not assign to static or instance fields of target program's classes and objects. But, BTrace class can assign to it's own static fields ("trace state" can be mutated). • Can not have instance fields and methods. Only static public void returning methods are allowed for a BTrace class. And all fields have to be static. • Can not have outer, inner, nested or local classes. • Can not have synchronized blocks or synchronized methods. • can not have loops (for, while, do..while) • Can not extend arbitrary class (super class has to be java.lang.Object) • Can not implement interfaces. • Can not contains assert statements.
- 27. Java Agents • An advanced technique for instrumenting code dynamically. • The foundation of modern profiling / debugging tools. • Two types of agents: Java and Native. • Pros: extremely powerful technique to collect state from a live app. • Cons: requires knowledge of creating verifiable bytecode.
- 28. Agent Types • Java agents are written in Java. Have access to the Instrumentation BCI API. • Native agents – written in C++. • Have access to JVMTI – the JVM’s low-level set of APIs and capabilities. – JIT compilation, Garbage Collection, Monitor acquisition, Exception callbacks, .. • More complex to write. • Platform dependent.
- 30. com.sun.tools.attach.VirtualMachine Attach at startup: java -Xmx2G -agentlib:myAgent -jar myapp.jar start To a live JVM using: com.sun.tools.attach.VirtualMachine Attach API.
- 34. ASMifying ASM Bytecode Outline plug-in