SlideShare a Scribd company logo
Аварийный дамп –
чёрный ящик упавшей JVM

Андрей Паньгин
Одноклассники, ведущий инженер
Java vs. C
•

 Crash!

array[999999] = 42;

array[999999] = 42;

 ArrayIndexOutOfBoundsException

•

nullRef->callMethod();

 NullPointerException

•

nullRef.callMethod();

 Crash!

int y = x / 0;

int y = x / 0;

 ArithmeticException

 Crash!
Java пуленепробиваемая?
Нет, только не это!
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#

A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x00002b47f02da0c3, pid=20644, tid=1096538432
JRE version: 6.0_26-b03
Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64)
Problematic frame:
V [libjvm.so+0x8400c3]
An error report file with more information is saved as:
/one/bin/hs_err_pid20644.log
If you would like to submit a bug report, please visit:
http://java.sun.com/webapps/bugreport/crash.jsp
Фатальные ошибки
1. Проблема в native коде
2. Некорректное использование закрытого API
(например, sun.misc.Unsafe)

3. Баг JVM (да, встречали не раз!)
4. Проблема с «железом» (и такое бывало!)
Сломаем JVM?
Заголовок
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGFPE (0x8) at pc=0x00007f1c1197b585, pid=3898, tid=139758846732032
#
# JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 compressed oops)
#

код сигнала
адрес инструкции

id процесса

id потока

сборка JRE и JVM
Фрейм
# Problematic frame:
# C [libdiv.so+0x585]

Java_demo1_NativeDiv_div+0x5

адрес инструкции (относительно DLL и символа)
тип фрейма
С

Native C frame

V

VM function

v

VM generated stub

j

Interpreted Java frame

J

Compiled Java frame
Сигнал
siginfo:si_signo=SIGFPE: si_errno=0, si_code=1 (FPE_INTDIV), si_addr=0x00007f1c1197b585

адрес инструкции или
обращения к памяти
(для SEGV, SIGBUS)

код сигнала
Linux

Windows

SIGSEGV

EXCEPTION_ACCESS_VIOLATION
EXCEPTION_STACK_OVERFLOW

SIGBUS

EXCEPTION_ACCESS_VIOLATION
EXCEPTION_DATATYPE_MISALIGNMENT

SEGV_MAPERR

отсутствующая страница

SEGV_ACCERR

нет прав доступа к странице

SIGILL

EXCEPTION_ILLEGAL_INSTRUCTION
EXCEPTION_PRIV_INSTRUCTION

BUS_ADRALN

невыровненный адрес

SIGFPE

EXCEPTION_INT_*
EXCEPTION_FLT_*

BUS_ADRERR

несуществующий адрес

ILL_ILLOPC

неверная инструкция

ILL_PRVOPC

привилегированная инструкция

FPE_INTDIV

целочисленное деление на 0

FPE_FLTDIV

вещественное деление на 0

причина сигнала
Регистры CPU
Registers:
RAX=0x00000000c5448e8f,
RSP=0x00007f1c2469d798,
R8 =0x0000c5448e8f8389,
R12=0x0000000000000000,
RIP=0x00007f1c1197b585,

RBX=0x00000000bd250ff0, RCX=0x0000000000000000, RDX=0x00000000ffffffff
RBP=0x00007f1c2469d7f0, RSI=0x00007f1c2469d800, RDI=0x00007f1c1c0099e8
R9 =0x00000850b3941760, R10=0x00007f1c1901270c, R11=0x00007f1c234de260
R13=0x00000000bd250ff0, R14=0x00007f1c2469d818, R15=0x00007f1c1c009800
EFLAGS=0x0000000000010287, CSGSFS=0x0000000000000033
Верхушка стека
Top of Stack: (sp=0x00007f1c2469d798)
0x00007f1c2469d798:
00007f1c19012738
0x00007f1c2469d7a8:
0000000000000000
0x00007f1c2469d7b8:
0000000000000000
0x00007f1c2469d7c8:
00000000bd251480
0x00007f1c2469d7d8:
00000000bd250ff0
0x00007f1c2469d7e8:
00007f1c2469d810
0x00007f1c2469d7f8:
00007f1c190061d4
0x00007f1c2469d808:
00007f1c1900ecd6

00000000bd1a53e8
00007f1c2469d7b0
00007f1c2469d818
0000000000000000
0000000000000000
00007f1c2469d860
00000000eb64ad68
0000000000000000
Машинный код
Instructions: (pc=0x00007f1c1197b585)
0x00007f1c1197b565:
48 85 c0 74 0e 5d
0x00007f1c1197b575:
1f 40 00 5d c3 90
0x00007f1c1197b585:
f7 f9 c3 90 90 90
0x00007f1c1197b595:
48 83 ec 08 48 8b

48
90
90
05

8d
90
90
78

3d
90
90
08

c6
90
90
20

08
90
90
00

20
89
55
48

00
d0
48
83

ff
c1
89
f8

e0
fa
e5
ff

0f
1f
53
74

до
после

• Дизассемблер
– http://www.onlinedisassembler.com

• Описание инструкций
– http://www.intel.com/content/www/us/en/processors/
architectures-software-developer-manuals.html
Дизассемблер
.data:0x00000018
.data:0x00000019
.data:0x0000001a
.data:0x0000001b
.data:0x0000001d
.data:0x00000020
.data:0x00000022
.data:0x00000023
.data:0x00000024
.data:0x00000025
.data:0x00000026
.data:0x00000027

смещение

90
90
90
89d0
c1fa1f
f7f9
c3
90
90
90
90
90

машинный
код

nop
nop
nop
mov
sar
idiv
ret
nop
nop
nop
nop
nop

eax,edx
edx,0x1f
ecx

декодированные
инструкции

место падения
idiv r/m32
signed divide edx:eax by r/m32
Значения регистров
Register to memory mapping:

RAX=0x00000000c5448e8f is an unallocated location in the heap
RBX=0x00000000bd250ff0 is an oop
{method}
- klass: {other class}
RCX=0x0000000000000000 is an unknown value
RDX=0x00000000ffffffff is an unallocated location in the heap
RSP=0x00007f1c2469d798 is pointing into the stack for thread: 0x00007f1c1c009800
RBP=0x00007f1c2469d7f0 is pointing into the stack for thread: 0x00007f1c1c009800
RSI=0x00007f1c2469d800 is pointing into the stack for thread: 0x00007f1c1c009800
RDI=0x00007f1c1c0099e8 is an unknown value
R8 =0x0000c5448e8f8389 is an unknown value
R9 =0x00000850b3941760 is an unknown value
R10=0x00007f1c1901270c is at code_begin+620 in an Interpreter codelet
method entry point (kind = native) [0x00007f1c190124a0, 0x00007f1c19012d00] 2144 bytes
R11=0x00007f1c234de260: <offset 0x8a7260> in libjvm.so at 0x00007f1c22c37000
R12=0x0000000000000000 is an unknown value
R13=0x00000000bd250ff0 is an oop
{method}
- klass: {other class}
R14=0x00007f1c2469d818 is pointing into the stack for thread: 0x00007f1c1c009800
R15=0x00007f1c1c009800 is a thread
Стек

границы стека

Stack: [0x7f1c2459e000,0x7f1c2469f000],

указатель стека

sp=0x7f1c2469d798,

free space=1021k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libdiv.so+0x585] Java_demo1_NativeDiv_div+0x5
j demo1.NativeDiv.access$000(II)I+2
смещение в байтах
j demo1.NativeDiv$1.run()V+20
j demo1.NativeDiv.runLoop(ILjava/lang/Runnable;)V+8
j demo1.NativeDiv.main([Ljava/lang/String;)V+21
индекс байткода
v ~StubRoutines::call_stub
V [libjvm.so+0x5f8405] JavaCalls::call_helper()+0x365
V [libjvm.so+0x5f6e68] JavaCalls::call()+0x28
V [libjvm.so+0x62f8d9] jni_invoke_static()+0x219
V [libjvm.so+0x638962] jni_CallStaticVoidMethod+0x162
C [libjli.so+0x36d9] JavaMain+0x7e9

тип фрейма

декодированный адрес
(DLL, символ + смещение)
Потоки
Java Threads: ( => current thread )
0x7f1c1c096000 JavaThread "Service Thread" daemon
[_thread_blocked, id=3910, stack(0x7f1c11c7e000,0x7f1c11d7f000)]
0x7f1c1c093800 JavaThread "C2 CompilerThread1" daemon
[_thread_blocked, id=3909, stack(0x7f1c11d7f000,0x7f1c11e80000)]
0x7f1c1c090800 JavaThread "C2 CompilerThread0" daemon
[_thread_blocked, id=3908, stack(0x7f1c18014000,0x7f1c18115000)]
0x7f1c1c08e800 JavaThread "Signal Dispatcher" daemon
[_thread_blocked, id=3907, stack(0x7f1c18115000,0x7f1c18216000)]
0x7f1c1c06f800 JavaThread "Finalizer" daemon
[_thread_blocked, id=3906, stack(0x7f1c188f9000,0x7f1c189fa000)]
0x7f1c1c06b800 JavaThread "Reference Handler" daemon
[_thread_blocked, id=3905, stack(0x7f1c189fa000,0x7f1c18afb000)]
=>0x7f1c1c009800 JavaThread "main"
[_thread_in_native, id=3899, stack(0x7f1c2459e000,0x7f1c2469f000)]

указатель Thread*

статус
in_java
in_native
in_vm
blocked
trans

thread id

границы стека

тип и название
Heap
Heap
PSYoungGen
total 18944K, used 327K [0x0eb600000, 0x0ecb00000, 0x100000000)
eden space 16384K, 2% used [0x0eb600000,0x0eb651f28,0x0ec600000)
from space 2560K, 0% used [0x0ec880000,0x0ec880000,0x0ecb00000)
to
space 2560K, 0% used [0x0ec600000,0x0ec600000,0x0ec880000)
ParOldGen
total 41984K, used 0K [0x0c2200000, 0x0c4b00000, 0x0eb600000)
object space 41984K, 0% used [0x0c2200000,0x0c2200000,0x0c4b00000)
PSPermGen
total 21504K, used 2378K [0x0bd000000, 0x0be500000, 0x0c2200000)
object space 21504K, 11% used [0x0bd000000,0x0bd2529f0,0x0be500000)

текущие границы

предел
Code Cache
Code Cache [0x2aaaab977000, 0x2aaaabbe7000, 0x2aaaae977000)
total_blobs=701 nmethods=360 adapters=295 free_code_cache=48574272 largest_free_block=20224

nmethods + adapters + stubs

свободное место

количество скомпилированных методов
События компилятора
Compilation events (10 events):
Event: 0.135 Thread 0x7fb4090800
Event: 0.136 Thread 0x7fb4093800
Event: 0.140 Thread 0x7fb4090800
Event: 0.140 Thread 0x7fb4090800
Event: 0.144 Thread 0x7fb4090800
Event: 0.144 Thread 0x7fb4090800
Event: 0.145 Thread 0x7fb4090800
Event: 0.145 Thread 0x7fb4093800
Event: 0.146 Thread 0x7fb4090800
Event: 0.150 Thread 0x7fb4090800

конец
начало

5
6
nmethod
7
nmethod
8
nmethod
nmethod
10 %
nmethod

java.util.Random::nextInt (7 bytes)
demo1.NativeDiv$1::run (28 bytes)
5 0x7fb1060090 code [0x7fb10601e0, 0x7fb10602d8]
java.util.Random::nextInt (60 bytes)
7 0x7fb105fb50 code [0x7fb105fca0, 0x7fb105fe98]
demo1.NativeDiv::access$000 (6 bytes)
8 0x7fb1062290 code [0x7fb10623e0, 0x7fb1062448]
6 0x7fb105f510 code [0x7fb105f680, 0x7fb105f888]
demo1.NativeDiv::runLoop @ 2 (20 bytes)
10% 0x7fb10649d0 code [0x7fb1064b40, 0x7fb1064ec8]

on-stack replacement

границы скомпилированного кода
Прочие события
GC Heap History (10 events):

Deoptimization events (10 events):
Internal exceptions (10 events):
Events
Event:
Event:
Event:
Event:
Event:
Event:
Event:
Event:
Event:
Event:

(10 events):
0.113 loading
0.114 loading
0.114 loading
0.114 loading
0.115 loading
0.115 loading
0.115 loading
0.115 loading
0.119 loading
0.119 loading

class
class
class
class
class
class
class
class
class
class

0x7f03b86cb520
0x7f03b86cb520
0x7f03b869aa50
0x7f03b869aa50
0x7f03b8682cd0
0x7f03b8682cd0
0x7f03b8682c70
0x7f03b8682c70
0x7f03b40a6c70
0x7f03b40a6c70

done
done
done
done

done
Карта памяти
Dynamic libraries:
00400000-00401000 r-xp 00000000 08:03 156103
00600000-00601000 rw-p 00000000 08:03 156103
00bc6000-00be7000 rw-p 00000000 00:00 0
7f1bdc000000-7f1bdc021000 rw-p 00000000 00:00
7f1bdc021000-7f1be0000000 ---p 00000000 00:00
7f1c1197b000-7f1c1197c000 r-xp 00000000 00:15
7f1c11b7c000-7f1c11b7d000 rw-p 00001000 00:15
7f1c11b7e000-7f1c11c7e000 rw-p 00000000 00:00
7f1c11c81000-7f1c11d7f000 rw-p 00000000 00:00
7f1c14000000-7f1c14021000 rw-p 00000000 00:00
7f1c14021000-7f1c18000000 ---p 00000000 00:00
7f1c20c45000-7f1c20e02000 r--s 039d3000 08:03
7f1c24494000-7f1c244b6000 r-xp 00000000 08:03
7f1c246b7000-7f1c246b9000 rw-p 00023000 08:03

область памяти

права доступа
read
write
execute
private
shared

/usr/java/jdk1.7.0_40/bin/java
/usr/java/jdk1.7.0_40/bin/java
[heap]
0
0
287
287
0
0
0
0
156934
265734
265734

inode
смещение
в файле
id устройства
(/dev/hda3)

/media/crash/lib/libdiv.so
/media/crash/lib/libdiv.so
[stack:3911]
[stack:3910]

/usr/java/jdk1.7.0_40/jre/lib/rt.jar
/lib/x86_64-linux-gnu/ld-2.15.so
/lib/x86_64-linux-gnu/ld-2.15.so

отображаемый файл
Окружение
VM Arguments:
jvm_args: -Djava.library.path=/media/crash/lib
java_command: demo1.NativeDiv
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/jdk1.7.0_40/bin
USERNAME=root
SHELL=/bin/bash
DISPLAY=:0
Память
/proc/meminfo:
MemTotal:
MemFree:
Buffers:
Cached:
SwapCached:
Active:
Inactive:
Unevictable:
Mlocked:
SwapTotal:
SwapFree:
Dirty:
Writeback:
AnonPages:
Mapped:
Shmem:
Slab:
SReclaimable:
SUnreclaim:

4048960
2783932
136360
437332
0
500384
405308
0
0
1500156
1500156
252
0
332152
89372
4920
289216
269068
20148

kB
kB
kB
kB
kB
kB
kB
kB
kB
kB
kB
kB
kB
kB
kB
kB
kB
kB
kB
CPU
CPU:total 4 (4 cores per cpu, 1 threads per core) family 6 model 58 stepping 9, cmov, cx8,
fxsr, mmx, sse, sse2, sse3, ssse3, tsc
/proc/cpuinfo:
processor
vendor_id
cpu family
model
model name
stepping
cpu MHz
cache size
physical id
siblings
core id
cpu cores
fpu
fpu_exception
cpuid level
wp
flags
clflush mmx fxsr sse

: 0
: GenuineIntel
: 6
: 58
: Intel(R) Core(TM) i7-3517U CPU @ 1.90GHz
: 9
: 2369.955
: 6144 KB
: 0
: 4
: 0
: 4
: yes
: yes
: 5
: yes
: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl pni ssse3 lahf_lm
Аварийный дамп – чёрный ящик упавшей JVM. Андрей Паньгин
x86-64 calling convention
•

http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/
src/cpu/x86/vm/assembler_x86.hpp
JNIEnv*

C args

c0

jobject this или
jclass holder

c1

c2

c3

c4

c5

Windows RCX

RDX

R8

R9

RDI*

RSI*

Linux

RSI

RDX

RCX

R8

R9

j0

j1

j2

j3

j4

RDI

Java args j5

* не C arg
Ошибки бывают у всех
Настройка отчёта об ошибке
• -XX:ErrorFile=./hs_err_pid%p.log
• -XX:OnError="cat hs_err_pid%p.log | mail my@email.com"
• -XX:+ShowMessageBoxOnError

• -XX:+CreateMinidumpOnCrash (только Windows)
• -XX:+UseOSErrorReporting
• -XX:+SuppressFatalErrorMessage
Спасибо!
• Блог
– http://habrahabr.ru/company/odnoklassniki/blog/

• Наш Open Source
– https://github.com/odnoklassniki

• Контакты
– andrey.pangin@odnoklassniki.ru

• Работа в Одноклассниках
– http://v.ok.ru

More Related Content

PDF
Hacklu11 Writeup
nkslides
 
PDF
Extreme JavaScript Performance
Thomas Fuchs
 
PDF
Rich and Snappy Apps (No Scaling Required)
Thomas Fuchs
 
PDF
ipython notebook poc memory forensics
Vincent Ohprecio
 
PDF
Java, Up to Date Sources
輝 子安
 
PDF
Systems and methods for electronic communications
Tal Lavian Ph.D.
 
PPTX
PyconKR 2018 Deep dive into Coroutine
Daehee Kim
 
Hacklu11 Writeup
nkslides
 
Extreme JavaScript Performance
Thomas Fuchs
 
Rich and Snappy Apps (No Scaling Required)
Thomas Fuchs
 
ipython notebook poc memory forensics
Vincent Ohprecio
 
Java, Up to Date Sources
輝 子安
 
Systems and methods for electronic communications
Tal Lavian Ph.D.
 
PyconKR 2018 Deep dive into Coroutine
Daehee Kim
 

What's hot (20)

PDF
Алексей Кутумов, Coroutines everywhere
Sergey Platonov
 
PPTX
12c Mini Lesson - Improved Error Handling in PLSQL
Connor McDonald
 
PDF
Ssaw08 0624
Atsushi Tadokoro
 
PDF
Towards Reusable Components With Aspects [ICSE 2008]
Kevin Hoffman
 
PDF
HotSpot template interpreter memos
ytoshima
 
PPTX
Random stability in systemVerilog and UVM based testbench
Kashyap Adodariya
 
PDF
TC74VCX244FT PSpice Model (Free SPICE Model)
Tsuyoshi Horigome
 
PDF
OpenIot & ELC Europe 2016 Berlin - How to develop the ARM 64bit board, Samsun...
Chanwoo Choi
 
PDF
The Ring programming language version 1.10 book - Part 16 of 212
Mahmoud Samir Fayed
 
PDF
The Ring programming language version 1.8 book - Part 65 of 202
Mahmoud Samir Fayed
 
PPTX
Hypercritical C++ Code Review
Andrey Karpov
 
PDF
openFrameworks、サウンド機能・音響合成、ofxMaxim, ofxOsc, ofxPd, ofxSuperCollider
Atsushi Tadokoro
 
PPTX
Der perfekte 12c trigger
syntegris information solutions GmbH
 
PDF
The Ring programming language version 1.6 book - Part 55 of 189
Mahmoud Samir Fayed
 
PDF
20120822 joxa
ericbmerritt
 
KEY
20110424 action scriptを使わないflash勉強会
Hiroki Mizuno
 
DOCX
Exception Handling in Scala
Nag Arvind Gudiseva
 
DOCX
Assignment
Ayesha Bhatti
 
PDF
SCC2017「両利きスマホアプリ開発のススメ」資料
Kenji Sakashita
 
TXT
Digital Voltmeter displaying voltage level on a seven segment display and com...
Karthik Rathinavel
 
Алексей Кутумов, Coroutines everywhere
Sergey Platonov
 
12c Mini Lesson - Improved Error Handling in PLSQL
Connor McDonald
 
Ssaw08 0624
Atsushi Tadokoro
 
Towards Reusable Components With Aspects [ICSE 2008]
Kevin Hoffman
 
HotSpot template interpreter memos
ytoshima
 
Random stability in systemVerilog and UVM based testbench
Kashyap Adodariya
 
TC74VCX244FT PSpice Model (Free SPICE Model)
Tsuyoshi Horigome
 
OpenIot & ELC Europe 2016 Berlin - How to develop the ARM 64bit board, Samsun...
Chanwoo Choi
 
The Ring programming language version 1.10 book - Part 16 of 212
Mahmoud Samir Fayed
 
The Ring programming language version 1.8 book - Part 65 of 202
Mahmoud Samir Fayed
 
Hypercritical C++ Code Review
Andrey Karpov
 
openFrameworks、サウンド機能・音響合成、ofxMaxim, ofxOsc, ofxPd, ofxSuperCollider
Atsushi Tadokoro
 
Der perfekte 12c trigger
syntegris information solutions GmbH
 
The Ring programming language version 1.6 book - Part 55 of 189
Mahmoud Samir Fayed
 
20120822 joxa
ericbmerritt
 
20110424 action scriptを使わないflash勉強会
Hiroki Mizuno
 
Exception Handling in Scala
Nag Arvind Gudiseva
 
Assignment
Ayesha Bhatti
 
SCC2017「両利きスマホアプリ開発のススメ」資料
Kenji Sakashita
 
Digital Voltmeter displaying voltage level on a seven segment display and com...
Karthik Rathinavel
 
Ad

Similar to Аварийный дамп – чёрный ящик упавшей JVM. Андрей Паньгин (9)

PDF
Рахманов Александр "Что полезного в разборе дампов для .NET-разработчиков?"
Yulia Tsisyk
 
PDF
Debugging TV Frame 0x01
Dmitry Vostokov
 
PPTX
Don't dump thread dumps
Tier1app
 
PPTX
TroubleshootingJVMOutages-3CaseStudies.pptx
Tier1 app
 
PPTX
TroubleshootingJVMOutages-3CaseStudies (1).pptx
Tier1 app
 
PPTX
Windows kernel debugging workshop in florida
Sisimon Soman
 
PDF
Debugging TV Frame 0x33
Dmitry Vostokov
 
PPTX
Secrets of building a debuggable runtime: Learn how language implementors sol...
Dev_Events
 
PPTX
Don't dump thread dumps
Tier1 App
 
Рахманов Александр "Что полезного в разборе дампов для .NET-разработчиков?"
Yulia Tsisyk
 
Debugging TV Frame 0x01
Dmitry Vostokov
 
Don't dump thread dumps
Tier1app
 
TroubleshootingJVMOutages-3CaseStudies.pptx
Tier1 app
 
TroubleshootingJVMOutages-3CaseStudies (1).pptx
Tier1 app
 
Windows kernel debugging workshop in florida
Sisimon Soman
 
Debugging TV Frame 0x33
Dmitry Vostokov
 
Secrets of building a debuggable runtime: Learn how language implementors sol...
Dev_Events
 
Don't dump thread dumps
Tier1 App
 
Ad

More from odnoklassniki.ru (16)

PDF
Distributed systems at ok.ru #rigadevday
odnoklassniki.ru
 
PDF
Тестирование аварий. Андрей Губа. Highload++ 2015
odnoklassniki.ru
 
PDF
Тюним память и сетевой стек в Linux: история перевода высоконагруженных серве...
odnoklassniki.ru
 
PDF
Распределенные системы в Одноклассниках
odnoklassniki.ru
 
PDF
Add a bit of ACID to Cassandra. Cassandra Summit EU 2014
odnoklassniki.ru
 
PDF
Кадры решают все, или стриминг видео в «Одноклассниках». Александр Тоболь
odnoklassniki.ru
 
PDF
За гранью NoSQL: NewSQL на Cassandra
odnoklassniki.ru
 
PDF
Платформа для видео сроком в квартал. Александр Тоболь.
odnoklassniki.ru
 
PDF
Франкенштейнизация Voldemort или key-value данные в Одноклассниках. Роман Ан...
odnoklassniki.ru
 
PDF
Being closer to Cassandra by Oleg Anastasyev. Talk at Cassandra Summit EU 2013
odnoklassniki.ru
 
PPTX
Управление тысячами серверов в Одноклассниках. Алексей Чудов.
odnoklassniki.ru
 
PDF
Класс!ная Cassandra
odnoklassniki.ru
 
PDF
Java Runtime: повседневные обязанности JVM
odnoklassniki.ru
 
PDF
Незаурядная Java как инструмент разработки высоконагруженного сервера
odnoklassniki.ru
 
PPT
Cистема внутренней статистики Odnoklassniki.ru
odnoklassniki.ru
 
PPTX
Как, используя Lucene, построить высоконагруженную систему поиска разнородных...
odnoklassniki.ru
 
Distributed systems at ok.ru #rigadevday
odnoklassniki.ru
 
Тестирование аварий. Андрей Губа. Highload++ 2015
odnoklassniki.ru
 
Тюним память и сетевой стек в Linux: история перевода высоконагруженных серве...
odnoklassniki.ru
 
Распределенные системы в Одноклассниках
odnoklassniki.ru
 
Add a bit of ACID to Cassandra. Cassandra Summit EU 2014
odnoklassniki.ru
 
Кадры решают все, или стриминг видео в «Одноклассниках». Александр Тоболь
odnoklassniki.ru
 
За гранью NoSQL: NewSQL на Cassandra
odnoklassniki.ru
 
Платформа для видео сроком в квартал. Александр Тоболь.
odnoklassniki.ru
 
Франкенштейнизация Voldemort или key-value данные в Одноклассниках. Роман Ан...
odnoklassniki.ru
 
Being closer to Cassandra by Oleg Anastasyev. Talk at Cassandra Summit EU 2013
odnoklassniki.ru
 
Управление тысячами серверов в Одноклассниках. Алексей Чудов.
odnoklassniki.ru
 
Класс!ная Cassandra
odnoklassniki.ru
 
Java Runtime: повседневные обязанности JVM
odnoklassniki.ru
 
Незаурядная Java как инструмент разработки высоконагруженного сервера
odnoklassniki.ru
 
Cистема внутренней статистики Odnoklassniki.ru
odnoklassniki.ru
 
Как, используя Lucene, построить высоконагруженную систему поиска разнородных...
odnoklassniki.ru
 

Recently uploaded (20)

PPTX
What-is-the-World-Wide-Web -- Introduction
tonifi9488
 
PPTX
AI and Robotics for Human Well-being.pptx
JAYMIN SUTHAR
 
PDF
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
 
PDF
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
PPTX
Agile Chennai 18-19 July 2025 | Emerging patterns in Agentic AI by Bharani Su...
AgileNetwork
 
PDF
Security features in Dell, HP, and Lenovo PC systems: A research-based compar...
Principled Technologies
 
PPTX
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
PDF
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
PPTX
Agile Chennai 18-19 July 2025 Ideathon | AI Powered Microfinance Literacy Gui...
AgileNetwork
 
PDF
Automating ArcGIS Content Discovery with FME: A Real World Use Case
Safe Software
 
PDF
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
PPTX
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
PDF
Unlocking the Future- AI Agents Meet Oracle Database 23ai - AIOUG Yatra 2025.pdf
Sandesh Rao
 
PDF
Doc9.....................................
SofiaCollazos
 
PPTX
The Future of AI & Machine Learning.pptx
pritsen4700
 
PDF
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
 
PDF
Presentation about Hardware and Software in Computer
snehamodhawadiya
 
PDF
GDG Cloud Munich - Intro - Luiz Carneiro - #BuildWithAI - July - Abdel.pdf
Luiz Carneiro
 
PDF
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
PDF
Research-Fundamentals-and-Topic-Development.pdf
ayesha butalia
 
What-is-the-World-Wide-Web -- Introduction
tonifi9488
 
AI and Robotics for Human Well-being.pptx
JAYMIN SUTHAR
 
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
 
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
Agile Chennai 18-19 July 2025 | Emerging patterns in Agentic AI by Bharani Su...
AgileNetwork
 
Security features in Dell, HP, and Lenovo PC systems: A research-based compar...
Principled Technologies
 
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
Agile Chennai 18-19 July 2025 Ideathon | AI Powered Microfinance Literacy Gui...
AgileNetwork
 
Automating ArcGIS Content Discovery with FME: A Real World Use Case
Safe Software
 
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
Unlocking the Future- AI Agents Meet Oracle Database 23ai - AIOUG Yatra 2025.pdf
Sandesh Rao
 
Doc9.....................................
SofiaCollazos
 
The Future of AI & Machine Learning.pptx
pritsen4700
 
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
 
Presentation about Hardware and Software in Computer
snehamodhawadiya
 
GDG Cloud Munich - Intro - Luiz Carneiro - #BuildWithAI - July - Abdel.pdf
Luiz Carneiro
 
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
Research-Fundamentals-and-Topic-Development.pdf
ayesha butalia
 

Аварийный дамп – чёрный ящик упавшей JVM. Андрей Паньгин

  • 1. Аварийный дамп – чёрный ящик упавшей JVM Андрей Паньгин Одноклассники, ведущий инженер
  • 2. Java vs. C •  Crash! array[999999] = 42; array[999999] = 42;  ArrayIndexOutOfBoundsException • nullRef->callMethod();  NullPointerException • nullRef.callMethod();  Crash! int y = x / 0; int y = x / 0;  ArithmeticException  Crash!
  • 4. Нет, только не это! # # # # # # # # # # # # # # # # A fatal error has been detected by the Java Runtime Environment: SIGSEGV (0xb) at pc=0x00002b47f02da0c3, pid=20644, tid=1096538432 JRE version: 6.0_26-b03 Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64) Problematic frame: V [libjvm.so+0x8400c3] An error report file with more information is saved as: /one/bin/hs_err_pid20644.log If you would like to submit a bug report, please visit: http://java.sun.com/webapps/bugreport/crash.jsp
  • 5. Фатальные ошибки 1. Проблема в native коде 2. Некорректное использование закрытого API (например, sun.misc.Unsafe) 3. Баг JVM (да, встречали не раз!) 4. Проблема с «железом» (и такое бывало!)
  • 7. Заголовок # # A fatal error has been detected by the Java Runtime Environment: # # SIGFPE (0x8) at pc=0x00007f1c1197b585, pid=3898, tid=139758846732032 # # JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 compressed oops) # код сигнала адрес инструкции id процесса id потока сборка JRE и JVM
  • 8. Фрейм # Problematic frame: # C [libdiv.so+0x585] Java_demo1_NativeDiv_div+0x5 адрес инструкции (относительно DLL и символа) тип фрейма С Native C frame V VM function v VM generated stub j Interpreted Java frame J Compiled Java frame
  • 9. Сигнал siginfo:si_signo=SIGFPE: si_errno=0, si_code=1 (FPE_INTDIV), si_addr=0x00007f1c1197b585 адрес инструкции или обращения к памяти (для SEGV, SIGBUS) код сигнала Linux Windows SIGSEGV EXCEPTION_ACCESS_VIOLATION EXCEPTION_STACK_OVERFLOW SIGBUS EXCEPTION_ACCESS_VIOLATION EXCEPTION_DATATYPE_MISALIGNMENT SEGV_MAPERR отсутствующая страница SEGV_ACCERR нет прав доступа к странице SIGILL EXCEPTION_ILLEGAL_INSTRUCTION EXCEPTION_PRIV_INSTRUCTION BUS_ADRALN невыровненный адрес SIGFPE EXCEPTION_INT_* EXCEPTION_FLT_* BUS_ADRERR несуществующий адрес ILL_ILLOPC неверная инструкция ILL_PRVOPC привилегированная инструкция FPE_INTDIV целочисленное деление на 0 FPE_FLTDIV вещественное деление на 0 причина сигнала
  • 10. Регистры CPU Registers: RAX=0x00000000c5448e8f, RSP=0x00007f1c2469d798, R8 =0x0000c5448e8f8389, R12=0x0000000000000000, RIP=0x00007f1c1197b585, RBX=0x00000000bd250ff0, RCX=0x0000000000000000, RDX=0x00000000ffffffff RBP=0x00007f1c2469d7f0, RSI=0x00007f1c2469d800, RDI=0x00007f1c1c0099e8 R9 =0x00000850b3941760, R10=0x00007f1c1901270c, R11=0x00007f1c234de260 R13=0x00000000bd250ff0, R14=0x00007f1c2469d818, R15=0x00007f1c1c009800 EFLAGS=0x0000000000010287, CSGSFS=0x0000000000000033
  • 11. Верхушка стека Top of Stack: (sp=0x00007f1c2469d798) 0x00007f1c2469d798: 00007f1c19012738 0x00007f1c2469d7a8: 0000000000000000 0x00007f1c2469d7b8: 0000000000000000 0x00007f1c2469d7c8: 00000000bd251480 0x00007f1c2469d7d8: 00000000bd250ff0 0x00007f1c2469d7e8: 00007f1c2469d810 0x00007f1c2469d7f8: 00007f1c190061d4 0x00007f1c2469d808: 00007f1c1900ecd6 00000000bd1a53e8 00007f1c2469d7b0 00007f1c2469d818 0000000000000000 0000000000000000 00007f1c2469d860 00000000eb64ad68 0000000000000000
  • 12. Машинный код Instructions: (pc=0x00007f1c1197b585) 0x00007f1c1197b565: 48 85 c0 74 0e 5d 0x00007f1c1197b575: 1f 40 00 5d c3 90 0x00007f1c1197b585: f7 f9 c3 90 90 90 0x00007f1c1197b595: 48 83 ec 08 48 8b 48 90 90 05 8d 90 90 78 3d 90 90 08 c6 90 90 20 08 90 90 00 20 89 55 48 00 d0 48 83 ff c1 89 f8 e0 fa e5 ff 0f 1f 53 74 до после • Дизассемблер – http://www.onlinedisassembler.com • Описание инструкций – http://www.intel.com/content/www/us/en/processors/ architectures-software-developer-manuals.html
  • 14. Значения регистров Register to memory mapping: RAX=0x00000000c5448e8f is an unallocated location in the heap RBX=0x00000000bd250ff0 is an oop {method} - klass: {other class} RCX=0x0000000000000000 is an unknown value RDX=0x00000000ffffffff is an unallocated location in the heap RSP=0x00007f1c2469d798 is pointing into the stack for thread: 0x00007f1c1c009800 RBP=0x00007f1c2469d7f0 is pointing into the stack for thread: 0x00007f1c1c009800 RSI=0x00007f1c2469d800 is pointing into the stack for thread: 0x00007f1c1c009800 RDI=0x00007f1c1c0099e8 is an unknown value R8 =0x0000c5448e8f8389 is an unknown value R9 =0x00000850b3941760 is an unknown value R10=0x00007f1c1901270c is at code_begin+620 in an Interpreter codelet method entry point (kind = native) [0x00007f1c190124a0, 0x00007f1c19012d00] 2144 bytes R11=0x00007f1c234de260: <offset 0x8a7260> in libjvm.so at 0x00007f1c22c37000 R12=0x0000000000000000 is an unknown value R13=0x00000000bd250ff0 is an oop {method} - klass: {other class} R14=0x00007f1c2469d818 is pointing into the stack for thread: 0x00007f1c1c009800 R15=0x00007f1c1c009800 is a thread
  • 15. Стек границы стека Stack: [0x7f1c2459e000,0x7f1c2469f000], указатель стека sp=0x7f1c2469d798, free space=1021k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libdiv.so+0x585] Java_demo1_NativeDiv_div+0x5 j demo1.NativeDiv.access$000(II)I+2 смещение в байтах j demo1.NativeDiv$1.run()V+20 j demo1.NativeDiv.runLoop(ILjava/lang/Runnable;)V+8 j demo1.NativeDiv.main([Ljava/lang/String;)V+21 индекс байткода v ~StubRoutines::call_stub V [libjvm.so+0x5f8405] JavaCalls::call_helper()+0x365 V [libjvm.so+0x5f6e68] JavaCalls::call()+0x28 V [libjvm.so+0x62f8d9] jni_invoke_static()+0x219 V [libjvm.so+0x638962] jni_CallStaticVoidMethod+0x162 C [libjli.so+0x36d9] JavaMain+0x7e9 тип фрейма декодированный адрес (DLL, символ + смещение)
  • 16. Потоки Java Threads: ( => current thread ) 0x7f1c1c096000 JavaThread "Service Thread" daemon [_thread_blocked, id=3910, stack(0x7f1c11c7e000,0x7f1c11d7f000)] 0x7f1c1c093800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3909, stack(0x7f1c11d7f000,0x7f1c11e80000)] 0x7f1c1c090800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=3908, stack(0x7f1c18014000,0x7f1c18115000)] 0x7f1c1c08e800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3907, stack(0x7f1c18115000,0x7f1c18216000)] 0x7f1c1c06f800 JavaThread "Finalizer" daemon [_thread_blocked, id=3906, stack(0x7f1c188f9000,0x7f1c189fa000)] 0x7f1c1c06b800 JavaThread "Reference Handler" daemon [_thread_blocked, id=3905, stack(0x7f1c189fa000,0x7f1c18afb000)] =>0x7f1c1c009800 JavaThread "main" [_thread_in_native, id=3899, stack(0x7f1c2459e000,0x7f1c2469f000)] указатель Thread* статус in_java in_native in_vm blocked trans thread id границы стека тип и название
  • 17. Heap Heap PSYoungGen total 18944K, used 327K [0x0eb600000, 0x0ecb00000, 0x100000000) eden space 16384K, 2% used [0x0eb600000,0x0eb651f28,0x0ec600000) from space 2560K, 0% used [0x0ec880000,0x0ec880000,0x0ecb00000) to space 2560K, 0% used [0x0ec600000,0x0ec600000,0x0ec880000) ParOldGen total 41984K, used 0K [0x0c2200000, 0x0c4b00000, 0x0eb600000) object space 41984K, 0% used [0x0c2200000,0x0c2200000,0x0c4b00000) PSPermGen total 21504K, used 2378K [0x0bd000000, 0x0be500000, 0x0c2200000) object space 21504K, 11% used [0x0bd000000,0x0bd2529f0,0x0be500000) текущие границы предел
  • 18. Code Cache Code Cache [0x2aaaab977000, 0x2aaaabbe7000, 0x2aaaae977000) total_blobs=701 nmethods=360 adapters=295 free_code_cache=48574272 largest_free_block=20224 nmethods + adapters + stubs свободное место количество скомпилированных методов
  • 19. События компилятора Compilation events (10 events): Event: 0.135 Thread 0x7fb4090800 Event: 0.136 Thread 0x7fb4093800 Event: 0.140 Thread 0x7fb4090800 Event: 0.140 Thread 0x7fb4090800 Event: 0.144 Thread 0x7fb4090800 Event: 0.144 Thread 0x7fb4090800 Event: 0.145 Thread 0x7fb4090800 Event: 0.145 Thread 0x7fb4093800 Event: 0.146 Thread 0x7fb4090800 Event: 0.150 Thread 0x7fb4090800 конец начало 5 6 nmethod 7 nmethod 8 nmethod nmethod 10 % nmethod java.util.Random::nextInt (7 bytes) demo1.NativeDiv$1::run (28 bytes) 5 0x7fb1060090 code [0x7fb10601e0, 0x7fb10602d8] java.util.Random::nextInt (60 bytes) 7 0x7fb105fb50 code [0x7fb105fca0, 0x7fb105fe98] demo1.NativeDiv::access$000 (6 bytes) 8 0x7fb1062290 code [0x7fb10623e0, 0x7fb1062448] 6 0x7fb105f510 code [0x7fb105f680, 0x7fb105f888] demo1.NativeDiv::runLoop @ 2 (20 bytes) 10% 0x7fb10649d0 code [0x7fb1064b40, 0x7fb1064ec8] on-stack replacement границы скомпилированного кода
  • 20. Прочие события GC Heap History (10 events): Deoptimization events (10 events): Internal exceptions (10 events): Events Event: Event: Event: Event: Event: Event: Event: Event: Event: Event: (10 events): 0.113 loading 0.114 loading 0.114 loading 0.114 loading 0.115 loading 0.115 loading 0.115 loading 0.115 loading 0.119 loading 0.119 loading class class class class class class class class class class 0x7f03b86cb520 0x7f03b86cb520 0x7f03b869aa50 0x7f03b869aa50 0x7f03b8682cd0 0x7f03b8682cd0 0x7f03b8682c70 0x7f03b8682c70 0x7f03b40a6c70 0x7f03b40a6c70 done done done done done
  • 21. Карта памяти Dynamic libraries: 00400000-00401000 r-xp 00000000 08:03 156103 00600000-00601000 rw-p 00000000 08:03 156103 00bc6000-00be7000 rw-p 00000000 00:00 0 7f1bdc000000-7f1bdc021000 rw-p 00000000 00:00 7f1bdc021000-7f1be0000000 ---p 00000000 00:00 7f1c1197b000-7f1c1197c000 r-xp 00000000 00:15 7f1c11b7c000-7f1c11b7d000 rw-p 00001000 00:15 7f1c11b7e000-7f1c11c7e000 rw-p 00000000 00:00 7f1c11c81000-7f1c11d7f000 rw-p 00000000 00:00 7f1c14000000-7f1c14021000 rw-p 00000000 00:00 7f1c14021000-7f1c18000000 ---p 00000000 00:00 7f1c20c45000-7f1c20e02000 r--s 039d3000 08:03 7f1c24494000-7f1c244b6000 r-xp 00000000 08:03 7f1c246b7000-7f1c246b9000 rw-p 00023000 08:03 область памяти права доступа read write execute private shared /usr/java/jdk1.7.0_40/bin/java /usr/java/jdk1.7.0_40/bin/java [heap] 0 0 287 287 0 0 0 0 156934 265734 265734 inode смещение в файле id устройства (/dev/hda3) /media/crash/lib/libdiv.so /media/crash/lib/libdiv.so [stack:3911] [stack:3910] /usr/java/jdk1.7.0_40/jre/lib/rt.jar /lib/x86_64-linux-gnu/ld-2.15.so /lib/x86_64-linux-gnu/ld-2.15.so отображаемый файл
  • 22. Окружение VM Arguments: jvm_args: -Djava.library.path=/media/crash/lib java_command: demo1.NativeDiv Launcher Type: SUN_STANDARD Environment Variables: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/jdk1.7.0_40/bin USERNAME=root SHELL=/bin/bash DISPLAY=:0
  • 24. CPU CPU:total 4 (4 cores per cpu, 1 threads per core) family 6 model 58 stepping 9, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, tsc /proc/cpuinfo: processor vendor_id cpu family model model name stepping cpu MHz cache size physical id siblings core id cpu cores fpu fpu_exception cpuid level wp flags clflush mmx fxsr sse : 0 : GenuineIntel : 6 : 58 : Intel(R) Core(TM) i7-3517U CPU @ 1.90GHz : 9 : 2369.955 : 6144 KB : 0 : 4 : 0 : 4 : yes : yes : 5 : yes : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl pni ssse3 lahf_lm
  • 26. x86-64 calling convention • http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/ src/cpu/x86/vm/assembler_x86.hpp JNIEnv* C args c0 jobject this или jclass holder c1 c2 c3 c4 c5 Windows RCX RDX R8 R9 RDI* RSI* Linux RSI RDX RCX R8 R9 j0 j1 j2 j3 j4 RDI Java args j5 * не C arg
  • 28. Настройка отчёта об ошибке • -XX:ErrorFile=./hs_err_pid%p.log • -XX:OnError="cat hs_err_pid%p.log | mail [email protected]" • -XX:+ShowMessageBoxOnError • -XX:+CreateMinidumpOnCrash (только Windows) • -XX:+UseOSErrorReporting • -XX:+SuppressFatalErrorMessage
  • 29. Спасибо! • Блог – http://habrahabr.ru/company/odnoklassniki/blog/ • Наш Open Source – https://github.com/odnoklassniki • Контакты – [email protected] • Работа в Одноклассниках – http://v.ok.ru