101 apend. troubleshooting tools v2

CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Linux Essenciais and System Administration

Key Knowledge Areas
Understand the correct way to troubleshoot the system and the network
Troubleshooting tools
Terms and Utilities
uptime
free
w
vmstat
top
df
ldd, ldconfig
ulimit
Iostat
/proc/*
debugfs
Strace
Logs – Syslog; syslog.conf; var/log
Network tools
2

Troubleshooting overview
3
Troubleshooting: a mystical art :/ in details
It is a thorough methodology used to track down the cause of problem.
Keywords: thorough and methodology
•Without a thorough and exhaustive approach, the issue might be overlooked
•Without a strong and methodical approach, the issue may be misdiagnosed
• Most Important: Only change one thing at a time
• Check #1 most likely cause: You
• Check logs for error messages
• After that, check configuration and permissions
• If all else fails, slowly, piece by piece, start removing complexity from the system to
narrow down the problem area.
• DOCUMENT EVERYTHING

LOGS
4
One of the easiest places to find the cause of a problem is in the log files.
Log files store informational messages from software.
The types of messages include debug info, status info, warnings, errors and more.
•Some Apps use the system-wide logging package.
•Others manage their own log files

system-wide LOGS: syslog
5
syslog - The system logger.
A framework consisting of a library, a daemon, a configuration file and logs.
Any application can use the library and log messages through syslog with simple
function calls. Log messages consist of 3 parts:
•Facility
•Level
•Message
Facility Describes the part of operating system that generated the message,
and is selected by the software:
auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0-local7
Level Represents the importance of the message, and is also chosen by the software:
emergency, alert, critical, error, warning, notice, info, debug

system-wide LOGS: /etc/syslog.conf
6
Massage
/etc/syslog.conf defines where all of the log messages should go.
Destinations include files, screens of logged in users, console, other syslog servers.
Basic file format:
•facility.level destination
Ex:
*.err /dev/console
mail.* /var/log/maillog
*.info;mail.none;authpriv.none /var/log/messages

system-wide LOGS: /var/log
7
• maillog messages from the email subsystem
• secure authentication and security messages
• cron cron messages
• boot.log boot messages
• Messages catch-all

apps own LOGS
8
not all software uses the syslog framework to handle it’s logging.
Some software manages it’s own logs.
Can be difficult to track down all of the log locations on an unfamiliar system.
Best way to handle this is to start from the init scripts.
LOCATING APPLICATION LOGS:
1.To track log file local for an apps, find it’s configuration file and see where the logs are being written.
2.Finding the configuration file might be difficult, so it’s best to start at source.
3.init starts all of system services, so there is an init script that is starting up the apps in question.
4.The init script almost always references the configuration file
5.If the configuration file location is known, scan it and find out where the logs are being written.
6.The format of the log file its completely dependent on the application.
Some will be similar to syslog, others, like Apache or Qmail, will be completely foreign looking.

troubleshooting
9
several tools to help the process of troubleshooting:
uptime; free; w; vmstat; top; df; ldd; ldconfig; ulimit; iostat
“CSI” troubleshooting:
/proc/*
debugfs
Strace
Looking through logs is fine, but really that’s a best case scenario.
The software and hardware rarely come out and announce problems and solutions in log
files.
No, it’s not easy! 
More often, users will encounter symptoms of a problem, and there are some tools that
COULD FIND and FIX the issue.

tools
10
uptime Reports system uptime along with load averages.
•Load Average: Average number of processes in run queue that are blocked.
•uptime reports three values: the load averaged over the last 1 minute, 5 minutes and 15
minutes. This is useful to get an idea of the load trend on the system.
free reports on memory and swap usage
•buffers: I/O buffers, directory cache
•cached: filesystem cache (data)
[root@dev1 ~]# uptime
16:09:55 up 682 days, 10:11, 1 user, load average: 0.00, 0.01, 0.00
[root@dev1 ~]#
Ex:
[[root@dev1 ~]# free
total used free shared buffers cached
Mem: 262316 214228 48088 0 1168
41728
-/+ buffers/cache: 171332 90984
Swap: 524280 74564 449716
[root@dev1 ~]#
Ex:

tools
11
w uptime report, with breakdown of logged-in users and process they are running
•JCPU: Combined CPU time of all processes attached to terminal (foreground and background)
•PCPU: CPU time of foreground process, listed in “what” column
vmstat Snapshot report covering several primary statistics.
•procs: number of running and blocked processes
•swap: swapped in and swapped out blocks of memory, per second
•io: blocks in and blocks out read/written per second
•system: interrupts and context switches per second
•cpu: user, system, idle, wait and time-stolen from a VM
[root@dev1 ~]# w
16:26:42 up 682 days, 10:28, 2 users, load average: 0.02, 0.05, 0.02
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 216-110-93-126.s 16:00 3:57 0.01s 0.01s -bash
root pts/9 216-110-93-126.s 16:22 0.00s 0.01s 0.00s w
Ex:
[root@dev1 ~]# vmstat
procs ----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 74564 3608 4456 70156 0 0 0 2 0 0 0 0 100 0 0
Ex:

tools
12
top Self-updating tool displays combination summary at top, followed by ordered
list of processes. Fully customizable.
• The summary includes uptime information, memory
• breakdowns, CPU utilization and process state summaries
top - 16:39:32 up 682 days, 10:41, 2 users, load average: 0.01, 0.00, 0.00
Tasks: 118 total, 1 running, 116 sleeping, 1 stopped, 0 zombie
Cpu(s): 0.1%us, 0.0%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.1%st
Mem: 262316k total, 258024k used, 4292k free, 7380k buffers
Swap: 524280k total, 74564k used, 449716k free, 67808k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 15 0 10316 648 592 S 0 0.2 0:06.24 init
2 root RT 0 0 0 0 S 0 0.0 0:04.88 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.19 ksoftirqd/0
Ex:

tools
13
df lists filesystem utilization
•Breaks down size and use information for each mounted filesystem
•-h is useful option to display in “human-friendly” format
[root@dev1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.4G 7.2G 1.8G 81% /
none 129M 0 129M 0% /dev/shm
Ex:
ldd List library dependencies
Ldconfig Update library location database
•/etc/ld.so.conf and /etc/ld.so.conf.d/*.conf for list of pathnames to search
for libraries, creates database for dynamic linker
[root@dev1 ~]# ldd /bin/bash
! libtermcap.so.2 => /lib64/libtermcap.so.2 (0x00002ac044572000)
! libdl.so.2 => /lib64/libdl.so.2 (0x00002ac044775000)
! libc.so.6 => /lib64/libc.so.6 (0x00002ac044979000)
! /lib64/ld-linux-x86-64.so.2 (0x00002ac044357000)
[root@dev1 ~]# cat /etc/ld.so.conf.d/mysql-x86_64.conf
/usr/lib64/mysql
[root@dev1 ~]# ldconfig
Ex:

tools
14
ulimit Sets resource limits
•Can limit open files, memory use, cpu time, subprocesses and more.
[root@dev1 ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 2112
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
max rt priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2112
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Ex:

tools
15
iostat IO statistics report
•Part of the sysstat package; not always installed
•Allows drilldown into IO system to view real time metrics on IO operations per filesystem
[root@dev1 ~]# iostat -x
Linux 2.6.18-xen (dev1) ! 12/10/09
avg-cpu: %user %nice %system %iowait %steal %idle
0.05 0.00 0.00 0.03 0.07 99.84
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda1 0.00 1.68 0.01 0.55 0.14 17.83 32.12 0.03 54.01 2.89 0.16
sda2 0.00 0.00 0.00 0.00 0.01 0.01 35.26 0.00 80.51 4.95 0.00
Ex:

/proc/*
16
/proc folder contains amounts of information useful for troubleshooting.
•/proc/meminfo: Memory utilization breakdown
•/proc/devices: Mapping major numbers to drivers
•/proc/dma: dma channel assignments
•/proc/ioports: io port assignments
/proc folder has detailed information on every process on the system.
Details on process status, environment, commandline, and more – man /proc

Debugfs and strace
17
debugfs: Very powerful filesystem debugging tool.
•Allows direct visualization and manipulation of the filesystem internals
•Extremely powerful, extremely  dangerous.
strace: Traces each library call a process makes.
•Extremely useful to see what a process is doing
•Can find errors, bugs, permission issues and more

Networking responsibilities
18
Troubleshooting NETWORK tools
Networking systems together is often a difficult task, further complicated by large
networks and special requirements.
For this reason, networking is it’s own area of expertise.
The network engineer is responsible for everything up to and including the cable
and plug connecting to the server.
The systems engineer is responsible for everything within the server, up to and
including the network card interfacing to the cable.

Basic network troubleshooting
19
Basic network troubleshooting is verifying three aspects of network performance:
•LAN access
•Inter-LAN access
•DNS service
ping: “Packet Internet Groper”
Using IP/ICMP echo requests and echo replies, times response time between two machines.
ping 192.168.0.1
Times reported are Round Trip Times (RTT) and represent the time between sending a
request and receiving a response.
traceroute
Traces the route a message takes to get from source machine to the destination.
netstat
Network statistics - details on open and recently closed network connections
iptraf
network statistics tool

Basic Network troubleshooting
20
nmap
Network mapper - for seeing what services are showing on a particular machine
tcpdump
A tool to dump raw network traffic for analysis
wireshark
GUI interface to a tcpdump-like tool
ntop
Top-like command for network connections
ngrep
grep for network connections

Fim de sessão
21

101 apend. troubleshooting tools v2

More Related Content

What's hot (19)

Viewers also liked (13)

Similar to 101 apend. troubleshooting tools v2 (20)

More from Acácio Oliveira (20)

Recently uploaded (20)

101 apend. troubleshooting tools v2