Handling Performance Issues - Linux

The first thing we need to troubleshoot performance issues on any server is Patience,bcuz servers may not be responding well due to slowness or heavy process running on the server.
Here we are going to see how to troubleshoot the Redhat Linux performance issues.The performance issue mainly raises due to system resource shortage. If the application is not properly configured according to the system resources ,then definitely  the system will encounter these kind of issues.As a system admin,we need to figure out what kind of resource shortage we have and what are process are making the system in to such a  situations. 
Key System resources in Redhat Linux:
1. CPU 
2. Memory
3. Swap
4. Filesystem (Disk or LUN)
5. Network

1.CPU:
CPU utilization can be monitor using various in build Linux tools. top,vmstat  and “sar -u” are few of them.
To get the current CPU utilization details,

VMSTAT

[root@vinodhkumar ~]# vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 402432  45308 449980    0    0    92    54   74  112  1  2 94  3  0
 0  0      0 402424  45308 450008    0    0     0     0   54   82  1  0 99  0  0
 0  0      0 402424  45308 450008    0    0     0     0   33   52  0  0 100  0  0
 0  0      0 402432  45316 450004    0    0     0    72   59  109  0  1 99  0  0
 0  0      0 402432  45316 450008    0    0     0     0   37   59  0  0 100  0  0

SAR

[root@vinodhkumar ~]# sar -u 5 5
Linux 2.6.32-279.el6.x86_64 (vinodhkumar)     11/25/2013      _x86_64_        (1 CPU)
10:34:11 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:34:16 AM     all      0.00      0.00      0.80      0.00      0.00     99.20
10:34:21 AM     all      0.40      0.00      2.00      0.00      0.00     97.60
10:34:26 AM     all      0.00      0.00      1.41      0.00      0.00     98.59
10:34:31 AM     all      0.00      0.00      0.40      0.00      0.00     99.60
10:34:36 AM     all      0.00      0.00      0.80      1.00      0.00     98.20
Average:         all      0.08      0.00      1.08      0.20      0.00     98.63


TOP



top - 10:36:09 up  1:16,  4 users,  load average: 0.00, 0.01, 0.04
Tasks: 156 total,   1 running, 155 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.3%us,  1.5%sy,  0.0%ni, 93.9%id,  3.3%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   1250268k total,   841816k used,   408452k free,    45384k buffers
Swap:  2523128k total,        0k used,  2523128k free,   450152k cached


High CPU consuming process:



[root@vinodhkumar~]# top
top - 10:42:11 up  1:22,  4 users,  load average: 0.76, 0.22, 0.08
Tasks: 156 total,   2 running, 154 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1250268k total,   889884k used,   360384k free,    80212k buffers
Swap:  2523128k total,        0k used,  2523128k free,   450164k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6010 root      20   0  109m 1244  888 R 98.1  0.1   0:32.21 find
 1094 root      20   0     0    0    0 S  1.3  0.0   0:02.40 flush-253:0
 6031 root      20   0 15028 1296  964 R  0.7  0.1   0:00.03 top
 2680 root      20   0 40336  616  364 S  0.3  0.0   0:01.20 udisks-daemon
    1 root      20   0 19348 1564 1252 S  0.0  0.1   0:02.07 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.01 kthreadd


Using PS command,
[root@vinodhkumar ~]# ps -eo pcpu,args | sort -k 1 -r | head -8
%CPU COMMAND
90.5 find / -name temp_myname*
 0.2 /usr/sbin/vmtoolsd
 0.2 /usr/lib/vmware-tools/sbin64/vmtoolsd -n vmusr --blockFd 3
 0.1 /usr/sbin/restorecond -u
 0.0 [watchdog/0] 0.0 [vmmemctl] 0.0 /usr/sbin/wpa_supplicant -c /etc/wpa_supplicant/wpa_supplicant.conf -B -u -f /var/log/wpa_supplicant.log -P /var/run/wpa_supplicant.pid