Linux System performance checking
1. VMSTAT - vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity.
eg: vmstat -SM 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 218 309 48 922 0 0 103 103 878 1020 17 5 77 2 0
1 0 218 309 48 922 0 0 0 0 495 1677 3 1 96 0 0
FIELD DESCRIPTION FOR VM MODE
Procs
r: The number of runnable processes (running or waiting for run time).
b: The number of processes in uninterruptible sleep.
Memory
swpd: the amount of virtual memory used.
free: the amount of idle memory.
buff: the amount of memory used as buffers.
cache: the amount of memory used as cache.
inact: the amount of inactive memory. (-a option)
active: the amount of active memory. (-a option)
Swap
si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).
IO
bi: Blocks received from a block device (blocks/s).
bo: Blocks sent to a block device (blocks/s).
System
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.
CPU
These are percentages of total CPU time.
us: Time spent running non-kernel code. (user time, including nice time)
sy: Time spent running kernel code. (system time)
id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.
st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.
2. IOSTAT - Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.
eg: iostat -kx 3
avg-cpu: %user %nice %system %iowait %steal %idle
15.91 0.41 4.37 1.50 0.00 77.81
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.55 5.60 4.52 4.78 116.76 116.07 50.11 0.59 63.04 19.66 104.05 6.40 5.95
dm-0 0.00 0.00 4.23 4.74 103.21 94.90 44.19 0.60 66.74 20.44 107.99 6.45 5.78
dm-1 0.00 0.00 1.76 5.26 7.08 21.03 8.01 4.71 672.15 13.89 892.31 0.50 0.35
CPU Utilization Report
The first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors. The report has the following format:
%user
Show the percentage of CPU utilization that occurred while executing at the user level (application).
%nice
Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
%system
Show the percentage of CPU utilization that occurred while executing at the system level (kernel).
%iowait
Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
%steal
Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
%idle
Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
Device Utilization Report
Device:
This column gives the device (or partition) name as listed in the /dev directory.
tps
Indicate the number of transfers per second that were issued to the device. A transfer is an I/O request to the device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.
Blk_read/s (kB_read/s, MB_read/s)
Indicate the amount of data read from the device expressed in a number of blocks (kilobytes, megabytes) per second. Blocks are equivalent to sectors and therefore have a size of 512 bytes.
Blk_wrtn/s (kB_wrtn/s, MB_wrtn/s)
Indicate the amount of data written to the device expressed in a number of blocks (kilobytes, megabytes) per second.
Blk_read (kB_read, MB_read)
The total number of blocks (kilobytes, megabytes) read.
Blk_wrtn (kB_wrtn, MB_wrtn)
The total number of blocks (kilobytes, megabytes) written.
rrqm/s
The number of read requests merged per second that were queued to the device.
wrqm/s
The number of write requests merged per second that were queued to the device.
r/s
The number (after merges) of read requests completed per second for the device.
w/s
The number (after merges) of write requests completed per second for the device.
rsec/s (rkB/s, rMB/s)
The number of sectors (kilobytes, megabytes) read from the device per second.
wsec/s (wkB/s, wMB/s)
The number of sectors (kilobytes, megabytes) written to the device per second.
avgrq-sz
The average size (in sectors) of the requests that were issued to the device.
avgqu-sz
The average queue length of the requests that were issued to the device.
await
The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
r_await
The average time (in milliseconds) for read requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
w_await
The average time (in milliseconds) for write requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
svctm
The average service time (in milliseconds) for I/O requests that were issued to the device. Warning! Do not trust this field any more. This field will be removed in a future sysstat version.
%util
Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.
3. MPSTAT - Report processors related statistics.
eg: mpstat 2 2
03:06:47 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
03:06:49 PM all 2.77 0.00 0.63 0.13 0.00 0.00 0.00 0.00 0.00 96.47
03:06:51 PM all 3.13 0.00 0.50 0.13 0.00 0.00 0.00 0.00 0.00 96.24
Average: all 2.95 0.00 0.57 0.13 0.00 0.00 0.00 0.00 0.00 96.36
Report CPU utilization.
The following values are displayed:
CPU
Processor number. The keyword all indicates that statistics are calculated as averages among all processors.
%usr
Show the percentage of CPU utilization that occurred while executing at the user level (application).
%nice
Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
%sys
Show the percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time spent servicing hardware and software interrupts.
%iowait
Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
%irq
Show the percentage of time spent by the CPU or CPUs to service hardware interrupts.
%soft
Show the percentage of time spent by the CPU or CPUs to service software interrupts.
%steal
Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
%guest
Show the percentage of time spent by the CPU or CPUs to run a virtual processor.
%gnice
Show the percentage of time spent by the CPU or CPUs to run a niced guest.
%idle
Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
4. ULIMIT - The ulimit and sysctl programs allow to limit system-wide resource use. This can help a lot in system administration, e.g. when a user starts too many processes and therefore makes the system unresponsive for other users.
eg: ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 14726
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
5. LSOF - lists on its standard output file information about files opened by processes
A file is opened when the process or program stored in it is executed and closed when it is no longer required or the associated process or program has terminated. To determine the information such as which files are open, which processes are using them, and who the owners are, the lsof (list open files) command is used.
eg: lsof -np PID i.e lsof -np 22266
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
vlc 22266 naveen cwd DIR 253,0 4096 3307693 /home/naveen
vlc 22266 naveen rtd DIR 253,0 4096 192 /
vlc 22266 naveen txt REG 253,0 15840 71886477 /usr/bin/vlc
6. STRACE - trace system calls and signals.
eg: strace -p PID i.e. strace -p 22266
Process 22266 attached
rt_sigtimedwait([HUP INT QUIT TERM], NULL, NULL, 8
Understanding Linux process
Viewing Processes
Heading
|
Description
|
UID
|
User ID of the process owner.
|
PID
|
Process ID of the process.
|
PPID
|
Process ID of the parent process.
|
C
|
The process priority.
|
STIME
|
The process start time.
|
TTY
|
The terminal on which the process was started. Console represents the system console and ? indicates that the process is a daemon.
|
TIME
|
Total execution time for the process.
|
CMD
|
The name of the command or the program.
|
Listing a Specific Process
Listing of linux process can be done using various command like,
top
To determine the linux process, it can be based on
- Determining Processes by Ownership
Determining Processes by Ownership
Process States
- The running state determines that the process is currently being executed by the system CPU.
- The sleeping state shows that the process is currently waiting for input from a user or another process.
- The waiting state means that the process has received the input it has been waiting for and it is now ready to run as soon as its turn arrives.
- The stopped state indicates that the process is currently halted and will not run even when its turn comes, unless it is sent a signal.
- The zombie state designates that the process is dead. A zombie process exists in the process table just as any other process entry, but takes up no resources. The entry for a zombie process is retained until the parent process permits it to die. A zombie process is also called a defunct process.
Process Niceness & How to Set it
Altering Niceness of a Running Process
Listing Open Files
Column
|
Description
|
COMMAND
|
Displays the first nine characters of the command or process name.
|
PID
|
Displays the PID of the process.
|
USER
|
Displays the owner of the process.
|
FD
|
Displays the file descriptor of the file. Some of the values in this field would be:
cwd = current working directory; rtd = root directory;
txt = text file; mem = memory-mapped file; pd = parent directory
|
TYPE
|
Displays the node type of the file.
|
DEVICE
|
Displays the major and minor numbers of the device on which the file is located
|
SIZE
|
Displays the file size or offset in bytes.
|
NODE
|
Displays the inode number of the file.
|
NAME
|
Displays the file name or the file system name where the file resides.
|
Memory & CPU Utilization by a particular Process/Command in Linux/UNIX
Example:1 Show how much CPU & Memory Consumed by Chrome:
Example :2 CPU and Memory Consumed By MYSQL daemon
Example:3 Memory & CPU utilized by Apache Daemon-HTTPD
There are two commands commonly used to view currently running processes. These are ps (process status) and top.The ps command without any options or arguments, lists processes specific to the terminal where this command is run:
#ps
PID TTY TIME CMD
7494 pts/0 00:00:00 bash
9662 pts/0 00:00:00 ps
The output has four columns that show the PID of the process in the first column, theterminal the process belongs to in the second column, the cumulative time the process is given by the system CPU in the third column, and the name of the actual command or program being executed in the last column.
Two options -e (every) and -f (full) are popularly used to generate detailed information on every process running on the system. There are a number of additional options available with the ps command. Check the man pages for details.
# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 10:49 ? 00:00:03 /usr/lib/systemd/systemd --switc
root 2 0 0 10:49 ? 00:00:00 [kthreadd]
root 3 2 0 10:49 ? 00:00:00 [ksoftirqd/0]
The output shows more details about the running processes. Below Table describes the content type of each column.
Notice that in the ps output above there are scores of daemon processes running in the background that have no association with any terminals. Also notice the PID and PPID numbers. The smaller the number, the earlier it is started. The process with PID 0 is started first at system boot, followed by the process with PID 1, and so on. Each PID has an associated PPID in the 3 rd column. The owner of each process is also shown along with the name of the command or program.
Information about each running process is kept and maintained in a process table, which the ps and other commands read to display information.
The second method for viewing the process information is the “top command”, which also displays the CPU, memory, and swap utilization. A sample output from a running top session is shown below :
#top
The pidof command can be used to list the PID of a specific process if you know the name of the process. For example, to list the PID of the crond daemon, run the command as follows:
# pidof chrome
9621 9375 8559 8546 7233 7217 5394 5365 5323 4522 4498 4473 4447 4427 4405 4334 4332 4262 4254 4242
ps
htop
Processes can be listed by their ownership or group membership. The pgrep command is used for this purpose. For example, to list all the processes owned by root, use any of the following:
# ps -U root
# pgrep -U root
After a process is spawned, it does not run continuously. It may be in a non-running condition for a while or waiting for some other process to feed it with information so that it can continue to run.
There are five process states and each process is in one state at any given time. These states are running, sleeping, waiting, stopped, and zombie, and are explained below:
The priority of a process (niceness) is determined using the nice value. The system assigns a nice value to a process at initiation to establish a priority. There are 40 nice values with -20 being the highest and +19 the lowest. Most system-started processes use the default nice value of 0. A child process inherits the nice value of its parent process.
Use the ps command and specify the -l option to determine the niceness of running processes. See the associated nicenesses for each process under the NI column:
eg: $ ps -efl
To determine the default niceness, use the nice command without any options or arguments:
eg: $ nice
0
A different priority may be assigned to a program or command at its startup. For example, to run the top command at a lower priority of +3:
# nice -3 top
Use the ps command with the -l option, or view the output of the top command, and validate the niceness of the process. It should be +3.
To run the same program at a higher priority with the niceness of -3, specify the value with a pair of dashes:
# nice --3 top
The niceness of a running program may be altered using the renice command. For example, to change the nice value of top while it is running from -3 to -6, specify the PID (908) with the renice command:
#renice -6 908
To alter the nice values of all the processes owned by members of a particular group, use the -g option with renice. Similarly, to alter the nice values of all the processes owned by a particular user, use the -u option with it. Run the renice command without any options to view its usage.
A file is opened when the process or program stored in it is executed and closed when it is no longer required or the associated process or program has terminated. To determine the information such as which files are open, which processes are using them, and who the owners are, the lsof (list open files) command is used. Without any options, this command displays a list of all open files.
# lsof
The command generated nine columns in the output; these are listed and explained in the below table :
with help of “ps” and “pidof” commads we can determine how much cpu and memory is consumed or used by particular process and command.
Where “ps(process status)” displays information about the current active processes.Pidof finds the process id's (pids) of the named programs. It prints those id's on the standard output.
#ps -p $(pidof chrome | sed 's/\s/,/g') -o %cpu,%mem,cmd
# ps -p $(pidof mysqld | sed 's/\s/,/g') -o %cpu,%mem,cmd
# ps -p $(pidof httpd | sed 's/\s/,/g') -o %cpu,%mem,cmd
To see every process with a user-defined format:
#ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu
# ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm
# ps -Ao pid,tt,user,fname,tmout,f,wchan
