Wednesday 24 February 2016

Find command example in linux

Find command used to search both files and directory in a linux system.



  1. How to find core files

# find / -name core
  1. How to view 10 days old files
find /path/to/files* -mtime +10
  1. How to find file those having 2gb size
#find . -size +20000k -exec du -h {} \;
  1. to search inode number and seeing content inside that file
                         find / -inum number -exec cat number {} \;      or find . -inum 1610620106 -exec cat i {} \;
  1. to search inode number and delete inode number file
                               find . -inum number -exec rm -i {} \;
  1. Find the inode number file eg:123 and move that file to aaa folder
                     i=123; find . -inum $i -exec mv -i {} aaa \;
         7.how to find a particular file in particular directory and search for a particular string occurance

find /tmp/aaa/ -type f -and -name "1.txt" -exec grep -i "s" {} \;

(we can use "-and " "-not " "-or" for condition)


8. How to find a file which is bigger than +500k size

find / -type f -size +500k 


9. How to search directory or file based on ownership

for directory:
find /tmp/ -type d -user test

for file:
find /tmp/ -type f -user test -name "filename"

10. How to search for file which has 777 permission

find /tmp/ -perm 777 (return all the file with 777 permission)

or

find /tmp/ -perm 777 -name "1.txt" (return only particular 1.txt file which matches 777 permission )

11. How to find total number of file in a certain directory

If you want to exclude subdirectories, you need a heavier duty tool than ls.
$ find targetdir -maxdepth 1 -type f | wc -l  # find all files in current target directory and exclude sub-directory

find targetdir -type f | wc -l     #  find all files including files in sub and sub-sub-directory

-type f ensures that the find command only returns regular files for counting (no subdirectories).
By default, the find command traverses into subdirectories for searching. -maxdepth 1 prevents find from traversing into subdirectories. If you do want to count files in the subdirectories, just remove -maxdepth 1 from the command line.
Note that the find command does NOT classify a symbolic link as a regular file. Therefore, the above find -type f command does not return symbolic links. As a result, the final count excludes all symbolic links.
To include symbolic links, add the -follow option to find.
$ find targetdir -follow  -maxdepth 1 -type f | wc -l
To find number of directories present in a certain directory
$ find targetdir -maxdepth 1 -type d | wc -l        # leaves sub sub-directories
$ find targetdir -type d | wc -l     #  find all sub and sub-sub-directory










Monday 15 February 2016

Raid5 configuration using 3 disk

Configuring Raid 5 in 3 disk



Raid 5: Combines striping and parity. Data is written across all disks as in RAID 0, but parity data is also written to one of the disks. Should a hard drive failure occur, this parity data can be used to recover the data from the failed drive, including while the data is being accessed and the drive is still missing from the array.

Select the 3 disk and format the three disk in raid using fdisk command and update the changes to kernel using partprobe command.


Note: here I am creating new raid with name md0 and guessing that we have 3 raid disk with /dev/sda(1,2,3)


1. Creating RAID 5 


mdadm --create /dev/md0 --level=5 --raid-disk=3 /dev/sda1 /dev/sda2 /dev/sda3



2. Format the raid partition


mke2fs -j /dev/md0


3. Create a mount point for raid


mkdir /data


4.Mount the raid 


mount /dev/md0  /data


5. To permanently mount the raid devices, add the mount point in /etc/fstab at end of the file as below,


/dev/md0 /data ext3 defaults 0 0 



RAID MANAGEMENT IN LINUX



 To verify the active raid that we have created in previous step by


mdadm --detail /dev/md0


 To check the running status of raid devces


cat /proc/mdstat




RAID Failover


1.  assume that one hard disk(/dev/sda2) form raid device gets fail

mdadm /dev/md0 --fail /dev/sda2



2. To verify the raid device status 


mdadm --detail /dev/md0


and for run time check


cat /proc/mdstat


3. Remove faulty devices


mdadm /dev/md0 --remove /dev/sda2



4. Adding a new harddisk


Note: before adding the disk(/dev/sda4) to the raid, kindly format and add as raid partition using fdisk and partprobe command. then


mdadm /dev/md0 --add /dev/sd4






REMOVE RAID 


1. unmount raid partition


umount /data


2. stop the raid device


mdadm --stop /dev/md0


3. Remove the raid device


mdadm --remove /dev/md0



4. Remove the entry from /etc/fstab



5. to remove raid partition form hard disk


use fdisk command and and select disk partition and use "d" to delete and "w" to write the changes.













Wednesday 10 February 2016

Linux System performance checking & Understanding Linux process

Linux System performance checking

1.  VMSTAT -  vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity.

eg: vmstat -SM 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0    218    309     48    922    0    0   103   103  878 1020 17  5 77  2  0
 1  0    218    309     48    922    0    0     0     0  495 1677  3  1 96  0  0

FIELD DESCRIPTION FOR VM MODE
   Procs
       r: The number of runnable processes (running or waiting for run time).
       b: The number of processes in uninterruptible sleep.

   Memory
       swpd: the amount of virtual memory used.
       free: the amount of idle memory.
       buff: the amount of memory used as buffers.
       cache: the amount of memory used as cache.
       inact: the amount of inactive memory.  (-a option)
       active: the amount of active memory.  (-a option)

   Swap
       si: Amount of memory swapped in from disk (/s).
       so: Amount of memory swapped to disk (/s).

   IO
       bi: Blocks received from a block device (blocks/s).
       bo: Blocks sent to a block device (blocks/s).

   System
       in: The number of interrupts per second, including the clock.
       cs: The number of context switches per second.

CPU
       These are percentages of total CPU time.
       us: Time spent running non-kernel code.  (user time, including nice time)
       sy: Time spent running kernel code.  (system time)
       id: Time spent idle.  Prior to Linux 2.5.41, this includes IO-wait time.
       wa: Time spent waiting for IO.  Prior to Linux 2.5.41, included in idle.
       st: Time stolen from a virtual machine.  Prior to Linux 2.6.11, unknown.

2. IOSTAT - Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.

eg: iostat -kx 3

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.91    0.41    4.37    1.50    0.00   77.81

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               1.55     5.60    4.52    4.78   116.76   116.07    50.11     0.59   63.04   19.66  104.05   6.40   5.95
dm-0              0.00     0.00    4.23    4.74   103.21    94.90    44.19     0.60   66.74   20.44  107.99   6.45   5.78
dm-1              0.00     0.00    1.76    5.26     7.08    21.03     8.01     4.71  672.15   13.89  892.31   0.50   0.35

CPU Utilization Report
              The  first  report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors.  The report has the following format:

              %user
                     Show the percentage of CPU utilization that occurred while executing at the user level (application).

              %nice
                     Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.

              %system
                     Show the percentage of CPU utilization that occurred while executing at the system level (kernel).

              %iowait
                     Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

              %steal
                     Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.

              %idle
                     Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

Device Utilization Report
 Device:
                     This column gives the device (or partition) name as listed in the /dev directory.

 tps
          Indicate the number of transfers per second that were issued to the device. A transfer is an  I/O  request  to the  device.  Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.

              Blk_read/s (kB_read/s, MB_read/s)
                     Indicate the amount of data read from the device expressed in a number of blocks  (kilobytes,  megabytes)  per second. Blocks are equivalent to sectors and therefore have a size of 512 bytes.

              Blk_wrtn/s (kB_wrtn/s, MB_wrtn/s)
                     Indicate  the  amount of data written to the device expressed in a number of blocks (kilobytes, megabytes) per second.

              Blk_read (kB_read, MB_read)
                     The total number of blocks (kilobytes, megabytes) read.

              Blk_wrtn (kB_wrtn, MB_wrtn)
                     The total number of blocks (kilobytes, megabytes) written.
rrqm/s
                     The number of read requests merged per second that were queued to the device.

              wrqm/s
                     The number of write requests merged per second that were queued to the device.

              r/s
                     The number (after merges) of read requests completed per second for the device.

              w/s
                     The number (after merges) of write requests completed per second for the device.

              rsec/s (rkB/s, rMB/s)
                     The number of sectors (kilobytes, megabytes) read from the device per second.

              wsec/s (wkB/s, wMB/s)
                     The number of sectors (kilobytes, megabytes) written to the device per second.

              avgrq-sz
                     The average size (in sectors) of the requests that were issued to the device.

              avgqu-sz
                     The average queue length of the requests that were issued to the device.

              await
                     The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the  time spent by the requests in queue and the time spent servicing them.

r_await
                     The average time (in milliseconds) for read requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.

              w_await
                     The average time (in milliseconds) for write requests issued to the device to be  served.  This  includes  the time spent by the requests in queue and the time spent servicing them.

              svctm
                     The  average  service  time (in milliseconds) for I/O requests that were issued to the device. Warning! Do not trust this field any more.  This field will be removed in a future sysstat version.

              %util
                     Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for  the device). Device saturation occurs when this value is close to 100%.


3. MPSTAT - Report processors related statistics.

eg: mpstat 2 2

03:06:47 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
03:06:49 PM  all    2.77    0.00    0.63    0.13    0.00    0.00    0.00    0.00    0.00   96.47
03:06:51 PM  all    3.13    0.00    0.50    0.13    0.00    0.00    0.00    0.00    0.00   96.24
Average:     all    2.95    0.00    0.57    0.13    0.00    0.00    0.00    0.00    0.00   96.36

Report CPU utilization
The following values are displayed:

              CPU
                     Processor number. The keyword all indicates that statistics are calculated as averages among all processors.

              %usr
                     Show the percentage of CPU utilization that occurred while executing at the user level (application).

              %nice
                     Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.

              %sys
                     Show  the  percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time spent servicing hardware and software interrupts.

              %iowait
                     Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

              %irq
                     Show the percentage of time spent by the CPU or CPUs to service hardware interrupts.


 %soft

                     Show the percentage of time spent by the CPU or CPUs to service software interrupts.



              %steal

                     Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.


              %guest
                     Show the percentage of time spent by the CPU or CPUs to run a virtual processor.

              %gnice
                     Show the percentage of time spent by the CPU or CPUs to run a niced guest.

              %idle
                     Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.


4. ULIMIT - The ulimit and sysctl programs allow to limit system-wide resource use. This can help a lot in system administration, e.g. when a user starts too many processes and therefore makes the system unresponsive for other users.

eg: ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 14726
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


5. LSOFlists  on  its standard output file information about files opened by processes
A file is opened when the process or program stored in it is executed and closed when it is no longer required or the associated process or program has terminated. To determine the information such as which files are open, which processes are using them, and who the owners are, the lsof (list open files) command is used. 

eg: lsof -np PID  i.e  lsof -np 22266
COMMAND   PID   USER   FD      TYPE             DEVICE  SIZE/OFF      NODE NAME
vlc     22266 naveen  cwd       DIR              253,0      4096   3307693 /home/naveen
vlc     22266 naveen  rtd       DIR              253,0      4096       192 /
vlc     22266 naveen  txt       REG              253,0     15840  71886477 /usr/bin/vlc

6. STRACE  -  trace system calls and signals.

eg: strace -p PID  i.e.  strace -p 22266
Process 22266 attached
rt_sigtimedwait([HUP INT QUIT TERM], NULL, NULL, 8


Understanding Linux process

Viewing Processes
Heading
Description
UID
User ID of the process owner.
PID
Process ID of the process.
PPID
Process ID of the parent process.
C
The process priority.
STIME
The process start time.
TTY
The terminal on which the process was started. Console represents the system console and ? indicates that the process is a daemon.
TIME
Total execution time for the process.
CMD
The name of the command or the program.
Listing a Specific Process
Listing of linux process can be done using various command like,
top
To determine the linux process, it can be based on
  • Determining Processes by Ownership
  • Process States
  • Process Priority
  • Listing Open Files
Determining Processes by Ownership
Process States

  • The running state determines that the process is currently being executed by the system CPU.
  • The sleeping state shows that the process is currently waiting for input from a user or another process.
  • The waiting state means that the process has received the input it has been waiting for and it is now ready to run as soon as its turn arrives.
  • The stopped state indicates that the process is currently halted and will not run even when its turn comes, unless it is sent a signal.
  • The zombie state designates that the process is dead. A zombie process exists in the process table just as any other process entry, but takes up no resources. The entry for a zombie process is retained until the parent process permits it to die. A zombie process is also called a defunct process.
Process Niceness & How to Set it
Altering Niceness of a Running Process
Listing Open Files

Column
Description
COMMAND
Displays the first nine characters of the command or process name.
PID
Displays the PID of the process.
USER
Displays the owner of the process.
FD
Displays the file descriptor of the file. Some of the values in this field would be:
cwd = current working directory; rtd = root directory;
txt = text file; mem = memory-mapped file; pd = parent directory
TYPE
Displays the node type of the file.
DEVICE
Displays the major and minor numbers of the device on which the file is located
SIZE
Displays the file size or offset in bytes.
NODE
Displays the inode number of the file.
NAME
Displays the file name or the file system name where the file resides.
Memory & CPU Utilization by a particular Process/Command in Linux/UNIX
Example:1 Show how much CPU & Memory Consumed by Chrome:
Example :2 CPU and Memory Consumed By MYSQL daemon
Example:3 Memory & CPU utilized by Apache Daemon-HTTPD

There are two commands commonly used to view currently running processes. These are ps (process status) and top.The ps command without any options or arguments, lists processes specific to the terminal where this command is run:
#ps
  PID TTY          TIME CMD
 7494 pts/0    00:00:00 bash
 9662 pts/0    00:00:00 ps
The output has four columns that show the PID of the process in the first column, theterminal the process belongs to in the second column, the cumulative time the process is given by the system CPU in the third column, and the name of the actual command or program being executed in the last column.
Two options -e (every) and -f (full) are popularly used to generate detailed information on every process running on the system. There are a number of additional options available with the ps command. Check the man pages for details.
# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 10:49 ?        00:00:03 /usr/lib/systemd/systemd --switc
root         2     0  0 10:49 ?        00:00:00 [kthreadd]
root         3     2  0 10:49 ?        00:00:00 [ksoftirqd/0]
The output shows more details about the running processes. Below Table describes the content type of each column.

Notice that in the ps output above there are scores of daemon processes running in the background that have no association with any terminals. Also notice the PID and PPID numbers. The smaller the number, the earlier it is started. The process with PID 0 is started first at system boot, followed by the process with PID 1, and so on. Each PID has an associated PPID in the 3 rd column. The owner of each process is also shown along with the name of the command or program.
Information about each running process is kept and maintained in a process table, which the ps and other commands read to display information.
The second method for viewing the process information is the “top command”, which also displays the CPU, memory, and swap utilization. A sample output from a running top session is shown below :
#top

The pidof command can be used to list the PID of a specific process if you know the name of the process. For example, to list the PID of the crond daemon, run the command as follows:
# pidof chrome
9621 9375 8559 8546 7233 7217 5394 5365 5323 4522 4498 4473 4447 4427 4405 4334 4332 4262 4254 4242




ps
htop


Processes can be listed by their ownership or group membership. The pgrep command is used for this purpose. For example, to list all the processes owned by root, use any of the following:
# ps -U root
# pgrep -U root
After a process is spawned, it does not run continuously. It may be in a non-running condition for a while or waiting for some other process to feed it with information so that it can continue to run.
There are five process states and each process is in one state at any given time. These states are runningsleepingwaitingstopped, and zombie, and are explained below:
The priority of a process (niceness) is determined using the nice value. The system assigns a nice value to a process at initiation to establish a priority. There are 40 nice values with -20 being the highest and +19 the lowest. Most system-started processes use the default nice value of 0. A child process inherits the nice value of its parent process.
Use the ps command and specify the -l option to determine the niceness of running processes. See the associated nicenesses for each process under the NI column:
eg: $ ps -efl
To determine the default niceness, use the nice command without any options or arguments:
eg: $ nice
0
A different priority may be assigned to a program or command at its startup. For example, to run the top command at a lower priority of +3:
# nice -3 top
Use the ps command with the -l option, or view the output of the top command, and validate the niceness of the process. It should be +3.
To run the same program at a higher priority with the niceness of -3, specify the value with a pair of dashes:
# nice --3 top
The niceness of a running program may be altered using the renice command. For example, to change the nice value of top while it is running from -3 to -6, specify the PID (908) with the renice command:
#renice -6 908
To alter the nice values of all the processes owned by members of a particular group, use the -g option with renice. Similarly, to alter the nice values of all the processes owned by a particular user, use the -u option with it. Run the renice command without any options to view its usage.
A file is opened when the process or program stored in it is executed and closed when it is no longer required or the associated process or program has terminated. To determine the information such as which files are open, which processes are using them, and who the owners are, the lsof (list open files) command is used. Without any options, this command displays a list of all open files.
# lsof

The command generated nine columns in the output; these are listed and explained in the below table :

with help of “ps” and “pidof” commads we can determine how much cpu and memory is consumed or used by particular process and command.
Where “ps(process status)” displays information about the current active processes.Pidof finds the process id's (pids) of the named programs. It prints those id's on the standard output.
#ps -p $(pidof chrome | sed 's/\s/,/g') -o %cpu,%mem,cmd

ps -p $(pidof mysqld | sed 's/\s/,/g') -o %cpu,%mem,cmd


ps -p $(pidof httpd | sed 's/\s/,/g') -o %cpu,%mem,cmd



To see every process with a user-defined format:

#ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu 
# ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm
#  ps -Ao pid,tt,user,fname,tmout,f,wchan