• VMware

    Learn about VMware virtualization for its products like vsphere ESX and ESXi, vCenter Server, VMware View, VMware P2V and many more

  • Linux

    Step by step configuration tutorials for many of the Linux services like DNS, DHCP, FTP, Samba4 etc including many tips and tricks in Red Hat Linux.

  • Database

    Learn installation and configuration of databases like Oracle, My SQL, Postgresql, etc including many other related tutorials in Linux.

  • Life always offers you a second chance ... Its called tomorrow !!!

    Saturday, February 22, 2014

    Tutorial for Monitoring Tools SAR and KSAR with examples in Linux

    sar is a short term abbreviation for System Activity Report.

    Important points for sar

    • It can be used for realtime monitoring of Linux system performance.
    • The  sar  command writes to standard output based on the values in  the  count  and  interval parameters
    • The  collected  data  can also  be saved in the file specified by the -o filename flag, in addition to being displayed onto the screen.
    • You can select information about specific system activities using flags. Not specifying any flags selects  only  CPU activity.
    • The sar command extracts and writes to standard output records previously saved in a  file.  This file  can be either the one specified by the -f flag.
    • All data is captured in binary form and saved to a file (datafile) with default location /var/log/sa/saXX where XX specified the day of the month

    Package required (Sysstat)

    Before you start running the sar command make sure that the required package for the same is installed in your machine i.e.sysstat package

    For Red Hat Linux
    Check if the rpm exists already
    # rpm -qa | grep sysstat
    If it is not installed you can install the same using
    # yum install sysstat

    If we try to query the installed packages for sysstat
    [deepak@test1 ~]$ rpm -ql sysstat
    /etc/cron.d/sysstat 
    /usr/bin/iostat  
    /usr/bin/mpstat
    /usr/bin/sadf
    /usr/bin/sar
    /usr/lib64/sa
    /usr/lib64/sa/sa1
    /usr/lib64/sa/sa2
    /usr/lib64/sa/sadc

    Package
    Usage
    sysstat
    To define the maximum no. of days to save sar reports
    iostat
    Report Central Processing Unit (CPU) statistics and input/output statistics
    mpstat
    Report processors related statistics
    sadf
    Display data collected by sar in multiple formats
    sar
    Collect, report, or save system activity information
    sa
    summarizes accounting information
    sa1
    Collect and store binary data in the system activity daily data file
    Sa2
    Write a daily report in the /var/log/sa directory
    sadc
    System activity data collector
    So as you see now sar has been installed

    To check the version
    [deepak@test1 ~]$ sar -V
    sysstat version 7.0.2
    (C) Sebastien Godard

    Collect sar reports on a daily basis using cron jobs (This entry is made by default as soon as you install sysstat package)
    [root@test1 ~]# cat /etc/cron.d/sysstat
    # run system activity accounting tool every 10 minutes
    */10 * * * * root /usr/lib64/sa/sa1 1 1
    # generate a daily summary of process accounting at 23:53
    53 23 * * * root /usr/lib64/sa/sa2 -A

    IMPORTANT NOTE:
    Do not confuse when I say sa1 and sa2 both creates a file inside /var/log/sa/. The important difference is that sa1 creates a binary file with the syntax saXX and sa2 creates a report (normal file) at the end of the day as provided in the cron job. This file is human readable and its syntax will be sarXX where XX is the day of the month in both the cases.
    Syntax:
    # sar {argument} {interval} {count}

    CPU usage reports

    Even if you don't use any argument along with sar command it will show you the overall CPU utilization. But the best part about sar is that you can use it to view realtime activity for any performance related feature in Linux using the count and interval parameter along with sar command

    For example:
    [deepak@test1 ~]$ sar
    Linux 2.6.18-348.el5 (test1)   02/20/14

    00:00:01          CPU     %user     %nice   %system   %iowait    %steal     %idle
    00:10:01          all     18.03      0.00      9.38      2.24      0.00     70.35
    00:20:02          all     17.04      0.00      6.90      2.70      0.00     73.37
    00:30:01          all     11.90      0.00      4.33      1.76      0.00     82.01
    00:40:01          all      8.67      0.00      2.60      0.21      0.00     88.51
    00:50:01          all      7.99      0.00      2.35      0.62      0.00     89.04
    01:00:01          all      8.78      0.00      2.54      0.29      0.00     88.38
    01:10:01          all      9.44      0.00      4.71      0.65      0.00     85.19
    01:20:01          all     14.26      0.00      4.69      0.50      0.00     80.55
    01:30:01          all      7.94      0.00      4.52      0.36      0.00     87.18
    01:40:01          all     27.02      0.00      6.04      0.30      0.00     66.64

    Average:          all     17.90      0.00     10.60      1.54      0.00     69.96
    Explanation: CPU utilization reports using (-u) argument. Since no interval and count argument is specified, it will show the report activity for complete day.

    For example:
    [deepak@test1 ~]$ sar -u 2 5
    Linux 2.6.18-348.el5 (test1)   02/20/14

    05:24:03          CPU     %user     %nice   %system   %iowait    %steal     %idle
    05:24:05          all     20.05      0.00      4.24      3.50      0.00     72.21
    05:24:07          all     19.88      0.00      5.72      3.12      0.00     71.28
    05:24:09          all     11.22      0.00      7.72      1.94      0.00     79.12
    05:24:11          all     21.12      0.00      4.45      3.37      0.00     71.05
    05:24:13          all     20.45      0.00      4.44      3.61      0.00     71.49
    Average:          all     18.54      0.00      5.32      3.11      0.00     73.03
    Explanation: In the above command we have specified an interval argument of 2 and count argument of 5 so it will give us a realtime report of cpu utilization for every 2 seconds for a total of 5 output along with average of all.

    Value
    Meaning
    %user
    Percentage of CPU utilization that occurred while executing at the user level (application).
    %nice
    Percentage of CPU utilization that occurred while executing at the user level with nice priority.
    %system
    Percentage of CPU utilization that occurred while executing at the system level (kernel).
    %iowait
    Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
    %steal
    Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
    %idle
    Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

    Individual CPU reports

    [deepak@test1 ~]$ sar -P ALL | less
    2Linux 2.6.18-348.el5 (test1)     02/20/14

    00:00:01          CPU     %user     %nice   %system   %iowait    %steal     %idle
    00:10:01          all     18.03      0.00      9.38      2.24      0.00     70.35
    00:10:01            0      7.37      0.00     11.23      0.94      0.00     80.46
    00:10:01            1      8.29      0.00      6.42      0.87      0.00     84.41
    00:10:01            2      6.17      0.00      8.05      1.24      0.00     84.53
    00:10:01            3      8.40      0.00      6.62      1.31      0.00     83.68
    00:10:01            4      7.51      0.00      6.40      0.56      0.00     85.53

    Average:          CPU     %user     %nice   %system   %iowait    %steal     %idle
    Average:          all     17.90      0.00     10.60      1.54      0.00     69.96
    Average:            0      9.69      0.00      8.76      0.39      0.00     81.16
    Average:            1     11.33      0.00      7.51      0.57      0.00     80.59
    Average:            2     11.28      0.00      7.30      0.54      0.00     80.88
    Average:            3     10.96      0.00      7.14      0.68      0.00     81.22
    Average:            4     10.66      0.00      7.30      0.45      0.00     81.58
    Explanation: Report CPU utilization for all the processors (If no interval and count value is specified it will show the statistics of complete day from the time monitoring started)

    [deepak@test1 ~]$ sar -P 1 2 4
    Linux 2.6.18-348.el5 (test1)   02/20/14

    05:31:27          CPU     %user     %nice   %system   %iowait    %steal     %idle
    05:31:29            1      0.00      0.00      0.00      0.00      0.00    100.00
    05:31:31            1      0.00      0.00      0.00      0.00      0.00    100.00
    05:31:33            1      0.00      0.00      0.50      0.00      0.00     99.50
    05:31:35            1      1.49      0.00      0.00      0.00      0.00     98.51
    Average:            1      0.37      0.00      0.12      0.00      0.00     99.50
    Explanation: Report CPU utilization of CPU 1 for an interval of 2 seconds (count = 4) along with an average value

    Memory and swap space usage Reports

    [deepak@test1 ~]$ sar -r 2 4
    Linux 2.6.18-348.el5 (test1)   02/20/14

    05:43:01    kbmemfree   kbmemused  %memused kbbuffers    kbcached    kbswpfree  kbswpused   %swpused  kbswpcad
    05:43:03     18146452   245967972     93.13    256956    225320936   33548444      5980      0.02       264
    05:43:05     18085944   246028480     93.15    257016    225376624   33548444      5980      0.02       264
    05:43:07     18027132   246087292     93.17    257016    225433756   33548444      5980      0.02       264
    05:43:09     17964680   246149744     93.20    257040    225494048   33548444      5980      0.02       264
    Average:     18056052   246058372     93.16    257007    225406341   33548444      5980      0.02       264
    Explanation: The above command will show 4 consecutive outputs for memory and swap space utilization statistics at an interval of 2 seconds.

    Values
    Meaning
    kbmemfree
    Amount of free memory available in kilobytes
    kbmemused
    Amount of used memory in kilobytes
    %memused
    Percentage of used memory
    kbbuffers
    Amount of memory used as buffers by the kernel in kilobytes
    kbcached
    Amount of memory used to cache data by the kernel in kilobytes
    kbswpfree
    Amount of free swap space in kilobytes
    kbswpused
    Amount of used swap space in kilobytes
    %swpused
    Percentage of used swap space
    kbswpcad
    Amount of cached swap memory in kilobytes.

    Network Utilization Statistics

    [deepak@test1 ~]$ sar -n DEV 1 1
    Linux 2.6.18-348.el5 (test1)   02/20/14

    06:07:37   IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
    06:07:38      lo    125.00    125.00    288.00    288.00      0.00      0.00      0.00
    06:07:38    eth0    671.00    794.00    387.00    437.00      0.00      0.00      1.00
    06:07:38    eth1      0.00      0.00      0.00      0.00      0.00      0.00      1.00
    06:07:38    eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    06:07:38    eth3    384.00    651.00    413.00    730.00      0.00      0.00      0.00
    6:07:38    bond0    671.00    794.00    387.00    437.00      0.00      0.00      2.00
    06:07:38   bond1     70.00     74.00    213.00    593.00      0.00      0.00      2.00

    Average:   IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
    Average:      lo    125.00    125.00    288.00    288.00      0.00      0.00      0.00
    Average:    eth0    671.00    794.00    387.00    437.00      0.00      0.00      1.00
    Average:    eth1      0.00      0.00      0.00      0.00      0.00      0.00      1.00
    Average:    eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    Average:    eth3    384.00   4651.00    413.00    730.00      0.00      0.00      0.00
    Average:   bond0    671.00    794.00    387.00    437.00      0.00      0.00      2.00
    Average:   bond1     70.00     74.00    213.00    593.00      0.00      0.00      2.00
    Explanation: Report network statistics. With the DEV keyword, statistics from the network devices are reported.

    Value
    Meaning
    IFACE
    Name of the network interface for which statistics are reported
    rxpck/s
    Total number of packets received per second
    txpck/s
    Total number of packets transmitted per second
    rxbyt/s
    Total number of bytes received per second
    txbyt/s
    Total number of bytes transmitted per second
    rxcmp/s
    Number of compressed packets received per second (for cslip etc.)
    txcmp/s
    Number of compressed packets transmitted per second
    rxmcst/s
    Number of multicast packets received per second

    No. of Process created per second Reports

    [deepak@test1 ~]$ sar -c 1 3
    Linux 2.6.18-348.el5 (test1)   02/20/14

    06:10:29       proc/s
    06:10:30      1152.00
    06:10:31      1059.00
    06:10:32      1082.18
    Average:      1097.67
    Explanation: Report process creation activity using -c argument. The above command shows 3 consecuitve output for every second.

    Load Average Reports

    [deepak@test1 ~]$ sar -q 2 4
    Linux 2.6.18-348.el5 (test1)   02/20/14

    06:20:19      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    06:20:21            7      2464      8.40      9.63     10.46
    06:20:23           13      2467      8.21      9.57     10.44
    06:20:25            7      2465      8.21      9.57     10.44
    06:20:27            7      2460      8.21      9.57     10.44
    Average:            8      2464      8.26      9.59     10.45
    Explanation: Report queue length and load averages with 4 output at an interval of 2 seconds for each output.

    Value
    Meaning
    runq-sz
    Run queue length (number of processes waiting for run time)
    plist-sz
    Number of processes and threads in the process list
    ldavg-1
    System load average for the last minute
    ldavg-5
    System load average for the past 5 minutes
    ldavg-15
    System load average for the past 15 minutes

    Store the sar output to a file

    [root@test1 ~]# sar -u 1 3 -o test.txt
    Linux 2.6.18-194.26.1.el5 (test1)     02/20/14

    05:50:26          CPU     %user     %nice   %system   %iowait    %steal     %idle
    05:50:27          all      0.00      0.00      0.06      0.12      0.00     99.81
    05:50:28          all      0.00      0.00      0.06      0.12      0.00     99.81
    05:50:29          all      0.00      0.00      0.06      0.56      0.00     99.38
    Average:          all      0.00      0.00      0.06      0.27      0.00     99.67
    Explanation: You can save the readings to a file in binary form which can be viewed again using sar -f parameter

    Collect report from a file (created above)

    [root@test1 ~]# sar -f test.txt
    Linux 2.6.18-194.26.1.el5 (test1)     02/20/14

    05:50:26          CPU     %user     %nice   %system   %iowait    %steal     %idle
    05:50:27          all      0.00      0.00      0.06      0.12      0.00     99.81
    05:50:28          all      0.00      0.00      0.06      0.12      0.00     99.81
    05:50:29          all      0.00      0.00      0.06      0.56      0.00     99.38
    Average:          all      0.00      0.00      0.06      0.27      0.00     99.67


    Collect sar reports from file for specific date

    By default all the sar reports are stored inside /var/lg/sa/saXX as binary file. To read those files use the below command with show arguments
    [deepak@test1 ~]$ sar -f  /var/log/sa/sa19  -q
    Linux 2.6.18-348.el5 (test1)   02/19/14

    00:00:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    00:10:01           14      3319     14.28     16.29     15.79
    00:20:01            9      3230     18.34     17.97     16.63
    00:30:01           15      3174     11.57     11.52     13.80
    00:40:01            7      2949     12.50     12.58     13.07
    23:30:01            6      2499     10.09     11.53     12.10
    23:40:02            7      2591     19.20     16.10     13.74
    23:50:01           19      3515     29.81     22.92     18.09
    Average:            7      2510      9.79      9.72      9.66
    Explanation: Using the above command you are collecting report for load average (-q) for 19th day(sa19) of the current month. I have skimmed the output here. Now since I have not used any other time related argument it will show me the load average output for the complete day.

    Collect sar reports from file for specific date and specific time

    [deepak@test1 ~]$ sar -f  /var/log/sa/sa19  -q -s 19:00:00 -e 20:00:00
    Linux 2.6.18-348.el5 (test1)   02/19/14

    19:00:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    19:10:01            1      2286      2.36      1.81      1.59
    19:20:01            0      2284      0.48      1.32      1.55
    19:30:01            2      2286      0.39      0.48      0.97
    19:40:01            1      2290      2.72      2.13      1.54
    19:50:01            0      2288      2.13      2.58      2.08
    Average:            1      2287      1.62      1.66      1.55
    Explanation: Using the above command I have given a time argument with start(-s) and end(-e) time to get the reports only for that particular time interval

    Collect Monthly reports

    By default you can view reports for only the current month or as it is set inside /etc/sysconfig/sysstat

    To view the default settings
    [deepak@test1 ~]$ cat /etc/sysconfig/sysstat
    # How long to keep log files (days), maximum is a month
    HISTORY=7

    So as you see the maximum time you can keep your report is for 1 month but still if you want to save reports for more than a month follow the below steps.
    [root@test1 201404]# cat /etc/sysconfig/sysstat
    # sysstat-9.0.4 configuration file.
    # How long to keep log files (in days).

    # If value is greater than 28, then log files are kept in
    # multiple directories, one for each month.
    HISTORY=30

    # cd /var/log/sa/
    [root@test1
    sa]# ls -l
    total 8
    drwxr-xr-x 2 root root 4096 Apr  2 10:10
    201403
    drwxr-xr-x 2 root root 4096 Apr  2 10:10 201404
    lrwxrwxrwx 1 root root   11 Apr  2 10:10
    sa02 -> 201404/sa02
    lrwxrwxrwx 1 root root   12 Apr  2 10:10 sar02 -> 201404/sar02
    As you can see different directories are getting created for different month and also separate sa and sar files (softlinks) are created inside sa directory pointing to their respective paths.

    Collect Graphical reports using ksar

    If you do not feel comfortable using CLI interface then there is an alternate option to use Graphical interface for all the reports.

    For this purpose you will have to download kar package which can be done using the belw link
    KSAR download

    Once the package is downloaded. Extract it at any preferred location.
    Move into the ksar directory created after extraction step, you should see the below directories/file.

    Double click "Ksar.jar" java file. After which the below windows should come up.
    Click on Data and select "Launch SSH Command.."

    In the next pop up window provide the host name you want to connect using the below mentioned syntax
    username@server-name
    For exampledeepak@192.168.0.5

    Click Yes to establish a connection with the host

    Provide the password for the host you want to connect

    Use the shown command as it will give you the complete graphical report for the whole day

    Once the data import is finished you will see the below popup window. Click OK

    Now you can select different parameters as shown in the left panel of the ksar to view all the reports. I have shown few examples using few screenshots

    Interface traffic for eth0

    Load average Statistics

    Memory usage Statistics

    CPU usage Statistics

    Swap Memory usage statistics


    References:

    2 comments: