Tutorial for Monitoring Tools SAR and KSAR with examples in Linux

In my last article I have explained about Understanding Load Average and monitoring network bandwidth using iperf and netperf.


sar is a short term abbreviation for System Activity Report.

Important points for sar

  • It can be used for realtime monitoring of Linux system performance.
  • The  sar  command writes to standard output based on the values in  the  count  and  interval parameters
  • The  collected  data  can also  be saved in the file specified by the -o filename flag, in addition to being displayed onto the screen.
  • You can select information about specific system activities using flags. Not specifying any flags selects  only  CPU activity.
  • The sar command extracts and writes to standard output records previously saved in a  file.  This file  can be either the one specified by the -f flag.
  • All data is captured in binary form and saved to a file (datafile) with default location /var/log/sa/saXX where XX specified the day of the month

Package required (Sysstat)

Before you start running the sar command make sure that the required package for the same is installed in your machine i.e.sysstat package

For Red Hat Linux
Check if the rpm exists already
# rpm -qa | grep sysstat
If it is not installed you can install the same using

# yum install sysstat

If we try to query the installed packages for sysstat
[deepak@test1 ~]$ rpm -ql sysstat
/etc/cron.d/sysstat 
/usr/bin/iostat  
/usr/bin/mpstat
/usr/bin/sadf
/usr/bin/sar
/usr/lib64/sa
/usr/lib64/sa/sa1
/usr/lib64/sa/sa2
/usr/lib64/sa/sadc

Package
Usage
sysstat
To define the maximum no. of days to save sar reports
iostat
Report Central Processing Unit (CPU) statistics and input/output statistics
mpstat
Report processors related statistics
sadf
Display data collected by sar in multiple formats
sar
Collect, report, or save system activity information
sa
summarizes accounting information
sa1
Collect and store binary data in the system activity daily data file
Sa2
Write a daily report in the /var/log/sa directory
sadc
System activity data collector

So as you see now sar has been installed

To check the version
[deepak@test1 ~]$ sar -V
sysstat version 7.0.2
(C) Sebastien Godard
Collect sar reports on a daily basis using cron jobs (This entry is made by default as soon as you install sysstat package)
[root@test1 ~]# cat /etc/cron.d/sysstat
# run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

IMPORTANT NOTE : Do not confuse when I say sa1 and sa2 both creates a file inside /var/log/sa/. The important difference is that sa1 creates a binary file with the syntax saXX and sa2 creates a report (normal file) at the end of the day as provided in the cron job. This file is human readable and its syntax will be sarXX where XX is the day of the month in both the cases.

Syntax:
# sar {argument} {interval} {count}

CPU usage reports

Even if you don't use any argument along with sar command it will show you the overall CPU utilization. But the best part about sar is that you can use it to view realtime activity for any performance related feature in Linux using the count and interval parameter along with sar command

For example:
[deepak@test1 ~]$ sar
Linux 2.6.18-348.el5 (test1)   02/20/14

00:00:01          CPU     %user     %nice   %system   %iowait    %steal     %idle
00:10:01          all     18.03      0.00      9.38      2.24      0.00     70.35
00:20:02          all     17.04      0.00      6.90      2.70      0.00     73.37
00:30:01          all     11.90      0.00      4.33      1.76      0.00     82.01
00:40:01          all      8.67      0.00      2.60      0.21      0.00     88.51
00:50:01          all      7.99      0.00      2.35      0.62      0.00     89.04
01:00:01          all      8.78      0.00      2.54      0.29      0.00     88.38
01:10:01          all      9.44      0.00      4.71      0.65      0.00     85.19
01:20:01          all     14.26      0.00      4.69      0.50      0.00     80.55
01:30:01          all      7.94      0.00      4.52      0.36      0.00     87.18
01:40:01          all     27.02      0.00      6.04      0.30      0.00     66.64

Average:          all     17.90      0.00     10.60      1.54      0.00     69.96
Explanation: CPU utilization reports using (-u) argument. Since no interval and count argument is specified, it will show the report activity for complete day.

For example:
[deepak@test1 ~]$ sar -u 2 5
Linux 2.6.18-348.el5 (test1)   02/20/14

05:24:03          CPU     %user     %nice   %system   %iowait    %steal     %idle
05:24:05          all     20.05      0.00      4.24      3.50      0.00     72.21
05:24:07          all     19.88      0.00      5.72      3.12      0.00     71.28
05:24:09          all     11.22      0.00      7.72      1.94      0.00     79.12
05:24:11          all     21.12      0.00      4.45      3.37      0.00     71.05
05:24:13          all     20.45      0.00      4.44      3.61      0.00     71.49
Average:          all     18.54      0.00      5.32      3.11      0.00     73.03 Explanation: In the above command we have specified an interval argument of 2 and count argument of 5 so it will give us a realtime report of cpu utilization for every 2 seconds for a total of 5 output along with average of all.

Value
Meaning
%user
Percentage of CPU utilization that occurred while executing at the user level (application).
%nice
Percentage of CPU utilization that occurred while executing at the user level with nice priority.
%system
Percentage of CPU utilization that occurred while executing at the system level (kernel).
%iowait
Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
%steal
Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
%idle
Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

Individual CPU reports

[deepak@test1 ~]$ sar -P ALL | less
2Linux 2.6.18-348.el5 (test1)     02/20/14

00:00:01          CPU     %user     %nice   %system   %iowait    %steal     %idle
00:10:01          all     18.03      0.00      9.38      2.24      0.00     70.35
00:10:01            0      7.37      0.00     11.23      0.94      0.00     80.46
00:10:01            1      8.29      0.00      6.42      0.87      0.00     84.41
00:10:01            2      6.17      0.00      8.05      1.24      0.00     84.53
00:10:01            3      8.40      0.00      6.62      1.31      0.00     83.68
00:10:01            4      7.51      0.00      6.40      0.56      0.00     85.53

Average:          CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:          all     17.90      0.00     10.60      1.54      0.00     69.96
Average:            0      9.69      0.00      8.76      0.39      0.00     81.16
Average:            1     11.33      0.00      7.51      0.57      0.00     80.59
Average:            2     11.28      0.00      7.30      0.54      0.00     80.88
Average:            3     10.96      0.00      7.14      0.68      0.00     81.22
Average:            4     10.66      0.00      7.30      0.45      0.00     81.58 Explanation: Report CPU utilization for all the processors (If no interval and count value is specified it will show the statistics of complete day from the time monitoring started)

[deepak@test1 ~]$ sar -P 1 2 4
Linux 2.6.18-348.el5 (test1)   02/20/14

05:31:27          CPU     %user     %nice   %system   %iowait    %steal     %idle
05:31:29            1      0.00      0.00      0.00      0.00      0.00    100.00
05:31:31            1      0.00      0.00      0.00      0.00      0.00    100.00
05:31:33            1      0.00      0.00      0.50      0.00      0.00     99.50
05:31:35            1      1.49      0.00      0.00      0.00      0.00     98.51
Average:            1      0.37      0.00      0.12      0.00      0.00     99.50 Explanation: Report CPU utilization of CPU 1 for an interval of 2 seconds (count = 4) along with an average value

Memory and swap space usage Reports

[deepak@test1 ~]$ sar -r 2 4
Linux 2.6.18-348.el5 (test1)   02/20/14

05:43:01    kbmemfree   kbmemused  %memused kbbuffers    kbcached    kbswpfree  kbswpused   %swpused  kbswpcad
05:43:03     18146452   245967972     93.13    256956    225320936   33548444      5980      0.02       264
05:43:05     18085944   246028480     93.15    257016    225376624   33548444      5980      0.02       264
05:43:07     18027132   246087292     93.17    257016    225433756   33548444      5980      0.02       264
05:43:09     17964680   246149744     93.20    257040    225494048   33548444      5980      0.02       264
Average:     18056052   246058372     93.16    257007    225406341   33548444      5980      0.02       264

Explanation: The above command will show 4 consecutive outputs for memory and swap space utilization statistics at an interval of 2 seconds.

Values
Meaning
kbmemfree
Amount of free memory available in kilobytes
kbmemused
Amount of used memory in kilobytes
%memused
Percentage of used memory
kbbuffers
Amount of memory used as buffers by the kernel in kilobytes
kbcached
Amount of memory used to cache data by the kernel in kilobytes
kbswpfree
Amount of free swap space in kilobytes
kbswpused
Amount of used swap space in kilobytes
%swpused
Percentage of used swap space
kbswpcad
Amount of cached swap memory in kilobytes.

Network Utilization Statistics

[deepak@test1 ~]$ sar -n DEV 1 1
Linux 2.6.18-348.el5 (test1)   02/20/14

06:07:37   IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
06:07:38      lo    125.00    125.00    288.00    288.00      0.00      0.00      0.00
06:07:38    eth0    671.00    794.00    387.00    437.00      0.00      0.00      1.00
06:07:38    eth1      0.00      0.00      0.00      0.00      0.00      0.00      1.00
06:07:38    eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:07:38    eth3    384.00    651.00    413.00    730.00      0.00      0.00      0.00
6:07:38    bond0    671.00    794.00    387.00    437.00      0.00      0.00      2.00
06:07:38   bond1     70.00     74.00    213.00    593.00      0.00      0.00      2.00

Average:   IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
Average:      lo    125.00    125.00    288.00    288.00      0.00      0.00      0.00
Average:    eth0    671.00    794.00    387.00    437.00      0.00      0.00      1.00
Average:    eth1      0.00      0.00      0.00      0.00      0.00      0.00      1.00
Average:    eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    eth3    384.00   4651.00    413.00    730.00      0.00      0.00      0.00
Average:   bond0    671.00    794.00    387.00    437.00      0.00      0.00      2.00
Average:   bond1     70.00     74.00    213.00    593.00      0.00      0.00      2.00
Explanation: Report network statistics. With the DEV keyword, statistics from the network devices are reported.

Value
Meaning
IFACE
Name of the network interface for which statistics are reported
rxpck/s
Total number of packets received per second
txpck/s
Total number of packets transmitted per second
rxbyt/s
Total number of bytes received per second
txbyt/s
Total number of bytes transmitted per second
rxcmp/s
Number of compressed packets received per second (for cslip etc.)
txcmp/s
Number of compressed packets transmitted per second
rxmcst/s
Number of multicast packets received per second

No. of Process created per second Reports

[deepak@test1 ~]$ sar -c 1 3
Linux 2.6.18-348.el5 (test1)   02/20/14

06:10:29       proc/s
06:10:30      1152.00
06:10:31      1059.00
06:10:32      1082.18
Average:      1097.67 Explanation: Report process creation activity using -c argument. The above command shows 3 consecuitve output for every second.

Load Average Reports

[deepak@test1 ~]$ sar -q 2 4
Linux 2.6.18-348.el5 (test1)   02/20/14

06:20:19      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
06:20:21            7      2464      8.40      9.63     10.46
06:20:23           13      2467      8.21      9.57     10.44
06:20:25            7      2465      8.21      9.57     10.44
06:20:27            7      2460      8.21      9.57     10.44
Average:            8      2464      8.26      9.59     10.45 Explanation: Report queue length and load averages with 4 output at an interval of 2 seconds for each output.

Value
Meaning
runq-sz
Run queue length (number of processes waiting for run time)
plist-sz
Number of processes and threads in the process list
ldavg-1
System load average for the last minute
ldavg-5
System load average for the past 5 minutes
ldavg-15
System load average for the past 15 minutes

Store the sar output to a file

[root@test1 ~]# sar -u 1 3 -o test.txt
Linux 2.6.18-194.26.1.el5 (test1)     02/20/14

05:50:26          CPU     %user     %nice   %system   %iowait    %steal     %idle
05:50:27          all      0.00      0.00      0.06      0.12      0.00     99.81
05:50:28          all      0.00      0.00      0.06      0.12      0.00     99.81
05:50:29          all      0.00      0.00      0.06      0.56      0.00     99.38
Average:          all      0.00      0.00      0.06      0.27      0.00     99.67 Explanation: You can save the readings to a file in binary form which can be viewed again using sar -f parameter

Collect report from a file (created above)

[root@test1 ~]# sar -f test.txt
Linux 2.6.18-194.26.1.el5 (test1)     02/20/14

05:50:26          CPU     %user     %nice   %system   %iowait    %steal     %idle
05:50:27          all      0.00      0.00      0.06      0.12      0.00     99.81
05:50:28          all      0.00      0.00      0.06      0.12      0.00     99.81
05:50:29          all      0.00      0.00      0.06      0.56      0.00     99.38
Average:          all      0.00      0.00      0.06      0.27      0.00     99.67

Collect sar reports from file for specific date

By default all the sar reports are stored inside /var/lg/sa/saXX as binary file. To read those files use the below command with show arguments
[deepak@test1 ~]$ sar -f  /var/log/sa/sa19  -q
Linux 2.6.18-348.el5 (test1)   02/19/14

00:00:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
00:10:01           14      3319     14.28     16.29     15.79
00:20:01            9      3230     18.34     17.97     16.63
00:30:01           15      3174     11.57     11.52     13.80
00:40:01            7      2949     12.50     12.58     13.07
23:30:01            6      2499     10.09     11.53     12.10
23:40:02            7      2591     19.20     16.10     13.74
23:50:01           19      3515     29.81     22.92     18.09
Average:            7      2510      9.79      9.72      9.66 Explanation: Using the above command you are collecting report for load average (-q) for 19th day(sa19) of the current month. I have skimmed the output here. Now since I have not used any other time related argument it will show me the load average output for the complete day.

Collect sar reports from file for specific date and specific time

[deepak@test1 ~]$ sar -f  /var/log/sa/sa19  -q -s 19:00:00 -e 20:00:00
Linux 2.6.18-348.el5 (test1)   02/19/14

19:00:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
19:10:01            1      2286      2.36      1.81      1.59
19:20:01            0      2284      0.48      1.32      1.55
19:30:01            2      2286      0.39      0.48      0.97
19:40:01            1      2290      2.72      2.13      1.54
19:50:01            0      2288      2.13      2.58      2.08
Average:            1      2287      1.62      1.66      1.55 Explanation: Using the above command I have given a time argument with start(-s) and end(-e) time to get the reports only for that particular time interval

Collect Monthly reports

By default you can view reports for only the current month or as it is set inside /etc/sysconfig/sysstat

To view the default settings
[deepak@test1 ~]$ cat /etc/sysconfig/sysstat
# How long to keep log files (days), maximum is a month
HISTORY=7
So as you see the maximum time you can keep your report is for 1 month but still if you want to save reports for more than a month follow the below steps.

[root@test1 201404]# cat /etc/sysconfig/sysstat
# sysstat-9.0.4 configuration file.
# How long to keep log files (in days).

# If value is greater than 28, then log files are kept in
# multiple directories, one for each month.
HISTORY=30
# cd /var/log/sa/
[root@test1
sa]# ls -l
total 8
drwxr-xr-x 2 root root 4096 Apr  2 10:10
201403
drwxr-xr-x 2 root root 4096 Apr  2 10:10 201404
lrwxrwxrwx 1 root root   11 Apr  2 10:10
sa02 -> 201404/sa02
lrwxrwxrwx 1 root root   12 Apr  2 10:10 sar02 -> 201404/sar02

As you can see different directories are getting created for different month and also separate sa and sar files (softlinks) are created inside sa directory pointing to their respective paths.

Collect Graphical reports using ksar

If you do not feel comfortable using CLI interface then there is an alternate option to use Graphical interface for all the reports.

For this purpose you will have to download kar package which can be done using the belw link
KSAR download

Once the package is downloaded. Extract it at any preferred location.
Move into the ksar directory created after extraction step, you should see the below directories/file.

Double click "Ksar.jar" java file. After which the below windows should come up.
Click on Data and select "Launch SSH Command.."

In the next pop up window provide the host name you want to connect using the below mentioned syntax
username@server-name
For exampledeepak@192.168.0.5

Click Yes to establish a connection with the host

Provide the password for the host you want to connect

Use the shown command as it will give you the complete graphical report for the whole day

Once the data import is finished you will see the below popup window. Click OK

Now you can select different parameters as shown in the left panel of the ksar to view all the reports. I have shown few examples using few screenshots

Interface traffic for eth0

Load average Statistics

Memory usage Statistics

CPU usage Statistics

Swap Memory usage statistics

References: