Understanding Load Average in Linux and when to be worried about it?

The one of the most important task of any Linux Admin includes performance monitoring which includes a parameter "Load Average" or "CPU Load".

Load Average is the value which represents the load on your system for a specific period of time. Also it can be considered the ratio of the number of active tasks to the number of available CPUs.

 

How to check?

You can use either top or uptime command to view the load average. The output would look like as shown below

# uptime
00:07:00 up 4 days, 6:14, 1 user, load average: 0.11, 0.14, 0.09

# top
top - 00:07:12 up 4 days, 6:15, 1 user, load average: 0.09, 0.13, 0.09

 

What are the three values?

As you can see three values representing the load average column. These show the load on your system over a significant period of time (one or current, five and fifteen minutes averages).

 

How do you know your system has a high load?

The most important question as in most cases I have seen how do you determine your system has high load.
Does a high value represents high load average and that your system requires atention?
What is the threshold value for load average?
How can we conclude if the load average value is good or bad?

Before I answer these question first let us understand about multi core and multi processor CPU.

A Central Processing Unit in earlier days used to be having only one processor and the core concept was not their in those days. But with the advancement in technology and the urge of higher speed to meet up demands of IT industry multiple processor were integrated in the same CPU making it multi-processor.

However increasing the no. of processor did increased the working speed of many tasks and performance but it also leads to increase in size, complexity and heat issues. So, in order to continue improvement of performance the core concept was introduced.

Instead of having two CPUs and a motherboard capable of hosting them, two CPUS are taken together and combined to form a dual core processor which will utilize an individual socket using less power and size capable of performing the same amount of task as dual processor CPU.

Bottom Line is that Load value depends on the no. of cores in your machine. For example a dual core is relevant to 2 processor or 2 cores and quad core is relevant to 4 processor or four cores as the maximum value for load.

 

How do I check the no. of cores on my Linux system?

Number of physical CPUs
The following command will show how many active physical processors a system has.
Example: If this number is 2, one could potentially open up the system chassis and remove 2 physical processors with one's hands.

# grep physical.id /proc/cpuinfo | sort -u | wc -l
2

 

Number of cores per CPU
On a system with multi-core processors, the following command should report the number of CPU cores per physical processor (though in rare cases it might not).
Example: If this number is 8 and physical CPUs is 2, then each of the 2 physical processors has 8 CPU cores, leading to a total of 16 cores.

# grep cpu.cores /proc/cpuinfo | sort -u
cpu cores : 8

 

Number of logical processors
This last command will show the total number of "logical" processors seen by the Linux kernel. This number is usually the most important of the three stats. It is the effective number of processors -- as far as the operating system is concerned, this is the number of distinct CPUs that can do work at any given microsecond.

Example: Continuing with the above example scenario, the number seen below could be 16 instead of 8. Simply put, if this command shows a different number than the total number of CPU cores, it's because hyper-threading is enabled on the CPUs, further dividing each core (in this example, into 2 usable "threads").

# grep processor /proc/cpuinfo | wc -l
16

How to check if Hyper Threading is enabled or disabled?

Coming back to our primary question

 

Understanding Load Average

If the number of active tasks utilising CPU is less as compared to available CPU cores then the load average can be considered normal but if the no. of active tasks starts increasing with respect to available CPU cores then the load average will start rising.

For example in my case I am using a dummy script to utilise all my CPU and increase the load

top - 20:29:27 up  1:41,  2 users,  load average: 11.11, 3.74, 1.44
Tasks: 257 total,  17 running, 240 sleeping,   0 stopped,   0 zombie
%Cpu(s):100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 13173656+total, 27243444 free, 10403308+used,   460040 buff/cache
KiB Swap:  4190204 total,  4190204 free,        0 used. 26855740 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
22360 root      20   0  196724   5684    736 R 100.0  0.0   1:07.32 python
22361 root      20   0  196724   5684    736 R 100.0  0.0   1:07.33 python
22363 root      20   0  196724   5688    736 R 100.0  0.0   1:07.32 python
22364 root      20   0  196724   5688    736 R 100.0  0.0   1:07.32 python
22369 root      20   0  196724   5700    736 R 100.0  0.0   1:07.30 python
22371 root      20   0  196724   5704    736 R 100.0  0.0   1:07.29 python
22373 root      20   0  196724   5704    736 R 100.0  0.0   1:07.28 python
22374 root      20   0  196724   5708    736 R 100.0  0.0   1:07.26 python
22359 root      20   0  196724   5684    740 R 100.0  0.0   1:07.30 python
22362 root      20   0  196724   5688    740 R 100.0  0.0   1:07.32 python
22365 root      20   0  196724   5688    736 R 100.0  0.0   1:07.32 python
22366 root      20   0  196724   5692    736 R 100.0  0.0   1:07.32 python
22367 root      20   0  196724   5692    736 R 100.0  0.0   1:07.32 python
22368 root      20   0  196724   5692    736 R 100.0  0.0   1:07.32 python
22370 root      20   0  196724   5700    736 R 100.0  0.0   1:07.23 python
22372 root      20   0  196724   5704    736 R 100.0  0.0   1:07.29 python

So as per the no. of cores I calculated i.e 16 cores and seeing the value 11.11 Although this is high but I shouldn't be worried much unless it crosses the red line value i.e. 16 for my case.

I hope the article was useful.

 

Related Articles
Tutorial for Monitoring Tools SAR and KSAR with examples in Linux
9 examples to help you understand top command usage in Unix/Linux
15 tips to enhance security of your Linux machine
How to check all the currently running services in Linux

 

11 thoughts on “Understanding Load Average in Linux and when to be worried about it?”

  1. In the section "How do I check the no. of cores on my Linux system?" we can also check the cpu cores by using "top" command and Press 1 (one), when the top command is running, which will break the CPU down and show details for all the individual CPUs running on the system.

    Reply
  2. Hello Dipanjan,

    As what you are saying is about the logical CPU which can not be mistaken with no of core your system has. As explained in my calculation above the logical CPU or processor is calculated as

    2 (physical ID) x 4 (core-id) x 2 (hyper threading) = 16 processors

    From what I wrote above
    The hyper threading presents 2 logical CPUs to the operating system for each actual core so it effectively doubles the no. of logical CPU in your system

    # less /proc/cpuinfo | grep "cpu cores" | uniq
    cpu cores : 4

    With ht enabled it becomes 4 x 2 = 8 cores

    I hope I make sense.

    Reply
  3. Hyperthreading does not magically give you more physical cores. It does NOT effectively double the number of logical cores on your system.

    Hyperthreading is nice because it does reduce the cost of switching processes on a core, but it cannot reduce that cost to 0. A hyperthreaded core will perform better than a single core, but not nearly as fast as 2 physically separate cores.

    Here is a good article explaining how much of a performance boost you might be able to expect: dasher.com/blog/will-hyper-threading-improve-processing-performance/

    Reply
  4. I agree with you it just gives an impression to the running processes of doubling the physical processor creating logical processors the same way we use virtual CPUs in VMware with a single core CPU server.

    Since the article was about Load Average so I skipped explaining much for hyper threading but thanks for the info.

    Reply
  5. Hi Deepak Prasad
    Thanks for sharing information on load average.
    I have a question load average value from TOP command output.
    If you execute the TOP command in RHEL OS it will list all VCPU(including hyperthread) instead only CPU cores then i feel load average value also might have calculated based on all VCPUs.
    How do we make sure RHEL code only taking CPU core while calculating load average?
    For example i have a server with 2 CPU and each processor has 2 core and hyperthreading enabled.
    In this case the total VCPUs is 8.
    The top command will list 8 VCPUs , here load average below 8 should be normal.
    Plese give your suggestion.

    Thanks
    Ramachandiran V

    Reply

Leave a Comment