LFCS: Monitor Linux Processes Resource Usage and Set Process Limits on a Per-User Basis – Part 14

Monitor Linux ,Due to latest changes in the LFCS certification examination objectives powerful from February second, 2016, we are including the wished articles to the LFCS series posted here. To prepare for this examination, you’re strongly recommended to undergo the LFCE series as properly.

Monitor Linux ,Every Linux gadget administrator desires to recognise how to affirm the integrity and availability of hardware, resources, and key techniques. In addition, setting resource limits on a in step with-consumer basis need to also be part of his / her ability set.

In this article we can explore some approaches to ensure that the device both hardware and the software program is behaving efficiently to keep away from potential problems which could purpose surprising production downtime and money loss.

Monitor Linux ,Linux Reporting Processors Statistics

With mpstat you could view the sports for every processor in my view or the machine as a whole, both as a one-time photograph or dynamically.

In order to apply this device, you will need to put in sysstat:

# yum update && yum install sysstat              [On CentOS based systems]
# aptitutde update && aptitude install sysstat   [On Ubuntu based systems]
# zypper update && zypper install sysstat        [On openSUSE systems]

Read more approximately sysstat and it’s utilities at Learn Sysstat and Its Utilities mpstat, pidstat, iostat and sar in Linux

Once you have set up mpstat, use it to generate reviews of processors information.

To show 3 worldwide reports of CPU utilization (-u) for all CPUs (as indicated by means of -P ALL) at a 2-2nd c programming language, do:

# mpstat -P ALL -u 2 3
Monitor Linux ,Sample Output
Linux 3.19.0-32-generic (tecmint.com) 	Wednesday 30 March 2016 	_x86_64_	(4 CPU)

11:41:07  IST  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:41:09  IST  all    5.85    0.00    1.12    0.12    0.00    0.00    0.00    0.00    0.00   92.91
11:41:09  IST    0    4.48    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   94.53
11:41:09  IST    1    2.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00   97.00
11:41:09  IST    2    6.44    0.00    0.99    0.00    0.00    0.00    0.00    0.00    0.00   92.57
11:41:09  IST    3   10.45    0.00    1.99    0.00    0.00    0.00    0.00    0.00    0.00   87.56

11:41:09  IST  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:41:11  IST  all   11.60    0.12    1.12    0.50    0.00    0.00    0.00    0.00    0.00   86.66
11:41:11  IST    0   10.50    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   88.50
11:41:11  IST    1   14.36    0.00    1.49    2.48    0.00    0.00    0.00    0.00    0.00   81.68
11:41:11  IST    2    2.00    0.50    1.00    0.00    0.00    0.00    0.00    0.00    0.00   96.50
11:41:11  IST    3   19.40    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   79.60

11:41:11  IST  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:41:13  IST  all    5.69    0.00    1.24    0.00    0.00    0.00    0.00    0.00    0.00   93.07
11:41:13  IST    0    2.97    0.00    1.49    0.00    0.00    0.00    0.00    0.00    0.00   95.54
11:41:13  IST    1   10.78    0.00    1.47    0.00    0.00    0.00    0.00    0.00    0.00   87.75
11:41:13  IST    2    2.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   97.00
11:41:13  IST    3    6.93    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00   92.57

Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all    7.71    0.04    1.16    0.21    0.00    0.00    0.00    0.00    0.00   90.89
Average:       0    5.97    0.00    1.16    0.00    0.00    0.00    0.00    0.00    0.00   92.87
Average:       1    9.24    0.00    1.16    0.83    0.00    0.00    0.00    0.00    0.00   88.78
Average:       2    3.49    0.17    1.00    0.00    0.00    0.00    0.00    0.00    0.00   95.35
Average:       3   12.25    0.00    1.16    0.00    0.00    0.00    0.00    0.00    0.00   86.59

To view the same statistics for a specific CPU (CPU 0 in the following example), use:

# mpstat -P 0 -u 2 3
Monitor Linux Sample Output
Linux 3.19.0-32-generic (tecmint.com) 	Wednesday 30 March 2016 	_x86_64_	(4 CPU)

11:42:08  IST  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:42:10  IST    0    3.00    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00   96.50
11:42:12  IST    0    4.08    0.00    0.00    2.55    0.00    0.00    0.00    0.00    0.00   93.37
11:42:14  IST    0    9.74    0.00    0.51    0.00    0.00    0.00    0.00    0.00    0.00   89.74
Average:       0    5.58    0.00    0.34    0.85    0.00    0.00    0.00    0.00    0.00   93.23

The output of the above commands shows these columns:

  • CPU: Processor number as an integer, or the phrase all as a mean for all processors.
  • %usr: Percentage of CPU utilization while jogging user stage applications.
  • %first-rate: Same as %usr, but with satisfactory precedence.
  • %sys: Percentage of CPU usage that took place while executing kernel packages. This does now not consist of time spent dealing with interrupts or coping with hardware.
  • %iowait: Percentage of time while the given CPU (or all) became idle, in the course of which there has been a useful resource-extensive I/O operation scheduled on that CPU. A extra special clarification (with examples) may be located right here.
  • %irq: Percentage of time spent servicing hardware interrupts.
  • %tender: Same as %irq, but with software interrupts.
  • %steal: Percentage of time spent in involuntary wait (thieve or stolen time) while a digital machine, as visitor, is “prevailing” the hypervisor’s attention even as competing for the CPU(s). This fee have to be kept as small as viable. A high cost on this area means the virtual system is stalling – or soon could be.
  • %guest: Percentage of time spent walking a virtual processor.
  • %idle: percentage of time when CPU(s) have been not executing any responsibilities. If you look at a low value on this column, this is a demonstration of the system being positioned below a heavy load. In that case, you will need to take a more in-depth observe the procedure listing, as we are able to speak in a minute, to determine what is causing it.

To put the place the processor under a somewhat high load, run the following commands and then execute mpstat (as indicated) in a separate terminal:

# dd if=/dev/zero of=test.iso bs=1G count=1
# mpstat -u -P 0 2 3
# ping -f localhost # Interrupt with Ctrl + C after mpstat below completes
# mpstat -u -P 0 2 3

Finally, compare to the output of mpstat under “normal” circumstances:

As you may see inside the photograph above, CPU zero changed into beneath a heavy load all through the first examples, as indicated by the %idle column.

In the subsequent phase we can talk a way to discover those aid-hungry techniques, how to reap greater facts approximately them, and the way to take suitable motion.

Monitor Linux ,Reporting Linux Processes

To listing techniques sorting them by way of CPU usage, we will use the widely recognized playstation command with the -eo (to select all strategies with user-described format) and –type (to specify a custom sorting order) alternatives, like so:

# ps -eo pid,ppid,cmd,%cpu,%mem --sort=-%cpu

The above command will most effective display the PID, PPID, the command related to the technique, and the proportion of CPU and RAM utilization sorted via the share of CPU usage in descending order. When performed in the course of the creation of the .Iso document, here’s the primary few traces of the output:

Once we’ve got diagnosed a system of hobby (such as the only with PID=2822), we can navigate to /proc/PID (/proc/2822 in this situation) and do a listing list.

This listing is where several documents and subdirectories with targeted statistics approximately this unique method are saved while it’s far going for walks.

For example:

  • /proc/2822/io carries IO facts for the process (variety of characters and bytes examine and written, amongst others, for the duration of IO operations).
  • /proc/2822/attr/present day suggests the modern-day SELinux safety attributes of the technique.
  • /proc/2822/cgroup describes the control businesses (cgroups for brief) to which the method belongs if the CONFIG_CGROUPS kernel configuration alternative is enabled, which you may verify with:
# cat /boot/config-$(uname -r) | grep -i cgroups

If the option is enabled, you should see:


Using cgroups you may manipulate the quantity of allowed resource utilization on a consistent with-procedure foundation as explained in Chapters 1 via four of the Red Hat Enterprise Linux 7 Resource Management guide, in Chapter nine of the openSUSE System Analysis and Tuning guide, and inside the Control Groups segment of the Ubuntu 14.04 Server documentation.

The /proc/2822/fd is a directory that contains one symbolic hyperlink for every report descriptor the manner has opened. The following picture suggests this data for the technique that became started in tty1 (the primary terminal) to create the .Iso photo:

The above picture shows that stdin (file descriptor zero), stdout (report descriptor 1), and stderr (document descriptor 2) are mapped to /dev/0, /root/take a look at.Iso, and /dev/tty1, respectively.

More information approximately /proc may be found in “The /proc filesystem” record stored and maintained by way of Kernel.Org, and inside the Linux Programmer’s Manual.

Setting Resource Limits on a Per-User Basis in Linux

If you aren’t careful and permit any user to run a limiteless number of techniques, you could eventually experience an surprising system shutdown or get locked out as the system enters an unusable country. To prevent this from going on, you should place a limit on the range of techniques customers can begin.

To try this, edit /and so forth/safety/limits.Conf and add the following line at the bottom of the report to set the limit:

*   	hard	nproc   10

The first discipline can be used to suggest either a person, a set, or all of them (*), whereas the second field enforces a hard restrict on the range of method (nproc) to ten. To observe adjustments, logging out and again in is enough.

Thus, permit’s see what occurs if a sure user apart from root (either a valid one or now not) attempts to start a shell fork bomb. If we had not implemented limits, this would first of all release two instances of a feature, and then replica each of them in a neverending loop. Thus, it’d eventually bringing your device to a move slowly.

However, with the above limit in area, the fork bomb does not prevail however the consumer will still get locked out till the gadget administrator kills the procedure related to it:

TIP: Other possible restrictions made possible by ulimit are documented in the limits.conf file.

Linux Other Process Management Tools

In addition to the equipment mentioned previously, a machine administrator may additionally need to:

a) Modify the execution precedence (use of system assets) of a procedure the use of renice. This manner that the kernel will allocate more or much less gadget sources to the manner based totally at the assigned precedence (more than a few normally referred to as “niceness” in a range from -20 to 19).

The lower the price, the greater the execution precedence. Regular users (other than root) can only modify the niceness of procedures they very own to a better cost (meaning a lower execution priority), whereas root can alter this cost for any process, and might boom or lower it.

The simple syntax of renice is as follows:

# renice [-n] <new priority> <UID, GID, PGID, or empty> identifier

If the argument after the new precedence cost isn’t present (empty), it’s far set to PID by default. In that case, the niceness of manner with PID=identifier is set to .

B) Interrupt the ordinary execution of a process whilst wished. This is typically called “killing” the procedure. Under the hood, this means sending the technique a sign to complete its execution well and release any used sources in an orderly way.

To kill a process, use the kill command as follows:

# kill PID

Alternatively, you may use pkill to terminate all procedures of a given owner (-u), or a set proprietor (-G), or even those tactics that have a PPID in not unusual (-P). These options may be observed by means of the numeric illustration or the real name as identifier:

# pkill [options] identifier

For example,

# pkill -G 1000

will kill all processes owned by group with GID=1000.


# pkill -P 4993 

Will kill all methods whose PPID is 4993.

Before running a pkill, it is a superb idea to check the results with pgrep first, possibly the usage of the -l choice as nicely to list the methods’ names. It takes the equal options but best returns the PIDs of techniques (with out taking any similarly motion) that might be killed if pkill is used.

# pgrep -l -u gacanepa

This is illustrated in the next image:


In this text we have explored some approaches to monitor aid utilization so as to affirm the integrity and availability of crucial hardware and software program components in a Linux system.

We have also learned the way to take suitable motion (either by way of adjusting the execution priority of a given technique or by terminating it) beneath unusual occasions.

We desire the concepts explained in this educational had been useful.