Detailed CPU Usage Overview With psacct

Published on Author gryzli

If you ever asked yourself the following questions, this might be the right place for you:

How to check how much cpu is consuming each system user ?

How to check how much cpu is consumed by each command ?

Check when a command has been executed and how much cpu it did ?

How to check all commands executed by a given user ?

What is the total cpu my system is consuming ?

These are pretty standard sysadmin questions, which arise soon or later in your day-to-day practice.

There is a pretty nice interface inside your linux kernel, called “Process accounting“, which you can benefit a lot by using “psacct“.

 

What information Process Accounting could give us ?

If you are using recent Linux distribution, for example Centos 6 or Centos 7, the structure of each process accounting record, contains the following fields (it’s the acct_v3 one):

 

By enabling process accounting, you will get such structure for each process that has finished it’s execution.

So basically you have access to pretty much information about each process, which could be effectively used for debugging,troubleshooting or performance optimizations later on.

In order to make your life easier, there is the ‘psacct’ package, which makes handling this information pretty easy.

 

Installing psacct under Centos 6/7

The first thing you need to do is to install psacct and enable it (which will also enable the process accounting inside your kernel).

The process is easy enough:

# Install

yum install psacct

 

# Enable Centos 7

systemctl enable psacct

# Enable Centos 6

chkconfig psacct on

 

Finally start the process

# Start Centos 7/ Centos 6

service psacct start

 

Effectively Using psacct To Monitor CPU Usage

Now that you have installed psacct, you will have access to some nice commands, most important of which are:

sa, lastcomm, ac and dump-acct

Before going any further, you should know the following:

*) Process accounting is made on a daily basis. Whatever information you see it is for the current day

*) By default linux kernel is using “/var/account/pacct” file for writing the process information. Which means, that all the tools provided by ‘psacct’ are working with this file

*) There is logrotate config installed, which makes sure to rotate your ‘pacct’ file on a daily basis and keep the older files in the same directory “/var/account/” for a historic analysis when needed

 

Using sa  to check cpu usage summary

sa is the tool to use for checking summary information for your cpu consumption. Most of the time you will use it to get summary either by ‘user’ or by ‘command’

Showing cpu usage summary by command

The most important stats from this output are:

column1:  Total number of executions for the given command

column2:  Total number of time elapsed (the time, the process was running) in minutes

column3:  Total number of CPU time consumed by the command ( this is the sum of sys time + user time) in minutes

column6:  The command executed by execve() syscall, limited to 16 chararcters

 

You should know, that when you are executing scripts like perl, python, bash, php , the command that will be accounted is going to be the binary executable.

For example running “php some_script.php” will be seen just as “php”.

This makes life a bit harder, because you can’t see the actual script name, that has been executed.

 

Showing cpu usage summary by user

There is one more interesting execution variation of sa, which is sa -m

Here is how the output looks like:

Most important columns are:

Column1:  System username

Column4:  CPU time consumed by the user (for today) in minutes

 

Using lastcomm to check each command executed

Now as you know, how to see the CPU summaries, you may want to dig deeper, by observing when and what each user has executed. This is really nice technique for debugging the cpu usage by a given user or by command.

 

Showing All Commands By User (–user option)

Let’s say we want all commands executed by user “root”

The output contains the following:

Col1: Command nam

Col2: Flag. These are the meanings of the flags:
S — command executed by super-user
F — command executed after a fork but without a following exec
C — command run in PDP-11 compatibility mode (VAX only)
D — command terminated with the generation of a core file
X — command was terminated with the signal SIGTERM

Col3:  username

Col5: CPU time consumed in seconds (both system + user time)

Col6: Date/Time of the command execution

 

 

Showing All Commands By Command (–command option)

We could also print commands by command name, which is done with the –command option. 

Let say we need who and when executed tar:

 

You could also add multiple “–command” options if you like to view information for multiple commands at the same time. Same applies for the “–user” option as well.

Using dump-acct to dump information from /var/account/pacct

As I mentioned earlier, ‘/var/account/pacct’ is generated directly by the kernel and is in binary format. Sometimes you may need to parse/dump the contents of this file. One easy way to do this (without writing software) is to use the psacct provided dump-acct command.

 

Here is an example:

 

The output from the dump-acct is in the following format:

The cpu related timings “ac_utime, ac_stime, ac_etime” are all measured in CPU ticks. On most systems , 1 cpu second is equal to 100 CPU ticks.

 

Pitfalls

There are some important pitfalls, you should consider before/during using accounting tools.

Command names shortening

Currently all command names are shortened to 16 characters (at least on Centos 6/7 systems).  The limitation comes from the acct v3 structure itself.

 

CPU by long-running processes

As stated earlier, accounting works only for processes, that has finished their execution. It is important to understand, that when you are checking CPU summary with (sa or sa -m), you don’t see the CPU time consumed, by still or long-running processes, like daemons are.

Usually such processes are: php, httpd, nginx, mysqld, dovecot, exim, ….etc….

Also keep in mind, that when long-running process has terminated, the accounting subsystem, will account all the CPU that has been made by this process. So even that you know, you are seeing accounted cpu times, for the current day, this doesn’t apply for terminated long-running processes.

 

Using sa/lastcomm too frequently is expensive !

If you are planning on automating and scraping the results from sa/lastcomm for some reason, keep in mind that running sa/sa -m/ lastcomm is expensive, especially on a systems with huge accounting files.

Before doing such thing, make sure to calculate the CPU time, that your sa command will consume (time sa -m );