A couple of days ago, Xavier posted about Atuin to gain more insight into the command history. Atuin does a great job of better organizing what is usually handled by "bash_history" and collecting meaningful additional data. Our reader David commented that this can also be done quite well with Linux's kernel process accounting feature, and I think he is very right. I really like Linux process accounting for a number of reasons, so here is a quick introduction.
Process accounting is a kernel feature. You will not see a specific process responsible for it. Instead, the "accton" command signals the kernel to start logging process data to a specific location (usually /var/log/account/pacct). Once a process terminates, the kernel will log respective details to the binary log file.
1 - Installation
I don't think process accounting is enabled by default on any Linux system. It does add a little additional overhead, but some users may shy away from it because it requires additional disk writes to collect the information. Memory and other CPUs should not be significantly impacted by process accounting. On my not very busy Proxmox system, it uses about 50 MB/day of disk space. So nothing that should be noticeable for most systems.
Installation usually comes down to installing the respective package for your distribution. On Debian based distributions, it is just
apt install acct
This will typically also configure the startup scripts, but it can't hurt to run
systemctl enable --now acct
That is it. Wait a little bit, and you will see the log.
2 - How to read the logs
Logs are saved in a binary format. The "lastcomm" command can be used to display the log in a readable format. For example:
ip6tables-save S root __ 0.00 secs Wed Aug 12 06:25
iptables-restor S root __ 0.00 secs Wed Aug 12 06:25
iptables-save S root __ 0.00 secs Wed Aug 12 06:25
check_ssh 100107 __ 0.00 secs Wed Aug 12 06:25
cron F 100000 __ 0.00 secs Wed Aug 12 06:25
sh S 100000 __ 0.00 secs Wed Aug 12 06:25
debian-sa1 100000 __ 0.00 secs Wed Aug 12 06:25
These are a few lines from my Proxmox server. It logs the process name, Flags (S=super user, F=forked process, D=generated core dump, X=terminated by signal), User name (or ID), CPU execution time, and finally the timestamp at which the process was started. The output may be modified slightly depending on the command-line arguments used.
3 - Remote Logging
Unlike most Linux logs, these logs are not created by syslog. However, you may still read them with syslog to forward them to a central log collector/SIEM. Syslog-ng for example include a "s_pacct" processor for process accounting logs. You enable it with this configuration:
source s_pacct {
pacct(file("/var/log/account/pacct"));
};
4 - Other useful tools
The "sa" command can be used to easily extract summaries from accounting data. For example, a breakdown by CPU time used by different processes
# sa -c | head -10
329063 100.00% 259245.37re 100.00% 70.29cp 100.00% 0avio 24320k
469 0.14% 55.38re 0.02% 52.65cp 74.90% 0avio 124752k ffmpeg
488 0.15% 5.43re 0.00% 4.57cp 6.51% 0avio 5338k apt-get
2820 0.86% 3.60re 0.00% 3.34cp 4.74% 0avio 13295k ceph
1231 0.37% 1.96re 0.00% 1.92cp 2.73% 0avio 1974k ps
312 0.09% 1907.83re 0.74% 1.36cp 1.93% 0avio 88176k named
7 0.00% 39483.07re 15.23% 0.76cp 1.08% 0avio 6348k systemd-journal
123 0.04% 34804.71re 13.43% 0.34cp 0.48% 0avio 20367k **other
8 0.00% 0.68re 0.00% 0.27cp 0.38% 0avio 3926k store
366 0.11% 0.58re 0.00% 0.26cp 0.36% 0avio 2235k dpkg-deb*
5 - Containers
Process accounting is a kernel feature, and the kernel must be compiled and configured to support process accounting. If you are running Linux containers in Proxmox (the platform I am using), process accounting will not work unless the container is privileged. But it does not have to work. The container processes are logged by the host, which I think is actually better. This way, the logs are more easily centralized, and they can't be tampered with from inside the container.
6 - Conclusion
I think Linux kernel process accounting is a very neat and often overlooked feature. You may be able to do more fine-grained inspection with eBPF, but process accounting is "ready to go and useful" with little work. It does not log command line options, which may be an issue in incident response. But it is a very good supplement to other features like bash_history files, and it captures processes that bash_history would never see.
--
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu
Twitter|