Analyzing Unexpected Memory Consumption on a MySQL Slave Server: Hidden Systemd Session Files and tmpfs

Phenomenon

Monitoring alerts indicated that a machine's free memory fell below 10%. The top command showed the following (partial) output:

[root@mysql-slaver ~]# top
top - 13:45:43 up 1835 days, 20:52,  2 users,  load average: 0.02, 0.03, 0.05
Tasks: 210 total,   1 running, 208 sleeping,   1 stopped,   0 zombie
%Cpu(s):  0.5 us,  0.6 sy,  0.0 ni, 98.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32780028 total,   905684 free, 19957900 used, 11916444 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  3448260 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                            
 2677 mysql     20   0   20.1g  15.1g   3392 S   0.0 48.2 430:17.58 mysqld                                                             
10549 polkitd   20   0 3277476   3.1g    632 S   0.3  9.9 146:47.24 redis-server                                                       
18183 root      20   0  877308 215868   1892 T   2.7  0.7   2736:45 xxxxxx
  442 root      20   0  160244  93016  88552 S   0.3  0.3 314:14.86 systemd-journal                                                    
32537 root      20   0  731620  58360  54588 S   0.3  0.2  29:09.61 rsyslogd

The total memory is 32 GB, with 19 GB used and 11 GB in buff/cache, leaving only about 3 GB available. MySQL and Redis together consume roughly 18.2 GB, yet the remaining 8 GB of “used” memory is unaccounted for.

Analysis

Running free -m gave a clearer view:

[root@MySQL-slaver ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:          32011       19490         881        8762       11639        3366
Swap:             0           0           0

The shared column shows about 8 GB, which originates from tmpfs (Shmem in /proc/meminfo).

Inspecting df -h reveals the tmpfs mounts:

[root@MySQL-slaver ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G   16K   16G   1% /dev/shm
tmpfs            16G  8.6G  7.1G  55% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup

The /run mount consumes 8.6 GB, matching the shared memory figure. Digging deeper with du -am /run|sort -rn -k1|head -10 shows the biggest consumers:

[root@MySQL-slaver ~]# du -am /run|sort -rn -k1|head -10
8761    /run
7137    /run/systemd
7126    /run/systemd/users
1624    /run/log/journal/89308070e0c04c6a86bf577f4064efca
1624    /run/log/journal
1624    /run/log

The /run/systemd/users directory unusually occupies ~7 GB. Listing its contents shows only a tiny file, prompting suspicion of hidden files:

[root@MySQL-slaver ~]# ls -l /run/systemd/users
total 44
-rw-r--r-- 1 root root 41056 Mar 23 14:14 0

Using find /run/systemd/users uncovers hundreds of thousands of hidden files (names starting with .#), many dating back to 2018.

[root@MySQL-slaver ~]# find /run/systemd/users|wc -l
337632

Examining one of these files shows it stores private session data for the root user:

[root@MySQL-slaver ~]# less /run/systemd/users/.#03DvfrF
# This is private data. Do not parse.

NAME=root
STATE=active
RUNTIME=/run/user/0
SLICE=user-0.slice
DISPLAY=4231719
REALTIME=1521010223727718
MONOTONIC=79029110787
SESSIONS=4232100 4232099 4232098 ......

Querying loginctl list-sessions confirms that the root user has over 2 000 active sessions, most created by crond in 2018 and never terminated.

[root@MySQL-slaver ~]# loginctl list-sessions
   SESSION        UID USER             SEAT            
     24597          0 root                             
    146401          0 root                             
    133160          0 root                             
     82494          0 root                             
     82514          0 root                             
    106049          0 root   
......
[root@MySQL-slaver ~]# loginctl list-sessions|awk '{print $3}'|sort|uniq -c
      1 
      1 listed.
   2131 root
      1 USER

These stale sessions occupy the hidden files and contribute to the memory pressure.

Solution

Two practical options are suggested:

Since the server mainly runs MySQL (as a replica) and Redis, shrink innodb_buffer_pool_size to free memory, and later migrate Redis off the host before considering a reboot.

If a reboot is not possible, move the unused hidden files to another disk or delete the stale sessions with loginctl kill-session <ID>. The hidden files are not actively used (as shown by lsof), so removing them safely reclaims memory.

The author chose option 1 and reduced MySQL’s buffer pool size.

Keywords: memory, tmpfs, crond