A.J. Clark

Recently a concerned application administrator asked me why our Hobbit monitoring program was reporting that our Solaris 8 Broadvision portal servers are using a high amount of swap space. I decided to perform a little analysis to get a better understanding of what was happening and, more importantly ascertain if the swap usage is causing any performance problems. 

The server in question is a Sun Fire V440 with 8GB of physmem, 8GB swap and two UltraSPARC III 1.2Ghz processors. 

The swap(1M) command shows that there is over 6.4GB of swap used. The df(1M) command shows that /tmp (which uses swap for storage) is virtually empty too. 

# swap -s

total: 4633816k bytes allocated + 2120280k reserved = 6754096k used, 8108600k available

# df -k /tmp

Filesystem            kbytes    used   avail capacity  Mounted on

swap                 8110168    3592 8106576     1%    /tmp

The prstat(1M) command shows some interesting things:

• All processes are in a sleep state. This means that they are likely to have their Last Recently Used (LRU) memory pages “paged out” to swap. This paging is done by the Solaris scheduler (sched)
• Combined, the processes are requesting more memory (13GB) than that of which is physically available (8GB) to the system. Thus pages of the processes’ memory are ‘paged out’ to swap. This would bring the swap usage to around 5/6GB – as evidenced by swap(1M). Remember that in Solaris “virtual memory” is composed of both real RAM and SWAP which means this box has 16GB of virtual memory (8GB physmem 8GB swap)

# prstat -a -s rss

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 14808 bvstg     569M  427M sleep   58    0   0:00.00 0.7% bvsmgr/18
  2090 bvqa      822M  357M sleep   58    0   0:25.47 0.0% contentdistribu/5
 14734 bvstg     370M  269M sleep   50    0   0:37.24 0.0% bvsmgr/16
 14802 bvstg     355M  259M sleep   58    0   0:55.39 0.0% bvsmgr/16
 14626 bvstg     346M  252M sleep   58    0   0:00.02 0.0% bvsmgr/16
  2123 bvqa      280M  249M sleep   58    0   0:13.25 0.0% procserver/6
   328 bvqa      330M  222M sleep   38    0   0:00.37 0.0% bvsmgr/16
   317 bvqa      309M  190M sleep   58    0   0:00.01 0.0% bvsmgr/15
 12755 bvqaprod  322M  169M sleep   58    0   0:00.02 0.0% bvsmgr/14
 28147 bvqa3     315M  133M sleep   59    0   0:00.13 0.0% bvsmgr/14
  2016 tomusr    338M  124M sleep   10    0   0:00.03 0.0% java/27
  2047 tomusr    332M  117M sleep   58    0   0:00.03 0.0% java/27
  2061 tomusr    326M  111M sleep    0    0   0:00.03 0.0% java/26
  2029 tomusr    326M  111M sleep   21    0   0:00.03 0.0% java/27
 15358 bvstg     148M   98M sleep   58    0   0:00.08 0.8% java/11
 28072 bvqa3     288M   70M sleep   58    0   0:00.00 0.0% bvsmgr/13
 14350 bvqaprod   83M   65M sleep   58    0   0:00.01 0.0% genericdb/6
 15372 bvstg     109M   59M sleep   58    0   0:00.02 0.1% java/12
 14433 bvqaprod  285M   58M sleep   58    0   0:00.03 0.0% sched_srv/12
  4017 bvqa      281M   54M sleep   58    0   0:00.02 0.0% sched_srv/12
 20370 bvstg     281M   52M sleep   58    0   0:00.01 0.0% sched_srv/12
 26662 bvstg      92M   48M sleep   53    2   0:00.00 0.0% java/12
  2104 bvqa       56M   47M sleep   59    0   0:00.11 0.0% frtsobj/22
 14767 bvstg      55M   39M sleep   58    0   0:01.03 0.0% cntdb/7
 14641 bvstg      57M   39M sleep   58    0   0:01.02 0.0% cntdb/7
 28085 bvqa3      54M   38M sleep   58    0   0:01.20 0.0% cntdb/7
   233 bvqa       55M   37M sleep   58    0   0:00.20 0.0% cntdb/7
 12767 bvqaprod   54M   36M sleep   58    0   0:00.08 0.0% cntdb/7
   477 root       37M   32M sleep   58    0   0:00.00 0.7% aex-pluginmanag/16
 20498 bvqa       78M   28M sleep   58    0   0:00.00 0.0% java/12

 NPROC USERNAME  SIZE   RSS MEMORY      TIME  CPU
    71 bvstg    4107M 2015M    25%   1:52.28 0.1%
    53 bvqa     2980M 1449M    18%   5:24.40 0.0%
    94 bvqa3    4013M 1022M    13%   0:12.24 0.0%
     4 tomusr   1323M  463M   5.9%   0:00.12 0.0%
    17 bvqaprod  994M  400M   5.0%   0:13.41 0.0%
    44 root      194M  122M   1.5%   2:10.51 0.2%
    12 bb         13M   11M   0.1%   0:05.20 0.0%
     1 daemon   2584K 1776K   0.0%   0:00.00 0.0%

To make sure that these processes were not all sharing a large amount of shared memory (shmsys) the pmap(1M) command was used. The results show that most of the address space is Private.

# pmap -x 14808
14808:  bvsmgr p_1221_32 -install_name bvsmgr/sarug31532/ccwstg/BV_SessionMana
 Address   Kbytes Resident Shared Private Permissions       Mapped File
00010000      48      40      40       - read/exec         bvsmgr
0002A000      24      24       -      24 read/write/exec   bvsmgr
... <snip> ...
FF3E2000       8       8       -       8 read/write/exec   ld.so.1
FFBE6000      40      40       -      40 read/write          [ stack ]
--------  ------  ------  ------  ------
total Kb  558488  530304  111344  418960

To get a sample of freemem/vs freeswap sar(1M) was used. The sar(1M) output shows that the amount of free memory pages is increasing (reclaimed)

# sar -r 5 5
21:04:04 freemem freeswap
21:04:09  293980 16213779
21:04:14  294135 16216864
21:04:19  294136 16216864
21:04:24  294136 16216864
21:04:29  294725 16249363

Average   294222 16222736

The output of the vmstat(1M) command shows a couple of interesting things:

• No processes are in a RUN/BLOCK/SWAP state (evidenced by prstat(1M) above)
• The reclaim column shows that free memory is being relclaimed
• The scan rate column shows no activity. Presumably because all processes are in sleep state.
• The disk/md3 column shows that no I/O activity is happening on the swap slice
• Low activity is occuring within the minor faults column. This suggests that the application is requesting data from pages of memory which have been ‘paged out’ – but this is occurring at a very low rate. Probably due to the low overall load of the box.

# vmstat 5 5
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m3 m5   in   sy   cs us sy id
 0 0 0 9647152 3462488 161 513 246 3 2 0 0  0  0  0  4  482  482  559  5  3 91
 0 0 0 8108784 2353456 37 104 0 0  0  0  0  0  0  0  1  345 5054 6750  0  2 98
 0 0 0 8108784 2353456 32 79 0  0  0  0  0  0  0  0  0  357 5050 6793  0  1 99
 0 0 0 8108784 2353456 32 79 0  0  0  0  0  5  0  0  0  385 5052 6848  0  1 99
 0 0 0 8108784 2353472 93 247 0 0  0  0  0  0  0  0  0  349 6593 6975  1  2 98

Note: vmstat -S was not used.

Summary

Inactive pages are being swapped out because the box does not have enough physical memory to store them all in RAM. Although even if there was enough physical memory inactive pages would still be swapped as soon as freemem fell below lotsfree. The paging is encouraged by the fact that all the processes are in a sleep state and the load is low. 

As long as the load does not suddenly increase exponentially it is not necessary to increase the physical memory.