RAM and Virtual Memory: diagnosis overview (Linux)

RAM Overview

Memory Shortages
UNIX memory shortages impact performance once virtual memory is impacted. Therefore, when we are investigating possible memory shortages, we are investigating virtual memory as well. This is an overview of diagnosing memory issues.

Memory Diagnosis (Linux)

Summary of Tools

The following are used to diagnose memory usage issues in Linux:
  • /proc/meminfo
  • free
  • sar (arguments: -r -W -B -R)
  • ps aux
  • top
  • vmstat

Note that memory is expressed in page size. The page sizes can be determined by one of these commands:
getconf PAGESIZE
getconf PAGE_SIZE

The /proc file system (also known as procfs) provides system information. The kernel reports information by writing it into virtual files under /proc, including information about processes. Tools like ps get information about processes by reading these virtual files. /proc/meminfo has memory usage information. Many of the utilities SAs rely on to for memory information get their data from /proc/meminfo.

You can get memory, cache and swap information with this command:
egrep 'Mem|Cache|Swap' /proc/meminfo

Each running process has a subdirectory named after the PID. The status virtual file in this subdirectory provides information about the status of the process. The maps file has information on how the process's virtual memory is used. To see the information about these files, run:
  • cat /proc//status
  • cat /proc//maps

Memory-specific fields in /proc//status:
  • VmSize: Virtual memory size. (Same as vsz parameter in ps). Because Linux only allocates physical memory when an application tries to use it, this value may be much greater than the amount of physical memory the application is actually using.
  • VmLck: Locked memory size. Amount of memory used by this process that cannot be swapped to disk.
  • VmHWM: Peak resident set size ("high water mark").
  • VmRSS: Resident set size. The amount of physical memory this process is using. (Same as rss statistic in ps).
  • VmData, VmStk, VmExe: Size of data (virtual size of data usage, the dsiz statistic in ps), stack, and text segments.
  • VmLib: Shared library code size (the size of the libraries this process is using).
  • VmPTE: Page table entries size (since Linux 2.6.10).

Memory-specific fields in /proc//maps:
  • address: the address space in the process that it occupies
  • perms: r = read, w = write, x = execute, s = shared, p = private (copy on write)
  • dev: the device (major:minor)
  • inode: 0 indicates that no inode is associated with the memory region (uninitialized data)
  • pathname

Most SAs will run free first to diagnose memory usage issues. Here is some sample output:

$ free
              total       used        free    shared    buffers    cached
Mem:        4040360    4012200       28160         0     176628   3571348
-/+ buffers/cache:      264224     3776136
Swap:       4200956      12184     4188772

Explanations of this example (values in KB):
  • 4040360: the total amount of available memory
  • 264224: are used by processes ("used" less "buffers" and "cached", i.e., 4012200 - 176628 - 3571348)
  • 3776136: amount free for other applications ("total" less amount used by processes (see previous line), i.e., 4040360 - 264224)
  • 176628: buffers (file system metadata)
  • 3571348: cache (pages with actual contents of files or block devices)
  • 28160: this reflects memory used for buffers and cache. Since the use of buffers and cache improve performance by reducing repeated I/O and because buffers and cache are easily cleared, you will see a low value in this field except, perhaps, when the server has been recently rebooted and has not started doing any work

These arguments can provide more readable output:
  • -b,-k,-m,-g show output in bytes, KB, MB, or GB
  • -t display total for RAM + swap

These arguments provide a running update:
  • -s update every [delay] seconds
  • -c update [count] times

sar reports data that has already been gathered by two programs, called sa1 and sa2. They are run as cron jobs. The output from these jobs are stored in /var/log/sa. The following setting in /etc/sysconfig/sysstat indicates that output should be saved for 28 days:

If the cron jobs are not running, sar will not have any data to report.

The following reports are used for diagnosing memory issues.
  • sar -r
  • sar -W
  • sar -B
  • sar -R

ps aux
The ps command with arguments aux will provide memory usage per process. Try piping the output to awk:

ps aux | awk '{ print $4, $11 }' | sort -n

This will show %MEM (field 4) and the command (field 11) and sort by %MEM.

The ps command also has sorting functionality. Here are two examples, the first is to sort by PID (process ID) and the second by memory usage:
ps aux --sort -pid
ps aux --sort -rss

To reverse sort order, remove the "-" from our argument:
ps aux --sort pid
ps aux --sort rss

The output from ps aux is in the following columns:
  • USER: username running the process
  • PID:  process ID
  • %CPU: percentage of time spent running during the entire lifetime of a process
  • %MEM: RAM usage
  • VSZ: Virtual memory size of the process in K
  • RSS: Physical memory in use by process in K (does not include swap space)
  • TTY: associated terminal
  • STAT: process state
  • START: Start time of process. If more than 24 hours old, will show a date.
  • TIME: Cumulative CPU time used
  • COMMAND:  Command associated with process, including arguments

Memory-specific Options for ps
The ps command can retrieve information on amount and type of memory a process is using with this command:

ps [-o vsz,rss,tsiz,dsiz,majflt,minflt,pmem,command]

These options are:
  • -o: Allows you to specify exactly which process statistics you want to track. These are specified in a comma-seperated list (no spaces).
  • vsz: virtual memory size of the process in KiB (1024-byte units).
  • rss: resident set size, the non-swapped physical memory that a task has used (in kiloBytes).
  • tsiz: Text size, the virtual size (not the physical size) of the program code. A good indication of the size of the program.
  • dsiz: Data size, the virtual size of the program's usage. A good indication of the size of the data structures and stack of the application.
  • majflt: The number of major page faults (caused Linux to read a page from disk) that have occurred with this process. These faults are caused by reading text or data from the program image from storage.
  • minflt: The number of minor page faults (caused Linux to read a page from memory instead of cache) that have occurred with this process. These faults are caused by memory allocation.
  • pmem: %mem, ratio of the process's resident set size  to the physical memory on the machine, expressed as a percentage
  • command: the actual command that invoked the process

Click here for information on vmstat

Suggestions for Future Learning
This information will be in the second edition of UNIX For Application Support Staff . The ETA for the second edition is December 1, 2016.

RAM Overview
Technical articles