Oom Killer

From GoBlueMich Wiki
Jump to navigation Jump to search

OOM Killer set on some servers; it is designed to help minimized downtime after the server begins to reach an Out of Memory State.

This helps find basics about OOMs when there is not a load monitoring script set up yet.

Recent OOM_Killer kills

You may be able to see patterns of many more php processes than http, http having high memory usage, or other common problems if there are oom_killer logs.

Oom Killer Count

grep oom_kill /var/log/messages -c

What Was Killed

grep "kernel: Out of memory"  /var/log/messages

PHP Count vs httpd

Compare counts of php processes versus httpd processes:

for i in $(grep "`date +%b' '%e`".*oom_kill /var/log/messages| awk '{print $3}'); do grep "${i}" /var/log/messages |grep  httpd -c;done
for i in $(grep "`date +%b' '%e`".*oom_kill /var/log/messages| awk '{print $3}'); do grep "${i}" /var/log/messages |grep  php -c;done

If there are many more php processes than http then you may need want to increase the httpd Timeout or reduce php settings such as memory_limit or max_execution_time.

General User/Proc Counts

Review all listed processes:


for i in $(grep "`date +%b' '%e`".*oom_kill /var/log/messages| awk '{print $3}'|sed s'/.$//'); do grep "${i}" /var/log/messages|awk '/kernel:/ {print $7,$NF}' | sort |uniq -c |sort -rnk1;done

Top Procs Being Killed

Today's top 5 processes which are listed having the most memory usage per minute timestamp:

for i in $(grep "`date +%b' '%e`".*oom_kill /var/log/messages| awk '{print $3}'|sed s'/.$//'); do grep "${i}" /var/log/messages|grep kernel: | sort -rnk 11|head -n5;done
for i in $(grep "`date +%b' '%e`".*oom_kill /var/log/messages| awk '{print $3}'|sed s'/.$//'); do grep "${i}" /var/log/messages|grep kernel: | sort -rnk 10 | head -n5;done

Worst Top Procs Being Killed

When there are many oom_killer instances in a given day this will parse the most occurring as the largest processes.

for i in $(grep "`date +%b' '%e`".*oom_kill /var/log/messages| awk '{print $3}'|sed s'/.$//'); do grep "${i}" /var/log/messages|grep kernel: | sort -rnk 11|head -n5;done |awk '{print $NF}'|sort|uniq -c|sort -rnk1|head

Procs Being Killed

Review today's occurrences of common load issue processes mentioned in oom_killer logs:

for i in $(grep "`date +%b' '%e`".*oom_kill /var/log/messages| awk '{print $3}'|sed s'/.$//'); do grep "${i}" /var/log/messages|grep kernel: | grep -i 'upcp\|backup\|cpanellog\|find\|pkg\|easy' |head -n5;done

Sysctl Variables Affecting Oom Killer (CentOS 7)

vm.admin_reserve_kbytes

The amount of free memory in the system that should be reserved for users with the capability cap_sys_admin.

vm.admin_reserve_kbytes defaults to min(3% of free pages, 8MB)

That should provide enough for the admin to log in and kill a process, if necessary, under the default overcommit 'guess' mode.

Systems running under overcommit never should increase this to account for the full Virtual Memory Size of programs used to recover. Otherwise, root may not be able to log in to recover the system.

How do you calculate a minimum useful reserve?

sshd or login + bash (or some other shell) + top (or ps, kill, etc.)

For overcommit 'guess', we can sum resident set sizes (RSS). On x86_64 this is about 8MB.

For overcommit 'never', we can take the max of their virtual sizes (VSZ) and add the sum of their RSS. On x86_64 this is about 128MB.

Changing this takes effect whenever an application requests memory. This is measured in KiB.

  • 131072KiB = 128MiB
  • 8192KiB = 8MiB

vm.oom_kill_allocating_task

This enables or disables killing the OOM-triggering task in out-of-memory situations.

If this is set to zero, the OOM killer will scan through the entire tasklist and select a task based on heuristics to kill. This normally selects a rogue memory-hogging task that frees up a large amount of memory when killed.

If this is set to non-zero, the OOM killer simply kills the task that triggered the out-of-memory condition. This avoids the expensive tasklist scan.

If panic_on_oom is selected, it takes precedence over whatever value is used in oom_kill_allocating_task.

The default value is 0.


vm.overcommit_kbytes

When vm.overcommit_memory is set to 2, the committed address space is not permitted to exceed swap plus this amount of physical RAM. See below.

Note: vm.overcommit_kbytes is the counterpart of vm.overcommit_ratio. Only one of them may be specified at a time. Setting one disables the other (which then appears as 0 when read).


vm.overcommit_memory

This value contains a flag that enables memory overcommitment.

When this flag is 0, the kernel attempts to estimate the amount of free memory left when userspace requests more memory.

When this flag is 1, the kernel pretends there is always enough memory until it actually runs out.

When this flag is 2, the kernel uses a "never overcommit" policy that attempts to prevent any overcommit of memory. Note that vm.user_reserve_kbytes affects this policy.

This feature can be very useful because there are a lot of programs that malloc() huge amounts of memory "just-in-case" and don't use much of it.

The default value is 0.


vm.overcommit_ratio

When vm.overcommit_memory is set to 2, the committed address space is not permitted to exceed swap plus this percentage of physical RAM. See above.


Sysctl Variables Affecting Oom Killer (CentOS 5-6)

admin_reserve_kbytes

The amount of free memory in the system that should be reserved for users with the capability cap_sys_admin.

admin_reserve_kbytes defaults to min(3% of free pages, 8MB)

That should provide enough for the admin to log in and kill a process, if necessary, under the default overcommit 'guess' mode.

Systems running under overcommit never should increase this to account for the full Virtual Memory Size of programs used to recover. Otherwise, root may not be able to log in to recover the system.

How do you calculate a minimum useful reserve?

sshd or login + bash (or some other shell) + top (or ps, kill, etc.)

For overcommit 'guess', we can sum resident set sizes (RSS). On x86_64 this is about 8MB.

For overcommit 'never', we can take the max of their virtual sizes (VSZ) and add the sum of their RSS. On x86_64 this is about 128MB.

Changing this takes effect whenever an application requests memory. This is measured in KiB.

  • 131072KiB = 128MiB
  • 8192KiB = 8MiB

oom_kill_allocating_task

This enables or disables killing the OOM-triggering task in out-of-memory situations.

If this is set to zero, the OOM killer will scan through the entire tasklist and select a task based on heuristics to kill. This normally selects a rogue memory-hogging task that frees up a large amount of memory when killed.

If this is set to non-zero, the OOM killer simply kills the task that triggered the out-of-memory condition. This avoids the expensive tasklist scan.

If panic_on_oom is selected, it takes precedence over whatever value is used in oom_kill_allocating_task.

The default value is 0.


overcommit_kbytes

When overcommit_memory is set to 2, the committed address space is not permitted to exceed swap plus this amount of physical RAM. See below.

Note: overcommit_kbytes is the counterpart of overcommit_ratio. Only one of them may be specified at a time. Setting one disables the other (which then appears as 0 when read).


overcommit_memory

This value contains a flag that enables memory overcommitment.

When this flag is 0, the kernel attempts to estimate the amount of free memory left when userspace requests more memory.

When this flag is 1, the kernel pretends there is always enough memory until it actually runs out.

When this flag is 2, the kernel uses a "never overcommit" policy that attempts to prevent any overcommit of memory. Note that user_reserve_kbytes affects this policy.

This feature can be very useful because there are a lot of programs that malloc() huge amounts of memory "just-in-case" and don't use much of it.

The default value is 0.


overcommit_ratio

When overcommit_memory is set to 2, the committed address space is not permitted to exceed swap plus this percentage of physical RAM. See above.


List Defaults for Variables (CentOS 5-6)

NOTE: For CentOS 7+, use the CentOS 5-6 version of variable (remove "vm." at front of variable)

  • Display the defaults of those variables. Note these in ticket.
echo -e "\n";for variable in $(echo -e "admin_reserve_kbytes \n oom_kill_allocating_task \n overcommit_memory \n overcommit_ratio \n swappiness"); do echo "$variable" && cat /proc/sys/vm/$variable && echo -e "\n"; done


Make Changes to Variable Values (CentOS 5-7)

Setting value via procfs

You can use standard echo command to write data to variables (this temporary change):

echo "value" > /proc/sys/vm/variable

Temporary on the command line

Use sysctl command with -w option when you want to change a sysctl setting:

sysctl -w variable=value

Configuration file /etc/sysctl.conf

This is recommended way. First open /etc/sysctl.conf file, enter:

vim /etc/sysctl.confNow add value:
$variable = value

Close and save the changes. Type the following command to load sysctl settings from the file /etc/sysctl.conf file:

sysctl -pOR
sysctl -p /etc/sysctl.conf

This method will load settings permanently at boot time from /etc/sysctl.conf file.

Template:Warning

Example Changes

Template:Warning


  • Edit conf file as root
vim /etc/sysctl.conf


  • EXAMPLE Variables:


  • CenOS 5-6
admin_reserve_kbytes=131072
oom_kill_allocating_task=1
swappiness=10

Additional Changes if you know what you're doing

overcommit_memory=2
overcommit_ratio=100
  • CentOS 7+
vm.admin_reserve_kbytes=131072
vm.oom_kill_allocating_task=1
vm.swappiness=10

Additional Changes if you know what you're doing

vm.overcommit_memory=2
vm.overcommit_ratio=100

Note: These changes will not take affect until reboot unless you run:

sysctl -p

To make live changes (not persistent), echo variable to `/proc/sys/vm/$variable` .

Ex:

  • Check current setting:
cat /proc/sys/vm/admin_reserve_kbytes
8192
  • Change current setting:
echo 131072 > /proc/sys/vm/admin_reserve_kbytes

Analogy

An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused.