kernel: Out of socket memory

Solution for this is to increase the TCP memory. This can be done by adding the following parameters to /etc/sysctl.conf.

net.core.wmem_max=12582912
net.core.rmem_max=12582912
net.ipv4.tcp_rmem= 10240 87380 12582912
net.ipv4.tcp_wmem= 10240 87380 12582912

These figures are just an example and need to be tuned per system basis. On the similar lines tcp_max_orphans sysctl variable value can be increased but it has memory overhead of ~64K per orphan entry and needs careful tuning.

• For more information on tuning socket buffers refer to: How to tune the TCP Socket Buffers?

There are three factors which may cause the problem,

1. The networking behavior of your system. for example, how many TCP socket created on your system.
2. How much system RAM in your system.
3. The following two system kernel parameters.

/proc/sys/net/ipv4/tcp_mem
/proc/sys/net/ipv4/tcp_max_orphans

an example in 1GB RAM system
# cat /proc/sys/net/ipv4/tcp_max_orphans 
32768

# cat /proc/sys/net/ipv4/tcp_mem 
98304     131072     196608

The meaning of the two kernel parameters,

1. tcp_max_orphans -- Maximal number of TCP sockets not attached to any user file handle held by system. If this number is exceeded orphaned connections are reset immediately and warning is printed. The default value of this parameter on RHEL5.2 is 32768.
2. tcp_mem -- vector of 3 INTEGERs: min, pressure, max.

• min: below this number of pages TCP is not bothered about its memory appetite.
• pressure: when amount of memory allocated by TCP exceeds this number of pages, TCP moderates its memory consumption and enters memory pressure mode, which is exited when memory consumption falls under "min". The memory pressure mode presses down the TCP receive and send buffers for all the sockets as much as possible, until the low mark is reached again.  
• max: number of pages allowed for queuing by all TCP sockets.

If the number of orphan socket is more than the value of tcp_max_orphans, there may trigger the messages "kernel: Out of socket memory".

If the total number of memory page which are assigned to all the system TCP socket is more than the max value of tcp_mem., there may trigger the messages "kernel: Out of socket memory".

Both the situations above will trigger the messages "kernel: Out of socket memory".

How much memory is use by TCP/UDP across the system?