Linux kernel: neighbour table overflow

ARP cache is overflowing. Most likely reason - too much traffic on the network.

kernel error

dmesg |grep "neighbor table overflow"

check gc_thresh

sysctl -a | grep net.ipv4.neigh.default.gc_thresh

default value

/proc/sys/net/ipv4/neigh/default/gc_stale_time:60
/proc/sys/net/ipv4/neigh/default/gc_thresh1:128
/proc/sys/net/ipv4/neigh/default/gc_thresh2:512
/proc/sys/net/ipv4/neigh/default/gc_thresh3:1024

gc_stale_time 决定检查一次相邻层记录的有效性的周期。当相邻层记录失效时，将在给它发送数据前，再解析一次。缺省值是60秒。
gc_thresh1 存在于ARP高速缓存中的最少层数，如果少于这个数，垃圾收集器将不会运行。缺省值是128。
gc_thresh2 保存在ARP高速缓存中的最多的记录软限制。垃圾收集器在开始收集前，允许记录数超过这个数字5秒。缺省值是 512。
gc_thresh3 保存在ARP高速缓存中的最多记录的硬限制，一旦高速缓存中的数目高于此，垃圾收集器将马上运行。缺省值是1024。

gc_stale_time (since Linux 2.2)
       Determines how often to check for stale neighbor entries.  When a neighbor entry is considered stale, it is resolved again before sending data to  it.
       Defaults to 60 seconds.

gc_thresh1 (since Linux 2.2)
       The  minimum  number  of  entries  to keep in the ARP cache.  The garbage collector will not run if there are fewer than this number of entries in the
       cache.  Defaults to 128.

gc_thresh2 (since Linux 2.2)
       The soft maximum number of entries to keep in the ARP cache.  The garbage collector will allow the number of entries to exceed this for 5 seconds  be‐
       fore collection will be performed.  Defaults to 512.

gc_thresh3 (since Linux 2.2)
       The  hard  maximum number of entries to keep in the ARP cache.  The garbage collector will always run if there are more than this number of entries in
       the cache.  Defaults to 1024.

analysis

arp -v

## sum the arp record number
arp -an | wc -l

best practice

/etc/sysctl.conf

## works best with <= 500 client computers ##
# Force gc to clean-up quickly
net.ipv4.neigh.default.gc_interval = 3600

# Set ARP cache entry timeout
net.ipv4.neigh.default.gc_stale_time = 3600

# Setup DNS threshold for arp 
net.ipv4.neigh.default.gc_thresh3 = 4096
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh1 = 1024

sysctl -p

reference

man 7 arp
https://openai.com/blog/scaling-kubernetes-to-2500-nodes
https://www.cyberciti.biz/faq/centos-redhat-debian-linux-neighbor-table-overflow

Disclaimer

License under CC BY-NC 4.0
Copyright issue feedback me#imzye.me, replace # with @
Not all the commands and scripts are tested in production environment, use at your own risk
No privacy information is collected here

Try iOS App

Yi's Blog