Calico verbose panic log: ip6tables-legacy-save failed
Problem
When you start the new version of Calico (e.g. 3.22) with docker, you may see the following panic logs repeatly from Calico log output when ipv6 related kernel parameter (/proc/sys/net/ipv6/conf/all/disable_ipv6) is disabled.
2022-05-31 02:35:15.284 [PANIC][14115] felix/table.go 769: ip6tables-legacy-save command failed after retries ipVersion=0x6 table="raw"
panic: (*logrus.Entry) 0xc00143d4f0
goroutine 498 [running]:
github.com/sirupsen/logrus.Entry.log({0xc0001321e0, 0xc0011704b0, {0x0, 0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0}, 0x0, ...)
/go/pkg/mod/github.com/projectcalico/logrus@v1.0.4-calico/entry.go:128 +0x56c
github.com/sirupsen/logrus.(*Entry).Panic(0xc000a18be0, {0xc0018c2b58, 0x1, 0x1})
/go/pkg/mod/github.com/projectcalico/logrus@v1.0.4-calico/entry.go:173 +0xfb
github.com/sirupsen/logrus.(*Entry).Panicf(0xc000a18be0, {0x2d636b2, 0x2d21148}, {0xc0018c2c10, 0x70222c5d2232332f, 0x2273656c69666f72})
/go/pkg/mod/github.com/projectcalico/logrus@v1.0.4-calico/entry.go:221 +0x70
github.com/projectcalico/calico/felix/iptables.(*Table).getHashesAndRulesFromDataplane(0xc00005f600)
/go/src/github.com/projectcalico/calico/felix/iptables/table.go:769 +0x3cc
github.com/projectcalico/calico/felix/iptables.(*Table).loadDataplaneState(0xc00005f600)
/go/src/github.com/projectcalico/calico/felix/iptables/table.go:606 +0x196
github.com/projectcalico/calico/felix/iptables.(*Table).Apply(0xc00005f600)
/go/src/github.com/projectcalico/calico/felix/iptables/table.go:990 +0x373
github.com/projectcalico/calico/felix/dataplane/linux.(*InternalDataplane).apply.func3(0x0)
/go/src/github.com/projectcalico/calico/felix/dataplane/linux/int_dataplane.go:1858 +0x52
created by github.com/projectcalico/calico/felix/dataplane/linux.(*InternalDataplane).apply
/go/src/github.com/projectcalico/calico/felix/dataplane/linux/int_dataplane.go:1857 +0x671
2022-05-31 02:35:16.721 [WARNING][14248] felix/table.go 814: iptables save failed error=exit status 1
Reason and explanation
New version of Calico supports ipv6 by relying on the following Linux kernel modules, and load them by running command ip6tables-legacy-save. However, these modules cannot be loaded properly if this command is triggered inside docker with isolated namespace.
ip6t_rpfilter
ip6table_nat
ip6table_raw
ip6table_mangle
ip6table_filter
vip6_tables
Solution
Manually run one of the ip6tables-* command (e.g. ip6tables-legacy-save) as root user, or use modprobe to load the corresponding modules.
sudo modprobe ip6t_rpfilter
sudo modprobe ip6table_nat
sudo modprobe ip6table_raw
sudo modprobe ip6table_mangle
sudo modprobe ip6table_filter
sudo modprobe ip6_tables
ipv6 related parameter
net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.default.disable_ipv6
net.ipv6.conf.lo.disable_ipv6