September 13, 2015

Linode iptables-persistent not starting on Debian Jessie

Today I noticed that my Linode does not restore its iptables configuration after reboot. This is how the investigation progressed. I have opened a ticket with Linode to update the relevant documentation.

In Debian Linux, there is a dedicated package to save and restore iptables configuration called iptables-persistent and implemented as two plugins (IPv4 and IPv6) for the netfilter-persistent framework. This, by the way, obsoletes the advice given in the Securing your server Linode guide, which instructs the administrator to create a new network startup script.

The netfilter-persistent service was activated, but since the configuration was not updating the service was apparently failing. Trying to run it manually produced this:

$ sudo systemctl start netfilter-persistent
A dependency job for netfilter-persistent.service failed. See 'journalctl -xn' for details.
$ sudo journalctl -xn
-- Logs begin at Sun 2015-09-13 16:17:33 EDT, end at Sun 2015-09-13 16:43:55 EDT. --
Sep 13 16:43:51 systemd-modules-load[3452]: could not open moddep file '/lib/modules/...-x86_64-linode/modules.dep.bin'
Sep 13 16:43:51 systemd-modules-load[3452]: Failed to lookup alias 'loop': Function not implemented
Sep 13 16:43:51 systemd[1]: systemd-modules-load.service: main process exited, code=exited, status=1/FAILURE
Sep 13 16:43:51 systemd[1]: Failed to start Load Kernel Modules.
-- Subject: Unit systemd-modules-load.service has failed
-- Defined-By: systemd
-- Support:
-- Unit systemd-modules-load.service has failed.
-- The result is failed.
Sep 13 16:43:51 systemd[1]: Dependency failed for netfilter persistent configuration.
-- Subject: Unit netfilter-persistent.service has failed
-- Defined-By: systemd
-- Support:
-- Unit netfilter-persistent.service has failed.
-- The result is dependency.
Sep 13 16:43:51 systemd[1]: Unit systemd-modules-load.service entered failed state.

This is already enough to find Debian bug 769356 which explains the problem. netfilter-persistent depends on modules.builtin, which includes the loop kernel module:

$ cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

However, Linodes initialized with a Debian template use a custom kernel that does not provide any builtin modules, so the configuration in /etc/modules is not correct. There are several fix options: correct /etc/modules by commenting out the loop module, use a different kernel if the loop module is required, or changing the netfilter-persistent service description to not depend on builtin modules. The first option is the easiest choice.

The other relevant page in Linode’s documentation is Control network traffic with iptables, which I have asked them to update following this investigation.