I can netboot Ubuntu 20.04 on a Raspberry Pi 4b over NFS, using a Synology DS 1618+ as a TFTP and NFS server. But I would like to protect the root file system with overlayroot so several machines can run simultaneously from the same root. This just will not work for me. I asked a question a few days ago on Ask Ubuntu but haven't really received any useful insights. I also understand more about the problem now and would like to rephrase the question here for a broader audience.
The Problem
Although I can netboot Ubuntu 20.04 on an RPi4 using NFS, as soon as I enable overlayroot (overlayroot="tmpfs:recurse=0") the system starts up in a degraded state (systemctl is-system-running). It seems clear that this has something to do with overlayroot.
In this state, only root can log in. No other users get past the login/password prompt.
Examination of syslog reveals that the first thing to go wrong during boot is the startup of system-networkd, which fails with the message ("Operation not supported"). Closer examination reveals that systemd-networkd tries to run as a user (system-network). Since system-netwworkd doesn't start, neither do a number of other services:
UNIT LOAD ACTIVE SUB DESCRIPTION
● atd.service loaded failed failed Deferred execution scheduler
● avahi-daemon.service loaded failed failed Avahi mDNS/DNS-SD Stack
● systemd-networkd.service loaded failed failed Network Service
● systemd-resolved.service loaded failed failed Network Name Resolution
● systemd-timesyncd.service loaded failed failed Network Time Synchronization
● systemd-networkd.socket loaded failed failed Network Service Netlink Socket
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
6 loaded units listed.Configuration
DHCPd
I have a pair of ISC DHCP servers which have been running well for over a year. I made the following alterations to their configuration to support network booting:
# DHCP configuration for PXE boot of RPI4
option tftp-server-name "192.168.8.20"; #option 66
option bootfile-name "bootcode.bin"; #option 67
option vendor-class-identifier code 60 = string;
option vendor-encapsulated-options code 43 = string;
option space RPi code width 1 length width 1;
option RPi.discovery code 6 = unsigned integer 8;
option RPi.menu-prompt code 10 = text;
option RPi.menu-item code 9 = text;
option vendor-class-identifier "PXEClient";
option vendor-encapsulated-options "Raspberry Pi Boot";
vendor-option-space RPi;
option RPi.discovery 3;
option RPi.menu-prompt "PXE";
option RPi.menu-item "Raspberry Pi Boot";
filename "pxelinux.0";
next-server 192.168.8.20;
option tftp-server-address 192.168.8.20;I will admit that I don't entirely understand this, but it works :-) I'm sure there's stuff in there that the RPi4 doesn't use.
Raspberry Pi 4b
I have set up the RPi4 boot loader as follows:
$ sudo rpi-eeprom-config
[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=1
BOOT_ORDER=0xf21
TFTP_PREFIX=1
TFTP_PREFIX_STR=RPi4-Ubuntu/Synology DS 1618+ NAS
TFTP Server
At startup the RPi4 sends the TFTP_PREFIX_STR above and receives the following files back from the TFTP server on the Synology NAS:
bcm2710-rpi-2-b.dtb bootcode.bin initrd.img
bcm2710-rpi-3-b.dtb boot.scr overlay_map.dtb
bcm2710-rpi-3-b-plus.dtb cmdline.txt overlays
bcm2710-rpi-cm3.dtb config.txt start4cd.elf
bcm2710-rpi-zero-2.dtb fixup4cd.dat start4db.elf
bcm2711-rpi-400.dtb fixup4.dat start4.elf
bcm2711-rpi-4-b.dtb fixup4db.dat start4x.elf
bcm2711-rpi-cm4.dtb fixup4x.dat start_cd.elf
bcm2837-rpi-3-a-plus.dtb fixup_cd.dat start_db.elf
bcm2837-rpi-3-b.dtb fixup.dat start.elf
bcm2837-rpi-3-b-plus.dtb fixup_db.dat start_x.elf
bcm2837-rpi-cm3-io3.dtb fixup_x.dat vmlinuzThese are from the Ubuntu 20.04 installation image for RPi (partition boot). I have made the following modifications to config.txt:
[pi4]
# Run as fast as firmware / board allows
arm_boost=1
[all]
arm_64bit=1
device_tree_address=0x03000000
enable_uart=1
cmdline=cmdline.txt
kernel=vmlinuz
initramfs initrd.img followkernel
include syscfg.txt
include usercfg.txtand cmdline.txt:
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.8.20:/volume3/pxe/nfs/RPi4-Ubuntu/OS,tcp,rw ip=dhcp rootfstype=nfs elevator=deadline rootwaitNFS Server
The NFS server serves up the nfsroot mentioned in cmdline.txt as the root (/). This is basically the rootfs partition from the Ubuntu bistro. It also serves up the files from the Ubuntu distro's boot partition, which are mounted at /boot.
exports -v shows the following:
/volume3/pxe 192.168.0.0/21(rw,async,no_wdelay,hide,crossmnt,no_subtree_check,insecure_locks,anonuid=1024,anongid=100,sec=sys,insecure,root_squash,no_all_squash)
/volume3/pxe 192.168.8.0/21(rw,async,no_wdelay,hide,crossmnt,no_subtree_check,insecure_locks,anonuid=1024,anongid=100,sec=sys,insecure,root_squash,no_all_squash)
/volume3/pxe 192.168.32.0/21(rw,async,no_wdelay,hide,crossmnt,no_subtree_check,insecure_locks,anonuid=1024,anongid=100,sec=sys,insecure,root_squash,no_all_squash)
/volume3/pxe 192.168.72.0/21(rw,async,no_wdelay,hide,crossmnt,no_subtree_check,insecure_locks,anonuid=1024,anongid=100,sec=sys,insecure,root_squash,no_all_squash)During the boot process, it seems that Synology NFS and the RPi4 will only use NFSv3. In /etc/fstab, however, they both seem happy to use NFSv4/NFSv4.1.
Back to the Pi
With this constellation the RPi boots and runs well over the net with an NFS file system as root. However, at this point, wishing to protect the root file system from being written to, I proceed to modify the configuration file of overlayroot at /etc/overlayroot.conf so it looks like the following:
overlayroot_cfgdisk="disabled"
#overlayroot=""
overlayroot="tmpfs:recurse=0,debug=1"(I have omitted the 160+ lines of comment before this.)
When I reboot the problem described above ensues.
Help! Please?
I have tried the above with 32-bit and 64-bit Ubuntu 22.04 with exactly the same result. I tried Raspberry OS 11 (64-bit), but it doesn't offer overlayroot as an option. Its alternative, built into raspi-config, doesn't work with NFS. So I didn't pursue that.
I greatly simplified the TFTP configuration for Ubuntu, however, based on what Raspberry OS does.
I wouldn't be surprised if the problem had something to do with the Synology NFS server, but I don't really know where to start looking.
I certainly don't preclude the idea that I have made a mistake somewhere, so I'd be grateful if anyone could point any such out.
As soon as Ubuntu 22.04 comes out, I'll try that too.
There are lots of people out in the Net who have talked about doing this. I've read a lot of articles and tried to take their messages onboard. I've seen one or two allusions to possible problems with NFS root file systems and overlayroot. But I've not seen anything that describes exactly what I'm trying to do.
