# This file is part of Ian Kelling's automated-distro-installer # Copyright (C) 2024 Ian Kelling # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. PXE install w multi-boot, btrfs & Libreboot support Some things are specific to my home network, and uses files with secrets that are not in this repo. I use this for bare metal and vms, and two scripts which can run post boot so I use them on vps distributed image as well. Features people may find useful: installs encrypted trisquel, debian, ubuntu, arch, and parabola (archlike install is likely broken, I've only done pxe boots recently), in a multi-boot setup using multiple subvolumes of a single btrfs filesystem. Utilizes multiple disks, with scripts to automatically decrypt on intentional reboots, but not after shutdown or power loss. Normal install mode for fai is using pxe, but on a libreboot system, there is no pxe. The pxe in a normal computer is nonfree firmware. Alternatives to normal pxe that I've tried: * libreboot + seabios + ipxe * Use a live cd to call pxe-kexec, this is described later in this file. * Use the fai autodiscover iso. This is more automated, so nicer. * Use an install method above to setup a gnu/linux disk partition that coordinates with libreboot grub to acts like a pxe boot using kexec. The boot process takes a bit longer than normal pxe. This is the bootstrap partition in my scripts. Things I haven't tried: * The bios chip has enough room for an initrd. This could be setup to work like the partition I use to kexec, but it would be faster, and not require installing to disk. The partititioning and filesystem script is at fai/config/hooks/partition.DEFAULT. Disks are grouped as ssd or hdd and raided in raid 1 or raid 0 per configuration. The base partitions are divided into boot, swap, and root, (only boot is unencrypted). There are scripts to resize those partitions post-provision and while the system is running. People who use fai may find these things as useful examples: it uses dnsmasq (on a openwrt machine) for dhcp instead of the isc dhcp. fai-wrapper is a small script to use basic fai classes outside of fai. It does not use the fai partitioning tool, but the script is inspired from it and works outside of fai. It supports running a fai server on debian within android via Maru. It also automates configuration of an openwrt router after manual initial installation. After provisionining is done, I sync files using btrfs, or unison for vps, then automate further setup using a different set of scripts, https://iankelling.org/git/?p=distro-setup;a=tree. My network is a wndr3700v2 router with openwrt on it and a few pcs/laptops. Since fai requires a debian server as the fai server, there are also scripts to automate a debian install using pxe and preseeding, which can be done from any distro. Some of the scripts have dependencies for some simple obvious utility scripts from https://iankelling.org/git, and of course there are some hostnames that are specific to my network. # Per-host/install configuration Before doing a fai install, you will need to populate a class file. I use one called 51-multi-boot, which you can see example of in fai/config/class/50-host-classes. Before doing a fai install, you will need to populate /q/root/luks and /q/root/shadow, see their references. You might also want to copy existing /etc/ssh/*host* to /p/c/machine_specific/HOST/filesystem/etc/ssh host-* luks keyfiles generated like: h=demohost; head -c 2048 /dev/urandom | od | se dd of=/q/root/luks/host-$h Configuration of which luks key to use is in fai/config/hooks/partition.DEFAULT Configuration of which (if any) shadow file to use is in fai/config/distro-install-common/end and which shadow file / luks file(s) to copy into the new machine depends on fai-redep arguments. Also, setup dns in bind and wrt-setup-local. After install, btrbk to setup data, and then distro-begin && distro end. See notes in distro-begin for other configuration. # Scripts (meant to be used directly): # Setup the environment for the install # create tiny autodiscover cd # todo: with fai-revm at least, this complains about missing vmlinuz. need to fix this. fai-redep && sudo fai-cd -g $PWD/grub.cfg.autodiscover -f -A $BASEFILE_DIR/autodiscover.iso # create normal fai cd (replace TARGET_HOSTNAME) fai-redep -t TARGET_HOSTNAME && sudo fai-cd -M -g $PWD/grub.cfg.netinst-noreboot -f $BASEFILE_DIR/netinst.iso # note, may need to set hostname, depending on config, # and some other things for environment not on your lan # for example see fai/config/class/LINODE.var. See linode notes below. mymk-basefile # Create basefiles for various distros archlike-pxe # Setup pxe boot server from an archlike base image fai-redep # Deploy fai configuration to host "faiserver" faiserver-uninstall # uninstall fai-server faiserver-setup # install fai-server on the current machine myfai-chboot # setup fai tftp and nfs. useful for doing pxe-kexec pxe-server # disable/enable pxe dhcp, tfp, and nfs. calls myfai-chboot wrt-setup # setup my router in general: dhcp, dns, etc. # Script to do a distro install faiserver-revm # using pxe & preseed, create a vm which is a fai server dsfull # install & post-install a new fai distro arch-init-remote # install arch after it's been booted into it's setup env live-kexec # Kexec this or a remote machine using host faiserver. also useful to run as curl live-kexec|bash # Test scripts arch-revm # test arch install on a fresh vm fai-revm # test fai install on a fresh vm # Scripts to call after a distro install for various reasons chboot # Set grub to boot into a different distro (installed earlier) install-chboot # reinstall chboot to /boot subvols, for chboot updates. eboot # reboot without automatic disk decryption fai-wrapper # use fai classes outside of fai. sourced, not called. faiserver-disable # Disable the fai nfs server exports fresize # resize swap or boot partitions in a host # NAT/forward/vpn tftp I tried to get this working, but failed. tftp server in theory can be forwarded over a vpn, eg on a wireguard tunnel. However, I found that when actually pxe booting, it wouldn't work, only the 1st filename would be requested, eg, in the logs: Jun 20 23:51:02 kd in.tftpd[4021350]: RRQ from 10.2.0.12 filename pxelinux.0 To get that far, nating tftp requires some special attention in iptables, like so: https://unix.stackexchange.com/questions/579508/iptables-rules-to-forward-tftp-via-nat iptables -t raw -A PREROUTING -p udp --dport 69 -s 209.51.188.0/24 -j CT --helper tftp modprobe nf_nat_tftp to test tftp from a client machine: tftp SERVER_IP -c get pxelinux.0 rm -fv pxelinux.0 # Common problems ## kernel mismatch very early error, no remote logs: ERROR: the running kernel does not match the kernel modules inside the nfsroot. ERROR: Kernel modules directory /lib/modules/5.10.0-8-amd not available. Only found /lib/modules/5.10.0-15-amd64 solution: if running from fai-cd, recreate autodiscover cd as noted above in setup. # What good logs look like: logging nfs traffic from server s rpcdebug -m nfsd -s all normal nfs mount & umount logs look like: journalctl -ef | gr nfs Jun 20 22:15:36 kd rpc.mountd[2025725]: authenticated mount request from 10.32.2.1:865 for /srv/fai/nfsroot (/srv/fai/nfsroot) Jun 20 22:15:36 kd kernel: nfsd: exp_rootfh(/srv/fai/nfsroot [00000000e8c53e54] *:dm-0/5521225) Jun 20 22:15:36 kd kernel: nfsd: fh_compose(exp 00:1b/5521225 fai/nfsroot, ino=5521225) Jun 20 22:15:36 kd kernel: nfsd: FSINFO(3) 28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000 Jun 20 22:15:36 kd kernel: nfsd: fh_verify(28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000) Jun 20 22:15:36 kd kernel: nfsd: PATHCONF(3) 28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000 Jun 20 22:15:36 kd kernel: nfsd: fh_verify(28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000) Jun 20 22:15:36 kd kernel: nfsd: GETATTR(3) 28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000 Jun 20 22:15:36 kd kernel: nfsd: fh_verify(28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000) Jun 20 22:15:36 kd kernel: nfsd: FSINFO(3) 28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000 Jun 20 22:15:36 kd kernel: nfsd: fh_verify(28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000) Jun 20 22:15:36 kd kernel: nfsd: GETATTR(3) 28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000 Jun 20 22:15:36 kd kernel: nfsd: fh_verify(28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000) Jun 20 22:15:45 kd rpc.mountd[2025725]: authenticated unmount request from 10.32.2.1:986 for /srv/fai/nfsroot (/srv/fai/nfsroot) normal tftpd logs from: after setting -vv in TFTP_OPTIONS in /etc/default/tftpd-hpa journalctl -u tftpd-hpa Jun 20 23:51:02 kd in.tftpd[4021350]: RRQ from 10.2.0.12 filename pxelinux.0 Jun 20 23:51:02 kd in.tftpd[4021351]: RRQ from 10.2.0.12 filename ldlinux.c32 Jun 20 23:51:02 kd in.tftpd[4021352]: RRQ from 10.2.0.12 filename pxelinux.cfg/a913a477-fca6-234d-a928-6bb011decd05 Jun 20 23:51:02 kd in.tftpd[4021352]: sending NAK (1, File not found) to 10.2.0.12 Jun 20 23:51:02 kd in.tftpd[4021353]: RRQ from 10.2.0.12 filename pxelinux.cfg/01-52-54-00-9c-ef-ad Jun 20 23:51:02 kd in.tftpd[4021353]: sending NAK (1, File not found) to 10.2.0.12 Jun 20 23:51:02 kd in.tftpd[4021354]: RRQ from 10.2.0.12 filename pxelinux.cfg/0A02000C Jun 20 23:51:02 kd in.tftpd[4021355]: RRQ from 10.2.0.12 filename vmlinuz-5.10.0-15-amd64 Jun 20 23:51:03 kd in.tftpd[4021356]: RRQ from 10.2.0.12 filename initrd.img-5.10.0-15-amd64 # Replacing a raid 10 disk pxe-server -S HOST fai # btrfs replace or delete. prefer replace. to setup partitions on replacement drive: scp fai-wrapper HOST: ssh root@HOST . fai-wrapper export SPECIAL_DISK=/dev/REPLACEMENT_DEV /var/lib/fai/config/hooks/partition.DEFAULT ssh root@HOST for x in /target/* /target; do umount $x; done cat >p PASSWORD HERE(ctrl-d ctrl-d) cd /dev/disk/by-id/ for d in ata*part1; do cryptsetup luksOpen -d /root/p $d crypt_dev_$d; done x=(/dev/mapper/*part1); mount -o subvol=root_trisquelflidas $x /mnt # btrfs fi show /mnt # btrfs replace start -f /dev/mapper/OLD_DEV /dev/mapper/NEW_DEV /mnt # btrfs replace status /mnt # nohup btrfs dev delete /dev/sde1 /mnt mount -o subvol=boot_trisquelflidas /dev/sda3 /mnt/boot # also replace or delete disk for boot for x in dev proc sys; do mount -o bind /$x /mnt/$x; done chroot /mnt /bin/bash # replace disk in fstab # replace disk in /etc/crypttab update-grub update-initramfs -u mount /a /a/exe/keyscript-on exit reboot # Expected output in fai logs ## On focal: fai.log:updatebase.UBUNTU FAILED with exit code 1. the real error is dpkg-reconfigure locales, seems to be related to a workaround for < 20.04, relevant comment: # in case the locales are already included inside the base file (Ubuntu) in config/hooks/instsoft.DEBIAN ## For flidas, when installing systemd, this error happens, and it's a superflous upstream bug based on reading the post install script: addgroup: The group `systemd-journal' already exists as a system group. Exiting. Operation failed: No such file or directory ## On nabia/newer, python is removed, now its python3, and its easier to just let the package get removed than do host class package config. fai.log:WARNING: These unknown packages are removed from the installation list: python python-minimal Similar to python, linux-image-amd64 is the debian package name for the kernel, linux-image-generic is for ubuntu, but the DEBIAN class is defined on ubuntu and its easier to just let the package get removed with this warning: fai.log:WARNING: These unknown packages are removed from the installation list: linux-image-amd64 Also, cryptsetup-initramfs is new to buster/nabia, it gets removed on earlier versions. ## parted error fai.log:Error: /dev/vda: unrecognised disk label This is from parted -m $d unit MiB print. It happens when there are no partitions yet. ######## notes on creating a lan with just 2 computers ######## ## below assumes eth0 is the ethernet device used to connect to the target computer. # this is not strictly needed. I had my connection die at some point, # and I suspected this might help. # based on # https://support.qacafe.com/knowledge-base/how-do-i-prevent-network-manager-from-controlling-an-interface/ cat > /etc/NetworkManager/conf.d/99-fai-tmp.conf <<'EOF' [main] plugins=keyfile [keyfile] unmanaged-devices=interface-name:eth0 EOF ser restart NetworkManager cat >> /etc/network/interfaces <<'EOF' iface eth0 inet static address 10.0.44.1/24 EOF ifup eth0 # note turn off fsf vpn, so route to coresite is the normal route. echo 1 > /proc/sys/net/ipv4/ip_forward m s iptables -t nat -A POSTROUTING -o $(ip -4 route get 8.8.8.8 | sed -nr 's,^.* dev\s+(\S+).*,\1,p') -j MASQUERADE change /p/c/machine_specific/vps/bind-initial/db.b8.nz faiserver 10.0.44.1 TARGET_HOSTNAME 10.0.44.2 apt install isc-dhcp-server cat >> /etc/default/isc-dhcp-server <<'EOF' INTERFACESv4="eth0" EOF edit ./dhcpd.conf to change mac address and target host name. s cp /b/fai/dhcpd.conf /etc/dhcp/ ser restart isc-dhcp-server edit /a/bin/fai/fai/config/class/51-multi-boot pxe-server -d TARGET fai Then do a pxe boot on the target host ##### linode notes ###### * create 2 disks, installer (3000 mb, raw), boot (remaining, raw) * create 2 profiles w direct boot, no helpers: * installer (sda=boot, sdb=installer, boot dev=sdb) * boot (sda=boot) * Boot into rescue mode, ssh in with lish, curl url_to_some_fai_cd_created_image | dd of=/dev/sda poweroff * boot into installer. * Lish shows console, at the end of install, it gives prompt because logs failed to save remotely, check the logs, then reboot into boot profile if all is well. If that doesn't happen, turn off lassie in settings. ###### ubuntu notes ###### For someone who really needed ubuntu on host tp, otherwise they would end up on a non-gnu os, and I didn't want to figure out how to get all the default software installed, I did the following: # On remote host: # install etiona cd /b/fai # set 51-multi-boot to set classes outside of fai-wrapper conditional, including NOWIPE . fai-wrapper ./fai/config/hooks/partition.DEFAULT # on remote host # install ubuntu 20.04 using virt-install sudo -i virt-install --os-variant=ubuntu16.04 --cdrom ubuntu-20.04-desktop-amd64.iso --disk path=u2004.qcow2 -r 2048 --vcpus 1 -n u2004 qemu-img create -o preallocation=metadata -f qcow2 u2004.qcow2 15G # alternatively, also tried a physical install, because I know the virtual install ends up # with some differen things, like some spice service. then pulled the data out with rsync -ahSAX --numeric-ids --exclude=proc --exclude=sys --exclude=dev --exclude=tmp --exclude=run root@tp:/ .; mkdir proc sys dev tmp modprobe nbd qemu-nbd --connect=/dev/nbd0 u1804.qcow2 -f qcow2 qemu-nbd --connect=/dev/nbd0 u2004.qcow2 -f qcow2 mount /dev/nbd0p1 /mnt/1 # bionic mount /dev/nbd0p5 /mnt/1 # focal mount -o bind /mnt/root/root_ubuntubionic /mnt/2 mount -o bind /mnt/root/root_ubuntufocal /mnt/2 mkdir -p /mnt/2/boot mount -o bind /mnt/boot/boot_ubuntubionic /mnt/2/boot mount -o bind /mnt/boot/boot_ubuntufocal /mnt/2/boot # S = sparse, A = acls, X = xattrs rsync -ahSAX --numeric-ids /mnt/1/ /mnt/2 cd /mnt/2 cp /tmp/fai/crypttab etc sed -i "s#/root/keyscript,#decrypt_keyctl,#" etc/crypttab cp /tmp/fai/fstab etc echo "tmpfs /tmp tmpfs nodev,nosuid,size=50%,mode=1777 0 0" >> etc/fstab chrbind chroot . mv /etc/resolv.conf /etc/resolv.conf.old echo nameserver 1.1.1.1 >/etc/resolv.conf # install programs from /a/bin/fai/fai/config/package_config/STANDARD: apt install -y openssh-client openssh-server cryptsetup keyutils btrfs-progs console-setup kbd pciutils usbutils unattended-upgrades initramfs-tools-core dropbear-initramfs mv /etc/resolv.conf.old /etc/resolv.conf exit d=etc/initramfs-tools mkdir -p $d/root/.ssh etc/dropbear-initramfs root/.ssh chmod 700 $d/root $d/root/.ssh root/.ssh cp -p /root/.ssh/authorized_keys $d/root/.ssh/authorized_keys cp -p /root/.ssh/authorized_keys etc/dropbear-initramfs cp -p /root/.ssh/authorized_keys root/.ssh/authorized_keys chroot . sed -ri 's/^ *GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="rd.luks.crypttab=no"/' /etc/default/grub grub-install --no-floppy $(grub-probe -tdrive -d /dev/sda) update-grub grub-bios-setup -d /boot/grub/i386-pc -s /dev/sda exit umount proc umount dev umount sys reboot # for switching the boot to root2 zboot # for switching back, efibootmgr, if there is a problem with the root filesystem detection, # boot into the debian bootstrap distro, run partition.DEFAULT using comments for mktab arg. # then manually run iboot and then reboot. # pine rock64 notes # the only useful image is ubuntu 18.04 ayafun or something. # using emmc usb: s mount /dev/sdb7 /mnt/1 s cp `which qemu-arm-static` /mnt/1/usr/bin s chroot /mnt/1 qemu-arm-static /bin/bash usermod --login iank --move-home --home /home/iank rock46 groupmod --new-name iank rock64 passwd iank # boot it s apt-get update s apt dist-upgrade ### How to merge upstream fai-config git checkout upstream cd path-to-fai-config git pull --stat # the following needs modification if there was deletions or renames rsync --exclude /.git -rlpgoDcvi . /b/fai/fai/config/ cd /b/fai/fai/config/ # where XXXXX is the git commit hash # note, several files which just had trailing space changes will get ignored. git commit -am "update upstream to XXXXX" git checkout master git merge upstream # fix conflicts git commit # TODO Change arch to archlike and to support arch and parabola # License The license for the project is GPLv2 or later, mostly because fai is and I periodically merge the upstream example config, which contains small scripts. Also, there is a modified encrypt.upstream, which is from the cryptsetup package in arch, which is under the same license.