PXE install w multi-boot, btrfs & Libreboot support Some things are specific to my home network, and uses files with secrets that are not in this repo. I use this for bare metal and vms, and two scripts which can run post boot so I use them on vps distributed image as well. Features people may find useful: installs encrypted trisquel, debian, ubuntu, arch, and parabola (archlike install is likely broken, I've only done pxe boots recently), in a multi-boot setup using multiple subvolumes of a single btrfs filesystem. Utilizes multiple disks, with scripts to automatically decrypt on intentional reboots, but not after shutdown or power loss. Normal install mode for fai is using pxe, but on a libreboot system, there is no pxe. The pxe in a normal computer is nonfree firmware. Alternatives to normal pxe that I've tried: * libreboot + seabios + ipxe * Use a live cd to call pxe-kexec, this is described later in this file. * Use the fai autodiscover iso. This is more automated, so nicer. * Use an install method above to setup a gnu/linux disk partition that coordinates with libreboot grub to acts like a pxe boot using kexec. The boot process takes a bit longer than normal pxe. This is the bootstrap partition in my scripts. Things I haven't tried: * The bios chip has enough room for an initrd. This could be setup to work like the partition I use to kexec, but it would be faster, and not require installing to disk. The partititioning and filesystem script is at fai/config/hooks/partition.DEFAULT. Disks are grouped as ssd or hdd and raided in raid 1 or raid 0 per configuration. The base partitions are divided into boot, swap, and root, (only boot is unencrypted). There are scripts to resize those partitions post-provision and while the system is running. People who use fai may find these things as useful examples: it uses dnsmasq (on a openwrt machine) for dhcp instead of the isc dhcp. fai-wrapper is a small script to use basic fai classes outside of fai. It does not use the fai partitioning tool, but the script is inspired from it and works outside of fai. It supports running a fai server on debian within android via Maru. It also automates configuration of an openwrt router after manual initial installation. After provisionining is done, I sync files using btrfs, or unison for vps, then automate further setup using a different set of scripts, https://iankelling.org/git/?p=distro-setup;a=tree. My network is a wndr3700v2 router with openwrt on it and a few pcs/laptops. Since fai requires a debian server as the fai server, there are also scripts to automate a debian install using pxe and preseeding, which can be done from any distro. Some of the scripts have dependencies for some simple obvious utility scripts from https://iankelling.org/git, and of course there are some hostnames that are specific to my network. # Per-host/install configuration Before doing a fai install, you will need to populate a class file. I use one called 51-multi-boot, which you can see example of in fai/config/class/50-host-classes. Before doing a fai install, you will need to populate /q/root/luks and /q/root/shadow, see their references. You might also want to copy existing /etc/ssh/*host* to /p/c/machine_specific/HOST/filesystem/etc/ssh host-* luks keyfiles generated like: head -c 2048 /dev/urandom | od | s dd of=/q/root/luks/host-demohost Configuration of which luks key to use is in fai/config/hooks/partition.DEFAULT Configuration of which (if any) shadow file to use is in fai/config/distro-install-common/end and which shadow file / luks file(s) to copy into the new machine depends on fai-redep arguments. # Scripts (meant to be used directly): # Setup the environment for the install # create tiny autodiscover cd # todo: with fai-revm at least, this complains about missing vmlinuz. need to fix this. fai-redep && sudo fai-cd -g $PWD/grub.cfg.autodiscover -f -A $BASEFILE_DIR/autodiscover.iso # create normal fai cd (replace TARGET_HOSTNAME) fai-redep -t TARGET_HOSTNAME && sudo fai-cd -M -g $PWD/grub.cfg.netinst-noreboot -f $BASEFILE_DIR/netinst.iso # note, may need to set hostname, depending on config, # and some other things for environment not on your lan # for example see fai/config/class/LINODE.var. See linode notes below. mymk-basefile # Create basefiles for various distros archlike-pxe # Setup pxe boot server from an archlike base image fai-redep # Deploy fai configuration to host "faiserver" faiserver-uninstall # uninstall fai-server faiserver-setup # install fai-server on the current machine myfai-chboot # setup fai tftp and nfs. useful for doing pxe-kexec pxe-server # disable/enable pxe dhcp, tfp, and nfs. calls myfai-chboot wrt-setup # setup my router in general: dhcp, dns, etc. # Script to do a distro install faiserver-revm # using pxe & preseed, create a vm which is a fai server dsfull # install & post-install a new fai distro arch-init-remote # install arch after it's been booted into it's setup env live-kexec # Kexec this or a remote machine using host faiserver. also useful to run as curl live-kexec|bash # Test scripts arch-revm # test arch install on a fresh vm fai-revm # test fai install on a fresh vm # Scripts to call after a distro install for various reasons chboot # Set grub to boot into a different distro (installed earlier) install-chboot # reinstall chboot to /boot subvols, for chboot updates. eboot # reboot without automatic disk decryption fai-wrapper # use fai classes outside of fai. sourced, not called. faiserver-disable # Disable the fai nfs server exports fresize # resize swap or boot partitions in a host # Replacing a raid 10 disk pxe-server -S HOST fai # btrfs replace or delete. prefer replace. to setup partitions on replacement drive: scp fai-wrapper HOST: ssh root@HOST . fai-wrapper export SPECIAL_DISK=/dev/REPLACEMENT_DEV /var/lib/fai/config/hooks/partition.DEFAULT ssh root@HOST for x in /target/* /target; do umount $x; done cat >p PASSWORD HERE(ctrl-d ctrl-d) cd /dev/disk/by-id/ for d in ata*part1; do cryptsetup luksOpen -d /root/p $d crypt_dev_$d; done x=(/dev/mapper/*part1); mount -o subvol=root_trisquelflidas $x /mnt # btrfs fi show /mnt # btrfs replace start -f /dev/mapper/OLD_DEV /dev/mapper/NEW_DEV /mnt # btrfs replace status /mnt # nohup btrfs dev delete /dev/sde1 /mnt mount -o subvol=boot_trisquelflidas /dev/sda3 /mnt/boot # also replace or delete disk for boot for x in dev proc sys; do mount -o bind /$x /mnt/$x; done chroot /mnt /bin/bash # replace disk in fstab # replace disk in /etc/crypttab update-grub update-initramfs -u mount /a /a/exe/keyscript-on exit reboot # Expected output in fai logs On focal, fai.log:updatebase.UBUNTU FAILED with exit code 1. the real error is dpkg-reconfigure locales, seems to be related to a workaround for < 20.04, relevant comment: # in case the locales are already included inside the base file (Ubuntu) in config/hooks/instsoft.DEBIAN For flidas, when installing systemd, this error happens, and it's a superflous upstream bug based on reading the post install script: addgroup: The group `systemd-journal' already exists as a system group. Exiting. Operation failed: No such file or directory On nabia/newer, python is removed, now its python3, and its easier to just let the package get removed than do host class package config. fai.log:WARNING: These unknown packages are removed from the installation list: python python-minimal Similar to python, linux-image-amd64 is the debian package name for the kernel, linux-image-generic is for ubuntu, but the DEBIAN class is defined on ubuntu and its easier to just let the package get removed with this warning: fai.log:WARNING: These unknown packages are removed from the installation list: linux-image-amd64 Also, cryptsetup-initramfs is new to buster/nabia, it gets removed on earlier versions. # linode notes * create 2 disks, installer (3000 mb, raw), boot (remaining, raw) * create 2 profiles w direct boot, no helpers: * installer (sda=boot, sdb=installer, boot dev=sdb) * boot (sda=boot) * Boot into rescue mode, ssh in with lish, curl url_to_some_fai_cd_created_image | dd of=/dev/sda poweroff * boot into installer. * Lish shows console, at the end of install, it gives prompt because logs failed to save remotely, check the logs, then reboot into boot profile if all is well. If that doesn't happen, turn off lassie in settings. # ubuntu notes For someone who really needed ubuntu on host tp, otherwise they would end up on a non-gnu os, and I didn't want to figure out how to get all the default software installed, I did the following: # On remote host: # install etiona cd /b/fai # set 51-multi-boot to set classes outside of fai-wrapper conditional, including NOWIPE . fai-wrapper ./fai/config/hooks/partition.DEFAULT # on remote host # install ubuntu 20.04 using virt-install sudo -i virt-install --os-variant=ubuntu16.04 --cdrom ubuntu-20.04-desktop-amd64.iso --disk path=u2004.qcow2 -r 2048 --vcpus 1 -n u2004 qemu-img create -o preallocation=metadata -f qcow2 u2004.qcow2 15G # alternatively, also tried a physical install, because I know the virtual install ends up # with some differen things, like some spice service. then pulled the data out with rsync -ahSAX --numeric-ids --exclude=proc --exclude=sys --exclude=dev --exclude=tmp --exclude=run root@tp:/ .; mkdir proc sys dev tmp modprobe nbd qemu-nbd --connect=/dev/nbd0 u1804.qcow2 -f qcow2 qemu-nbd --connect=/dev/nbd0 u2004.qcow2 -f qcow2 mount /dev/nbd0p1 /mnt/1 # bionic mount /dev/nbd0p5 /mnt/1 # focal mount -o bind /mnt/root/root_ubuntubionic /mnt/2 mount -o bind /mnt/root/root_ubuntufocal /mnt/2 mkdir -p /mnt/2/boot mount -o bind /mnt/boot/boot_ubuntubionic /mnt/2/boot mount -o bind /mnt/boot/boot_ubuntufocal /mnt/2/boot # S = sparse, A = acls, X = xattrs rsync -ahSAX --numeric-ids /mnt/1/ /mnt/2 cd /mnt/2 cp /tmp/fai/crypttab etc sed -i "s#/root/keyscript,#decrypt_keyctl,#" etc/crypttab cp /tmp/fai/fstab etc echo "tmpfs /tmp tmpfs nodev,nosuid,size=50%,mode=1777 0 0" >> etc/fstab chrbind chroot . mv /etc/resolv.conf /etc/resolv.conf.old echo nameserver 1.1.1.1 >/etc/resolv.conf # install programs from /a/bin/fai/fai/config/package_config/STANDARD: apt install -y openssh-client openssh-server cryptsetup keyutils btrfs-progs console-setup kbd pciutils usbutils unattended-upgrades initramfs-tools-core dropbear-initramfs mv /etc/resolv.conf.old /etc/resolv.conf exit d=etc/initramfs-tools mkdir -p $d/root/.ssh etc/dropbear-initramfs root/.ssh chmod 700 $d/root $d/root/.ssh root/.ssh cp -p /root/.ssh/authorized_keys $d/root/.ssh/authorized_keys cp -p /root/.ssh/authorized_keys etc/dropbear-initramfs cp -p /root/.ssh/authorized_keys root/.ssh/authorized_keys chroot . sed -ri 's/^ *GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="rd.luks.crypttab=no"/' /etc/default/grub grub-install --no-floppy $(grub-probe -tdrive -d /dev/sda) update-grub grub-bios-setup -d /boot/grub/i386-pc -s /dev/sda exit umount proc umount dev umount sys reboot # for switching the boot to root2 zboot # for switching back, efibootmgr, if there is a problem with the root filesystem detection, # boot into the debian bootstrap distro, run partition.DEFAULT using comments for mktab arg. # then manually run iboot and then reboot. # pine rock64 notes # the only useful image is ubuntu 18.04 ayafun or something. # using emmc usb: s mount /dev/sdb7 /mnt/1 s cp `which qemu-arm-static` /mnt/1/usr/bin s chroot /mnt/1 qemu-arm-static /bin/bash usermod --login iank --move-home --home /home/iank rock46 groupmod --new-name iank rock64 passwd iank # boot it s apt-get update s apt dist-upgrade ### How to merge upstream fai-config git checkout upstream cd path-to-fai-config git pull --stat # the following needs modification if there was deletions or renames rsync --exclude /.git -rlpgoDcvi . /b/fai/fai/config/ cd /b/fai/fai/config/ # where XXXXX is the git commit hash # note, several files which just had trailing space changes will get ignored. git commit -am "update upstream to XXXXX" git checkout master git merge upstream # fix conflicts git commit # TODO Change arch to archlike and to support arch and parabola # License The license for the project is GPLv2 or later, mostly because fai is and I periodically merge the upstream example config, which contains small scripts. Also, there is a modified encrypt.upstream, which is from the cryptsetup package in arch, which is under the same license.