various fixes and add cmc secondary ap

[automated-distro-installer] / README

PXE install w multi-boot, btrfs & Libreboot support

Some things are specific to my home network, and uses files with secrets
that are not in this repo. I use this for bare metal and vms, and two
scripts which can run post boot so I use them on vps distributed image
as well.

Features people may find useful: installs encrypted trisquel, debian,
ubuntu, arch, and parabola (archlike install is likely broken, I've only
done pxe boots recently), in a multi-boot setup using multiple
subvolumes of a single btrfs filesystem.  Utilizes multiple disks, with
scripts to automatically decrypt on intentional reboots, but not after
shutdown or power loss.

Normal install mode for fai is using pxe, but on a libreboot system,
there is no pxe. The pxe in a normal computer is nonfree
firmware. Alternatives to normal pxe that I've tried:

* libreboot + seabios + ipxe

* Use a live cd to call pxe-kexec, this is described later in this file.

* Use the fai autodiscover iso. This is more automated, so nicer.

* Use an install method above to setup a gnu/linux disk partition that
  coordinates with libreboot grub to acts like a pxe boot using
  kexec. The boot process takes a bit longer than normal pxe. This is
  the bootstrap partition in my scripts.

Things I haven't tried:

* The bios chip has enough room for an initrd. This could be setup to
  work like the partition I use to kexec, but it would be faster, and
  not require installing to disk.

The partititioning and filesystem script is at
fai/config/hooks/partition.DEFAULT. Disks are grouped as ssd or hdd and
raided in raid 1 or raid 0 per configuration. The base partitions are
divided into boot, swap, and root, (only boot is unencrypted). There are
scripts to resize those partitions post-provision and while the system
is running.

People who use fai may find these things as useful examples: it uses
dnsmasq (on a openwrt machine) for dhcp instead of the isc
dhcp. fai-wrapper is a small script to use basic fai classes outside of
fai. It does not use the fai partitioning tool, but the script is
inspired from it and works outside of fai. It supports running a fai
server on debian within android via Maru.

It also automates configuration of an openwrt router after manual
initial installation.

After provisionining is done, I sync files using btrfs, or unison for
vps, then automate further setup using a different set of scripts,
https://iankelling.org/git/?p=distro-setup;a=tree.

My network is a wndr3700v2 router with openwrt on it and a few pcs/laptops.

Since fai requires a debian server as the fai server, there are also
scripts to automate a debian install using pxe and preseeding, which can
be done from any distro.

Some of the scripts have dependencies for some simple obvious utility
scripts from https://iankelling.org/git, and of course there are some
hostnames that are specific to my network.


# Per-host/install configuration

Before doing a fai install, you will need to populate a class file.  I
use one called 51-multi-boot, which you can see example of in
fai/config/class/50-host-classes.

Before doing a fai install, you will need to populate /q/root/luks and
/q/root/shadow, see their references. You might also want to copy
existing /etc/ssh/*host* to
/p/c/machine_specific/HOST/filesystem/etc/ssh

host-* luks keyfiles generated like:
h=demohost; head -c 2048 /dev/urandom | od | se dd of=/q/root/luks/host-$h

Configuration of which luks key to use is in
fai/config/hooks/partition.DEFAULT

Configuration of which (if any) shadow file to use is in
fai/config/distro-install-common/end
and which shadow file / luks file(s) to copy into the new machine depends
on fai-redep arguments.

Also, setup dns in bind and wrt-setup-local.

After install, btrbk to setup data, and then distro-begin && distro end.
See notes in distro-begin for other configuration.

# Scripts (meant to be used directly):


# Setup the environment for the install

# create tiny autodiscover cd
# todo: with fai-revm at least, this complains about missing vmlinuz. need to fix this.
fai-redep && sudo fai-cd -g $PWD/grub.cfg.autodiscover -f -A $BASEFILE_DIR/autodiscover.iso
# create normal fai cd (replace TARGET_HOSTNAME)
fai-redep -t TARGET_HOSTNAME && sudo fai-cd -M -g $PWD/grub.cfg.netinst-noreboot -f $BASEFILE_DIR/netinst.iso
# note, may need to set hostname, depending on config,
# and some other things for environment not on your lan
# for example see fai/config/class/LINODE.var. See linode notes below.

mymk-basefile # Create basefiles for various distros
archlike-pxe # Setup pxe boot server from an archlike base image
fai-redep # Deploy fai configuration to host "faiserver"
faiserver-uninstall # uninstall fai-server
faiserver-setup # install fai-server on the current machine
myfai-chboot # setup fai tftp and nfs. useful for doing pxe-kexec
pxe-server # disable/enable pxe dhcp, tfp, and nfs. calls myfai-chboot
wrt-setup  # setup my router in general: dhcp, dns, etc.


# Script to do a distro install

faiserver-revm # using pxe & preseed, create a vm which is a fai server
dsfull # install & post-install a new fai distro
arch-init-remote # install arch after it's been booted into it's setup env
live-kexec # Kexec this or a remote machine using host faiserver. also
             useful to run as curl live-kexec|bash


# Test scripts

arch-revm # test arch install on a fresh vm
fai-revm  # test fai install on a fresh vm


# Scripts to call after a distro install for various reasons

chboot # Set grub to boot into a different distro (installed earlier)
install-chboot # reinstall chboot to /boot subvols, for chboot updates.
eboot # reboot without automatic disk decryption
fai-wrapper # use fai classes outside of fai. sourced, not called.
faiserver-disable # Disable the fai nfs server exports
fresize # resize swap or boot partitions in a host


# NAT/forward/vpn tftp

I tried to get this working, but failed.

tftp server in theory can be forwarded over a vpn, eg on a wireguard tunnel.

However, I found that when actually pxe booting, it wouldn't work, only
the 1st filename would be requested, eg, in the logs:

Jun 20 23:51:02 kd in.tftpd[4021350]: RRQ from 10.2.0.12 filename pxelinux.0


To get that far, nating tftp requires some special attention in iptables, like so:

https://unix.stackexchange.com/questions/579508/iptables-rules-to-forward-tftp-via-nat
iptables -t raw -A PREROUTING -p udp --dport 69 -s 209.51.188.0/24 -j CT --helper tftp
modprobe nf_nat_tftp

to test tftp from a client machine:

tftp SERVER_IP -c get pxelinux.0
rm -fv pxelinux.0


# Common problems

## kernel mismatch very early error, no remote logs:

ERROR: the running kernel does not match the kernel modules inside the nfsroot.
ERROR: Kernel modules directory /lib/modules/5.10.0-8-amd not available. Only found /lib/modules/5.10.0-15-amd64

solution: if running from fai-cd, recreate autodiscover cd as noted above in setup.

# What good logs look like:

logging nfs traffic from server

s rpcdebug -m nfsd -s all


normal nfs mount & umount logs look like:

journalctl -ef | gr nfs

Jun 20 22:15:36 kd rpc.mountd[2025725]: authenticated mount request from 10.32.2.1:865 for /srv/fai/nfsroot (/srv/fai/nfsroot)
Jun 20 22:15:36 kd kernel: nfsd: exp_rootfh(/srv/fai/nfsroot [00000000e8c53e54] *:dm-0/5521225)
Jun 20 22:15:36 kd kernel: nfsd: fh_compose(exp 00:1b/5521225 fai/nfsroot, ino=5521225)
Jun 20 22:15:36 kd kernel: nfsd: FSINFO(3)   28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000
Jun 20 22:15:36 kd kernel: nfsd: fh_verify(28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000)
Jun 20 22:15:36 kd kernel: nfsd: PATHCONF(3) 28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000
Jun 20 22:15:36 kd kernel: nfsd: fh_verify(28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000)
Jun 20 22:15:36 kd kernel: nfsd: GETATTR(3)  28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000
Jun 20 22:15:36 kd kernel: nfsd: fh_verify(28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000)
Jun 20 22:15:36 kd kernel: nfsd: FSINFO(3)   28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000
Jun 20 22:15:36 kd kernel: nfsd: fh_verify(28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000)
Jun 20 22:15:36 kd kernel: nfsd: GETATTR(3)  28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000
Jun 20 22:15:36 kd kernel: nfsd: fh_verify(28: 00070001 00543f49 00000000 d185f7b0 58d1a3c6 00000000)
Jun 20 22:15:45 kd rpc.mountd[2025725]: authenticated unmount request from 10.32.2.1:986 for /srv/fai/nfsroot (/srv/fai/nfsroot)

normal tftpd logs from:

after setting -vv in TFTP_OPTIONS in /etc/default/tftpd-hpa

journalctl -u tftpd-hpa

Jun 20 23:51:02 kd in.tftpd[4021350]: RRQ from 10.2.0.12 filename pxelinux.0
Jun 20 23:51:02 kd in.tftpd[4021351]: RRQ from 10.2.0.12 filename ldlinux.c32
Jun 20 23:51:02 kd in.tftpd[4021352]: RRQ from 10.2.0.12 filename pxelinux.cfg/a913a477-fca6-234d-a928-6bb011decd05
Jun 20 23:51:02 kd in.tftpd[4021352]: sending NAK (1, File not found) to 10.2.0.12
Jun 20 23:51:02 kd in.tftpd[4021353]: RRQ from 10.2.0.12 filename pxelinux.cfg/01-52-54-00-9c-ef-ad
Jun 20 23:51:02 kd in.tftpd[4021353]: sending NAK (1, File not found) to 10.2.0.12
Jun 20 23:51:02 kd in.tftpd[4021354]: RRQ from 10.2.0.12 filename pxelinux.cfg/0A02000C
Jun 20 23:51:02 kd in.tftpd[4021355]: RRQ from 10.2.0.12 filename vmlinuz-5.10.0-15-amd64
Jun 20 23:51:03 kd in.tftpd[4021356]: RRQ from 10.2.0.12 filename initrd.img-5.10.0-15-amd64



# Replacing a raid 10 disk

pxe-server -S HOST fai

# btrfs replace or delete. prefer replace. to setup partitions on replacement drive:
scp fai-wrapper HOST:
ssh root@HOST
. fai-wrapper
export SPECIAL_DISK=/dev/REPLACEMENT_DEV
/var/lib/fai/config/hooks/partition.DEFAULT


ssh root@HOST
for x in /target/* /target; do umount $x; done
cat >p
PASSWORD HERE(ctrl-d ctrl-d)
cd /dev/disk/by-id/
for d in ata*part1; do cryptsetup luksOpen -d /root/p $d crypt_dev_$d; done
x=(/dev/mapper/*part1); mount -o subvol=root_trisquelflidas $x /mnt
# btrfs fi show /mnt
# btrfs replace start -f /dev/mapper/OLD_DEV /dev/mapper/NEW_DEV /mnt
# btrfs replace status /mnt
# nohup btrfs dev delete /dev/sde1 /mnt
mount -o subvol=boot_trisquelflidas /dev/sda3 /mnt/boot
# also replace or delete disk for boot
for x in dev proc sys; do mount -o bind /$x /mnt/$x; done
chroot /mnt /bin/bash
# replace disk in fstab
# replace disk in /etc/crypttab
update-grub
update-initramfs -u
mount /a
/a/exe/keyscript-on
exit
reboot


# Expected output in fai logs


## On focal:

fai.log:updatebase.UBUNTU    FAILED with exit code 1.
the real error is dpkg-reconfigure locales, seems to be related
to a workaround for < 20.04, relevant comment:
# in case the locales are already included inside the base file (Ubuntu)
in config/hooks/instsoft.DEBIAN


## For flidas,

when installing systemd, this error happens, and it's
a superflous upstream bug based on reading the post install script:

addgroup: The group `systemd-journal' already exists as a system group. Exiting.
Operation failed: No such file or directory

## On nabia/newer,

python is removed, now its python3,
and its easier to just let the package get removed than
do host class package config.
fai.log:WARNING: These unknown packages are removed from the installation list: python python-minimal

Similar to python, linux-image-amd64 is the debian package name
for the kernel, linux-image-generic is for ubuntu, but the
DEBIAN class is defined on ubuntu and its easier to just let
the package get removed with this warning:
fai.log:WARNING: These unknown packages are removed from the installation list: linux-image-amd64
Also, cryptsetup-initramfs is new to buster/nabia, it gets removed
on earlier versions.

## parted error
fai.log:Error: /dev/vda: unrecognised disk label
This is from parted -m $d unit MiB print.
It happens when there are no partitions yet.


######## notes on creating a lan with just 2 computers ########


## below assumes eth0 is the ethernet device used to connect to the target computer.


# this is not strictly needed. I had my connection die at some point,
# and I suspected this might help.
# based on
# https://support.qacafe.com/knowledge-base/how-do-i-prevent-network-manager-from-controlling-an-interface/
cat > /etc/NetworkManager/conf.d/99-fai-tmp.conf <<'EOF'
[main]
plugins=keyfile

[keyfile]
unmanaged-devices=interface-name:eth0
EOF
ser restart NetworkManager


cat >> /etc/network/interfaces <<'EOF'
iface eth0 inet static
address 10.0.44.1/24
EOF

ifup eth0

# note turn off fsf vpn, so route to coresite is the normal route.
echo 1 > /proc/sys/net/ipv4/ip_forward
m s iptables -t nat -A POSTROUTING -o $(ip -4 route get 8.8.8.8 | sed -nr 's,^.* dev\s+(\S+).*,\1,p') -j MASQUERADE


change /p/c/machine_specific/vps/bind-initial/db.b8.nz
faiserver 10.0.44.1
TARGET_HOSTNAME 10.0.44.2

apt install isc-dhcp-server

cat >> /etc/default/isc-dhcp-server <<'EOF'
INTERFACESv4="eth0"
EOF

edit ./dhcpd.conf to change mac address and target host name.

s cp /b/fai/dhcpd.conf /etc/dhcp/
ser restart isc-dhcp-server

edit /a/bin/fai/fai/config/class/51-multi-boot

pxe-server -d TARGET fai

Then do a pxe boot on the target host



##### linode notes ######

* create 2 disks, installer (3000 mb, raw), boot (remaining, raw)
* create 2 profiles w direct boot, no helpers:
  * installer (sda=boot, sdb=installer, boot dev=sdb)
  * boot (sda=boot)
* Boot into rescue mode, ssh in with lish,
  curl url_to_some_fai_cd_created_image | dd of=/dev/sda
  poweroff
* boot into installer.
* Lish shows console, at the end of install, it gives prompt because
  logs failed to save remotely, check the logs, then reboot into boot
  profile if all is well. If that doesn't happen, turn off lassie in
  settings.



###### ubuntu notes  ######

For someone who really needed ubuntu on host tp, otherwise they would
end up on a non-gnu os, and I didn't want to figure out how to get all
the default software installed, I did the following:

# On remote host:
# install etiona
cd /b/fai
# set 51-multi-boot to set classes outside of fai-wrapper conditional, including NOWIPE
. fai-wrapper
./fai/config/hooks/partition.DEFAULT

# on remote host
# install ubuntu 20.04 using virt-install
sudo -i
virt-install --os-variant=ubuntu16.04 --cdrom ubuntu-20.04-desktop-amd64.iso --disk path=u2004.qcow2 -r 2048 --vcpus 1 -n u2004
qemu-img create -o preallocation=metadata -f qcow2 u2004.qcow2 15G
# alternatively, also tried a physical install, because I know the virtual install ends up
# with some differen things, like some spice service. then pulled the data out with
rsync -ahSAX --numeric-ids --exclude=proc --exclude=sys --exclude=dev --exclude=tmp --exclude=run root@tp:/ .; mkdir proc sys dev tmp

modprobe nbd
qemu-nbd --connect=/dev/nbd0 u1804.qcow2 -f qcow2
qemu-nbd --connect=/dev/nbd0 u2004.qcow2 -f qcow2
mount /dev/nbd0p1 /mnt/1 # bionic
mount /dev/nbd0p5 /mnt/1 # focal
mount -o bind /mnt/root/root_ubuntubionic /mnt/2
mount -o bind /mnt/root/root_ubuntufocal /mnt/2
mkdir -p /mnt/2/boot
mount -o bind /mnt/boot/boot_ubuntubionic /mnt/2/boot
mount -o bind /mnt/boot/boot_ubuntufocal /mnt/2/boot
# S = sparse, A = acls, X = xattrs
rsync -ahSAX --numeric-ids /mnt/1/ /mnt/2

cd /mnt/2
cp /tmp/fai/crypttab etc
sed -i "s#/root/keyscript,#decrypt_keyctl,#" etc/crypttab
cp /tmp/fai/fstab etc
echo "tmpfs     /tmp tmpfs     nodev,nosuid,size=50%,mode=1777   0    0" >> etc/fstab
chrbind
chroot .
mv /etc/resolv.conf /etc/resolv.conf.old
echo nameserver 1.1.1.1 >/etc/resolv.conf
# install programs from /a/bin/fai/fai/config/package_config/STANDARD:
apt install -y openssh-client openssh-server cryptsetup keyutils btrfs-progs console-setup kbd pciutils usbutils unattended-upgrades initramfs-tools-core dropbear-initramfs
mv /etc/resolv.conf.old /etc/resolv.conf
exit
d=etc/initramfs-tools
mkdir -p $d/root/.ssh etc/dropbear-initramfs root/.ssh
chmod 700 $d/root $d/root/.ssh root/.ssh
cp -p /root/.ssh/authorized_keys $d/root/.ssh/authorized_keys
cp -p /root/.ssh/authorized_keys etc/dropbear-initramfs
cp -p /root/.ssh/authorized_keys root/.ssh/authorized_keys
chroot .
sed -ri 's/^ *GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="rd.luks.crypttab=no"/' /etc/default/grub
grub-install --no-floppy $(grub-probe -tdrive -d /dev/sda)
update-grub
grub-bios-setup -d /boot/grub/i386-pc -s /dev/sda
exit
umount proc
umount dev
umount sys
reboot

# for switching the boot to root2
zboot
# for switching back, efibootmgr, if there is a problem with the root filesystem detection,
# boot into the debian bootstrap distro, run partition.DEFAULT using comments for mktab arg.
# then manually run iboot and then reboot.


# pine rock64 notes
# the only useful image is ubuntu 18.04 ayafun or something.
# using emmc usb:
s mount /dev/sdb7 /mnt/1
s cp `which qemu-arm-static` /mnt/1/usr/bin
s chroot /mnt/1 qemu-arm-static /bin/bash
usermod --login iank --move-home --home /home/iank rock46
groupmod --new-name iank rock64
passwd iank
# boot it
s apt-get update
s apt dist-upgrade


### How to merge upstream fai-config

git checkout upstream
cd path-to-fai-config
git pull --stat
# the following needs modification if there was deletions or renames
rsync --exclude /.git -rlpgoDcvi . /b/fai/fai/config/
cd /b/fai/fai/config/
# where XXXXX is the git commit hash
# note, several files which just had trailing space changes will get ignored.
git commit -am "update upstream to XXXXX"
git checkout master
git merge upstream
# fix conflicts
git commit


# TODO
Change arch to archlike and to support arch and parabola


# License

The license for the project is GPLv2 or later, mostly because fai is and
I periodically merge the upstream example config, which contains small
scripts. Also, there is a modified encrypt.upstream, which is from the
cryptsetup package in arch, which is under the same license.