1 PXE install w multi-boot, btrfs & Libreboot support
3 Some things are specific to my home network, and uses files with secrets
4 that are not in this repo. I use this for bare metal and vms, and two
5 scripts which can run post boot so I use them on vps distributed image
8 Features people may find useful: installs encrypted trisquel, debian,
9 ubuntu, arch, and parabola (archlike install is likely broken, I've only
10 done pxe boots recently), in a multi-boot setup using multiple
11 subvolumes of a single btrfs filesystem. Utilizes multiple disks, with
12 scripts to automatically decrypt on intentional reboots, but not after
13 shutdown or power loss.
15 Normal install mode for fai is using pxe, but on a libreboot system,
16 there is no pxe. The pxe in a normal computer is nonfree
17 firmware. Alternatives to normal pxe that I've tried:
19 * libreboot + seabios + ipxe
21 * Use a live cd to call pxe-kexec, this is described later in this file.
23 * Use the fai autodiscover iso. This is more automated, so nicer.
25 * Use an install method above to setup a gnu/linux disk partition that
26 coordinates with libreboot grub to acts like a pxe boot using
27 kexec. The boot process takes a bit longer than normal pxe. This is
28 the bootstrap partition in my scripts.
30 Things I haven't tried:
32 * The bios chip has enough room for an initrd. This could be setup to
33 work like the partition I use to kexec, but it would be faster, and
34 not require installing to disk.
36 The partititioning and filesystem script is at
37 fai/config/hooks/partition.DEFAULT. Disks are grouped as ssd or hdd and
38 raided in raid 1 or raid 0 per configuration. The base partitions are
39 divided into boot, swap, and root, (only boot is unencrypted). There are
40 scripts to resize those partitions post-provision and while the system
43 People who use fai may find these things as useful examples: it uses
44 dnsmasq (on a openwrt machine) for dhcp instead of the isc
45 dhcp. fai-wrapper is a small script to use basic fai classes outside of
46 fai. It does not use the fai partitioning tool, but the script is
47 inspired from it and works outside of fai. It supports running a fai
48 server on debian within android via Maru.
50 It also automates configuration of an openwrt router after manual
53 After provisionining is done, I sync files using btrfs, or unison for
54 vps, then automate further setup using a different set of scripts,
55 https://iankelling.org/git/?p=distro-setup;a=tree.
57 My network is a wndr3700v2 router with openwrt on it and a few pcs/laptops.
59 Since fai requires a debian server as the fai server, there are also
60 scripts to automate a debian install using pxe and preseeding, which can
61 be done from any distro.
63 Some of the scripts have dependencies for some simple obvious utility
64 scripts from https://iankelling.org/git, and of course there are some
65 hostnames that are specific to my network.
68 # Per-host/install configuration
70 Before doing a fai install, you will need to populate a class file. I
71 use one called 51-multi-boot, which you can see example of in
72 fai/config/class/50-host-classes.
76 Before doing a fai install, you will need to populate /q/root/luks and
77 /q/root/shadow, see their references. You might also want to copy
78 existing /etc/ssh/*host* to
79 /p/c/machine_specific/HOST/filesystem/etc/ssh
81 host-* luks keyfiles generated like:
82 head -c 2048 /dev/urandom | od | s dd of=/q/root/luks/host-demohost
84 Configuration of which luks key to use is in
85 fai/config/hooks/partition.DEFAULT
87 Configuration of which (if any) shadow file to use is in
88 fai/config/distro-install-common/end
89 and which shadow file / luks file(s) to copy into the new machine depends
90 on fai-redep arguments.
92 # Scripts (meant to be used directly):
95 # Setup the environment for the install
97 # create tiny autodiscover cd
98 # todo: with fai-revm at least, this complains about missing vmlinuz. need to fix this.
99 fai-redep && sudo fai-cd -g $PWD/grub.cfg.autodiscover -f -A $BASEFILE_DIR/autodiscover.iso
100 # create normal fai cd (replace TARGET_HOSTNAME)
101 fai-redep -t TARGET_HOSTNAME && sudo fai-cd -M -g $PWD/grub.cfg.netinst-noreboot -f $BASEFILE_DIR/netinst.iso
102 # note, may need to set hostname, depending on config,
103 # and some other things for environment not on your lan
104 # for example see fai/config/class/LINODE.var. See linode notes below.
106 mymk-basefile # Create basefiles for various distros
107 archlike-pxe # Setup pxe boot server from an archlike base image
108 fai-redep # Deploy fai configuration to host "faiserver"
109 faiserver-uninstall # uninstall fai-server
110 faiserver-setup # install fai-server on the current machine
111 myfai-chboot # setup fai tftp and nfs. useful for doing pxe-kexec
112 pxe-server # disable/enable pxe dhcp, tfp, and nfs. calls myfai-chboot
113 wrt-setup # setup my router in general: dhcp, dns, etc.
116 # Script to do a distro install
118 faiserver-revm # using pxe & preseed, create a vm which is a fai server
119 dsfull # install & post-install a new fai distro
120 arch-init-remote # install arch after it's been booted into it's setup env
121 live-kexec # Kexec this or a remote machine using host faiserver. also
122 useful to run as curl live-kexec|bash
127 arch-revm # test arch install on a fresh vm
128 fai-revm # test fai install on a fresh vm
131 # Scripts to call after a distro install for various reasons
133 chboot # Set grub to boot into a different distro (installed earlier)
134 install-chboot # reinstall chboot to /boot subvols, for chboot updates.
135 eboot # reboot without automatic disk decryption
136 fai-wrapper # use fai classes outside of fai. sourced, not called.
137 faiserver-disable # Disable the fai nfs server exports
138 fresize # resize swap or boot partitions in a host
141 # Replacing a raid 10 disk
143 # i expect better results with newer kernel and btrfs progs than the default stretch
146 pxe-server -S HOST fai
148 # btrfs replace or delete. prefer replace. to setup partitions on replacement drive:
149 scp fai-wrapper HOST:
152 export SPECIAL_DISK=/dev/REPLACEMENT_DEV
153 /var/lib/fai/config/hooks/partition.DEFAULT
157 for x in /target/* /target; do umount $x; done
159 PASSWORD HERE(ctrl-d ctrl-d)
161 for d in ata*part1; do cryptsetup luksOpen -d /root/p $d crypt_dev_$d; done
162 x=(/dev/mapper/*part1); mount -o subvol=root_trisquelflidas $x /mnt
164 # btrfs replace start -f /dev/mapper/OLD_DEV /dev/mapper/NEW_DEV /mnt
165 # btrfs replace status /mnt
166 # nohup btrfs dev delete /dev/sde1 /mnt
167 mount -o subvol=boot_trisquelflidas /dev/sda3 /mnt/boot
168 # also replace or delete disk for boot
169 for x in dev proc sys; do mount -o bind /$x /mnt/$x; done
170 chroot /mnt /bin/bash
171 # replace disk in fstab
172 # replace disk in /etc/crypttab
181 # Expected output in fai logs
184 fai.log:updatebase.UBUNTU FAILED with exit code 1.
185 the real error is dpkg-reconfigure locales, seems to be related
186 to a workaround for < 20.04, relevant comment:
187 # in case the locales are already included inside the base file (Ubuntu)
188 in config/hooks/instsoft.DEBIAN
191 For flidas, when installing systemd, this error happens, and it's
192 a superflous upstream bug based on reading the post install script:
194 addgroup: The group `systemd-journal' already exists as a system group. Exiting.
195 Operation failed: No such file or directory
197 On nabia/newer, python is removed, now its python3,
198 and its easier to just let the package get removed than
199 do host class package config.
200 fai.log:WARNING: These unknown packages are removed from the installation list: python python-minimal
202 Similar to python, linux-image-amd64 is the debian package name
203 for the kernel, linux-image-generic is for ubuntu, but the
204 DEBIAN class is defined on ubuntu and its easier to just let
205 the package get removed with this warning:
206 fai.log:WARNING: These unknown packages are removed from the installation list: linux-image-amd64
207 Also, cryptsetup-initramfs is new to buster/nabia, it gets removed
213 * create 2 disks, installer (3000 mb, raw), boot (remaining, raw)
214 * create 2 profiles w direct boot, no helpers:
215 * installer (sda=boot, sdb=installer, boot dev=sdb)
217 * Boot into rescue mode, ssh in with lish,
218 curl url_to_some_fai_cd_created_image | dd of=/dev/sda
220 * boot into installer.
221 * Lish shows console, at the end of install, it gives prompt because
222 logs failed to save remotely, check the logs, then reboot into boot
223 profile if all is well. If that doesn't happen, turn off lassie in
229 For someone who really needed ubuntu on host tp, otherwise they would
230 end up on a non-gnu os, and I didn't want to figure out how to get all
231 the default software installed, I did the following:
236 # set 51-multi-boot to set classes outside of fai-wrapper conditional, including NOWIPE
238 ./fai/config/hooks/partition.DEFAULT
241 # install ubuntu 20.04 using virt-install
243 virt-install --os-variant=ubuntu16.04 --cdrom ubuntu-20.04-desktop-amd64.iso --disk path=u2004.qcow2 -r 2048 --vcpus 1 -n u2004
244 qemu-img create -o preallocation=metadata -f qcow2 u2004.qcow2 15G
245 # alternatively, also tried a physical install, because I know the virtual install ends up
246 # with some differen things, like some spice service. then pulled the data out with
247 rsync -ahSAX --numeric-ids --exclude=proc --exclude=sys --exclude=dev --exclude=tmp --exclude=run root@tp:/ .; mkdir proc sys dev tmp
250 qemu-nbd --connect=/dev/nbd0 u1804.qcow2 -f qcow2
251 qemu-nbd --connect=/dev/nbd0 u2004.qcow2 -f qcow2
252 mount /dev/nbd0p1 /mnt/1 # bionic
253 mount /dev/nbd0p5 /mnt/1 # focal
254 mount -o bind /mnt/root/root_ubuntubionic /mnt/2
255 mount -o bind /mnt/root/root_ubuntufocal /mnt/2
257 mount -o bind /mnt/boot/boot_ubuntubionic /mnt/2/boot
258 mount -o bind /mnt/boot/boot_ubuntufocal /mnt/2/boot
259 # S = sparse, A = acls, X = xattrs
260 rsync -ahSAX --numeric-ids /mnt/1/ /mnt/2
263 cp /tmp/fai/crypttab etc
264 sed -i "s#/root/keyscript,#decrypt_keyctl,#" etc/crypttab
265 cp /tmp/fai/fstab etc
266 echo "tmpfs /tmp tmpfs nodev,nosuid,size=50%,mode=1777 0 0" >> etc/fstab
269 mv /etc/resolv.conf /etc/resolv.conf.old
270 echo nameserver 1.1.1.1 >/etc/resolv.conf
271 # install programs from /a/bin/fai/fai/config/package_config/STANDARD:
272 apt install -y openssh-client openssh-server cryptsetup keyutils btrfs-progs console-setup kbd pciutils usbutils unattended-upgrades initramfs-tools-core dropbear-initramfs
273 mv /etc/resolv.conf.old /etc/resolv.conf
275 d=etc/initramfs-tools
276 mkdir -p $d/root/.ssh etc/dropbear-initramfs root/.ssh
277 chmod 700 $d/root $d/root/.ssh root/.ssh
278 cp -p /root/.ssh/authorized_keys $d/root/.ssh/authorized_keys
279 cp -p /root/.ssh/authorized_keys etc/dropbear-initramfs
280 cp -p /root/.ssh/authorized_keys root/.ssh/authorized_keys
282 sed -ri 's/^ *GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="rd.luks.crypttab=no"/' /etc/default/grub
283 grub-install --no-floppy $(grub-probe -tdrive -d /dev/sda)
285 grub-bios-setup -d /boot/grub/i386-pc -s /dev/sda
293 # the only useful image is ubuntu 18.04 ayafun or something.
295 s mount /dev/sdb7 /mnt/1
296 s cp `which qemu-arm-static` /mnt/1/usr/bin
297 s chroot /mnt/1 qemu-arm-static /bin/bash
298 usermod --login iank --move-home --home /home/iank rock46
299 groupmod --new-name iank rock64
307 Change arch to archlike and to support arch and parabola
312 The license for the project is GPLv2 or later, mostly because fai is and
313 I periodically merge the upstream example config, which contains small
314 scripts. Also, there is a modified encrypt.upstream, which is from the
315 cryptsetup package in arch, which is under the same license.