1 PXE install w multi-boot, btrfs & Libreboot support
3 Some things are specific to my home network, and uses files with secrets
4 that are not in this repo. I use this for bare metal and vms, and two
5 scripts which can run post boot so I use them on vps distributed image
8 Features people may find useful: installs encrypted trisquel, debian,
9 ubuntu, arch, and parabola (archlike install is likely broken, I've only
10 done pxe boots recently), in a multi-boot setup using multiple
11 subvolumes of a single btrfs filesystem. Utilizes multiple disks, with
12 scripts to automatically decrypt on intentional reboots, but not after
13 shutdown or power loss.
15 Normal install mode for fai is using pxe, but on a libreboot system,
16 there is no pxe. The pxe in a normal computer is nonfree
17 firmware. Alternatives to normal pxe that I've tried:
19 * libreboot + seabios + ipxe
21 * Use a live cd to call pxe-kexec, this is described later in this file.
23 * Use the fai autodiscover iso. This is more automated, so nicer.
25 * Use an install method above to setup a gnu/linux disk partition that
26 coordinates with libreboot grub to acts like a pxe boot using
27 kexec. The boot process takes a bit longer than normal pxe. This is
28 the bootstrap partition in my scripts.
30 Things I haven't tried:
32 * The bios chip has enough room for an initrd. This could be setup to
33 work like the partition I use to kexec, but it would be faster, and
34 not require installing to disk.
36 The partititioning and filesystem script is at
37 fai/config/hooks/partition.DEFAULT. Disks are grouped as ssd or hdd and
38 raided in raid 1 or raid 0 per configuration. The base partitions are
39 divided into boot, swap, and root, (only boot is unencrypted). There are
40 scripts to resize those partitions post-provision and while the system
43 People who use fai may find these things as useful examples: it uses
44 dnsmasq (on a openwrt machine) for dhcp instead of the isc
45 dhcp. fai-wrapper is a small script to use basic fai classes outside of
46 fai. It does not use the fai partitioning tool, but the script is
47 inspired from it and works outside of fai. It supports running a fai
48 server on debian within android via Maru.
50 It also automates configuration of an openwrt router after manual
53 After provisionining is done, I sync files using btrfs, or unison for
54 vps, then automate further setup using a different set of scripts,
55 https://iankelling.org/git/?p=distro-setup;a=tree.
57 My network is a wndr3700v2 router with openwrt on it and a few pcs/laptops.
59 Since fai requires a debian server as the fai server, there are also
60 scripts to automate a debian install using pxe and preseeding, which can
61 be done from any distro.
63 Some of the scripts have dependencies for some simple obvious utility
64 scripts from https://iankelling.org/git, and of course there are some
65 hostnames that are specific to my network.
68 # Per-host/install configuration
70 Before doing a fai install, you will need to populate a class file. I
71 use one called 51-multi-boot, which you can see example of in
72 fai/config/class/50-host-classes.
76 Before doing a fai install, you will need to populate /q/root/luks and
77 /q/root/shadow, see their references. You might also want to copy
78 existing /etc/ssh/*host* to
79 /p/c/machine_specific/HOST/filesystem/etc/ssh
81 host-* luks keyfiles generated like:
82 head -c 2048 /dev/urandom | od | s dd of=/q/root/luks/host-demohost
84 Configuration of which luks key to use is in
85 fai/config/hooks/partition.DEFAULT
87 Configuration of which (if any) shadow file to use is in
88 fai/config/distro-install-common/end
89 and which shadow file / luks file(s) to copy into the new machine depends
90 on fai-redep arguments.
92 # Scripts (meant to be used directly):
95 # Setup the environment for the install
97 # create tiny autodiscover cd
98 # todo: with fai-revm at least, this complains about missing vmlinuz. need to fix this.
99 fai-redep && sudo fai-cd -g $PWD/grub.cfg.autodiscover -f -A $BASEFILE_DIR/autodiscover.iso
100 # create normal fai cd (replace TARGET_HOSTNAME)
101 fai-redep -t TARGET_HOSTNAME && sudo fai-cd -M -g $PWD/grub.cfg.netinst-noreboot -f $BASEFILE_DIR/netinst.iso
102 # note, may need to set hostname, depending on config,
103 # and some other things for environment not on your lan
104 # for example see fai/config/class/LINODE.var. See linode notes below.
106 mymk-basefile # Create basefiles for various distros
107 archlike-pxe # Setup pxe boot server from an archlike base image
108 fai-redep # Deploy fai configuration to host "faiserver"
109 faiserver-uninstall # uninstall fai-server
110 faiserver-setup # install fai-server on the current machine
111 myfai-chboot # setup fai tftp and nfs. useful for doing pxe-kexec
112 pxe-server # disable/enable pxe dhcp, tfp, and nfs. calls myfai-chboot
113 wrt-setup # setup my router in general: dhcp, dns, etc.
116 # Script to do a distro install
118 faiserver-revm # using pxe & preseed, create a vm which is a fai server
119 dsfull # install & post-install a new fai distro
120 arch-init-remote # install arch after it's been booted into it's setup env
121 live-kexec # Kexec this or a remote machine using host faiserver. also
122 useful to run as curl live-kexec|bash
127 arch-revm # test arch install on a fresh vm
128 fai-revm # test fai install on a fresh vm
131 # Scripts to call after a distro install for various reasons
133 chboot # Set grub to boot into a different distro (installed earlier)
134 install-chboot # reinstall chboot to /boot subvols, for chboot updates.
135 eboot # reboot without automatic disk decryption
136 fai-wrapper # use fai classes outside of fai. sourced, not called.
137 faiserver-disable # Disable the fai nfs server exports
138 fresize # resize swap or boot partitions in a host
141 # Replacing a raid 10 disk
143 pxe-server -S HOST fai
145 # btrfs replace or delete. prefer replace. to setup partitions on replacement drive:
146 scp fai-wrapper HOST:
149 export SPECIAL_DISK=/dev/REPLACEMENT_DEV
150 /var/lib/fai/config/hooks/partition.DEFAULT
154 for x in /target/* /target; do umount $x; done
156 PASSWORD HERE(ctrl-d ctrl-d)
158 for d in ata*part1; do cryptsetup luksOpen -d /root/p $d crypt_dev_$d; done
159 x=(/dev/mapper/*part1); mount -o subvol=root_trisquelflidas $x /mnt
161 # btrfs replace start -f /dev/mapper/OLD_DEV /dev/mapper/NEW_DEV /mnt
162 # btrfs replace status /mnt
163 # nohup btrfs dev delete /dev/sde1 /mnt
164 mount -o subvol=boot_trisquelflidas /dev/sda3 /mnt/boot
165 # also replace or delete disk for boot
166 for x in dev proc sys; do mount -o bind /$x /mnt/$x; done
167 chroot /mnt /bin/bash
168 # replace disk in fstab
169 # replace disk in /etc/crypttab
178 # Expected output in fai logs
181 fai.log:updatebase.UBUNTU FAILED with exit code 1.
182 the real error is dpkg-reconfigure locales, seems to be related
183 to a workaround for < 20.04, relevant comment:
184 # in case the locales are already included inside the base file (Ubuntu)
185 in config/hooks/instsoft.DEBIAN
188 For flidas, when installing systemd, this error happens, and it's
189 a superflous upstream bug based on reading the post install script:
191 addgroup: The group `systemd-journal' already exists as a system group. Exiting.
192 Operation failed: No such file or directory
194 On nabia/newer, python is removed, now its python3,
195 and its easier to just let the package get removed than
196 do host class package config.
197 fai.log:WARNING: These unknown packages are removed from the installation list: python python-minimal
199 Similar to python, linux-image-amd64 is the debian package name
200 for the kernel, linux-image-generic is for ubuntu, but the
201 DEBIAN class is defined on ubuntu and its easier to just let
202 the package get removed with this warning:
203 fai.log:WARNING: These unknown packages are removed from the installation list: linux-image-amd64
204 Also, cryptsetup-initramfs is new to buster/nabia, it gets removed
210 * create 2 disks, installer (3000 mb, raw), boot (remaining, raw)
211 * create 2 profiles w direct boot, no helpers:
212 * installer (sda=boot, sdb=installer, boot dev=sdb)
214 * Boot into rescue mode, ssh in with lish,
215 curl url_to_some_fai_cd_created_image | dd of=/dev/sda
217 * boot into installer.
218 * Lish shows console, at the end of install, it gives prompt because
219 logs failed to save remotely, check the logs, then reboot into boot
220 profile if all is well. If that doesn't happen, turn off lassie in
226 For someone who really needed ubuntu on host tp, otherwise they would
227 end up on a non-gnu os, and I didn't want to figure out how to get all
228 the default software installed, I did the following:
233 # set 51-multi-boot to set classes outside of fai-wrapper conditional, including NOWIPE
235 ./fai/config/hooks/partition.DEFAULT
238 # install ubuntu 20.04 using virt-install
240 virt-install --os-variant=ubuntu16.04 --cdrom ubuntu-20.04-desktop-amd64.iso --disk path=u2004.qcow2 -r 2048 --vcpus 1 -n u2004
241 qemu-img create -o preallocation=metadata -f qcow2 u2004.qcow2 15G
242 # alternatively, also tried a physical install, because I know the virtual install ends up
243 # with some differen things, like some spice service. then pulled the data out with
244 rsync -ahSAX --numeric-ids --exclude=proc --exclude=sys --exclude=dev --exclude=tmp --exclude=run root@tp:/ .; mkdir proc sys dev tmp
247 qemu-nbd --connect=/dev/nbd0 u1804.qcow2 -f qcow2
248 qemu-nbd --connect=/dev/nbd0 u2004.qcow2 -f qcow2
249 mount /dev/nbd0p1 /mnt/1 # bionic
250 mount /dev/nbd0p5 /mnt/1 # focal
251 mount -o bind /mnt/root/root_ubuntubionic /mnt/2
252 mount -o bind /mnt/root/root_ubuntufocal /mnt/2
254 mount -o bind /mnt/boot/boot_ubuntubionic /mnt/2/boot
255 mount -o bind /mnt/boot/boot_ubuntufocal /mnt/2/boot
256 # S = sparse, A = acls, X = xattrs
257 rsync -ahSAX --numeric-ids /mnt/1/ /mnt/2
260 cp /tmp/fai/crypttab etc
261 sed -i "s#/root/keyscript,#decrypt_keyctl,#" etc/crypttab
262 cp /tmp/fai/fstab etc
263 echo "tmpfs /tmp tmpfs nodev,nosuid,size=50%,mode=1777 0 0" >> etc/fstab
266 mv /etc/resolv.conf /etc/resolv.conf.old
267 echo nameserver 1.1.1.1 >/etc/resolv.conf
268 # install programs from /a/bin/fai/fai/config/package_config/STANDARD:
269 apt install -y openssh-client openssh-server cryptsetup keyutils btrfs-progs console-setup kbd pciutils usbutils unattended-upgrades initramfs-tools-core dropbear-initramfs
270 mv /etc/resolv.conf.old /etc/resolv.conf
272 d=etc/initramfs-tools
273 mkdir -p $d/root/.ssh etc/dropbear-initramfs root/.ssh
274 chmod 700 $d/root $d/root/.ssh root/.ssh
275 cp -p /root/.ssh/authorized_keys $d/root/.ssh/authorized_keys
276 cp -p /root/.ssh/authorized_keys etc/dropbear-initramfs
277 cp -p /root/.ssh/authorized_keys root/.ssh/authorized_keys
279 sed -ri 's/^ *GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="rd.luks.crypttab=no"/' /etc/default/grub
280 grub-install --no-floppy $(grub-probe -tdrive -d /dev/sda)
282 grub-bios-setup -d /boot/grub/i386-pc -s /dev/sda
289 # for switching the boot to root2
291 # for switching back, efibootmgr, if there is a problem with the root filesystem detection,
292 # boot into the debian bootstrap distro, run partition.DEFAULT using comments for mktab arg.
293 # then manually run iboot and then reboot.
297 # the only useful image is ubuntu 18.04 ayafun or something.
299 s mount /dev/sdb7 /mnt/1
300 s cp `which qemu-arm-static` /mnt/1/usr/bin
301 s chroot /mnt/1 qemu-arm-static /bin/bash
302 usermod --login iank --move-home --home /home/iank rock46
303 groupmod --new-name iank rock64
310 ### How to merge upstream fai-config
312 git checkout upstream
313 cd path-to-fai-config
315 # the following needs modification if there was deletions or renames
316 rsync --exclude /.git -rlpgoDcvi . /b/fai/fai/config/
317 cd /b/fai/fai/config/
318 # where XXXXX is the git commit hash
319 # note, several files which just had trailing space changes will get ignored.
320 git commit -am "update upstream to XXXXX"
328 Change arch to archlike and to support arch and parabola
333 The license for the project is GPLv2 or later, mostly because fai is and
334 I periodically merge the upstream example config, which contains small
335 scripts. Also, there is a modified encrypt.upstream, which is from the
336 cryptsetup package in arch, which is under the same license.