[work-notes] / work.org

* obs/i3 keybind reminders

f9: start/stop stream
s+f5: interlude
s+space: float a window & make it sticky to keep streaming it while I use another workspace

obof/obon # turn on/off automatic obs scene switching

mute mic: s+[
unmute mic: s+t

If you are viewing a tall window and want to show it to the audience,
go to the preview (click if the red lines aren't there), press
ctrl-f. Then reset with ctrl-r. If the source has a custom transform,
the procedure is different: first do ctrl-shift-c to copy the transform,
then ctrl-f, ten ctrl-shift-v to restore the transform.

** i3 keybinds to remember

shift+g i3 auto-layout-toggle
shift+b mark term
shift+e mark emacs
shift+6 [class="Emacs" title="^(?!#[a-zA-Z][a-zA-Z-]*$)"] move workspace current
shift+w fullscreen toggle
space toggle window float (useful for obs, keeping window visible)

** rarely used:
equal $ex "dunstctl close-all"
1 focus parent
shift+1 focus child
# change focus between tiling / floating windows
shift+65 focus mode_toggle

* TODO : Galene LibreJS

* low pri todos
** TODO add logcheck as a todo item in the prometheus project
** TODO bug tracker

*** savannah
Not easy to install.
No cli interface, but should be easily scriptable.


*** fossil

strange thing: they don't allow strangers to file bugs. need to
investigate how the distributed bug tracking works in practice.

missing javascript license, but doesn't look hard to fix.

*** probably not good programs

**** git-bug
barely maintained https://github.com/MichaelMure/git-bug
Not librejs marked. ReactJS webpack crap.


**** pagure

Not librejs marked.

**** rt
Not easy to install.

Their own use as bug tracker is not well maintained (it has spam
bugs). https://rt.bestpractical.com/

**** debbugs
can of worms. no easy interface.

**** radicle

javascript heavy, issues as git commits opens up a lot of questions &
problems that are unanswered by their documentation. It explicitly says
it doesn't support rewriting history, no, I think we ought to have
support for that.

https://radicle.xyz/

*** dead distributed projects

git-issue 2022 https://github.com/dspinellis/git-issue
sciit 2021 https://gitlab.com/sciit/sciit
bug 2019 https://github.com/driusan/bug
git-dit 2020 https://github.com/neithernut/git-dit
issue 2020 (unclear/unreliable distribution method) https://github.com/marekjm/issue
bugseverwhere 2017 https://gitlab.com/bugseverywhere/bugseverywhere
deft 2011 https://github.com/npryce/deft


* TODO add integrity check for backups
* TODO revisit missing backups script
* TODO test irc instant message notification in emacs bar
* yq

yq/README.md
wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq &&\
    chmod +x /usr/bin/yq

cat /a/f/ans/roles/prom/files/simple/etc/prometheus/rules/fsf.yml | yq '.groups[].rules[] | select(.alert).alert'
cat /a/f/ans/roles/prom/files/simple/etc/prometheus/rules/fsf.yml | yq '.groups[].rules[] | select(.alert).expr |@uri'
cat /a/f/ans/roles/prom/files/simple/etc/prometheus/rules/fsf.yml | yq '.groups[].rules[] | select(.alert).alert = "RedirectMatch \"^/f/" + .alert + "$\"" + " \"/graph?g0.expr=" + (.expr |@uri) | .alert + "&g0.tab=1\""' >/tmp/fsf-redirs.conf



* TODO check if wildebeest firewall rule for outbound ssh can go into ansible

* TODO check/fix enhanced tracking protection civicrm payment failure

* remote desktop

p install tigervnc-scraping-server

mkdir -p ~/.vnc
generated the pass by running vncpasswd

/usr/bin/X0tigervnc -display :0 -localhost=0 -AcceptSetDesktopSize=0 -rfbport 5900 -PasswordFile /home/iank/.vnc/passwd -SecurityTypes VncAuth,TLSVnc

xtigervncviewer -SecurityTypes VncAuth,TLSVnc -passwd /home/iank/.vnc/passwd bow:0

there's a wrapper script x0tigervncserver which puts it in the background, which I'd like to use, but I need the AcceptSetDesktopSize to avoid remote screen resolution being resized. looks like I can do that with an option:


/usr/share/perl5/TigerVNC/Config.pm
vncServerExtraArgs

just need to test out the perl syntax, and set it in

~/.vnc/tigervnc.conf


* TODO make sure we are watching SMART stats on community0p


* nagios

p install nagios4
a2enmod cgid
http://127.0.0.1/nagios4/

settings file
/etc/nagios4/nagios.cfg

ovrerview:
https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/config.html
nrpe is used to run processes on a monitored machine and get back data.

FSF uses check-mk for that now, but check-mk stopped working that way in
newer versions, now it wants to replace nagios entirely. We don't want that.



* TODO setup public inbox

* TODO patch gnu upload manual

to say about the fencepost debug file,
and to say about signing old key with new key,
and to not send mime signatures
and something else i wrote about before in an email.


* TODO ansiblize the gnu.org watchdog


* TODO make a libreplanet page documenting our discourse freedom fixes

* TODO alert when exim leaves around old processes
there is a message in the journal on restart.
logcheck could help here?

* TODO write alert for prometheus not running,

* TODO get logcheck working
* TODO redirect info@h-node.org
to where, is this old?


* TODO improve rt workflow

https://rt.gnu.org/Ticket/Update.html?Action=Comment&DefaultStatus=resolved&id=1767459
javascript:self.location=self.location+'&Status=resolved;Action=Take;id=1431087'
javascript:self.location=self.location+'&DefaultStatus=resolved;Action=Comment'
https://rt.gnu.org/Ticket/Display.html?id=1767459
https://rt.gnu.org/Ticket/Update.html?Action=Comment&DefaultStatus=resolved&id=1767459

* TODO email a patch to civicrm to increase bounce count
to 2 on ones that are normally 1, because of problems like this:
https://www.bleepingcomputer.com/news/google/gmail-hit-by-a-second-outage-within-a-single-day/
* TODO remove autofs stuff from gnuhope
* TODO get german server up and running
* TODO fix rt cc's etc

Thanks for connecting the dots here.

When people are CC'd on RT queue messages they get the original
message without ever seeing the RT queue id number.  And then later
when the subject line is changed or whatever that comes back with an
RT queue number.  But when I searched my mailbox for parts of that
subject line I couldn't find anything to connect it to.  I knew that
it might or might not contain the RT number but couldn't find anything
by the pieces of it.  This is a place where RT could be nicer.

Another problem is that if someone is CC'd on an RT message and RT
replies then it appears to me that it comes directly to me and I don't
see anyone else having been CC'd on the message.  This is a routine
thing where Karl and I might both be on a CC.  Then later I
subsequently feel I need to forward the message to Karl (or whomever)
so that they are not left out of the conversation.  And sometimes they
have been copied on the reply and sometimes they have not been.  This
is very confusing to me and another area where RT could be nicer.

In any case, thanks for updating me on the connection.  Now I know
what was going on there. Thanks! :-)

Bob

* TODO make ./update-zone easier
it can easily detect changed files with git and update those,
prompting to ask if the zones are right. Also, the log cat it does
is broken and should be fixed.
* TODO make cronjob to clear old duplicates in email sql table

* TODO look into List-Unsubscribe header for fsf newsletters
its an email address, i think we aren't processing it
My main objection was that the data requirement was extremely broad,
which they mostly addressed.

However, I have other big objections to this license.

1. Probably 99% of free software which is designed to be a service does
not implement data export functionality required by this license. Doing
so would often require hundreds of hours or programming.

2. You may submit data to a remote program, but the computing done is
not yours.

“any data that is an input to or an output
from the Work, where the presence of the data is necessary for
substantially identical use of the Work in an equivalent context chosen
by the Recipient, and ... (some condition) or has been assigned to the
Recipient"

Imagine a dating website software. You input your profile, output is:
every profile with a score of how well they match to you, however, you
only get displayed the top X matches.


* TODO check vault backport sources/preferences into ansible
* TODO fix emacs outline mode
to deal with the fact that comments get indented then not recognized

* TODO alert for spammers on lists0p
* TODO fix ticket about duplicate changes happening when running ansible
https://rt.gnu.org/Ticket/Display.html?id=1409745
* TODO mail reliability
** get alerts when mail system fails


* prometheus / ansible

** variables

This prints all vars, despite google saying otherwise.
- debug: var=vars


** TODO standardize on whether to use = or list item in yml
** prometheus

Node exporter can do tls & basic auth, but it is not worth it.
Better to just make an iptables rule to disallow all but the
prometheus server, and maybe some other ips used for testing.

*** for running scripts and exporting results, there are multiple ways
https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusScriptExporterWhy

pushgateway: seems best to avoid this, prometheus doesnt recommend it
unless the service is not tied to the specific host, afaik, all ours
are.
related: https://github.com/aecolley/client_bash

node exporters textfile collector: you run a cronjob and output to the
textfile. Use this for anything that you specifically want to collect
less than a 2 mins apart, prometheus considers metrics 5+ minutes old to
be stale.
https://github.com/prometheus/node_exporter

https://github.com/adhocteam/script_exporter
https://github.com/ricoberger/script_exporter

a few other ways are listed here:
https://nsrc.org/workshops/2021/sanog37/nmm/netmgmt/en/prometheus/ex-custom-metrics.htm

related:
https://github.com/prometheus-community/node-exporter-textfile-collector-scripts
https://prometheus.io/docs/instrumenting/exporters/

only exim exporters found on google:
https://github.com/gvengel/exim_exporter
https://github.com/fstab/exim_prometheus_exporter

useful for converting nagios check plugins to prometheus:
https://www.howtoforge.com/tutorial/write-a-custom-nagios-check-plugin/


useful general info to keep in mind:
https://prometheus.io/docs/concepts/metric_types/
https://prometheus.io/docs/concepts/data_model/
https://prometheus.io/docs/concepts/jobs_instances/
especially the example section:
https://prometheus.io/docs/instrumenting/exposition_formats/#text-format-details
for a boolean metric, 0 for false, 1 for true.
https://www.robustperception.io/booleans-logic-and-math


* TODO when lp registration form is going up,
make sure there is an opt-out for getting emails
* TODO fix topic in #fsf, etc to say how to identify fsf staff
by seeing cloaks.

* TODO make bash history writes and reads immediately for fsf

* TODO fix whitespace in work code
Note, I have changes in my local wtf to deal with this:
https://github.com/dlenski/wtf/issues/17


remove trailing whitespace, add final newline if needed

Done by the following command: this lists all files except .git, and
ignored files, then ignores symlnks and files that grep finds to be
binary, then runs wtf.py on them, https://github.com/dlenski/wtf .

git ls-files --exclude-standard -cmo --no-empty-directory | \
while read f; do if [[ -L $f ]] || ! grep -Iq . "$f"; then continue; fi;  wtf.py -i -E lf "$f"; done

Note, to avoid these in the first place, in emacs I have in my config
(ws-butler-global-mode), and (setq mode-require-final-newline t)


** TODO I should also research how this is done in vim, and
maybe add a commit hook to at least warn people
about whitespace.

* TODO locale in ansible
commit a7cbf81b9710030bb0a07e4fe0c5ce6279a0f46f
Author: Andrew Engelbrecht <andrew@fsf.org>
Date:   Tue Jan 23 18:10:44 2018 -0500

    added /etc/default/locale

    this is needed to set a proper locale for things like postgres
    databases, etc.


$ cat files/common/etc/default/locale
#  File generated by update-locale
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8


I think LANG should be set as it is, but not LC_ALL.

Reference:
https://wiki.debian.org/Locale

"End users should never set LC_ALL, at least not permanently"

"Using LC_ALL is strongly discouraged as it overrides everything. Please use it only when testing and never set it in a startup file. "

I've found LC_ALL to cause problems for me in the past when testing it
out.

* TODO review sshd config in ansible
rwp reported it has bad settings, like allowing X forwarding

* TODO make ticket for alert on eggs spamassin
* d8 bios chip

https://libreboot.org/docs/hardware/kgpe-d16.html
2MiB flash chips are included by default, on these boards. It’s on a
P-DIP 8 slot (SPI chip). The flash chip can be upgraded to higher sizes:
4MiB, 8MiB or 16MiB. With at least 8MiB, you could feasibly fit a
compressed linux+initramfs image (BusyBox+Linux system) into CBFS and
boot that, loading it into memory.


https://www.flashrom.org/Technology#DIP8:_Dual_In-line_Package.2C_8_pins
it is an EEPROM chip


https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?k=&pkeyword=&sv=0&pv16=6547&sf=1&FV=ffe00306%2C2380414%2C23805db%2C23805dc%2C23805dd%2C23805de%2C23805df%2C23805e0%2C1fec000a%2C1fec000b%2C1fec000d%2C1fec000e%2C1fec0011%2C1fec0012%2C1fec0015%2C1fec0006%2C1fec0009&quantity=&ColumnSort=0&page=1&pageSize=25

https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?k=&pkeyword=&sv=0&pv142=391&pv142=1639&pv142=1640&pv142=1641&pv142=1642&pv142=1643&pv142=1644&pv142=1645&pv142=1646&pv142=1647&pv142=1648&pv142=1651&pv142=1615&pv142=1616&pv142=1688&pv142=392&pv142=1708&pv142=1709&pv142=1710&pv142=1711&pv142=1712&pv142=1713&pv142=1714&pv142=1716&pv142=1718&pv142=1719&pv142=1484&pv142=1044&pv142=1499&pv142=1500&pv142=1501&pv142=1502&pv142=1503&pv142=1504&pv142=1505&pv142=1506&pv142=1507&pv142=1727&pv2043=6&pv2043=11&pv2043=9&pv2043=10&pv2043=21&pv2043=14&pv2043=13&pv2043=17&pv2043=18&pv16=12930&pv16=6547&sf=1&FV=ffe00306&quantity=&ColumnSort=0&page=1&pageSize=25

winbond
25Q16BVAIG
133

* TODO put approveGoodRevs into git from directory
* TODO complete alyssa's intern projects
* TODO update general-audit
with the +30 day thing for people who need recon,
and make sure to account for this member who intentionally has multiple
memberships
https://rt.gnu.org/Ticket/Display.html?id=1147159

(later: dunno what this is talking about)
* TODO put /usr/local/bin/mysql-postrotate.sh in ansible if it fixes
the postrotate problem. on my.fsf.org

* TODO ansible improvements

document the emails I sent to emba, asking for them to sign the machine
use policy, and handing off the vm.

document how to change volunteer keys

document how to change the list of files for volunteers

document how to change the list of files/folders that is exported for volunteers

files made in:
/a/work/ansible-configs/roles/kvmhost-ceph/files/usr/local/bin/create-vm-ceph-luks.sh
should also be in ansible.

* TODO periodically search for emails that got no response and follow up
* nonfree fsf firmware
processor microcode
printer firmware
usb conference phone
fiber optical converter
smart switch in data center
bios of a few machines we havnt upgraded yet

* TODO fix rss feed from header in r2e is FSF blogs: <author>
* lists archive


todo: fix archive command to add -verbose, send to a log in /home/mharc/log, rotate that log,
search that log for indexing errors.

todo: look into fixing the negative number error

cron entry:
*/15 * * * * mharc /home/mharc/bin/web-archive  >/dev/null 2>&1

*/15 * * * * mharc /home/mharc/bin/web-archive -verbose &> /home/mharc/log/web-archive-test2.log

mharc is used to configure namazu.

Alias /archive/html /home/mharc/html
ScriptAlias /archive/cgi-bin/ /home/mharc/cgi-bin/

a typical query url looks like this:
https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=test&submit=Search%21&idxname=gforth&max=20&result=normal&sort=score


mknmz command compiles the index into NMZ.* files in the current
directory, or the -O directory


Warning: Non-zero exit status returned from "/usr/bin/mknmz --mhonarc -f /home/mharc/cgi-bin/mknmzrc -T /home/mharc/cgi-bin/template -O /home/mharc/html/qemu-devel -Y --quiet /home/mharc/html/qemu-devel/2017-11": 256


/usr/bin/mknmz --mhonarc -f /home/mharc/cgi-bin/mknmzrc -T /home/mharc/cgi-bin/template -O /home/mharc/html/qemu-devel -Y /home/mharc/html/qemu-devel


Cgnu-reindex-failure of commit-gnuradio
^Cgnu-reindex-failure of commit-grub
^Cgnu-reindex-failure of commit-hurd

* Random rms notes

Reminder from John: rms will undermine and confuse ppl on things we do with gnu.

* TODO look into more appropriately / rt bounces
* TODO read about gnu webmasters
https://www.gnu.org/server/standards/README.webmastering.html
https://www.gnu.org/server/standards/README.editors.html
https://www.gnu.org/server/fsf-html-style-sheet.html

* TODO get notification on new tickets in sysadmin
because sometimes i want them. sometimes i won't.
* TODO file debian bug for exim dmarc
the default signed headers breaks debian mailing lists,
so change the default to what google uses
* bootloader / coreboot notes

 https://unix.stackexchange.com/questions/190865/is-it-possible-to-add-some-pxe-network-boot-option-to-grub
(07:02:41 PM) sudoman: http://ipxe.org/embed

https://www.coreboot.org/IPXE
seems to have a bunch of outdated build options, I skipped those.
Also, using cbfstool from that page appears to build the same image
as selecting equivalent options in the ncurses menu and just building
coreboot

for building coreboot, followed instructions plus
left default 2mb flash size based on googling and finding https://libreboot.org/docs/hardware/kgpe-d16.html

output of coreboot build is
./build/coreboot.rom

to install new rom, using flashrom from latest libreboot-util release,
sudo ./flashrom -p internal -w ./coreboot.rom

coreboot wiki says you can call buildgcc directly, but that doesn't build
everything you need, so it's a bunch of horseshit.


print info about a rom:
./build/cbfstool ./build/coreboot.rom print

flashing from office beaglebone
./flashrom -p linux_spi:dev=/dev/spidev1.0,spispeed=2048K -w ROMFILE

** seabios boot order

usefull command to have around:
screen /dev/ttyUSB1 115200

# https://www.seabios.org/Runtime_config
# build cbmem

cd coreboot/utils/cbmem
make
sudo ./cbmem -c |tee c
# flashing it
# https://www.coreboot.org/SeaBIOS


** libreboot update
https://libreboot.org/docs/#version


find appropriate rom, get size via
apt-get install flashrom
flashrom -p internal -V

if error, reboot, add kernel arg iomem=relaxed

download and extract from http://mirrors.mit.edu/libreboot/stable/20160907/rom/grub/
eg. depending on rom size,
wget http://mirrors.mit.edu/libreboot/stable/20160907/rom/grub/libreboot_r20160907_grub_x200_8mb.tar.xz
wget
http://mirrors.mit.edu/libreboot/stable/20160907/libreboot_r20160907_util.tar.xz



find probably x200_8mb_usqwerty_vesafb.rom (depending on size determined
earlier). rename it libreboot.rom.

get the mac address of eth0 or equivalent

move libreboot.rom to the following folder; this is where the executable for ich9gen is located:

mv libreboot_r20160907_grub_x200_8mb/x200_8mb_usqwerty_vesafb.rom libreboot_r20160907_util/ich9deblob/x86_64/libreboot.rom


./ich9gen --macaddress XX:XX:XX:XX:XX:XX
replace 8m with correct rom size,
dd if=ich9fdgbe_8m.bin of=libreboot.rom bs=1 count=12k conv=notrunc
mv libreboot.rom ../..
cd ../..
sudo ./flash update libreboot.rom
# equivalent flashrom command:
flashrom -p internal -w libreboot.rom

Ocassionally, coreboot changes the name of a given board. If flashrom complains about a board mismatch, but you are sure that you chose the correct ROM image, then run this alternative command:

$ sudo ./flash forceupdate libreboot.rom

You will see the flashrom program running for a little while, and you might see errors, but if it says Verifying flash... VERIFIED at the end, then it’s flashed, and should boot. If you see errors, try again (and again, and again). The message, Chip content is identical to the requested image is also an indication of a successful installation.


misc backup notes:

backup-config on vcs
backup-scripts on vcs and /root on monolith
backups go to /backup and
whizbackup exclude files are in /backup on monolith

* TODO put this transaction note somewhere
5th payment failure, recurring contribution will get marked as
cancelled, and we tell tc, or else they keep trying forever


* low pri todos
** TODO update https://libreboot.org/docs/install/index.html,
where it says
iomem=relaxed
put the actual complete error for seo.

** TODO document some lower proprity todos from john's meeting

** TODO make emacs meetup mailing list
** TODO follow up on slides email
** TODO send out command to technical-discuss to archive panic logs instead of delete
** TODO fix mu4e~view-browse-url-from-binding
it's broken for rt tickets
** TODO delete creds from this file which are in firefox
** TODO learn screen or the other one
** TODO new staff checklist, any new items to add?
** TODO think about rt priority system.
there are tags, tags in subject, and priority field
** TODO brains page review

how to handle different kinds of rt tickets.
  review, add to this.

  wishlist page, be familiar with it
** TODO record how staff use irc
andrew wants to try quasl irc client,
ruben uses weechat + addon + android client.
** TODO add my jabber contact info to my webpage
** TODO Add a link to donate to the FSF or join as a member to your email signature, and your RT signature.
** TODO sub to https://gluestick.office.fsf.org/recentchanges/index.atom
and https://brains.fsf.org/wiki/blogs/johns/
and any other
** TODO add spd setup to new host automation
** TODO Move tarantula:/nfs-root/NEW_HOST/root/.ssh/authorized_keys to authorized_keys.disabled
on all workstations, assuming nothing has gone wrong by doing it on
molly's workstation.

* misc
convert ipv6 ip to /64 in back

ip64() { IFS=: read -a ipa <<<$ip; ip=; for x in ${ipa[@]:0:4}; do [[ $x ]] || break; ip+=$x:; done; ip+=:/64; }

to run cfengine manually, either run on the target host:
cfagent --verbose --no-splay
or from the cfengine server,
ssh faiserver0 cfrun HOSTNAME

server form factors we have: supermicro 825, 113, 213

jeanie answers info@fsf.org and membership@fsf.org

fsf financial year starts oct 1st.

amt: pre-civicrm logmember database. might still be used for some financial
stuff. For access, ssh to amt.fsf.org, use history to connect to mysql
and mysql history to look up someone if needed.


** drupal access from cli

sudoman: iank: if you ever need to get access to drupal from the command line, you can do this:
(02:00:21 PM) sudoman: cd /var/www/site_name ; drush uli admin
(02:00:36 PM) sudoman: then edit the url, if necessary, replacing "default" with "example.com" and put that in a url bar


** searching talos licenses

/a/opt/talos-openbmc ALERT! $ git grep -E -i -e '^ *license *=' --and --not -e '= *["'"'"']\(? *(Apache-2.0|L?GPL[v-]?[123]\.[01]\+?|L?GPL[v-]?[123]\+?|MIT|BSD-[234]-Clause|BSD|CC-BY-3.0|X11|MPL-1.1|MIT-X|EPL-1.0|PSF|Artistic-2.0|Apache-2|ISC|MPL-2.0|Zlib|ClArtistic|copyleft-next-0.3.0|Artistic-1.0 \| GPL.*|IPL-1.0|SPL-1.0|NTP|BSD-0-Clause|SSPL-1|CC-BY-SA-3.0|BSL-1.0|gnuplot|PHP-3.0|GPL-2.0-with-OpenSSL-exception|tcl|openssl|OFL-1.1|IPA||SGI-1|BitstreamVera|netperf|iozone3|\$\{LICENSE_DEFAULT\} & BSD-2-Clause|MPLv1.1|zsh|ImageMagick|HDF5|GPL-2.0-with-GCC-exception|Artistic-1.0\|GPL.*|AGPL-3.0|Python-2.0|PD & MIT|MPL-1|GFDL-1.2|Artisticv1 \| GPLv1+|\(Apache-2.0|LGPL|PSFv2|Ruby|GPL|GPL-3.0-with-GCC-exception|MIT-style|FreeType|Khronos|nbench-byte|PD|radvd|Apache-2.0|Artisticv1 \| GPL.*|openldap|MIT license|CPL-1.0|BSD-1-Clause|ZPL-2.1|Artistic-1.0|read-edid|MIT license|Xdebug|ManishSingh)( *[|&]|["'"'"']$)' > /t/talos-openbmc

* map

** misc services/ hosts

tarantula:
for workstations: home directories and root filesystems. served over
nfs. also, dhcp server.

@fsf.org email: mail.fsf.org

main office ip. we have 14 static ips at the office, we don't use all of them.
74.94.156.211

rt version: it's shown in login screen,
4.2.13-5-gc649048
full text search was released on 4.4.2

** civicrm

log file:
/var/www/ConfigAndLog

current version:
also in the admin panel now
root@crmserver2p:/var/www/drupal-and-civi/sites/all# cat ./modules/civicrm/civicrm-version.php

devservers
crmserver1d  / mysqlserver2d
crmserver2d (no pii in this one, for volunteers to use)

mysqlserver1p: civicrm db

drupal users. through here you can masquerade, and also find people
based on username
https://my.fsf.org/admin/people

to go from a civi user page to a drupal user page, there is a field on
the civi page called "CiviCRM ID / User ID" with a value like: 198055  /
50312
the second number should be a link to their drupal profile.



** irc channels

fsf-office
fsfsys
fsfsys-private
** emails

sysadmin@gnu.org
sysadmin-nonrt@gnu.org
technical-discuss@fsf.org
fsf-office@fsf.org

other aliases:

mail.fsf.org:/etc/aliases-fsf.org



* exim notes

spam blocking in:
/usr/lib/mailman/Mailman/Cgi/subscribe.py

/usr/share/doc/exim4-base/spec.txt.gz

It is usually a good idea to test a new configuration for syntactic      #
correctness before installing it (for example, by running the command    #
"exim -C /config/file.new -bV

in debian, config file used is first found of:
CONFIGURE_FILE=/etc/exim4/exim4.conf:/var/lib/exim4/config.autogenerated
on newer than fsf systems, exim's generated config is
/var/lib/exim4/config.autogenerated
to view it after preprocessor/include file parsing (introduced in a ver sometime after flidas)
s exim4 -bP config
or on ancient exim:
s exim4 -bP configure_file
to view the options it's actually using, including defaults not
mentioned in the config, run this. however, it does not show acl's, and
i'm not sure what else it doesn't show
{ eval exim\ -bP\ {,routers}\; ; eval exim\ -bP\ {transports,authenticators}\; | sed '/^[^=]*:$/b;s/^/    /'; } >/tmp/x


force retry of all queued messages:
exiqgrep -i | xargs exim -M

smtp protocol overview
https://cr.yp.to/smtp/mail.html
interesting reference:
https://mailinabox.email/static/architecture.svg
https://bitlair.nl/Projects/Mailserver_with_Debian,_Exim,_spamassassin,_greylistd,_DKIM,_SRS,_SPF,_DMARC,_forwarding,_LDAP,_dovecot,_LMTP,_disk_crypto
https://github.com/andryyy/mailcow



# describes what all the exim processes are doing
exiwhat
# list of messages in queue
mailq  # aka exim -bp
# queue count
exim -bpc

# delete messages from queue, matching receiever
exiqgrep  -r edward@gnu.org -i| xargs exim -Mrm

exim -Mvl id    #view the message log for message id
exim -Mvh id    #view message id's headers
exim -Mvb id    #view message id's body

mailman won't let you post to subscribe unless you get first, and within a certain
window.

# look for exim log failures
zgrep ' ==\|\*\*' mainlog*gz | sed -r 's/^mainlog.//' | sort -g | less

exim log flags:
<=     message arrival. following address is the envelope sender address
(=     message fakereject
=>     normal message delivery
->     additional address in same delivery
>>     cutthrough message delivery
*>     delivery suppressed by -N
 **     delivery failed; address bounced
==     delivery deferred; temporary problem

A           authenticator name (and optional id and sender)
C           SMTP confirmation on delivery
            command list for “no mail in SMTP session”
CV          certificate verification status
D           duration of “no mail in SMTP session”
DN          distinguished name from peer certificate
DS          DNSSEC secured lookups
DT          on => lines: time taken for a delivery
F           sender address (on delivery lines)
H           host name and IP address
I           local interface used
K           CHUNKING extension used
id          message id for incoming message
P           on <= lines: protocol used
            on => and ** lines: return path
PRDR        PRDR extension used
PRX         on <= and => lines: proxy address
Q           alternate queue name
QT          on => lines: time spent on queue so far
            on “Completed” lines: time spent on queue
R           on <= lines: reference for local bounce
            on =>  >> ** and == lines: router name
S           size of message in bytes
SNI         server name indication from TLS client hello
ST          shadow transport name
T           on <= lines: message subject (topic)
            on => ** and == lines: transport name
U           local user or RFC 1413 identity
X           TLS cipher suite

testing acls:

exim -bh IP_ADDRESS
then manually enter smtp commands
http://www.samlogic.net/articles/smtp-commands-reference.htm
see below, org mode section on simulating messages.

testing routers, transport, rewrite, etc:

$ exim -bt -f iank@fsf.org x@gmail.com
R: smarthost for x@gmail.com
x@gmail.com
  router = fsfsmarthost, transport = remote_smtp_smarthost
  host mail.fsf.org [209.51.188.13]

clear out retry database:
s exim_tidydb -t 0m /var/spool/exim4 retry
note: m is for minutes, it could be d for days, it doesnt matter

clear out specific host in retry database:
s exim_dumpdb /var/spool/exim4 retry | gr some_host
# copy first space delimited word
s exim_fixdb /var/spool/exim4 retry
# paste, enter, d, enter



for testing expansions:
exim -be


misc exim notes:
useful exim docs:
/usr/share/doc/exim4-base/README.Debian.gz
/usr/share/doc/exim4-base/spec.txt.gz


also see brc file for testing exim.


dpatch patch-template -p 85-CVE_string2019 "string2019" \
 < string.patch  >debian/patches/85_CVE-string2019.dpatch
fakeroot debian/rules binary

** dmarc testing

I've setup my own strict dmarc domain, I'm using:

i@dmarctest.b8.nz

** simulating messages

for testing acls,

logwrite = test is good

for example, to test a failing dmarc message, run this on lists2d.fsf.org

while read -r line; do
  echo "$line"
  sleep 2
done <<'EOF'| exim -d+all -bhc 127.0.0.1
helo localhost
mail from:<mailman@lists.dev.fsf.org>
rcpt to:<ian@iankelling.org>
data
From: i@dmarctest.b8.nz
To: mailman@dev.fsf.org
Subject: Testing Exim

This is a test message.
.
quit
EOF

while read -r line; do
  echo "$line"
  sleep 2
done <<'EOF'| exim -d+all -bhc 127.0.0.1
helo localhost
mail from:<qemu-devel-bounces+testignore=je.b8.nz@nongnu.org>
rcpt to:<testignore@je.b8.nz>
data
From: ian@iankelling.org
To: testignore@je.b8.nz
Subject: Testing Exim

This is a test message.
.
quit
EOF



** sending to not all mx hosts for yahoo

# mx list:
exim -bem /tmp/t '${lookup dnsdb{>:mxh=yahoo.com}}'
# ip list
exim -bem /tmp/t '${lookup dnsdb {>:a=${lookup dnsdb{>:mxh=yahoo.com}}}}'

# setting ip list to a var
warn
set acl_m_yahoomx = ${lookup dnsdb {>:a=${lookup dnsdb{>:mxh=yahoo.com}}}}

# random int generated based on the message, modulo length of the list
exim -bem /tmp/t '${eval10: $received_time % ${listcount:00:11:22:33}}'

# picking from the list
exim -be '${listextract{0}{00:11:22}'
exim -be '${listextract{1}{00:11:22}'


# length of dns list:
exim -bem /tmp/t '${listcount:${sg{${lookup dnsdb{>:,#mx=yahoo.com}}}{[^:]+#}{}}}'
# exim -be '${reduce {${sg{${lookup dnsdb{>:mx=yahoo.com}}}{[^:]+ }{}}}{0}{${eval:$value + 1}}}' # old exim way
# random time rotating per message number modulo length of dns list
exim -bem /tmp/t '${eval10:($tod_epoch / 100000 + $received_time) % ${listcount:${sg{${lookup dnsdb{>:,#mx=yahoo.com}}}{[^:]+#}{}}}}'
# pick 1 from mx list
exim -be '${listextract{1}{${sg{${lookup dnsdb{>:mx=yahoo.com}}}{[^:]+ }{}}}}'
exim -be '${extract{1}{:}{${sg{${lookup dnsdb{>:mx=yahoo.com}}}{[^:]+ }{}}}}'
# pick random from mx list
exim -bem /tmp/t '${extract{${eval10:($tod_epoch / 100000 + $received_time) % ${reduce {${sg{${lookup dnsdb{>:mx=yahoo.com}}}{[^:]+ }{}}}{0}{${eval:$value + 1}}} + 1}}{:}{${sg{${lookup dnsdb{>:mx=yahoo.com}}}{[^:]+ }{}}}}'
# a record list of fsf.org
exim -be '${lookup dnsdb{>: a=fsf.org }}'
# max a record
exim -bem /tmp/t '${reduce {${lookup dnsdb{>: a=${extract{${eval10:($tod_epoch / 100000 + $received_time) % ${reduce {${sg{${lookup dnsdb{>:mx=yahoo.com}}}{[^:]+ }{}}}{0}{${eval:$value + 1}}} + 1}}{:}{${sg{${lookup dnsdb{>:mx=yahoo.com}}}{[^:]+ }{}}}} }}}{0}{${if gt {$item}{$value} {$item}{$value}}}}'
# max a record of random mx

# a record list from mx
exim -bem /tmp/t '${sort{${lookup dnsdb{>: a=${extract{${eval10:($tod_epoch / 100000 + $received_time) % ${reduce {${sg{${lookup dnsdb{>:,#mx=yahoo.com}}}{[^:]+#}{}}}{0}{${eval:$value + 1}}}}}{:}{${sg{${lookup dnsdb{>:mx=yahoo.com}}}{[^:]+ }{}}}} }}}{le}{$item}}'


# length of a record list:
exim -be '${reduce {   }{0}{${eval:$value + 1}}}'
# pick 1 from a record list
exim -be '${extract{0}{:}{${sort{${lookup dnsdb{>: a=fsf.org }}}{le}{$item}}}}'
# pick random from a record list
exim -be '${extract{0}{:}{${sort{${lookup dnsdb{>: a=fsf.org }}}{le}{$item}}}}'


** TODO figure out how the exim queue works, so many -qG processes
after just barely starting exim, and they seem to hang around long after
processing the queue. why?


* spamassassin reference
configs are in:
/usr/share/spamassassin
/etc/spamassassin

in t9, the manual lists default plugins. grepping, i see an additional
one:
Mail::SpamAssassin::Plugin::Rule2XSBody

todo: port over training info?

* reference
** import keyring
The following code adds the same keys with a high trust level in your trustdb (not the same as signing someone's key).

    for k in $(gpg --import fsf-keyring |& sed -rn 's,^gpg: key (.*):.*,\1,p'); do
      gpg --fingerprint -k $k | sed -nr 's, ,,g;s,$,:6:,;s,.*print=,,p;'; done | gpg --import-ownertrust


** license request on bug tracker

Missing LICENSE

I see you have no LICENSE file for this project.

I suggest releasing the code under the GPLv3 or AGPLv3 license so that
people are encouraged to make improvements and contribute them. Without
a license, sharing the code or any changes is a violation of copyright
law.

** misc

good ps command:

ps -faxuww

** ansible
default hosts is /etc/ansible/hosts

keywords:
https://docs.ansible.com/ansible/latest/reference_appendices/playbooks_keywords.html

With until, the default value for “retries” is 3 and “delay” is 5.
https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html

to test commands locally, run apx (bashrc)
and put something like this in /a/x.yml
---
- hosts: all

  tasks:
  - name: sleep1
    shell: sleep 10 && touch /tmp/t2
    async: 45
    poll: 1

  - name: sleep2
    shell: sleep 2 && touch /tmp/t1
    async: 45
    poll: 1


async loops, use
https://github.com/ansible/ansible/issues/44272



** asterisk debugging commands
see calls as they are made, etc:
asterisk -vvvvvr

from the asterisk shell, not sure what these do.
sip set debug on
show channels
* lists / mailman reference
to find test list

/var/lib/mailman/bin# ./list_lists | grep test
* dmarc
usr/lib/mailman/Mailman/Handlers/AvoidDuplicates.py

            elif ccaddrs.has_key(r.lower()):
                del ccaddrs[r.lower()]

usr/lib/mailman/Mailman/Utils.py
def IsDMARCProhibited(mlist, email):

https://en.wikipedia.org/wiki/DMARC
https://tools.ietf.org/html/rfc7489#section-3
https://dmarc.org/wiki/FAQ#senders

https://www.exim.org/exim-html-current/doc/html/spec_html/ch-support_for_dkim_domainkeys_identified_mail.html
https://www.ietf.org/rfc/rfc4871.txt

mailman test list
newlist -q mailman ian@iankelling.org jetdirpAbsEtpiHa


install mailman, follow
https://www.gnu.org/software/mailman/mailman-install/node16.html
better format /usr/share/doc/mailman/mailman-install.txt.gz
it implies you can follow this,
http://www.exim.org/howto/mailman21.html
but the mailman docs seem to cover it better.

/usr/lib/mailman/Mailman/mm_cfg.py
MTA=None   # Misnomer, suppresses alias output on newlist
ser restart mailman

web-conf -p 80 apache2 x2.office.fsf.org

edit
/etc/apache2/sites-enabled/x2.office.fsf.org.conf

Include /etc/mailman/apache.conf

s a2enmod cgid
ser restart apache2

browse
http://localhost/cgi-bin/mailman/admin/mailman/members


tee -a /etc/exim4/conf.d/main/000_localmacros <<'EOF'
# Home dir for your Mailman installation -- aka Mailman's prefix
# directory.
MAILMAN_HOME=/var/lib/mailman
MAILMAN_WRAP=MAILMAN_HOME/mail/mailman

# User and group for Mailman, should match your --with-mail-gid
# switch to Mailman's configure script.
MAILMAN_USER=list
MAILMAN_GROUP=list
EOF

s dd of=/etc/exim4/conf.d/router/099_exim4-config_mailman <<'EOF'
mailman_router:
  driver = accept
  require_files = MAILMAN_HOME/lists/$local_part/config.pck
  local_part_suffix_optional
  local_part_suffix = -admin : -bounces : -bounces+* : \
                      -confirm : -confirm+* : \
                      -join : -leave : \
                      -owner : -request : \
                      -subscribe : -unsubscribe
  transport = mailman_transport
EOF

s dd of=/etc/exim4/conf.d/transport/29_exim4-config_mailman <<'EOF'
mailman_transport:
  driver = pipe
  command = MAILMAN_WRAP \
            '${if def:local_part_suffix \
                  {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
                  {post}}' \
            $local_part
  current_directory = MAILMAN_HOME
  home_directory = MAILMAN_HOME
  user = MAILMAN_USER
  group = MAILMAN_GROUP
EOF

** testing for dmarc strict senders

wget -m ftp://lists.gnu.org/info-gnu
cd lists.gnu.org/info-gnu
sed -rn '/^From: /{s/.*@([^> ]*).*/\1/' * | sort -u | while -read -r l; do host -t txt _dmarc.$l; done