Conversation

Postmortem on last week's major Pinball Map outage (was out 2 hours). An upgrade from Ubuntu 24 to 26 introduced a breaking change to a systemd service we had set up.

Before=ufw.service on an iptables service caused an ordering issue with network services that resulted in networking dropping at boot. Same service had no (discernible) issue in 24.

We had to restore from an old snapshot. It was unfun.

What we learned:
- Make daily backups.
- Create a server setup runbook for quick and efficient installations.
- Create a brand new droplet on each major upgrade and just switch over to that once it's up.