We had to restore from an old snapshot. It was unfun.
What we learned:
- Make daily backups.
- Create a server setup runbook for quick and efficient installations.
- Create a brand new droplet on each major upgrade and just switch over to that once it's up.