Why is it nearly impossible so get a copy of everything archive.org has for a website?
They've removed the bulk "WARC" downloads and there's no wayback machine API. The only option is to use the undocumented "id_" trick to get the original HTML from web.archive.org and run a crawler.
... but web.archive.org bans you if you send more than 15 request per minute. Going slower still results in a lot of 429s.
This has to be the most annoying and most wasteful (on both sides) possible way to download anything. I'm trying to fetch less than a megabyte of text, and it's looking like it will take days.
Like, I get that AI scrapers are annoying, but things like bittorrent exist, and I'd be willing to create an account and pay to cover bandwidth and server time costs, but I can't.
They've removed the bulk "WARC" downloads and there's no wayback machine API. The only option is to use the undocumented "id_" trick to get the original HTML from web.archive.org and run a crawler.
... but web.archive.org bans you if you send more than 15 request per minute. Going slower still results in a lot of 429s.
This has to be the most annoying and most wasteful (on both sides) possible way to download anything. I'm trying to fetch less than a megabyte of text, and it's looking like it will take days.
Like, I get that AI scrapers are annoying, but things like bittorrent exist, and I'd be willing to create an account and pay to cover bandwidth and server time costs, but I can't.