I wrote archivescraper to scrape version information for betaversion lookups. It's based on ftpscraper and when I wrote it, I wanted to stay as close to ftpscraper as I could so as to make the jump from one to the other as small as possible.

archivescraper takes a while to run. In a fresh local dev environment, it can take 20+ minutes. It's kind of irritating and a time sink and I run it at least once a week.

Relatedly, I wrote a verifyprocessed job. That uses multiprocessing to reduce the time it takes to run significantly.

archivescraper has similar properties--the bulk of the time it takes to run is traversing links on a website which is predominantly slow HTTP conversations. That's pretty ideal for multiprocessing with lots of workers.

This bug covers taking what I did with verifyprocessing and applying it to archivescraper.

The last fresh run I did took 40 minutes.

Grabbing this to tinker with today. I think it's straight-forward except for error handling and reporting. That's a bit trickier.

willkg merged PR #4849: "bug 1536147: make archivescraper faster" in d8d1136.

This has been running on stage for a while and it's significantly faster. Yay!

We just pushed this to prod. Marking as FIXED.