The long-running Japanese software portal vector.co.jp is ending their hp.vector homepage service and deleting all hosted sites on December 20—this service hosts a ton of turn-of-the-millennium doujin and homebrew pages, including a lot of pasocom-relevant material for PC98, MSX, etc by many authors who never moved and/or went MIA and aren't going to back up their pages of their own accord, so if there's anything you want to save, don't sleep on it. (IIRC they never expanded their 5MB hosting cap, so we're not talking unwieldy levels of data.)
With the help of @dog and @asie, who were able to point me to the resources to build a pretty darn close to perfect list of seed URLs, we're expecting to get a full archive. It became a side project that I was able to successfully pitch as part of my work at Archive.
I'll have more updates soon as we make the material available, but the final product will be a collection with full-text search and potentially extended metadata, so more than just a bunch of unwieldy Wayback captures.
Just a real quick update to say that the crawling process is going well, but it's taking a bit longer than I originally anticipated. There were a lot more Vector homepages that had automated redirects on them than I was expecting, and our crawler follows those and tries to crawl them. Some of those redirects were to sites that caused the crawler to enter a "trap" where it builds up a big queue of bogus links. I was able to tame these with some help of a teammate, but there's still some more legit URLs to crawl. Hopefully soon!
Hey folks, I've found that the crawl has been digging up a ton of great stuff, so I'm continuing to run it. If you find that certain pages are missing (usually deep links within a site), it's likely that they'll eventually get filled in. But what we currently have available is around 40GB of the Vector Homepage Service and a lot of surrounding pages, including a lot of the main Vector software pages so we can capture as much of the actual software as possible.
I am hoping at some point to better organize this. We have the ability to apply metadata fields (most commonly Dublin Core fields for all my librarian friends out there, but custom fields are possible) for faceted filtering and a bunch of other options to make it easier to navigate. There's also full text search, although this probably isn't going to be perfect (at least not yet). But the data is being captured, so anything like that can be fixed/improved later.
