added a special ban for 101.44.*.* and 49.0.*.* (and soon more, i'm sure) because some asshole using the hauwei cloud™ cannot help but attempt to scrap the entire site

dragon warrior iii for the game boy color describes me as "stubborn", and i'm tempted to agree with that assessment
co-owner tcrf.net. i run an old forum, jul.
i've been around the internet since '01.
i generally feel like the internet
peaked somewhere around '07.
private: @xkeeper-PLUS
18+: @xkeeper-TI
plural / some kind of digital therian thing.
still discovering myself.
all of this is new to me.
added a special ban for 101.44.*.* and 49.0.*.* (and soon more, i'm sure) because some asshole using the hauwei cloud™ cannot help but attempt to scrap the entire site
i do not have an easy way of banning multiple ip ranges (even this one is a simple substr check against the first 5/7/8 characters) but that would be the ideal method
same with banning all of russia
One request per minute is rather slow for scrapping. When I was doing it, I was starting the crawler at least at one every 10 seconds but maybe as fast as one every 3 seconds?
it's been going on for at least 24 hours at this point, but it can't be anything else because it's going into deeply fucked urls you can only get to by following several links in, and robots.txt is there to say "no" to everything
if they're disregarding the robots.txt you pretty much have the right to take the gloves off imo
Ouch, ignoring robots.txt is rude. I hope the library I was using did follow it :s