EDIT: after learning about https://jort.link from @lifning (thanks!) i've made some changes to redirect mastodon traffic through them (as documented on their site) so that we aren't taking the brunt of it. still planning on fixing our own caching this week, but this should be an ok band-aid for now.
we're seeing issues related to the special caching we use for serving to Mastodon, which means we have had to temporarily start blocking requests from Mastodon to prevent performance issues.
the only impact (unless you're an instance admin) is that if you post a cohost link on mastodon, you won't get embed information. if you're an instance admin, you'll (probably) see an increase in failed requests to cohost (i will admit that i have no idea if this information is presented to admins).
we'll be unblocking mastodon server traffic once we've worked with fastly to fix the underlying issue this week. more technical details below the cut.
hopefully this is the last fire drill of the weekend. i'm tired. 
I'm pretty sure i've talked about this before, but an issue with mastodon's federation model is that things like opengraph metadata (what's used to display link embed details) don't get federated with the posts. this means that if someone posts a link to cohost, every single instance that sees that link will all make a request to us at the exact same fucking time.
while this isn't malicious (i consider it to be a consequence of questionable protocol design), it effectively manifests as a mini-DDoS every time someone links to us. we're not the first to mention this as an issue, it's just sort of A Thing.
in the past we've mitigated this by having a Special Caching Layer Just For Mastodon so that we don't have to fully rebuild the page every time, but for some reason this has stopped working consistently! given that more people than average are linking their other social media profiles, we're seeing occasional random spikes of up to 5x standard request load that are enough to temporarily overwhelm our autoscaling and bring the site down.
given that it's sunday and i don't want to deal with this too much right now, we're just doing the Easy Approach of blocking mastodon traffic at the firewall level. this week, i'll be working with our support contacts at fastly to figure out what went wrong with the caching layer and undo the blocking change.
thanks for using cohost! 
