hi folks! quick turnaround on this postmortem because we know Exactly What Happened and mostly Why It Happened
WHAT HAPPENED
While deploying tag silencing (live now by the way, check your settings page), a required database change led to the database needing to rebuild the index of posts by tag, taking it offline.
We knew that the index rebuild would happen, so we planned for it to occur all at once during the deployment process, which made the deployment process a little slower but we thought would leave us a working site afterwards. However, for unforeseen reasons, the index went offline and didn’t come back.
This meant that instead of being able to quickly grab a big list of posts in, for example, the "eggbug" tag, we had to go through every single post, individually, in order, until we found ones that were tagged with "eggbug". There are a lot of posts on cohost, so this was incredibly slow and caused serious performance issues, bringing the entire site to a halt.
We opted to roll back to the previous deploy, which we knew also wouldn’t work (the database change was breaking for old versions), but would error out quickly instead of thrashing the database. This was the cause of the e.filter is not a function error you may have seen.
I like how these posts demystify the process of making A Website from like occult sorcery that only the highest paid assholes on the planet are able to comprehend to just like, work that people do that has specific causes and effects and technical problems and solutions. it's a real breath of fresh air compared to the cult nonsense that a lot of tech people are pushing these days.
