Well, here we are. Before we launched, I had no idea if we'd even talk to the old man in Viridian to see the eggbug cameo, let alone get all the way to Cinnabar to catch one, let alone get to the "special message" at the end of the credits. About halfway through development, I'd already started bouncing reassurances off of the friends that had jumped aboard the project about how "well, even if this ends up being a flop and nobody does more than click it a few times and say 'oh neat', the technology we developed for it is still really impressive and able to be repurposed for other things..."
It took y'all less than 48 hours. good job!
So, while this is all fresh in my memory...
I had started hacking together a tech demo for this a few weekends ago, before I even had posting privileges on cohost, when I saw someone openly wondering when we'd see something like AO3 DoomGIF in a post. At that time, I had been in need of something on which to hyperfocus so I wouldn't totally lose it after the past few months of dealing with one crisis after another of ever-increasing severity, and, well... weird retro-gaming art installations are kind of my forte, so it was the first thing that stuck.
Because I'm fortunate enough to have the kinds of people in my life who see me working on something cool and immediately get excited about how they might be able to help, a few more folks got pulled into this thing's orbit, and we decided we should publish this thing under a demogroup, of sorts. Some of us could reasonably be described as Bell Labs fangirls, if you're wondering.
A few days later, cohost plays pokemon blue was born! ...by someone else. We were still gradually working on optimizing ours and building it to scale, and it would be a while. It looks like Twi took a very direct approach to attaching a JavaScript-based GB emulator to some DoomGIF code, and she ended up having trouble keeping it online and stable under the load. This was, however, a valiant first effort, and it was very helpful as confirmation that yes, we are right to be working so hard on making sure our version would be adequately ~web-scale~ (or at least cohost-scale).
Now, I have the privilege of having had experience working on backend tech for some particularly large-scale online games in my past (privilege in the "getting a job is easy with pale skin and a college degree" sense--I wouldn't wish the experience of working in corporate gamedev on anyone), which probably helped guide me to make some reasonable-enough design decisions that we were able to scale to several orders of magnitude more users than we actually ended up seeing when we were live, even despite running on some very modest decade-old server hardware. We didn't exactly go "viral," but we were prepared to, at least to the extent that we estimated we'd see if large swaths of the cohost crowd went totally wild for it.
We built in some mitigations against technical abuse, which of course is the easier kind of abuse to mitigate. We also built a post-hoc moderation interface, having anticipated that even though cohost has felt like a pretty positive and chill place so far, there was still a nonzero risk someone would, say, type in something dangerously inappropriate as a name, and we definitely wanted to make sure our post didn't get used as an unmoderated exception to an otherwise-moderated space.
Blast off at the speed of light
After three weeks of several of us having all our spare time consumed by this project (and some of the stressors forcing me into needing this distraction finally letting up), we finally decided that we should cut non-essential scope and pull the trigger on launching it so we wouldn't get caught up in neverending scope creep. (For the benefit of anyone reading who hasn't worked in game dev: Almost every game you've ever played has had this happen. That's why TCRF has so much to write about!) We had a big checklist of features sorted into essential/non-essential buckets, and the unchecked boxes in the "essential" column were dwindling. If the game ended up being chaotic to the point of taking on the order of weeks rather than days to conquer the E4 and COHORT, we would've gradually worked on improving the service itself, but the system as it stood was seemingly already viable and polished enough to be played and enjoyed.
So on launch night, we pulled everyone involved into a conference call (and had A/V issues, as is tradition) for running through the steps of our first go-live deployment, both to review the final text of the post and to have everyone present for any exciting or alarming moments that should command our attention. A surprising workaround for a cohost limitation got discovered then and there by @iliana that had us completely rewrite the user-facing workaround notes. A last-minute iOS bugfix got tackled by @viv. The nervous energy and excitement was almost overwhelming. We had our dashboard up with an eclectic combination of funny in-game stats and server-load, our admin interface up in case we needed to fiddle with anything, aaand... published the draft and boosted it from all our accounts.
We were so giddy to see some folks start filtering in and going through the new game cutscene, picking both of the cohost-bait player/rival name options I put in. Comments, boosts, follows, and florps started pouring in. People were in disbelief that it was real, and that it was working this well. We watched the first few hours together, with me mocking myself ad nauseum about violating Sir David Attenborough's code of non-interference whenever I wanted to push any buttons myself. And you all already beat Brock within that time! People were seemingly having fun with our little project, the highest praise I could've asked for for my work.
At one point shortly after that, someone took the wheel and walked us back south of Pewter. I wondered if we were going to fetch a Pikachu to deal with Misty, but no, we grabbed a Rattata instead just outside the forest, and started to nickname it... "Wait a minute. Wait a minute! Is Alex Zandra Van Chestein playing right now?" I exclaimed. This was notable not just because some of us in the Labs are fans of her writing, but also because she was one half of the thing that inspired @viv and me in the first place to start working together on weird retro-emulation projects (like this one!) half a decade ago.
But--oh dear, a genuine misinput (as in, I definitively know that both of the people who were on the controls at the time were sincerely trying to add some cute symbols to the end, no foul play here) resulted in the ♂️ symbol being appended to her name and locked in! My heart jumped a little, 'cause I absolutely did not want the fun little GBC experience we just launched to be involved in misgendering someone. Thankfully, we had built moderation tools to easily change nicknames in the game's working memory at runtime (dubbed "GameBlåhaj" by @iliana), so one of us quickly jumped on that before the name appeared anywhere else on the stream.
As it turned out, her girlfriend had been on the sticks using our toy to send a beautiful and vulnerable message of love to her girlfriend. The positive vibes were off the charts, system load levels were nominal, and we were encouraged by how cool the cohost community was being about more-or-less taking turns making progress and only bickering over controls the normal amount. We convinced each other that it was okay to get some sleep. Well, most of us. I stayed up another few hours because I was still rocking a bunch of restless excitement about what people would do next and anxiety that I'd have to intervene again somehow, but ultimately I've got a fatigue disorder to look after.
Incident response
I woke up the next morning to a message from @viv telling me that in the middle of the night, someone released all the Pokemon and tossed all the items. It may seem like an obvious oversight to have left releasing Pokemon in the game, given how the prior art of TPP went down, but, well, we had addressed many, many other potential avenues for bad behavior, and as folks in security know, playing defense means you have to get everything right, because the offense only needs you to get one thing wrong. (To come up for air for a moment, let's remember that the crux of this project was to make a GBC game playable and responsive with an <img src="screen.gif" /> and a bunch of <a href="button.php?..."> tags inside a cohost post.)
I stumbled out of bed and went straight to my PC (real life version) to get to work straight away at hacking out the ability to release Pokemon, grabbing the savestates from just before the incident and from the current state of the game, and trying to catch up on the hashtag to make sense of the situation. I discovered that our cute little #interactable GBC had been abused by someone to deliberately and genuinely hurt the feelings of other cohost users, saying in the hashtags of their post that they just had to do it (which, if it's not clear, is not true in my view--before you decide to prank a group of strangers, try to assume that one of them might just be having the worst day of their life already). This was obviously a pretty stressful thing to have happen on its own, but it gets worse--who's the victim in the crosshairs in the first post made by the perpetrator to brag about their actions? Why, it's Zandra again, being burned in Rattata-effigy, right under big bold text that says "FUCK YA LIFE." Oh shit, do we have a harassment situation on our hands?! Why oh why did I let myself go to sleep without taking the server down?! Fuck!
And before I get farther into this, I want to be very clear: I don't want anyone giving the person who did it a hard time. First because the consequences have been sorted out and it's largely water under the bridge now--but more importantly, while it may be an act of reckless immaturity for a bully to break everyone's toys to try and become the center of attention, we were also just as culpable for insufficiently childproofing the sharp edges of the playground we built in a public space. As it turned out, even though most of our experience of cohost was pretty positive, it's still a wide enough audience to include people who hold themselves to the high social behavior standards of Twitch chat, and we evidently needed to better account for the resulting inevitability that people out there will want to vandalize a public art installation if given the opportunity. Between this and the typo incident, we certainly learned a lot about how thoroughly we need to protect the things to which our players would grow attached in-game, and how proactively we should be vetting any text inputs, before we host something like this again.
Unlike many mainstream game developers, we absolutely did not want our project to become a source of compulsive anxiety for our players, worrying that if they didn't keep playing the game they'd lose everything again to the next self-described joker to come along. The past few months have been harrowing for a lot of people in our communties, and I wanted this thing to be a cozy-vibes re-tread of an old classic for everyone to enjoy, not a chaotic survival-horror stress-fest; not only was that story already told on Twitch, but, well, LGBTQ+ folk have enough of that genre going on in meatspace right now, and that's who made up most of our vocal playerbase.
At the end of the day, I count us as lucky. Our first fire-drill could've been much, much worse than, what, someone trying to be funny by hurting people without consent, but then later expressing an amount of genuine regret (in a footnote, at least) after some other members pointed out that it wasn't cool to trash someone's expression of love like that (which doesn't account for the other cohost members who were also affected, but it's something). It seemed like the intent was to be cruel, but not that cruel. There's worse shit out there that could've happened, for sure.
Still, it wasn't great that the majority of what showed up in our hashtag that day was copies of people replying to that fuck-your-life post again and again, making it generally be the first thing you'd see in the tag's page due to how cohost is structured. Not only was it far from the energy we'd hoped to bring to the community with all this work, but I was also already worried that we'd be causing cohost's staff a headache, as they had signed themselves up to build and moderate a website, not a weird old single-player videogame recontextualized into an online multiplayer game. And here's someone's proudly-shared screenshots of using our post as an excuse for behavior that might or might not be the standard they want for the cohost community--behavior to which we were now an accessory, if it wasn't--which at first glance (unintentionally, according to the author) looked like targeted harassment of an iconic light novel author.
Zandra herself said she was saddened, even though it was 'just a game' and all that. She'd even posted fanart of the ostensibly short-lived Rattata by the time I'd gotten my bearings. What are games for, if not for us to feel things about them, right? A contact offered to reach out to her for me to let her know we were restoring the data, and she was appreciative.
At any rate, I finished patching the game, with a little of my ambient frustration channeled into some light snark in the textbox and a patch-notes vaguepost (the show must go on), restarted the server with it and the restored save data, and with the biggest fire now put out, I could finally eat, take my morning meds and tea, and hop into an imminent meeting for dayjob.
Anxiety definitely followed me through the day as I would glance at people fighting over the controls to avoid letting someone toss items, and I could feel the stress our post was still causing people who were invested, so I put yet another quick patch in for that. Despite inventory space being super tight in gen 1, I made the judgement that it was long-term more important to protect against griefers, for the sake of both our players and my now-hypervigilant desire to patch up these vulnerabilities so I'd be able to sleep again that night. I also made shopkeeps (the only other way to get rid of items in bulk) buy items at full retail price, such that player resources would be a little more inconvenient for hostile actors to drain again.
At this point we were burnt out super hard on dealing with the aftermath, and very disheartened by the sour turn things took so soon for something we'd put so much effort into, and finding it kind of hard to be motivated to keep working on this thing. I was still visibly exuding enough hypervigilance-adrenaline about wanting to protect our players that my loved ones were getting worried about me, so I took a quick self-care break while @viv tried to enumerate what other possible venues of abuse remained about which we could do anything.
Triaging those things led to some promising implementation ideas, but ultimately we simply didn't have the time or motivation to make huge changes. We came to the conclusion that the main threat that we'd be able to mitigate with the energy available to us would be nicknames. Yes, we already had tooling for fixing that after-the-fact, but what if someone grabbed the controls overnight again and named something maliciously? In fact, we already had a close call with a Pokemon almost being named to insult the person who tried to softlock the game--and no matter how much a headache they'd given us, we would absolutely have edited that out too, as our work wasn't going to be used for harassing anyone if I could help it.
I sat in a call that night with @viv and walked her through my process of patching the game's code, and we bounced ideas off of each other (as we do) for how to best communicate what we were doing in-universe. Here's what we came up with:
The idea was that we'd switch the ROM out when we had the time and spoons to actively pay attention to what was going on with the version that had the ability to nickname things, so we could use our existing tools to patch memory as necessary, and then switch back when we didn't want it dominating our lives any more (but as fate would have it, y'all were on the E4 grind before we even got to that point, so the Name Rater never got to return from his break).
With that solved, we finally exhaled again and started trying to put a more positive spin on things: We'd already corked the major leaks in the boat as far as abuse vectors, so we could at least disengage safely without any irreversible harm. And hey, the harm that was caused already was largely reversible--our metrics collection and thorough archival practices gave us the precise timestamp at which the number of Pokemon plummeted, which meant I was able to quickly grab the right savestate and delegate out the task of resurrecting the boxes while I was frantically hacking away at SM83 assembly. The system worked!
[states]$ ls states_2022-07-22_19.24.29.tgz states_2022-07-23_04.27.27.tgz states_2022-07-23_16.28.05.tgz states_2022-07-22_19.26.58.tgz states_2022-07-23_05.27.30.tgz states_2022-07-23_17.28.08.tgz states_2022-07-22_20.27.01.tgz states_2022-07-23_08.27.39.tgz states_2022-07-23_18.28.11.tgz states_2022-07-22_21.27.05.tgz states_2022-07-23_09.27.43.tgz states_2022-07-23_19.28.15.tgz states_2022-07-22_22.27.08.tgz states_2022-07-23_10.27.46.tgz states_2022-07-23_20.28.18.tgz states_2022-07-22_23.27.11.tgz states_2022-07-23_11.27.49.tgz states_2022-07-23_21.28.21.tgz states_2022-07-23_00.27.14.tgz states_2022-07-23_12.27.52.tgz states_2022-07-23_22.28.24.tgz states_2022-07-23_01.27.17.tgz states_2022-07-23_13.27.55.tgz states_2022-07-23_23.28.27.tgz states_2022-07-23_02.27.20.tgz states_2022-07-23_14.27.59.tgz states_2022-07-24_00.28.31.tgz states_2022-07-23_03.27.24.tgz states_2022-07-23_15.28.02.tgz states_2022-07-24_01.28.31.tgz [states]$ tar tzf states_2022-07-24_01.28.31.tgz | tail -n 3 2022-07-24_01.19.33_save.state 2022-07-24_01.20.33_save.state 2022-07-24_01.21.33_save.state [states]$ for i in *.tgz ; do tar tzf $i ; done | wc -l 2597
People were still playing, albeit in smaller numbers, and making steady progress through the game--so the toxicity hadn't turned everyone off, nor had my hasty and somewhat heavy-handed approach to anti-griefing measures (heavy-handed for what I hope is a good reason--dissuading the people from participating whose actions dissuade the players I want from participating).
We were at Silph Co., which is probably the most labyrinthine the game gets, so I had no idea how much progress might be made by morning, but I did sleep a little bit better in the knowledge that we sufficiently disaster-proofed it (as much as you can disaster-proof gen 1).
Missing? No!
One of the many little easter-eggs I put into the game ahead of launch, of course, was everyone's favorite cohost-critter, EGGBUG. I had left some obtuse clues about how to obtain it in the post and in the dialogue the Old Man gives you in EGGBUG's first cameo in the Viridian tutorial, but I had no idea if people would catch on all the way and try it. Heck, for all I know whoever did it was on the same "mods are asleep, corrupt the game however I can" warpath. Whatever the case may be, waking up to see EGGBUG in the party was a far sight nicer than how the morning prior had gone--one of the little details I hid in the game and wasn't sure would even surface, was surfaced for all to see!
All would later also find out that evidently the egg- and bug- (and kiss-) themed moveset with which I'd equipped it was apparently kinda stacked? We all excitedly watched the first and second E4 attempts in bewilderment that you all had all made such short work of the game--I guess having folks filter in and out of the game in shifts in a 24-hour cycle would make short work of just about any RPG.
I was absolutely overjoyed when I saw the good job! appear on the screen. So much so that I almost felt bad pulling the plug on the game right afterward, but after this two-day-long rollercoaster on top of how much this had been dominating our lives before that (two days for y'all, the better part of a month for us), we needed a long break to regroup, rebuild, and come up with some new ideas before bringing it all back.
As far as a pilot run goes, I'd say it was a pretty solid success, and I seriously can't thank enough everyone who took part, shared it, gave us helpful feedback, gave us a reality check, or even just stared at it in awe and bewilderment. We hope you'll enjoy the next editions we've got in the works just as much.
Tune in next time for a walk through some of the details and polish you might've missed!
(Oh, and in case you missed it, we put a different game into the GBC for now, in case you wanna mess around with the tech!)
