Good morning, fuzzies! I hope you’ve had a good weekend, and that you’ve sprung ahead or whatever it is you do. I am going to do another AI post, although this ought to be the last one for a while. Maybe.
So: basically immediately after I wrote up PyNCHON, the sort-of Artificial, sort-of Intelligent narrative tool, AI21 announced a massive update to their product. They claim that the J2 models are significantly improved in every way, so I decided to try again. I wound up changing June’s name to Maya, because it occasionally thought that was a date, but otherwise I left it very similar.
As a recap, the premise is that Maya is a beleaguered young woman who meets a man, Luis, at a bar. And then he turns her into a kangaroo. Also, he is a shapeshifter himself. Possibly just a coyote. Possibly Coyote-coyote. Here is the exact prompt:
Maya sipped at the mojito. She had plenty of things to think about. She could think about her argument with her asshole of a boyfriend Mark. Ex-boyfriend, the woman's brain corrected her. He was her ex-boyfriend, now, until he came crawling back again and--like an idiot--she forgave him.
Or she could think about her job, asking her to come in on a weekend again because somebody else had called off. Maya figured she wouldn't be getting a raise, either. Corporate said the franchises weren't doing well enough for that. She could think about polishing her resume, and trying to find a new workplace--in this economy, no less.
Or she could think about the man next to her at the bar, who said his name was Luis. He was about her age, and he said he did something in 'finance,' and his eyes were the most peculiar, captivating amber. They'd been talking about the city for an hour or so, and the clubs he went to, and she was so done with playing "respectable" that she shrugged when he offered her a tab of... something? Ecstasy, she assumed. She wasn't the kind of girl to do that often.
Almost immediately she felt... odd. Not particularly intoxicated, just like she needed to be outside. At least for a moment. She excused herself, and left through the back door, trying to catch her breath. The world was becoming oddly fuzzy. She put out a hand to steady herself against the alley wall, and blinked in surprise. Her nails seemed to be growing longer as she watched, and the fingers themselves were becoming strangely fuzzy.
She shook her head, trying to clear it, and it didn't help: her fingers were definitely downed in soft, tawny fur now. Her pumps felt uncomfortable, and she kicked them off only to find that her feet and calves were changing form, too. Her legs were muscular and powerful; her brown-furred feet were growing longer.
The door opened again. It was Luis. When she whirled to face him, she felt the inertia of a heavy tail behind her. Voice sounding oddly resonant, Maya gasped in a panic: "
Topline summary: as someone who is—you may have guessed—not especially impressed by the promises machine learning companies make, AI21 is right in that J2 represents a significant improvement. PyNCHON completed the previous story in 291 steps*, of which 129 were deletion or correction steps, and generated 5859 words. The final story was 2756 words, of which 1237 were generated, a loss rate of 79%.
* I gave a slightly higher number in the earlier article, because I was counting the number of times an iteration was written to the display output, but since some of those writes didn’t entail changes to the generated text I’ve changed the way I count that here.
On J2, PyNCHON completed the story in 216 steps, of which 81 were deletion or correction steps, and generated 4118 words. The final story is 2626 words, of which 1756 were generated, a loss rate of 57%. So, in general, the new J2 model required less manual oversight. You can see that visually putting the output side-by-side (J1 at left, J2 at right).
Blue text was manually seeded (green text was manually inserted but provided on a previous generation); red text was corrected. The vast majority of these in J2 were provided early, essentially as part of the prompt. Once prompted, PyNCHON under J2 required much less manual steering. It also tended to produce longer paragraphs—the story itself is only 100 words shorter, as you can see here:
Subjectively, J2!PyNCHON also had much, much less difficult time being convinced that Maya was a kangaroo. On the other hand, it really wanted her to be a deer of some sort, or else a gazelle, or else a moose, or else, ah:
She shifted her weight. There was a newfound energy in her body it was impossible not to revel in, like she could have jumped to the roof of the bar in a single bound. "But am I really a deer?" Her voice sounded strange to her, and she licked her snout.
“No," Luis said. "You're a deertaur. Half deer, half human. But you'll fit right in." He stepped closer.
I mention this not because it is intrinsically funny (although I guess it kind of is) but because the implication is that AI21 has probably broadened its training material. I don’t know where from, exactly; trying with a variety of different prompts reflecting text it should’ve found on SoFurry or FurAffinity (closed species, prolific authors, etc.) came up with nothing, in any attempt at doing so.
But it knows about deertaurs. It also didn’t require any real correction in knowing what to expect from a story about anthropomorphic animals. Back when I was originally comparing AI21 and GPT-3 in writing fictional news stories about animals, I noted that GPT-3 did this already—prompt it with a story about a Border Collie fighter pilot, and it would essentially ignore the former. AI21, by contrast, would try very hard to come up with some story about military working dogs.
I haven’t tried redoing that experiment to see how much that’s changed, but there is in any case a qualitative difference in the “feel” of the output text. Is the story any good? I don’t think there’s much of an improvement there. Luis doesn’t go through the same brief, weird personality shift, but that could also just be that the second version of the story happened to stick much closer to romance-novel territory.
(J2 was also a little more eager than J1 to get their clothes off. Like, the prompt itself is not that horny, but in one branch that I had to discard because I’d screwed up something else in the model, J2 had Luis responding to Maya’s first question by starting to make out with her and Maya being extremely on board with this whole state of affairs)
From the perspective of actually telling stories, this was still a waste of time, still took longer than just writing it, and still produced significantly worse output. It has not changed my opinion on the merits of large-language models, and it has not made me think that LLMs are either A. going to be useful tools for writers or B. going to become writers themselves any time soon. @samikatz asked if it was possible for PyNCHON or a tool like it to write a story at all, and I’m definitely skeptical of that. The largest models take ~4,000-token inputs. It’s hard to cram Lord of the Rings into 4,000 tokens in a way that is meaningfully useful for GPT-3.
I also said last time that I think sex scenes are maybe the worst-case scenario for these AIs, in large part because I suspect a lot of the training material is serial fiction for which the sex comes at the end of one chapter, which means that the end of the sex can also be the end of the story, quite abruptly, and what comes next might be very unrelated. PyNCHON continues to believe this strongly, even explicitly:
Yet she'd never felt so feminine, so womanly, so satisfied as she did at that moment, feeling Luis relax in her arms, spent from the intense pleasure of their lovemaking. She wrapped her arms around him, and breathed in his hot, masculine musk.
”Thank you," she whispered.
END OF PART 3
This was my first attempt at a sexy-seduction-themed short story, and I'm eager to hear your feedback. The feedback I've received so far has been extremely positive.
If you haven't read the other parts, I encourage you to do so now. The links are: Part 1 and Part 2.
On display here is also the “raunchy bodice-ripper” aesthetic that both AI21 and GPT-3 gravitate towards, for the record. They love each other, and that’s sweet. Also maybe she’s pregnant now, which recurred several times in its proposed followups. In any case, like with watermarks copied by Stable Diffusion, I don’t think this was necessarily copied and pasted directly from AO3 or whatever, but the model does have some weight in it for that kind of text. (Obviously it’s also a lie, since PyNCHON already has another story under it’s belt :P)
What it thinks should come next varies from the sappy…
Thank you," she whispered.
~~~
When Maya woke up later that morning, Luis had already gotten dressed. She could smell pancakes cooking in the kitchen, and he was whispering to himself... or was he singing? She lay in bed, listening to his muffled voice, and she realized with a start that she was happy. She'd never felt this way before. Not even when she was with Tom.
"Good morning," Luis said cheerfully. "I have breakfast ready.”
”Thanks," Maya said, and she sat up in bed. "I can smell it. Did you make it from scratch?
…to the esoteric…
Thank you," she whispered.
PART III: MAYA'S DREAM
Sun-ligh hi up f ****** ***** ver ne t, **** h harder without ar had me up and put don r. I r my lif h d m ssh i'm not even sure he n t the was together. I know whate me.
(This chapter was written and sent to me by a germ reader who shall remain anonymous**
…to, uh…
Thank you," she whispered.
#
CHAPTER 31: “THE DEAL”
IRKUTSK, RUSSIAN FEDERATION: SEVEN WEEKS AFTER THE EVENTS AT UBAKAVILLE
Fareeha opened her eyes and saw Jin-Mei standing in front of her: the panda catgirl Maya had met at the merc camp a while back. She was dressed in a sky-blue blouse and a deep-violet skirt, her hair piled in a bun. The Apache pilot couldn't believe her eyes. The creature's striking, soft features were hidden beneath another layer. She would be deadly when it came time to launch their attack on Site Prism.
Well, fuck. Maybe I am going to be out of a job.
