SnepShark

Slowly making dogress

22, just graduated!

๐ŸŽฒ๐ŸŽฎ๐Ÿ’พฮ˜ฮ”โ€ฝ

Play my strange games over on snepshark.itch.io!

Header by LiveNLove7


Homepage + more links
snepshark.neocities.org/
Website League
pleasetf.me/SnepShark

vogon
@vogon

OpenAI recently launched an algorithm that purports to classify human- and AI-generated text, and for whatever reason they also decided to release actual precision/recall numbers which show how godawful it is. as OP points out, 30% of human-written text falls into the top 2 categories (as opposed to 54% of AI-generated text) but the fun thing is that in practice the numbers will be way worse than they even appear here for any application you're going to use it for, thanks to the magic of Bayes' Law.

if you have 1,000 students, 10 of whom commit AI-based plagiarism, 5 or 6 of them will be detected, compared to 297 honest students who are wrongfully accused (nearly 50 false positives for every true positive); if you decide to be more conservative and only use the top category as an indication of probable guilt, the numbers improve to a mere 3 plagiarists detected, against 89 wrongfully accused (it's correct almost 4% of the time!)


SnepShark
@SnepShark

With the detector spitting out a false negative ~40% of the time, it feels like it'd be very easy for the 10 cheaters to just repeatedly edit the essay and put it into OpenAI's tool until they get one of the unclear or unlikely results, making this even less effective in practice


You must log in to comment.

in reply to @vogon's post:

reminds me of the thing where a tool that works 99% of the time will still mess up for everyone hundredth person or so. these numbers are nightmares.

and then you'd have to expect that openAI wants to improve the "human-soundingness" of text generation, so isn't this tool just going to get worse???

I wonder how this compares to a human attempting to classify AI text. I feel like with some familiarization a human could do better, though probably not 100%.

Though honestly my main reaction is that AI writing is so bad it deserves a mediocre grade anyway. Instead of an F for cheating, enjoy your D+ for your paper that's full of factual inaccuracies and shows no understanding of the material deeper than being vaguely aware of its existence.