- Dev insisting it would have been infeasible to remake their game by actually paying people to do work so they used AI instead
- Polygon completely uncritically talking about how great it looks and sounds
abso fucking lutely not
My guess is that they did something like my 2023 paper with the UBC crew:
https://www.cs.ubc.ca/labs/imager/tr/2022/SubpixelDeblurring/
We looked at upsampling old pixel art in the context of de-blurring it. Our approach combines an initial upsampling (d) based on pix2pix (see: https://phillipi.github.io/pix2pix/ if you want the gritty details); a classical, non-AI algorithm then cleans the results (e). The paper claims "a few hundred hand-drawn sprites" and remastered versions created by rescanning the original assets were used. We got good results with 145 pieces of clip-art in the training set; 300 is more than enough. No extra random data sets from the Internet required.
"AI", to my mind, encompasses a lot of things. In the case of Broken Sword, and my paper, it's really not "AI" in the sense of an LLM trained on scraping the internet; instead, it should be read as "a very complex non-linear upsampling and convolution kernel filter" that is specifically optimized based on the target domain you're trying to work from. In that sense, it's much more about numerical optimization. It's not "shove everything into a tuned stable diffusion model and then fine tune it on your 300 inputs/outputs", mainly because despite what the stable diffusion model people want you to believe, this doesn't actually work.
So: what are the ethics of this?
No random data is being scraped from the Internet (presumably; I can only speak for myself here.) I am not a fan of papers or academic work that do this; I have loudly complained about a number of "data set" papers that just scrape vast quantities of models and publish the lot in a package with the idea that this is now a 'cleansed' training set. However, let us assume this is not the case here, because technically speaking I don't think it is. I may be wrong, in which case all bets are off: data scraping is contemptible. But I don't think I'm wrong.
Should artists have been hired, and were they put out of business by a machine? Maybe. I think we lost this fight some time in the Victorian era, and energy spent relitigating it is better spent overthrowing capitalism. Similarly, presumably the Broken Sword work was done either under an employment contract or as work-for-hire, at which point the original animator has no legal rights to the work, in the sense of copyright.
The question, therefore, is whether or not the animator has moral rights, and there are two standards for this in Canadian law. First, the integrity rights portion of moral rights mean that the author has the ability to preserve the intended meaning of the work and protect it from destruction or defamation. Second, and importantly, the alteration of a work in good faith to preserve its intended meaning or nature is not considered an infringement of moral rights under Canadian copyright law. There is a good summary of this over at the Heer Law blog: https://www.heerlaw.com/moral-rights-copyright-law - and I quote:
Canadian courts first examined the issue of moral rights infringement in Snow v. Eaton Centre Ltd., (1982) 70 C.P.R. (2d) 105 (Ont. H.C.J.). In this case, the Toronto Eaton Centre was found liable for infringing the plaintiff’s moral rights for putting festive ribbons on the plaintiff’s sculpture depicting sixty geese. The plaintiff argued that this modification was prejudicial to his reputation and compared it to “putting a wristwatch on Michelangelo’s David or earrings on the Venus de Milo.” The Ontario High Court of Justice—weighing the plaintiff’s opinion together with the opinions of other artists who were knowledgeable in the field—found that the plaintiff’s concern for his reputation was reasonable. The Court granted an injunction to compel the removal of the ribbons from the necks of the geese.
In 2003, the Ontario Superior Court of Justice considered the topic of moral rights again in a case between a photographer and a golf club, Ritchie v. Sawmill Creek Golf & Country Club Ltd. The photographer alleged his moral rights in his photographs displayed on the golf club’s website were infringed by the photos being enlarged beyond the size in which he provided them, and by his name having been removed from them. The court found the first argument unconvincing as the photos were not so enlarged as to be of markedly reduced quality and damaging to the plaintiff’s honor and reputation. As for the photographer’s moral rights of association “where reasonable in the circumstances” with his work, the court considered that, following a complaint to the RCMP that the golf club had infringed his copyright, it was no longer reasonable for the plaintiff to believe the club would continue to associate him with the photographs on their website.
So the question is: is this geese, or is this photographic enlargement?
I think ultimately this is going to be a question for the courts, which will be really interesting once we get there. Consider, if you will: we threw a mode into Dungeons of Dredmor which lets you upsample the original (drawn originally at far too small a resolution, because when we started game development I had a 1024x768 CRT monitor) sprites. One of those modes runs them through HQX, an upsampling filter developed by Max Stepin; the other mode is nearest-neighbour. Does HQX destroy the artistic integrity of the work? And assuming your ML model is entirely sourced from works you own or created, it is essentially applying a set of fixed transforms or rules to the input pixel art - is that any different than HQX, and if so how?
(I don't claim to have the right answers to this, by the way, and I do agree with Aura's original point that Polygon should not report this sort of thing uncritically. Poor media reporting around AI means that people can get away with "bad practices". But that doesn't mean there aren't "good practices", and achieving some sort of sensible ML praxis means we should ideate what the "good practices" are.)
yeah, it's a problem that the companies who are doing the most morally repugnant things - specifically scraping content they have no rights to, in order to train & repackage it as a product they can sell, are the loudest voices in the room (naturally, because they have something to sell you). Or that companies unnecessarily bolting AI onto products like search are earning everyone's hatred when the results they generate tell people how to make bleach muffins (which, of course they do, they're just generative language models, they don't know anything)
but like, we're training ML models to identify meteors in video imagery, right? it's often more consistent than a human at seeing things right down in the noise, it's faster than computationally expensive operations that can't be offloaded to a GPU, like hough transforms, and more consistent (if trained correctly) when the actual imagery doesn't quite match your target shape - eg. when there is significant lens distortion or when there are mid-span flares bright enough to saturate the detector, etc.
it's a great use of the technology because it's (nominally, not yet entirely proven) very accurate, less computationally heavy than other comparable approaches, and it's replacing work that no human would actually want to do, anyway1 - nobody wants to watch eight hours of video footage every night where nothing in particular happens, with enough constant vigilance to notice a cluster of pixels in a line light up by 15 brightness units for half a second
(now multiply that by networks of 20+ cameras)
i have a friend who is, rightly, i think, concerned that the frauds and thieves masquerading as AI & Machine Learning experts these days, with the loudest voices and something to sell (again, your stable diffusions, your generative large language models - the shit everyone hates) are going to turn public discourse against the entire field, if they haven't already. including the genuinely harmless-to-useful applications that don't just hoover up the entire creative output of humanity in order to repackage and sell it back to us. That they're going to usher in another decade of so-called AI winter and set the field back, and we'll lose the actually useful and positive applications like the meteor detector, or systems that can automatically transcribe speech-to-text for deaf people, and so on.
in conclusion, fuck capitalism
-
it's not like we're doing this manually now anyway, we have simple, fast, rise-in-brightness detectors, and slower hough transform detectors, matched filter detectors... but at some point in the past this used to be manual science work that got automated away by even these simpler algorithms
