There's much worry about the likelihood (and growing prevalence already) of people using chatbots to generate text and citing it as fact (either naïvely misunderstanding what the program does or deceptively using it to produce or support bullshit).
I find it important to reframe this not as “AIs making stuff up” but as “people using text generators (or other tools marketed as ‘AI’) to help them make stuff up”.
And I just thought of an example that might help drive home what the real problem is:
Someone who writes up a bunch of bullshit, and then edits it into a fake screenshot of ChatGPT.
Because you know some folks will fall for it. The (fake) screenshot shows that ChatGPT said it, so it must be true… right?