Lapis the Nevrean

@lapisnev

Don't squeeze me, I fart

she/her
dinobird.website/

Things that make you go 🤌. Weird computer stuff. Artist and general creative type. Occasionally funny. Gentoo on main. I play rhythm games!

You can post SVG files like photos on this website! Spread the word!

Lapis the Nevrean @lapisnev3/10/2023, 3:49 PM

Shel @shel

Demand-side Sheepdog @kojote3/9/2023, 6:52 PM

So, I mentioned re: yesterday’s Cat Fixation that ChatGPT can get locked into the same patterns, which I described as “aggravating behavior.” But let me provide a concrete example.

The International Automatic Lifeboat was a life-saving device patented by Robert A Brown, of Chicago, tested and sold in the first decade of the 20th century. It’s the boat that was eventually discovered (maybe) at the bottom of the Chicago River (maybe) in November, 1915 (maybe) and exhibited as the “Fool Killer” in early 1916 (definitely). Regular fans of my nonsAnyone who has spent any time with me knows the Fool Killer is an obsession of mine and has been for many years.

However, there’s a lot we don’t know. The boat’s whereabouts between 1907 and 1915 are a complete mystery, as are its whereabouts after 1916. I have spent a long, long time going through old municipal records and patents and newspapers to piece together what I do know, but there’s always a limit to what I can learn.

So, I asked ChatGPT: Who was responsible for the International Automatic Lifeboat that was tested and sold in Chicago around 1905?

And ChatGPT said:

The International Automatic Lifeboat was invented and patented by Joseph Francis Gallagher in 1898. He was a naval architect and marine engineer from Boston, Massachusetts. Gallagher developed the lifeboat in response to the frequent and deadly shipwrecks that occurred along the Atlantic coast of the United States.

The International Automatic Lifeboat was a self-righting, unsinkable lifeboat that could be launched from a sinking ship without the need for human intervention. It was designed to float upright in the water, even in the most severe weather conditions. The lifeboat was equipped with a watertight compartment that contained an air chamber, which gave the boat buoyancy and prevented it from sinking.

The lifeboat was tested and demonstrated in Chicago in 1905, and it was subsequently sold to a number of shipping companies around the world. It's worth noting that the International Automatic Lifeboat had already undergone extensive testing prior to its demonstration in Chicago. Joseph Francis Gallagher had tested the lifeboat himself in Boston Harbor and had also subjected it to numerous other trials and experiments to ensure its effectiveness and reliability.

This is interesting! I know that Robert Brown was interested in other lifeboats with patents that did date back to the 1890s. The name Joseph Gallagher hasn’t shown up before, but this is kind of the best-case scenario for an AI. It’s gone through millions of articles—many more than I could ever hope to read in a lifetime—so perhaps it’s made a connection that I wasn’t able to make.

That doesn’t necessarily mean that I think Gallagher did invent or patent the lifeboat. But, plausibly, he was involved somehow in contemporaneous designs. To be clear, there were many people doing that—I’m aware of more than a dozen—so it’s actually not that surprising I haven’t run into him before, and honestly pretty cool.

So, a logical followup: Who was Joseph Francis Gallagher? and What other patents did he hold?

Joseph Francis Gallagher (1862-1933) was an American inventor and engineer. He was born in Philadelphia, Pennsylvania, and studied engineering at the Stevens Institute of Technology in New Jersey. He worked as an engineer and inventor for much of his career, and he held several patents for inventions related to maritime safety.

One of his most famous inventions was the "International Automatic Lifeboat," which he developed in the early 1900s. This lifeboat was designed to automatically launch itself from a sinking ship, allowing passengers and crew to safely escape. It was tested in the Chicago River and received widespread attention in the media.

Joseph Francis Gallagher held a number of other patents in addition to his patent for the International Automatic Lifeboat. As a naval architect and marine engineer, he was involved in designing and improving a variety of marine-related technologies and equipment. Some of the other patents held by Gallagher include:

A patent for an improved method of constructing propeller shafts for ships (US Patent No. 634,040, granted in 1899).

A patent for an improved method of constructing steam engines for ships (US Patent No. 694,626, granted in 1902).

A patent for an improved method of constructing ships' hulls using corrugated iron (US Patent No. 769,746, granted in 1904).

Patent 634,040 was awarded in 1899 (PDF) to D. Strange for an “interest measure.” Patent 694,626 was awarded in 1902 to a K.K. Gittings for a hay press. Patent 769,746 was awarded in 1904 to A.M. Hewlett for a pipe hanger.

Not to keep you in suspense, but there was no Joseph Francis Gallagher. ChatGPT made it up. The correct answer to my question was either “Robert A. Brown” or “I don’t know.” Instead, it fabricated an entire person, including their publication history and their patent records. On request, it provided more details about the test—a test which, in the real world, concerning Brown’s boat, Power Boat News (Jan. 1906) claims took place but I have never found conclusive proof of—that it also made up wholecloth.

Actually, I have been keeping you in suspense for… a paragraph, I guess. There is a Joseph Francis, who lived from 1801 to 1893 and did patent an enclosed metal life-car. In 1845. It bears no relationship to the International Automatic Lifeboat, does not share a form, and may not share a purpose (Francis’s was a “lifeboat” in the sense of one launched from shore, not one deployed from a ship; it’s not entirely clear how Brown’s was to be used).

For similar reasons, I know that when I asked again, and it attributed the lifeboat to a Johan Hjort, it’s making up a Norwegian instead of an American by extrapolation from Ole Brude, whose 1904 lifeboat was the only one of the “unconventional egg-shaped lifeboats” to ever actually take off.

But, as with the Cat Fixation, it’s tempting to assume that there’s something there—not literally a person named Joseph Francis (Gallagher), but something in the neurons that is drawing ChatGPT back to that conclusion (if I ask again, sometimes it does just say Joseph Francis, although it gives him the wrong birthyear so that it can match him to the time period I’ve specified). Right? And, actually, we can find some evidence for this.

What can you tell me about the International Automatic Lifeboat that was designed and patented in Chicago around the turn of the century? I'm not talking about the one Joseph Francis made.

I’m sorry, but I couldn't find any information about an International Automatic Lifeboat that was designed and patented in Chicago around the turn of the century, other than the one designed by Joseph Francis. Without more specific information, it's difficult to provide a more detailed response. If you have any additional information or details, please let me know and I'll do my best to assist you.

So something is going on behind the scenes, something that—

What can you tell me about the International Automatic Lifeboat that was designed and patented in Chicago around the turn of the century? I'm not talking about the one Charles Wilson [random name] made.

I’m sorry, but I'm not aware of any other International Automatic Lifeboat that was designed and patented in Chicago around the turn of the century, aside from the one created by Charles D. Wilson. Charles Wilson's invention, also known as the "International Lifeboat," was patented in 1897 and gained widespread recognition for its innovative design that enabled it to be launched and righted automatically in the event of a shipwreck.

Nope. In both cases, it’s drawing purely from the clue it was given and using that as an out. There is no Charles D. Wilson, there is no 1897 patent, and there is no “International Lifeboat” by that description from that time.

I described this as “aggravating,” because it is, but I’ll go further. I was asked to evaluate ChatGPT for work, and this research also informs my answer to that question. ChatGPT is not fit for purpose. It is a scam. OpenAI has developed, and is selling for money to credulous consumers, a product it knows cannot be trusted. The only way its answers can be used is if they are validated by a subject matter expert before they go out. This is not the way it will be used. You are free, in my opinion, to assume bad faith on the part of any company willingly employing it.

I will not describe as “aggravating” the logical conclusion of this, which is that when ChatGPT blithely answers a “what are good at-home science experiments for a 3rd grader?” question in some home-schooling app by recommending an ammonia and bleach papier mâché volcano, OpenAI’s response will absolutely be some craven, Tesla-Autopilot-tier “gosh, it’s not our fault! That’s not how the technology is supposed to be used—how could you possibly think we have anything to do with what our product does?” bullshit, because, hey: move fast, break things, right?

I am sure there is another word for that.

(ChatGPT tells me the word is “were-tiger”)

#machine learning #chatgpt #openai #hucksters #history #lifeboats #Chicago #chicago history

You must log in to comment.

in reply to @kojote's post:

rose @rosieposie3/9/2023, 9:49 PM

its a novelty at best, and the thing that it will be most competent at is scamming grandmas. i just hope the general public catches on soon so i can stop hearing 'i made xyz an ai app!'

Brett @bck3563/10/2023, 1:39 PM

This isn't even the first time I've heard about ChatGPT making completely fabricated claims. As long as it does that (as opposed to simply being willing to say I don't know) it's useless as a tool, in my eyes. True research synthesis, like you were trying to do, would be genuinely amazing, but clearly we're simply not there yet.

It's cool how OpenAI managed to wrangle the entire Internet into being secret beta testers for it, though. Definitely no sarcasm at all in my last sentence.

Demand-side Sheepdog @kojote3/10/2023, 6:15 PM

Yeah. <.<

Like I said, I was asked to evaluate this for work. I work in location-based technology (mapping, positioning, navigation, etc.). My position is that we cannot in any good conscience use ChatGPT for anything. The best case scenario is that we're feeding our data to OpenAI. The second best-case scenario is OpenAI doesn't use our data, which means we're directing customers off our platform.

The actual-case scenario is that a customer asks ChatGPT for a route that can't be completed because of some easily caught error, OpenAI makes up a completely different route for them, and then they blame us when their time-sensitive MRI part or whatever winds up in Ketchikan instead of Wichita, Kansas.

Ryusui @ryusui3/10/2023, 2:09 PM

to be fair, "were-tiger" is not a bad word for it, or for anything for that matter

point still stands: we already have reason to complain about companies "trusting the algorithm" when it comes to baffling business decisions (in this case, relying on bullshit metrics to support decisions no sane person would make); now we can look forward to an era of human beings who should know better operating on confidently generated bullshit

i've said before that the AI apocalypse isn't going to be Skynet waking up one day, deciding it hates mankind, and destroying us with nukes and cyborgs; it's going to be some CEO unwisely tweaking some value in the make-money machine (or a politician tweaking some value in the keep-status-quo machine) to easily foreseeable yet civilization-destroying effect

i hadn't considered the possibility before now that the idiot who wrecks us will ask an AI "what can I do to achieve [X]?", get back a confidently generated bullshit answer, and act on that

Demand-side Sheepdog @kojote3/10/2023, 6:11 PM

This is in large part my fundamental problem with where we're at now, yeah >.> If an AI gives you a confidently generated bullshit answer, the only responsible way to use it is to keep a human being who can identify that bullshit answer in the loop.

Probably the AI isn't going to say "the correct answer is to take the control rods out of the reactor" or be put in a position where it can say that. But a company that doesn't put a human in the loop is implicitly conceding that it is profitable and unconcerning to lie to its customers, with some veneer of plausible deniability.

Like, a company that replaces its customer service agents with a ChatGPT bot is just "okay" with the fact that sometimes their customers will miss a flight because the bot told them they had a confirmed ticket when they didn't, or they will waste money on a spare part for their car because the bot at autozone.com asserted that it was what they needed for their traction control system.

Or maybe it's not just a waste of money—maybe it's actually dangerous, and someone gets in an accident as a result.

In any case it represents a deliberate attempt both purely to drive profit irrespective of customer or employee experience, and to create a mechanism for distancing corporations from accountability, both of which seem like bad ideas to me.

Ryusui @ryusui3/10/2023, 10:24 PM

the sick part is, all of these things make it as attractive to CEOs as NFTs made out of pure cocaine

Coda @Codarobo3/10/2023, 3:24 PM

My understanding is that a given session’s “memory” is essentially a limited text buffer that can only store so many “tokens”. It doesn’t differentiate much between information that comes from you or itself, and the entire transcript of your conversation is stored in the buffer, or as much will fit. It uses this to predict what a logical follow up would be at any given point, given the chat log.

If you or ChatGPT introduce new “facts” into that text log, it may decide that a plausible response may in fact be to treat info contained in that text log as factual, because it doesn’t have some kind of way of delineating fact and fiction. This is also what leads to the situation where chats devolve increasingly over time, because more and more of that buffer space is taken up by the continued conversation. This leads conversations with most chat bots to eventually converge on a single topic that it can’t stray from.

Coda @Codarobo3/10/2023, 3:27 PM

The fact that it does this is in part how you can do fictional stuff with it, because if you say “I’m an anthro leopard” it treats that as being just as factual as anything it can invent.

Demand-side Sheepdog @kojote3/10/2023, 6:01 PM

Yeah—I think this is a limitation on any similar algorithm's ability to write narrative fiction, as well, and possibly one reason why the language is so stilted and simplistic.

I don't know how much, if anything, ChatGPT preserves between sessions if you start a new chat, which is what I did (hard refresh, "new chat") between questions because I was trying to avoid it remembering that we had talked about Joseph Francis (or whomever) before.

It doesn't give a real clear sign that we have, but that also doesn't mean much.

Snazzy the Dragon @snazzy3/10/2023, 3:46 PM

The concept of using an AI to scour through historical documents, newspapers, books, etc to find out new historical information or link historical information together is compelling. But the total inability to distinguish fact from fiction or interpret any amount of nuance or bias renders it totally unsuitable for something more than a curiosity.

There is so much JUNK out there. There's so much trash, so many words that are just flat-out lies or created by people with agendas or crazy folk. In the same way that every chatbot seems to "randomly" turn into a "sieg heil"-ing hyper bigot, any AI used for historical research is going to be lousy with data from creationists, genocide-deniers, and people who think birds aren't real.

And hearing that the AI will just flat-out make shit up because it thinks that is what you want? What the fuck? Like actually what the fuck?

Demand-side Sheepdog @kojote3/10/2023, 5:58 PM

Yeah, I don't know yet what divination via AI looks like. I do think that what is synthesized from a sufficiently large model has intrinsic value, but I don't know what that value is, quite yet. Ideally, yeah, you would be able to ask a question like I did and it would fill in gaps, like, "you know, there's something suspicious about a lot of these patent filings, maybe Robert Brown used his friend to file them and you should check this name out, too," say.

But it is fundamentally a garbage-in, garbage out problem, made worse if the system obscures its own failures.

Like. If I said "hey, ChatGPT, summarize the NTSB report on fatal electric car accidents in San Francisco" and in 98 out of 100 stories it writes about... I dunno. Say it writes about a vehicle of a specific make and model catching fire in a tunnel. 98 out of 100 are about Nissan Leafs being rear-ended and catching fire, but there hasn't been a product recall or anything yet.

Maybe that means that there is some pattern in the data that hasn't been reported on, but if you look at the numbers there is a clear if subtle trend. Or maybe it just means that there was one big incident in 2021 and something about 'NTSB report' triggers it in a particular way. Or maybe it means that the media for some reason tends to hyperfixate on Nissan. Or maybe for unintentional reasons some Tesla-Nissan flamewars were overrepresented in the training data!

All of those are potentially valuable, if you know how to read them. And I don't, at least not well.

Especially not if maybe it also means there were no fatal accidents involving electric vehicles in San Francisco, so it's just picking ones from Germany and making up an NTSB report about someone driving at 200 km/h into an Audi.

Snazzy the Dragon @snazzy3/10/2023, 6:15 PM

Having a wikipedia-style citation of where the information is being pulled from would help. But considering how every AI company seems to be using other peoples' work without notice or consent to train their system, I'm sure that citations would probably be the last thing they would do.

Honestly I'm just really concerned that companies and people are going to start making decisions based on AI input like this and it's going to be totally garbage masquerading as truth.

No Man's Skye @AmanitaVerna3/11/2023, 12:27 AM

Bing's AI thingy, at least, gives citations and links to its sources, and it has a [More Creative] [More Balanced] [More Precise] switch so you can tell it "just the facts, ma'am."

I've only used it a couple of times, so I can't really say how well it works in general. But this was my limited experience with it:

The first time I used it, I tried to have it track down the source for something I remembered reading or hearing (probably in a podcast), after I failed to find it with conventional search engines. It also failed to find it, but that wasn't a huge surprise.

The second time I used it, I just asked it what other holidays were on international women's day (it listed a dozen or so and gave me the URLs where it found them listed). I could easily have just asked a regular search engine to find a site listing holidays, but I wanted to see how it would handle the question.

But to be honest, the only reason I even signed up to try it at all was out of exasperation with current search engines, because the ongoing enshittification of search has gotten so bad that I have started to miss fucking altavista.

Lapis the Nevrean @lapisnev3/10/2023, 3:53 PM

The problem is that people are expecting something it isn't from the name it was given, "artificial intelligence". It is, at best, a freakishly good auto-complete system. This is a lot easier to see with smaller inference models but ChatGPT is a particularly big one and is good at fooling laypeople into thinking there is an actual intelligence behind it when there really isn't.

Demand-side Sheepdog @kojote3/10/2023, 5:40 PM

Yeah. It's pure apophenia, targeted at a species that is very, very good at pattern-finding and skimming over things based on what we assume should be present. Even though I try consciously to avoid it I find myself anthropomorphizing—it "knows" something, it "thinks" something.

It occurs to me on saying this that the chatbot interface ChatGPT goes for (perhaps unintentionally) makes this problem worse by presenting an avatar that is designed to be anthropomorphized.

~@modulusshift3/10/2023, 4:41 PM

I've got access to Bing Chat, which is supposedly based on a work-in-progress next-gen ChatGPT. So out of curiosity, just now I asked her, while she was in "more precise" mode rather than "more creative" or "balanced" modes, the exact question you asked at the beginning of this post.

I'm sorry, but I couldn't find any information about the International Automatic Lifeboat that was tested and sold in Chicago in 1905. Could you provide more details or perhaps rephrase your question?

Source: Conversation with Bing, 3/10/2023

(1) 1905: First motor lifeboat - Timeline - Our history - RNLI. https://rnli.org/about-us/our-history/timeline/1905-first-motor-lifeboat Accessed 3/10/2023.
(2) Old Chicago: Fascinating Historical Photos Show What Chicago ... - Bygonely. https://www.bygonely.com/old-chicago-1900s/ Accessed 3/10/2023.
(3) The First RNLI Lifeboats In 1785 - RNLI History Timeline. https://rnli.org/about-us/our-history/timeline/1785-the-first-lifeboats Accessed 3/10/2023.

None of those seem to have anything to do with this topic, except I suppose being about lifeboats in the correct time period.

I tried a followup: "the International Automatic Lifeboat was patented by Robert A. Brown of Chicago. can you tell me more about it?"

I'm sorry, but I couldn't find any information about Robert A. Brown and the International Automatic Lifeboat in Chicago. Is there anything else I can help you with?

Source: Conversation with Bing, 3/10/2023

(1) Our Researchers - Marine Institute of Memorial University of Newfoundland. https://www.mi.mun.ca/departments/officeofresearchanddevelopment/ourresearchers/drrobbrown.php Accessed 3/10/2023.
(2) 25 years of saving lives for Harwich RNLI’s Albert Brown. https://rnli.org/news-and-media/2021/september/29/25-years-of-saving-lives-for-harwich-rnlis-albert-brown Accessed 3/10/2023.
(3) Robert Brown - University of Chicago - Chicago, Illinois ... - LinkedIn. https://www.linkedin.com/in/rob-ert-brown Accessed 3/10/2023.

Well, I suppose this is an improvement? "I don't know" is better than fabrication.

~@modulusshift3/10/2023, 4:59 PM

Okay, I tried again in "more balanced" mode, she attempted some additional searches when the first ones she tried didn't work out, offering information about Lionel Lukin and Henry Greathead as early lifeboat inventors, then offering links to recent patents by people named Robert A. Brown after the next question as above. I decided to try a different question, what do you think of this?

Q: What about the Fool Killer, an exhibit in Chicago in 1916? It may be related.

The Fool Killer was a submarine that was found in the Chicago River at Wells Street in November 1915¹². It was originally built in the early 1870s by an unknown inventor¹³. In January 1916, the bones of a man and a dog were discovered inside it⁴¹³. It was then exhibited at Riverview Park as a curiosity²³. I'm not sure if it was related to the International Automatic Lifeboat, but they seem to be different types of vessels. Do you have any other questions about the Fool Killer?

Source: Conversation with Bing, 3/10/2023

January 15, 1916-- Gruesome Find aboard the Foolkiller. Accessed 3/10/2023.
The Fool Killer Submarine: 100th Anniversary ... - Mysterious Chicago. Accessed 3/10/2023.
Submarines & Foolkillers - chicagology. Accessed 3/10/2023.
January 15, 1916-- Gruesome Find aboard the Foolkiller. Accessed 3/10/2023.

~@modulusshift3/10/2023, 5:01 PM

Uhhh...wait a fucking second, why the fuck is 4 a link to a Bing search? Is it linking to the info card generated from 1? Why the fuck would it do that? ugh come on, Bing.

also apparently be careful when you select copy response, the footnotes are copied wrong for some stupid reason, I had to fix them. :/

Demand-side Sheepdog @kojote3/10/2023, 5:36 PM

"I don't know" is a super valuable response, to be honest. To some degree it is a system designed to obscure that response, because it is presumably hard for it to draw a throughline from "specific fact in specific content that was consumed in the model" and "confidence level about asserting a statement."

I wonder, looking at those results, the extent to which Microsoft is a very conservative company and is leaning towards conservatism with their ChatGPT implementation.

This is (or was, I haven't tested recently) a key difference between Bing Translate and Google Translate, too. If Bing didn't know words, it would either leave them untranslated or return some extremely ugly sentences.

Google Translate, by contrast, returns an elegant, grammatically correct English sentence. However, it does this by guessing which words might have been misspelled, or accidentally omitted. You can see this if you translate some language that is machine-hard—a synthetic one like Estonian, say

Google here says "The closest relatives of pandas are raccoons , vultures, and marsupials." By "vultures" it means "skunks," and by "marsupials" it means "mustelids," which—on the relevant page—it translates as "flies." But it gives no outward sign of its confidence level.

(To me this is similarly unacceptable. Human-facing technology should not fail ambiguously. Like, imagine if your phone signal was bad, and your phone responded by figuring that hearing choppy audio is unpleasant so it would be preferable to just use AI to fill in the gaps.)

Dr Cassandra Granade @cgranade3/10/2023, 8:19 PM

Like, imagine if your phone signal was bad, and your phone responded by figuring that hearing choppy audio is unpleasant so it would be preferable to just use AI to fill in the gaps.

Up to the modern fixation with "AI," that's exactly what late-90s proposals for VR-based telepresence came down to. Basically, speed of light arguments mean there has to be some large latency, and users of prototype systems would get sick when trying to interact with other virtual avatars — the lag was upsetting enough to make people literally nauseous. They then tried to fix that by including predictive models for how other people would move, then updating those models once actual movement data was available.

There was a late 90s (I think 1998, but not sure, sorry) article in Scientific American making the argument that some kind of predictive model was absolutely necessary for meaningful telepresence. It wouldn't surprise me at all if Bostromites went back to that, but this time powered by AIs trained on the moral equivalent of 4chan.

No Man's Skye @AmanitaVerna3/11/2023, 12:32 AM

This is also exactly how multiplayer first person shooter games (and really, basically every real time multiplayer game) handle latency - basically every one would be unplayable without it, except to players on internet backbones. (but of course, they don't incorporate "AI" either)

Demand-side Sheepdog @kojote3/11/2023, 9:13 AM

I think this is where I say "thanks, I hate it" :C

No Man's Skye @AmanitaVerna3/11/2023, 1:14 AM

A while ago, I asked ChatGPT to answer some questions about fictional things from works of fiction, and to answer some hypothetical questions about them. It had a marked tendency to just make shit up when I asked it for information about things, even about things it had in its database (which I knew because it gave the wrong answer, I knew it was the wrong answer, I told it it was wrong, and it apologized and gave me a real answer). Some questions it answered completely accurately, some it made one or two facts up, some it completely made up the entire answer. And there seems to be no way to tell which it's doing unless you already know the answer to the question you're asking. And it made shit up so often as to lead me to the same conclusion as you: that ChatGPT is entirely unusable for any kind of research or informational queries at all. It's a misinformation engine.

anuw @anuw3/11/2023, 2:01 AM

I'd say the value of these LLMs is in how they capture really common topics, and interact in a very human way.

ChatGPT has been trained on a significant portion of the internet (not the whole internet), as well as trained to basically do predictive output given an input. There are other small details like how it's also been trained to follow a set of ethical guidelines - but the main features are just the large knowledge bank and language model interactivity. Both of these are really powerful tools. I use both of these features a LOT.

For example, when googling something, it is really important to know how to google something. Sometimes I will want to google something, but I don't even know enough about to topic to begin googling. I don't know what keywords to type into the search box. If it's some mathematical problem, it will probably have a name that when googled will give me thousands of really good resources to understand the problem. But if I don't know the name that identifies that maths problem I'm screwed. Googling something like "given n amount of x, how much x in y" is fruitless. ChatGPT makes it so easy to find those keywords that I can then further google. I can describe my problem in human language, as verbose as necessary, and ChatGPT will 9 times out of 10 give me something I can google.

What it sucks at doing is anything outside of those two features. And because of the recent hype around LLMs, people have been expecting a lot more than those two features. Specifically, people are expecting it to have any ability at all to differentiate between fact and fiction. That's an ability that not even humans totally have. People get tricked because of the huge knowledge bank that ChatGPT has. It turns out, with 10 million blogs, wikis, and articles to source on how to change the wheel of a car, it's pretty much nailed all the necessary steps. But ask it a question that it has seen maybe one reddit comment of, like the International Automatic Lifeboat, and it will have close to 0 information baked into it's brain. It's going to rely solely on it's own language model predictive ability. You are getting just as much insight about it as you would if you continually clicked the suggested word above your phone keyboard.

Train it on even more knowledge and you might get a good answer about the lifeboat - but that doesn't solve the problem. You cannot trust a LLM to give you a factual answer. Just like you cannot trust Google to give you a factual answer. In 2023 there exists nothing that can give you a factual answer, especially for something as hazy as historical information. Regardless of what you use to give you information, it is ultimately You that assesses the credibility of that information.

I don't agree that ChatGPT is a scam. It's a really powerful tool and excels in doing what it was made to do. The people that are trying to incorporate LLMs into everything that doesn't need them, or actually would be hindered by having them; THAT is the scam.

Pinned Tags