From the Atlantic piece about the OpenAI meltdown:
The more confident Sutskever grew about the power of OpenAI’s technology, the more he also allied himself with the existential-risk faction within the company. For a leadership offsite this year, according to two people familiar with the event, Sutskever commissioned a wooden effigy from a local artist that was intended to represent an “unaligned” AI—that is, one that does not meet a human’s objectives. He set it on fire to symbolize OpenAI’s commitment to its founding principles.
I NEED an interview with the local wood carver who got commissioned to make the unaligned AGI wicker man. I NEED this right now. Someone hunt down this guy and talk to him please.
Full piece here (web archive link), but it's overall not that interesting outside these details.
I see the current state of "AI" and think "well this is maybe a fun toy, but it's not really that impressive otherwise." but like the main AI tech at google or something will see the same thing and think we're a day away from the terminator. I'm pretty sure i'm not missing anything, but it's kinda odd for someone that's and expert to sorta get so caught up in the hype of AI
The people who finance AI research are venture capitalists (ie, idiots). These people will funnel money to AI companies that make extreme promises, especially if they can back those extreme promises with superficially impressive results (OpenAI fits this to a tee; big 'AGI' promise, superficially impressive results with generative networks).
Engineers at those companies then will tend to be people who either believe the extreme promises (ie, people who have AGI cult belief) or people who know they are heavily economically incentivized to go along with AGI cult belief.
For all that VCs claim to be 'data-driven' or 'rational', VC funding is a sieve that actively selects for irrational hype. If you're pursuing rapid 100x returns on your investment, and you're trying to achieve that by making lots and lots of bets, those bets on average have to be very irrational and unlikely to pan out.
We've seen this with other hype bubbles... there were presumably at one point people who had a realistic expectation of what VR tech could accomplish, but who got all the money and attention? It wasn't them, it was the people who were engaged in Snow Crash cult belief.