Has anyone considered that maybe generative AI is not necessarily super great?
On the inescapable deluge of meaningless slop.
Now available in an audio version read by me for paid subscribers.
Yesterday I was trying out a new email marketing service, Kit (formerly ConvertKit because tech companies love to rebrand). I was trying to set it up to do something very simple, send a custom welcome email when someone fills in a particular email signup form.
Just like, super basic email marketing stuff.
Except, it was not obvious how to do this in Kit. I could see a way, but it was slightly complicated, and involved using a paid feature, and I was trying to stick to the free plan.
However, this is not the story about how Kit offers an initially attractive free plan but sneakily hides an indispensable feature in their paid plan. (But to be clear, they do do this).
This is a story about what happened when I asked their support chat for help with the thing I was trying to do. I opened up the little help window and was introduced to "Sam", an AI chatbot that was their first line of support.
"Sam" was cheerful and friendly. "She" responded appropriately1 to my questions and had a seemingly encyclopedic knowledge of the features of the software.
I asked "Sam" how to do the thing I was trying to do and "she" answered with a lie. A complete fabrication. "She" told me how to do it in a way that seemed like it should work and would make sense to someone familiar with this category of software, but "she" invented a feature which just doesn't exist. So when I went to find the feature in question, it wasn't there.
I came back and said as much and "she" said, essentially "Oh, my bad. You're right, we don't have that feature, you have to do it this other way"
The other way was the way I had been trying to avoid.
So, to summarize: Within my first hour of using a piece of software I might very well end up spending hundreds, if not thousands, of dollars on over the lifetime of my business, their customer support lied to me, making up a feature that they just don't offer. Were I less comfortable with technology, I might have wasted more time looking for this non-existent feature, and in any case the user experience is astonishingly bad.
AI is very bad UX disguised as very good UX
UX (user experience) is the name tech people give for, shockingly, the experience a person has using a particular piece of technology. It's also the name of the field for working on these experiences.
Generative AI (ChatGPT and the like) seem like Gods gift to UX. You no longer have to figure anything out! You just ask chat! You only need to understand how to type in a prompt, hit send, and read a reply, and everything else can be derived from this very simple knowledge. This is part of why people are excited about GenAI. But underneath this veneer of good UX are countless examples like mine above, where the "good UX" actually leads you down an ultimately fruitless and frustrating path.
As another example, to go back to my exploration with Kit. I started looking into Kit because I was looking for a particular feature that I was finding it hard to search for (I won't bore you with the details, but unrelated to the functionality in the first example). So I described the feature to ChatGPT, which confidently told me that Kit did exactly what I needed. Turns out this was also a lie. Kit does not have the functionality I was looking for. ChatGPT just made this up. Yes, it serves me right for asking, but that's the problem! I was seduced by the appearance of good UX. It seemed much easier than googling a list of options, and trying to read between the lines of their vague, self-promoting feature lists, or trawling reddit threads to figure out which software did what I needed.
I could skip all that! Instead I could just describe the feature I needed, and this thing would respond with great conviction that Kit has exactly the feature I'm looking for. This is exactly what happened, and it was just not true. So, two different AI bots told me two different lies, each about a non-existent feature of this software.
AI offers idiot certainty
Play with an AI chatbot for 10 minutes and, if you're paying attention, you'll discover something. Whatever the AI chatbot says, it says it with great and cheerful certainty. And it’s certainty is unrelated to whether the thing it's saying is true. It's just unrelated. In life, when we encounter someone who is certain about something, that fact has some relationship with whether they're right. Yes, certainty is far from a guarantee that someone knows what they're talking about2, but it's generally a strong indicator.
So, "Someone is telling me something with certainty, that means it's probably true" is a heuristic which, day to day, actually serves us quite well. But of course, in the case of chatbots, there is no sense of truth. There's no model of the world about which a notion of truth could even make sense. What chatbots do is say plausible things. They say things that are structured in the same way as things other people have said. That's it. Their certainty is immaterial.
But it tricks us into believing it. I'm pretty skeptical about AI in general, I'm on the look out for these mistakes, and it still led me on a two layer wild goose chase.
I think the mechanism for why it speaks with such empty certainty is related to a wild discovery in neuroscience made in the middle of the 20th century. It's interesting stuff, and I go into the full story in this recent podcast episode
An inescapable deluge of meaningless slop.
When Midjourney and other AI image generators dropped a couple of years ago, I remember feeling concerned that the nauseating visual noise of AI was going to start polluting our visual world. That has come true to some extent (I recently saw a flyer in coffee shop that was clearly AI generated), though happily so far it's not as severe as it could have been. But I think a much worse thing is happening instead.
Because now, without your intervention, the top result of every major search engine is generated by AI. And increasingly, the first wave of interaction you'll have with service providers of all kinds will be with an AI bot. And these things create epistemic noise.
Steve Bannon famously advocated a policy strategy of "Flooding the zone with shit," overwhelming the sense making organs of the polity with so much noise, so much information, true, false, insane, reasonable, in a fire hose of meaning, that those sense making organs shut down.
Well, now that's happening automatically, and by well-intending tech companies of all scales. We now have a mechanical means of flooding the zone with shit, and Google and friends are happily deploying it en masse.
This is the thing that so disturbed me about my experience with "Sam" the chatbot: How little truth matters to Kit here. They must know that this machine is giving out false information; ALL AI CHATBOTS DO. It's unavoidable. It's intrinsic to the technology that it makes things up. They are making-things-up machines.
The CTO or whoever made the call to use the chatbot knew that it would spout lies and they just said "It doesn't matter".
Google knows that their AI results are often inaccurate. There's small print at the bottom of the AI generated search results that says "Generative AI is experimental". I seem to remember a month or two ago it said something more explicit like "AI results might be incorrect, always double check" or similar, but I might be hallucinating that one myself. Anyway, they know. They are completely unconcerned that the very tippety top of their flagship product, the intensely-coveted-for-decades TOP GOOGLE SEARCH RESULT is now a made up piece of slop from a making-things-up machine. Google became world dominant by caring about the quality of the search results you got. That's why we all use Google. And now they've just thrown that out. They've just decided not to care anymore about quality. As long as they can say something plausible in response to your search and, crucially, keep you on their site, that's all that matters to Google now.
This is an alarming deepening in the epistemic crisis that's been developing since around 2016. And this is the benign neglect of techno-feudalists we're talking about. Let's not even start thinking about deliberate misinformation campaigns. We've fully entered the era of mechanized information warfare.
So, this is all pretty depressing huh? Is there good news? Is there a way out? A way to fight it?
Yes! There is. There's a whole enormous, extraordinary, meaningful life that you can live outside the purview of this technology (and it's seductive neighbors, social media). There are, even as we speak, trees, just sitting there waiting to be enjoyed.
There are more books to be read than can be read in a lifetime. More films than anyone can watch. There are 8 billion living breathing miracle human beings out there for you to connect with, marvel at, chat with, make out with, dance with, get in fights with, make art with, and so much more.
The world is full of beauty. The machine profits by distracting you from this.
I offer coaching on developing a healthy relationship with technology and more soulful living. If you're interested in chatting with me about that, you can schedule a free call here:
https://robbiecarlton.as.me/?appointmentType=77408644
I wanted to write "She understood my questions." That was the English sentence that naturally presented itself in describing the experience. But of course, there was no one there to understand, and no understanding took place. While you're here, I considered using the pronoun it, which is the most appropriate pronoun to use in this situation, but I like "She" in quotes, to draw attention to the duplicity of presenting this interaction with an algorithm as though it were a person.
Some of you may be getting excited about the Dunning-Kruger effect and Bertrand Russell and things like that, suggesting there might be an inverse correlation between certainty and truth. But I think those are actually exceptional cases. In day to day life, it's vastly more common that when you encounter certainty ("I'm going to the dentist today", "That'll be $11 and 27 cents", "Ma'am, this is a Wendys"), the certainty is certainty about a true thing.
WORD