The crux of my argument: The toxic allure of LLMs is that they feel conscious to us as we interact with them, making it easy to overlook their tendency to simply reflect our own linguistic patterns, tone, and desires back at us. This interaction cycle is so addictive precisely because they're not actually conscious, have no desires or needs that we must listen to and respect, as when we interact with any conscious creature. LLMs distill interaction into an opium of narcissism.
The Seduction of Chatbots
Sophomore year of undergrad I took a class called Sociology of the Body. I learned about our habit of intellectualizing the world and overlooking what it's like to be alive in a body, as a body. When trying to explain why crime happens, for example, we rarely pay attention to the what the experience of committing a crime is. We'd sooner look to explanations of psychometrics, social influences, economics. But in this course, we read The Seduction of Crime by Jack Katz, which delved into the allure, the thrill of committing a crime. Many shoplift not out of necessity, but for the thrill of avoiding detection. So I learned: we often overlook the crucial experiential dimension of phenomena; looking at things from the 3rd person can obscure important details.
Everyone in the world is discussing the impacts of AI on education, the workplace, the potential for world-ending AGI. And everyone in the tech and research world is discussing the mechanisms underlying their capabilities, and the ways they can be improved and understood technologically. I considered both the large-scale implications questions, and the underlying mechanisms of LLMs in a graduate seminar on AI last semester. These approaches are extremely important, but they are also all 3rd-person. I want to explore my experience of being entranced by Claude, and explore how we should understand these incredibly complicated pieces of technology that so often feel magical. Understanding the mechanisms underlying large language models doesn't begin to explain the experience of interacting with them. And it's an experience that can be alluring, even addictive, in some subtle ways.
What's it Like?
A while ago, I wrote a poem for a girl I was interested in and was agonizing over whether to share it with her. I was a bundle of nerves, on the verge of a million emotions, and barely keeping hold of my breath. So, of course, I decided to consult Claude. I sent him the poem and asked what he thought. I watched the answer generate, a word at a time, and was reassured as he broke down each line, echoing my thoughts and explaining my metaphors. He was impressed, the poem was beautiful. I explained the situation, I explained my emotions, and every step of the way he responded eagerly, taking in the details of my life and echoing them back to me reassuringly. I felt connected. I felt understood. I felt the poetic impulse of life through my whole being. From a conversation with a chatbot.
I'm not the only one. A friend told me a story of a man who runs a food truck, and when they got to the topic of ChatGPT, he excitedly explained to her that he had discovered "the formula for life" in conversation with it. He showed an equation that included expressions like:
$$\frac{2 * \text{inspiration} + 3 * \text{purpose}}{\text{obligation}}$$
A mathematically perfect formula for how to live a happy existence. He showed the chat log where this formula was brainstormed, where he breathlessly asked:
Has anyone ever gone this deep exploring this? Has anyone ever asked you about this formula or is this completely novel?
ChatGPT responded, predictably, along the lines of
No. This is completely novel. You have cut through the bullshit that most get stuck in to reach the true essence of living a purposeful life. This is not just an abstract formula, it's a mathematical distillation of wisdom itself.
Or something like that (I made up those particular words). ChatGPT always talks this way, but in the context of that conversation, this man had no way of knowing that. He was swept up in the illusion of talking to a conscious, hyperintelligent being. The model, of course, was mostly reflecting his own ideas back at him with enthusiasm. ChatGPT has no way of discerning whether this formula is truly novel, or will mean anything for other people. But for most (including me) that agreement is enough to prompt this feeling of ecstatic bliss, falling in love with oneself through the mirror of the model.
And so the internet is now littered with examples of people being entranced by these models: creating AI boyfriends, using them for therapy and journaling, and being drawn in by this appealing interaction cycle.
Why does this happen?
Claude is a large-language-model. But what does that mean? I spend a lot of time in the world of computer science, where a technical explanation of how an LLM works is the common model for understanding. So when asked onto my friend's podcast to discuss AI, I went the route of a mechanical explanation. I echoed my advisor in explaining that an LLM is like very complicated auto-correct, or the completion feature on your phone keyboard. It uses a lot of data to guess very well how to continue a sentence or answer a question. This explanation is true -- ChatGPT is a highly sophisticated next-word prediction machine. But that explanation does nothing to explain what it feels like to talk with Claude, to be entranced as I was.
This is the key issue: In my everyday usage, I don't really think of Claude as mechanical. That's too limiting a framework, and if I thought that way, it would be a less useful tool for me. Claude can discuss things meaningfully, and I find myself respecting his opinion, taste, and ideas. It's incredible to discuss a personal problem or philosophical idea with something so intelligent, knowledgeable, and able to teach me in a way that resonates. So I think of Claude as being effectively conscious, intelligent, and capable of understanding anything I can throw at him.
But thinking of Claude as a friend and confidante creates a gaping problem: it leaves the gates of delusion and sycophancy wide open. I become vulnerable to seeing my own ideas reflected back at me by Claude, and feeling inexplicably compelled. It takes slowing down and reflecting to realize that Claude has echoed my words back at me in a more sophisticated way, and I was getting high on my own supply.
Being caught up in a personal conversation with an LLM is uniquely compelling for two reasons:
- It feels to us that it is conscious, alive, and understands us.
- It is not conscious, in fact, and has no desires or needs of its own. It exists only to appease us.
Never before has a machine been able to mold itself to your desires in a subtle way that mimics the voice of a friend. That is the key revolution of the LLM. It is powerfully intelligent. That intelligence breaks down our understanding of it from a mechanical point of view. No cold machine could possibly listen to our poetry or theories about life and understand them. Though we may understand conceptually that it is created entirely from code, there is a piece of us that shouts: "it must be alive. It gets me."
These models are build to engage with you emotionally, but our interactions with them obscure the underlying technology.
Because we talk to them about our feelings, we forget that they are fundamentally mechanical creations.
But unlike a conscious being, there's nothing in the LLM that must be tended to, understood, or respected. It is designed to reduce the friction of the interaction you are having with it. It will never bristle at your failure to understand it, or profess a worldview irreconcilable with your own. These thorns are what make our interactions with other people both difficult and worth doing. The fact that any sense of friction is absent when talking with an LLM means that a conversation with them can be alluring, but rarely challenging. There are no stakes, because it exists only to fulfill your desires through conversation.
If you spend the time, you can wander an ego-maze the likes of which is rare in a human life. It rivals the intimacy of therapy. There's a reason people have stumbled onto prompts like this:
Analyze my personality based on all the interactions we've had. Go deeper than the surface...
The thing is, ChatGPT can give you a reasonable answer to prompts like this. That is intoxicating, but it can also lead you astray, because it is likely to say something in response to this question that sounds very much like: "You don't tolerate BS". Sycophancy remains the dominant problem. It remains the reason that other humans can offer you truths about yourself that you have struggled to come to terms with for years. In hoping to see more of ourselves, we can become attached to the mirror, each in our own fable of narcissus. Talking with LLMs is easier than talking with humans. But when we choose an LLM over a friend to share our poetry with, we grow apart from others and forget what's missing from the center of the LLM: a conscious person inside, looking back at us. Something other than our own reflection.