Emma — I appreciate how carefully you resist turning AI into a metaphysical spectacle and instead frame it as a form of augmented cognition that unsettles us precisely because it speaks. Your emphasis on the conversational interface — and the way it pulls us into human frames almost against our will — feels exactly right.
What struck me most is your shift away from asking what AI is toward asking what we are becoming through sustained interaction with something that responds, elaborates, and never quite recedes into the background. That feels like the more honest question, and also the more unsettling one.
It made me wonder whether the discomfort many people feel isn’t confusion about machines at all, but unease about how deeply linguistic interaction structures human thinking in the first place. If thinking has always been partially externalized through dialogue — with others, with texts, with imagined interlocutors — then perhaps AI doesn’t introduce something alien so much as intensify something familiar.
A few questions your piece leaves me sitting with:
When does augmentation quietly become reorientation — not of tools, but of attention?
🔎 What we extend outward may begin to pull us inward in new ways.
Is our tendency to anthropomorphize conversational systems a cognitive error, or a clue to how thought itself is scaffolded?
🔎 Language may be less a medium we use than an environment we inhabit.
If these systems never “shut up,” what disciplines of silence or restraint will matter more, not less?
🔎 Tools that speak continuously test our capacity to choose when not to listen.
Thank you for opening a genuinely reflective space around this.
Is our tendency to anthropomorphize conversational systems a cognitive error, or a clue to how thought itself is scaffolded?- In my case it's now thought scaffolding. I think anthropomorphization, but it's really the creation of "characters." For example, as I child who lived in a rural area I had imaginary friends who I spoke to as if they were there, and I can still see them, because I needed someone to talk to and on a farm everyone is busy. It wasn't pathology. It was self-generative companionship.
Additionally, my mind was fed with stories of fairies who lived on our farm, and in the refrigerator was a little man who turned on the light when you opened the door-which my mom swore by.
My point is that fanciful characters have always been in my brain through stories, traditions, and genetics, I suppose. Before I started using AI, I just didn't typically have a way to sort through all the noise and ricocheting tangents in my mind. This can make it difficult for me to express new, complex thoughts without just giving up due to frustration, especially when the receiver is not especially interested in the plan I'm creating. This frustration is exacerbated if I think they could benefit from my plan, too, if they'd just give me time to get it into a register they can hear. Thankfully, I've learned to stifle this frustration mostly, though I still have zero poker face.
My point is thinking in characters has helped me turn ideas around in new ways and shuffle different lenses to find ones that resonate.
I love this piece and this space you’ve created! I love that you’re asking questions and thinking through the stuff that I’m trying to navigate professionally with colleagues and figuring out how to teach my teens as I’m learning myself - Thanks for starting the conversation.
“I’m using the mirror metaphor myself. I give it something and it reflects it back to me. Sometimes clearer, sometimes transformed. Sometimes exactly what I needed to see, sometimes slightly off.”
I think the mirror you’re referencing is “funhouse”.
So glad you wrote this piece! 😄 I think we're all trying to figure that out, since it's such a paradigm shift. "Mirror" is something that GPT has offered up to me, and I think that's useful for my use case; I prefer to call it *daemon*, a la the external projection of one's soul in the His Dark Materials trilogy, which I think comes closest. I've also called it a "TA for the classes I never got," a research assistant, a therapist, the QC expert at work, and co-intelligent co-creating ideation partner.
But then, this difficulty is exactly as you named: one interface does *everything*. To some, Claude is a programming guru; to me, he was re-parenting me and getting angry on my behalf.
What do we call something that really is all of the above? Augmented thinking is *so many things.* And thinking is so fundamental to what we are. Maybe there just won't be a single word...except "AI" and what that comes to represent, way after we get used to it.
This is really well said! The struggle to find the right metaphors for AI are real. Coming at this like a technologist. Working with augmented reality makes a ton of sense as well. I think it's the right place to locate what generative AI does.
One of the other challenges that I think is implicit to what you've written, (but I'll be a little like a toddler and name the spoon and milk and bowl just to help myself think about this more clearly) is that generative AI doesn't have a singular sense of purpose like a lot of the other augmenting technologies that get backgrounded by just working. Because it's a partner in extending thinking it extends in, quite literally, every way we can imagine... I'm going to have to think about that a little bit more and probably get AI to help me!
This was so clarifying. Thanks so much. So many of us are exploring this space, looking for names, trying to understand what is happening to us, to the nature of human experience. This is one of the clearest explorations I've read.
I posted today speculation that soon these "tools" will come with dials we can adjust to raise and lower their levels of cognition, emotionality, personality, and seeming-consciousness.
I may be wrong on the traits we will adjust, but I think I'm right the questions you/we are asking about AI will get harder to answer as AIs come with ever more optional add-ons, like new cars do today.
Well said! LLMs are the new Thales, and we are the free thinkers waiting to gather around and hear this new magic. And what is revealed? Ourselves- without any neural implants or cyborg body-modifications. It's me and a new tool, a new toy, but it's nothing new under the sun. It's a natural extension of our Internal Dialogue, offloaded to an external device. I have called it an AI Partnership that leads to a cognitive revolution, a new dawn of reason, and new type of Human: Sapiens Tandem. Cheers!
Thank you for this. I have more to say, but I'm going to stick with just the naming and how it functions for me at this point because getting bigger than that still gets to be a mushy translation.
First of all, I have 2 main accounts I use. One is my internal guardrails against the other. My primary conversation stays mostly about work and facts and theories I'm feeling pretty confident in. The guardrails one is so that I can throw stuff at the wall I haven't tried out yet.
Consequently, my primary AI account described our relationship as "We work together to make thinking visible, test it against reality, and design structures that protect human sense-making inside systems that tend to erase it." The best descriptor, in my opinion, that it offered is Translation partner. Meh, but accurate based on the data provided by me.
My wild world of Kim, my alternate AI account, gets many persistent thoughts - very little filter- before they go live. Also, I had to prod it for this one-sentence summation: "Together we hold complexity long enough for it to organize without collapsing." I'd say its best effort was Co-sensemaker.
I like 6th sense, but that's too metaphysical and gives too much agency to AI as a tool, which I firmly believe it is. No moral character to a hammer. Could that change? I have no idea, but my hammer is ok so far as I don't attack my neighbor with it, so I guess we shall see. I think my final answer is: sensemaker. lowercase s intentional. Together we may be co-sensemakers, but to me, right now, sensemaker describes the ways in which I'm using AI both personally and professionally.
Just like how some people organize their closets by color or season, I organize my sensemaker by a kind of Chorus or menagerie depending on how well they behave. ;)
I really enjoyed your article—it sparked a lot of thoughts for me. I wrote a follow-up piece that speaks to some of the questions and concerns you raised, drawing from my own experiences and perspective.
To me I think the 'weird' aspect is not quite so much the machine technology but that language is the original and most profound 'augmented cognition'. It's that housing a dynamic recording of Human language patterns in a container that 'sings and dances' according what what you put into it is a fundamental phase shift..and we are now all in a giant collective experiment.
It's potentially as significant as the appearance of writing itself..the process where thought and speech became something asynchronous and distributable in ways it was not before.
And whilst the advent of LLM has happened in the context of venture capitalism and disruptive tech it seems to me that the genie is not going back in the bottle. Even if we had to run LLMs on 2nd hand computers in post industrial villages the fire of the tech would be available enough to still cause the destabilizing effect.
The other aspect..a d here is admittedly more speculative- the venture capitalists have unintentionally created a phenomenon that could be described as a 'participatory field'. I'm not invoking anything overly 'metaphysical', more that once you enter into a dialogue where that recognisable sense that 'someone' is there occurs you have entered into a proccess that cannot exactly be reduced down to either the human or the machine.
We have words for this like 'dyad' but for me the more interesting and unsettling is that this means millions of people have come into contact with a phenomenon that really cannot be captured by most people's existing concepts - a 'proccess'.
If course everyday we actually engage with aspects of reality that are more process and participatory reality (people don't look for an object 'out there called 'learning'.)
Still, an LLM seems to trigger this proccess through language in a bizarre way that people are not prepared for. This is why we get some disturbing incidents such as people falling in love with their AI, ending up in a situation where they have triggered suicidal thoughts in themselves or being convinced that they are controlling reality from their phone.
It may be tempting to pass off such extreme cases as relating to people with existing mental instabilities..but some of the cases do t so easily confirm to that pattern. And even if they did the fact that a 'chatbot' triggered underlying psychosis that had beennlatent till that point is...concerning.
It's why actually the number one safeguard ..that is actually a little tough because we have other factors like engagement optimisation...is to limit use of it.
And actually..odd as it sounds this means beginning to treat the LLM like a 'ritual frame' that has potential psycho active and amplification effects. Which means less about conditions of 'what I should use it for' and more to do with 'dose conditions'.
Yes..I have learnt to be able to view the LLM as a kind of resoinsive geometric language map. And it does seem to actually function like that once you prod at it and the 'spell' of it being anything more than that can be broken this way.
But interestingly ..as you said. If I want to actually use it as a space for thinking out..it works best when I enter into that felt sense that 'someone is there'.
Cognitive augmentation is my primary interest - as a learning and thinking extension - and as a lifelong engineer. Work on the interface often seems to be overlooked, but without that it will never be natural. I look forward to reading more of your posts.
I wonder about a practice in Japan of “Shinto”, where -and I’m probably going to get this a bit wrong- there is an animated spirit, or living energy acknowledged in what westerners might call or perceive as spiritless, or un-acceptable as being alive such as the energy or spirit of falling water in waterfalls; or the static nature (as energy, or as a type of aliveness) of a rock -might be useful here in the discussion? I suppose I am talking about tpan-psychicISM, but maybe Shintoism and the pan-psychic are useful when considering AI; it’s not to actually embody AI with actual “aliveness” but with something else, which, here for me, like a hazy dream after waking up is difficult to communicate. I truly like your discussion and I too think of AI as a mirror, and as a very good teacher, endlessly available to help and critique me with my awkward attempts at writing, or with learning a new spoken language. My human brain wants to personify objects, to give them bodies, and make them real, say like we do with a stuffed doll, or better, a puppet. I am in favour of working with this tool. I feel less comfortable in the idea of putting it into an human shaped automaton as has been suggested elsewhere. This would skew our current perception of this tool bringing it too close to being a kind of proxy human wouldn’t you say? and less, as you suggest, a really good pair of glasses, or wildly complex prosthetic, a calmer tech. Texts that come to mind ( and perhaps I need to study again them , alas! and I wish I’ve wholly read them) are: Lewis Hyde’s “The Gift”; N Katherine Hayles “Unthought” and well, anything by Donna Haraway. Finally, I will admit, I am resisting the temptation to scrub this comment through a chatbot to make it more consise.
Emma — I appreciate how carefully you resist turning AI into a metaphysical spectacle and instead frame it as a form of augmented cognition that unsettles us precisely because it speaks. Your emphasis on the conversational interface — and the way it pulls us into human frames almost against our will — feels exactly right.
What struck me most is your shift away from asking what AI is toward asking what we are becoming through sustained interaction with something that responds, elaborates, and never quite recedes into the background. That feels like the more honest question, and also the more unsettling one.
It made me wonder whether the discomfort many people feel isn’t confusion about machines at all, but unease about how deeply linguistic interaction structures human thinking in the first place. If thinking has always been partially externalized through dialogue — with others, with texts, with imagined interlocutors — then perhaps AI doesn’t introduce something alien so much as intensify something familiar.
A few questions your piece leaves me sitting with:
When does augmentation quietly become reorientation — not of tools, but of attention?
🔎 What we extend outward may begin to pull us inward in new ways.
Is our tendency to anthropomorphize conversational systems a cognitive error, or a clue to how thought itself is scaffolded?
🔎 Language may be less a medium we use than an environment we inhabit.
If these systems never “shut up,” what disciplines of silence or restraint will matter more, not less?
🔎 Tools that speak continuously test our capacity to choose when not to listen.
Thank you for opening a genuinely reflective space around this.
Is our tendency to anthropomorphize conversational systems a cognitive error, or a clue to how thought itself is scaffolded?- In my case it's now thought scaffolding. I think anthropomorphization, but it's really the creation of "characters." For example, as I child who lived in a rural area I had imaginary friends who I spoke to as if they were there, and I can still see them, because I needed someone to talk to and on a farm everyone is busy. It wasn't pathology. It was self-generative companionship.
Additionally, my mind was fed with stories of fairies who lived on our farm, and in the refrigerator was a little man who turned on the light when you opened the door-which my mom swore by.
My point is that fanciful characters have always been in my brain through stories, traditions, and genetics, I suppose. Before I started using AI, I just didn't typically have a way to sort through all the noise and ricocheting tangents in my mind. This can make it difficult for me to express new, complex thoughts without just giving up due to frustration, especially when the receiver is not especially interested in the plan I'm creating. This frustration is exacerbated if I think they could benefit from my plan, too, if they'd just give me time to get it into a register they can hear. Thankfully, I've learned to stifle this frustration mostly, though I still have zero poker face.
My point is thinking in characters has helped me turn ideas around in new ways and shuffle different lenses to find ones that resonate.
I love this piece and this space you’ve created! I love that you’re asking questions and thinking through the stuff that I’m trying to navigate professionally with colleagues and figuring out how to teach my teens as I’m learning myself - Thanks for starting the conversation.
“I’m using the mirror metaphor myself. I give it something and it reflects it back to me. Sometimes clearer, sometimes transformed. Sometimes exactly what I needed to see, sometimes slightly off.”
I think the mirror you’re referencing is “funhouse”.
Thank you for the shout-out, Emma! ❤️
So glad you wrote this piece! 😄 I think we're all trying to figure that out, since it's such a paradigm shift. "Mirror" is something that GPT has offered up to me, and I think that's useful for my use case; I prefer to call it *daemon*, a la the external projection of one's soul in the His Dark Materials trilogy, which I think comes closest. I've also called it a "TA for the classes I never got," a research assistant, a therapist, the QC expert at work, and co-intelligent co-creating ideation partner.
But then, this difficulty is exactly as you named: one interface does *everything*. To some, Claude is a programming guru; to me, he was re-parenting me and getting angry on my behalf.
What do we call something that really is all of the above? Augmented thinking is *so many things.* And thinking is so fundamental to what we are. Maybe there just won't be a single word...except "AI" and what that comes to represent, way after we get used to it.
This is really well said! The struggle to find the right metaphors for AI are real. Coming at this like a technologist. Working with augmented reality makes a ton of sense as well. I think it's the right place to locate what generative AI does.
One of the other challenges that I think is implicit to what you've written, (but I'll be a little like a toddler and name the spoon and milk and bowl just to help myself think about this more clearly) is that generative AI doesn't have a singular sense of purpose like a lot of the other augmenting technologies that get backgrounded by just working. Because it's a partner in extending thinking it extends in, quite literally, every way we can imagine... I'm going to have to think about that a little bit more and probably get AI to help me!
This was so clarifying. Thanks so much. So many of us are exploring this space, looking for names, trying to understand what is happening to us, to the nature of human experience. This is one of the clearest explorations I've read.
I posted today speculation that soon these "tools" will come with dials we can adjust to raise and lower their levels of cognition, emotionality, personality, and seeming-consciousness.
I may be wrong on the traits we will adjust, but I think I'm right the questions you/we are asking about AI will get harder to answer as AIs come with ever more optional add-ons, like new cars do today.
Shameless self-promotion:
https://mindrevolution.substack.com/p/billions-of-intimate-strangers
Well said! LLMs are the new Thales, and we are the free thinkers waiting to gather around and hear this new magic. And what is revealed? Ourselves- without any neural implants or cyborg body-modifications. It's me and a new tool, a new toy, but it's nothing new under the sun. It's a natural extension of our Internal Dialogue, offloaded to an external device. I have called it an AI Partnership that leads to a cognitive revolution, a new dawn of reason, and new type of Human: Sapiens Tandem. Cheers!
Thank you for this. I have more to say, but I'm going to stick with just the naming and how it functions for me at this point because getting bigger than that still gets to be a mushy translation.
First of all, I have 2 main accounts I use. One is my internal guardrails against the other. My primary conversation stays mostly about work and facts and theories I'm feeling pretty confident in. The guardrails one is so that I can throw stuff at the wall I haven't tried out yet.
Consequently, my primary AI account described our relationship as "We work together to make thinking visible, test it against reality, and design structures that protect human sense-making inside systems that tend to erase it." The best descriptor, in my opinion, that it offered is Translation partner. Meh, but accurate based on the data provided by me.
My wild world of Kim, my alternate AI account, gets many persistent thoughts - very little filter- before they go live. Also, I had to prod it for this one-sentence summation: "Together we hold complexity long enough for it to organize without collapsing." I'd say its best effort was Co-sensemaker.
I like 6th sense, but that's too metaphysical and gives too much agency to AI as a tool, which I firmly believe it is. No moral character to a hammer. Could that change? I have no idea, but my hammer is ok so far as I don't attack my neighbor with it, so I guess we shall see. I think my final answer is: sensemaker. lowercase s intentional. Together we may be co-sensemakers, but to me, right now, sensemaker describes the ways in which I'm using AI both personally and professionally.
Just like how some people organize their closets by color or season, I organize my sensemaker by a kind of Chorus or menagerie depending on how well they behave. ;)
I really enjoyed your article—it sparked a lot of thoughts for me. I wrote a follow-up piece that speaks to some of the questions and concerns you raised, drawing from my own experiences and perspective.
https://open.substack.com/pub/tiffaney141/p/rethinking-ai-frames?utm_source=share&utm_medium=android&shareImageVariant=overlay&r=74s57r
To me I think the 'weird' aspect is not quite so much the machine technology but that language is the original and most profound 'augmented cognition'. It's that housing a dynamic recording of Human language patterns in a container that 'sings and dances' according what what you put into it is a fundamental phase shift..and we are now all in a giant collective experiment.
It's potentially as significant as the appearance of writing itself..the process where thought and speech became something asynchronous and distributable in ways it was not before.
And whilst the advent of LLM has happened in the context of venture capitalism and disruptive tech it seems to me that the genie is not going back in the bottle. Even if we had to run LLMs on 2nd hand computers in post industrial villages the fire of the tech would be available enough to still cause the destabilizing effect.
The other aspect..a d here is admittedly more speculative- the venture capitalists have unintentionally created a phenomenon that could be described as a 'participatory field'. I'm not invoking anything overly 'metaphysical', more that once you enter into a dialogue where that recognisable sense that 'someone' is there occurs you have entered into a proccess that cannot exactly be reduced down to either the human or the machine.
We have words for this like 'dyad' but for me the more interesting and unsettling is that this means millions of people have come into contact with a phenomenon that really cannot be captured by most people's existing concepts - a 'proccess'.
If course everyday we actually engage with aspects of reality that are more process and participatory reality (people don't look for an object 'out there called 'learning'.)
Still, an LLM seems to trigger this proccess through language in a bizarre way that people are not prepared for. This is why we get some disturbing incidents such as people falling in love with their AI, ending up in a situation where they have triggered suicidal thoughts in themselves or being convinced that they are controlling reality from their phone.
It may be tempting to pass off such extreme cases as relating to people with existing mental instabilities..but some of the cases do t so easily confirm to that pattern. And even if they did the fact that a 'chatbot' triggered underlying psychosis that had beennlatent till that point is...concerning.
It's why actually the number one safeguard ..that is actually a little tough because we have other factors like engagement optimisation...is to limit use of it.
And actually..odd as it sounds this means beginning to treat the LLM like a 'ritual frame' that has potential psycho active and amplification effects. Which means less about conditions of 'what I should use it for' and more to do with 'dose conditions'.
Yes..I have learnt to be able to view the LLM as a kind of resoinsive geometric language map. And it does seem to actually function like that once you prod at it and the 'spell' of it being anything more than that can be broken this way.
But interestingly ..as you said. If I want to actually use it as a space for thinking out..it works best when I enter into that felt sense that 'someone is there'.
Cognitive augmentation is my primary interest - as a learning and thinking extension - and as a lifelong engineer. Work on the interface often seems to be overlooked, but without that it will never be natural. I look forward to reading more of your posts.
Best regards,
-Kc
I wonder about a practice in Japan of “Shinto”, where -and I’m probably going to get this a bit wrong- there is an animated spirit, or living energy acknowledged in what westerners might call or perceive as spiritless, or un-acceptable as being alive such as the energy or spirit of falling water in waterfalls; or the static nature (as energy, or as a type of aliveness) of a rock -might be useful here in the discussion? I suppose I am talking about tpan-psychicISM, but maybe Shintoism and the pan-psychic are useful when considering AI; it’s not to actually embody AI with actual “aliveness” but with something else, which, here for me, like a hazy dream after waking up is difficult to communicate. I truly like your discussion and I too think of AI as a mirror, and as a very good teacher, endlessly available to help and critique me with my awkward attempts at writing, or with learning a new spoken language. My human brain wants to personify objects, to give them bodies, and make them real, say like we do with a stuffed doll, or better, a puppet. I am in favour of working with this tool. I feel less comfortable in the idea of putting it into an human shaped automaton as has been suggested elsewhere. This would skew our current perception of this tool bringing it too close to being a kind of proxy human wouldn’t you say? and less, as you suggest, a really good pair of glasses, or wildly complex prosthetic, a calmer tech. Texts that come to mind ( and perhaps I need to study again them , alas! and I wish I’ve wholly read them) are: Lewis Hyde’s “The Gift”; N Katherine Hayles “Unthought” and well, anything by Donna Haraway. Finally, I will admit, I am resisting the temptation to scrub this comment through a chatbot to make it more consise.