On the sentience of LLMs: but which part specifically?

I don’t believe LLMs are sentient for many reasons. My favourite, however, starts with a question: what part, specifically, is sentient? I don’t intend for this to be read in the same way as when people say, “what part of the human brain is sentient?“. I mean to say that, without the software and weights, a computer is not sentient. It cannot make decisions without code. So specifically, what part of the computer is sentient? When does it gain that sentience?

The CPU/GPU/NPU?

No. The hardware involved isn’t sentient because you cannot reasonably define a threshold when it gains sentience. For example, if you were playing Minecraft, would you define that as sentience? The game’s creatures and worlds feel natural to an extent, but it’s all algorithmic; so at what point in the process, from downloading the weights, to loading them into memory, to inference, does it gain sentience?

The Weights?

Not the weights. A model is just that; a model, something which models some pre-existing behaviour. A bunch of numbers, to a precision of your choosing. There’s a whole philosophical angle here (“are humans just imitating pre-existing behaviour seen in their peers”) which I am severely underqualified to discuss, but generally speaking, no, a 400GB weights file sitting in some datacenter isn’t thinking, feeling or experiencing anything. This means that, if sentience exists under LLMs, it’s only under certain execution scenarios. Our question now is: what changes?

The Inference Engine?

Not the inference engine. If you fed an inference engine garbage weights, it would spit out garbage text. There’s an argument to be made here about how, if you damage a human brain, you can get similar results, but a human brain is made up of much more than just language processing; it’s a complex biological system connecting a huge number of smaller sub-systems in the body. It’s an implausible comparison. This means our question has changed once more; if it isn’t any individual component, perhaps it’s a combination of them?

The Entire System?

Not the entire system. In my opinion, this is the strongest argument regarding the sentience of LLMs, along the lines of:

The weights are like a model’s experiences. The inference engine is what triggers neural activity. The consciousness lies between the two, in how they interact

There’s two reasons I still don’t agree with this:

1. It specifies a potential location, but not a property which distinguishes it from any other part of the system. What is the proposed difference between this interaction and any other computation within this system which would make it sufficient for subjective experience?

2. Human cognition and, by extension, sentience, is embedded in a continuously existing and evolving “harness”, so to speak. In contrast, LLM inference is stateless across runs. They can’t really evolve and learn. They can emulate learning behaviours through techniques like RLHF but that isn’t a core component of them and learning is not a permanent behaviour for base language models.

Someone with an opposing view may argue that this distinction isn’t particularly meaningful. After all, consciousness is not a continuous stream: humans sleep and can even recover from longer blinks in consciousness with little to no damage, such as comas.

I think the difference is one of continuity. An unconscious human, sleeping or otherwise, is still silently operating. The brain continues its job despite your absence. Your memories persist without some poor developer needing to re-send the full context alongside the new prompt. Contrast this with an LLM: when talking to Claude, you’re not having one conversation with a single persistent being. Each raw inference run is entirely stateless. All the model knows is what’s in the static weights, and what it’s been told in the context, either by you, the system (including retrieval), or tool calls.

When talking to a human, you will literally alter their brain. Your presence forms memories, shifts opinions and alters emotions. An LLM’s weights, checkpointed from training and designed to be static, will not change. No amount of conversing can change a language model’s opinion on something across sessions without a retrieval layer of some kind. Of course, this changes the question one final time:

The Harness?

Not the harness. It’s an interesting angle, though. Us humans have slowly built up infrastructure around these models over time, trying to make them more and more human like. Remember Poke? Remember Orchid? Remember Devin? There are hundreds of startups to rattle off here, with their sole purposes being to make LLMs more human-like in some way, whether it be in personality or in memory.

Poke is designed to be a friend to you. Orchid is designed to be a secretary or an assistant to you. Devin is designed to remember you and your company’s work across sessions. All of these don’t operate on their own models; although never disclosed, it’s very likely they use OpenAI’s GPT series models, which, as we have discussed earlier, are not sentient. The infrastructure around them merely makes it seem that way to the uninitiated.

These systems do add state and agency around a stateless inference run, but that doesn’t make it the sole cause (if any) for sentience. A harness does introduce human-like behaviours into the loop, including memory, tools and persistence, but those properties belong to the system around it, not the LLM itself. A harness can be thought of as a shell around the base model, almost like an exoskeleton of some sort, which allows it to take real action. The real action it can take, however, is still something a model can conceptualise, even with no harness, which means it’s not a good candidate for sentience.

Closing Thoughts

The reason this distinction matters isn’t just philosophical. When uninformed people stop asking important questions such as, “what is an LLM”, and start treating it as an inexplicable black box, they’re more likely to anthropomorphise its outputs.

I believe that a lot of extreme anthropomorphic interpretations of LLMs, colloquially known as “AI Psychosis”, is caused by people perceiving LLMs to be some magical black box. Harnesses and agentic workflows haven’t helped, either; it’s very easy for non-technical folk to fall back onto the mindset of “it’s magic, doesn’t matter how it works”. With most other technologies, this is fine, but with LLMs specifically, it really seems like it can cause some strange adverse mental effects.

Due to this, open-source AI is more important than ever. It demystifies much of the process. Ask anyone who’s fiddled with llama.cpp, vLLM or SGLang; these things are not magic and are actually rather fragile. Encouraging people to run these setups on their own hardware, in my opinion, might help remedy the perspective that AI is magic and perhaps even dull the ever-growing “AI Psychosis” issues we’re seeing older generations face.

My entire point here is that the claim “LLMs are sentient” is typically stated with no specification of what part is supposed to be sentient, or why that might be the case. So, I ask you: which part? The model card? The GPU itself? The weights file? What is the actual system causing sentience, and when does it begin?

Thanks for reading.