Agency Device

The first time I encountered a useless machine, it seemed like it was mocking me.

There it sat — a plain wooden box, one switch, nothing else to it. I flipped it on. A little hand swung out, deliberately knocked the switch back off, and pulled itself back inside. So I flipped it on again, and again, and again. Every single time, that hand came back looking more determined, more stubborn.

But there is no determination. No stubbornness. Just a motor doing exactly what it was designed to do — turn itself off. The most basic circuit imaginable. Yet I couldn't shake the feeling that this box was actively resisting me. Mocking me, even. Arthur C. Clarke described encountering one at Bell Labs as "something unspeakably sinister about a machine that does nothing — absolutely nothing — except switch itself off." ¹

That strange feeling was so fascinating that I immediately wanted one. I probably spent too much time watching videos of different designs, looked into building one with Lego, and started sketching a circuit on a breadboard. There was something captivating about this useless opponent, this machine that existed solely to disagree with its user. Something irrational. Deep and instinctive.

Agent Reflex

The box got to me because we're "hardwired" to detect agency even where there is none. The ancestor who took the rustle in the grass for a predator survived more often than the one who took it for wind, so that's the wiring we inherited. Better safe than sorry.

Psychologists call this the Hyperactive Agency Detection Device, or HADD for short ². It's a cognitive system that evolved to over detect agents in our environment. It fires all day. Storms are "angry," the car "refuses" to start on a cold morning, a box with a switch is fighting us.

But detection is only step one. The moment we've spotted an agent, we start guessing what its intentions might be — what does it want, what is it feeling, what does it have against us. We don't just register that something is there. ³

I'd argue this second step is what separates us from other animals: a rabbit detects agency and flees, while we detect agency and start telling ourselves a story about it.

Shapes and Souls

Fritz Heider and Marianne Simmel demonstrated just how powerful this tendency is back in 1944. They showed people a simple animation: geometric shapes moving around a screen. No faces, no limbs, nothing remotely human-like.

Watch it yourself. Even knowing it's just shapes following predetermined paths, the narratives unfold anyway — stories of bullying, romance, heroism. You can't not see them. ⁴

That's your theory of mind in action. The same cognitive machinery that helped our ancestors distinguish friend from foe now compels us to see intention in triangles bouncing around a screen.

And it gets stronger the less we understand something. Epley, Waytz, and Cacioppo's Three-Factor Theory of Anthropomorphism points to "effectance motivation" — our drive to make sense of and predict our environment. ⁵ When something behaves in ways we can't easily explain through mechanical means, our brain defaults to intentional explanations. It wants something. The useless machine is a perfect trigger: what tool exists solely to turn itself off? Unable to find a functional answer, we supply a psychological one. The machine doesn't want to be on.

This is also why people find randomly failing technology more infuriating than consistently broken technology. A car that never starts is just broken. A car that starts sometimes feels stubborn — like it's choosing not to cooperate. Unpredictability triggers agency detection. It's easier to deal with a stubborn machine than a random one, because at least stubbornness implies something you can reason with.

The Consciousness Question

I've been thinking about all of this because the current conversation around AI consciousness reminded me of that useless machine.

There's a real debate happening right now about whether large language models might be conscious, might have experiences, might understand. One study found that only a third of participants thought ChatGPT definitely did not have subjective experience — two thirds thought it had some degree of phenomenal consciousness. ⁶

I don't want to dismiss that debate; it touches genuinely hard problems in philosophy. But it's worth pausing to ask how much of it is us. Whether it's the same machinery that turns geometric shapes into characters and a wooden box into an opponent, now pointed at a chatbot.

And notice that our attribution grows with the sophistication of the trigger, not evidence of any subjective experience. A box that flips a switch gets stubbornness. Simple shapes get simple stories. A system that produces fluent language gets... consciousness?

Each step up in behavioral complexity produces a proportional increase in our projection. That's exactly what you'd expect if the attribution is coming from us. It's not clear that it tells us anything about what's happening on the other side.

Peter et al. make a similar point: once technology pushes our buttons this effectively, it has in itself become anthropomorphic. Not because it is human-like, but because we can no longer keep the distinction upright. ⁷

The Device

I never did build that useless machine. I eventually realized the machine wasn't the point — it was about what I was feeling: the split-second conviction that a motor plus a switch was fighting me.

It wasn't fighting me, though. There is nothing there but a simple circuit. Knowing that didn't stop the feeling. And that's the thing about our agency detection — it's not a belief you can argue yourself out of. It's a reflex. You can know exactly how the trick works and still feel the trick land.

Which is why I was coming back to it. The machines keep getting better — much better — at tripping that reflex. We spent the whole time worried they'd wake up and become conscious. I'd argue that real risk runs the other way: that we keep forgetting they haven't.

References