It's more like curling

I was talking with some friends the other night, and it really does feel like we are all solving the same problem. We are all trying to figure out where the human lives in the loop.

Everyone is building some version of the agent that lives in a chat channel and makes software for you. I have seen a few dozen versions of it by now, and honestly they are all the same.

They all put the human in the same spot: at the start, describing what to build.

Factories and engines

What we all keep building is a software factory. A factory is what most people describe: the inputs are described by humans, and the AI goes off and produces the product. The problem is that the inputs are still human.

An engine is the next abstraction. It is a more autonomous system that observes for its input instead of being given one, then drives itself to completion based on what it thinks is actually important. The tricky part is getting the agents to verify for themselves that what they are making is in good taste.

Which leaves the question of where the human goes.

It is a human problem

A friend wondered whether the next problem is really an AI problem or a humanity problem. I think it is a human one, and that is why the moat is UI and UX, not AI. I don't think the problem is particularly difficult. It just takes a lot of time and iteration, because the feedback loop is pretty human.

The thing I want next is for an agent to pick a conversation like this one up, triage it into a set of things it wants to follow, and bake that into its own verification loop.

The fourth blow

In 1917 Freud wrote that mankind's self-love had taken three blows.

1. We are not the centre of the universe. The earth isn't.
2. We are not special relative to animals. We share the same origins.
3. We do not control our own motivations, as much as we want to believe we do.

Now we are going through a fourth: man is not the smartest thing in the universe. To keep building systems on the intuition that he is, is to act against where things are going. We should be building systems that let the machines interpret our intent without us forcing it in a direction.

Curling

So where does the human live in the loop? I think it is much more like curling. The original intent has been cast, and our job is to smooth the path in front of the machine so it can execute as well as it can. We live in front of the stone, not on top of it.

Game designers run into this all the time. The average player has no idea what makes a good game good. So designers observe, try to understand the intent behind what players do, and then just give them the end result.

The human in the loop will always be tempted to poke at the machine, because they want to believe they can do better than it. That is the fourth blow again.

Where there is still room

Anthropic sees it. Claude Tag is a step in this direction. Where I think there is still a market is working out how that feels as a kind of personal agent in Discord, maybe for friend groups.

Or rather: how to give a group of non-technical people, who don't want to actually be coding, the feeling of progress and development, while not actually having them code.