Every idea gets an experiment

AI is the best thing to happen to curiosity in years. Anyone can now sketch an architecture before lunch, fit a statistical model after lunch and explain a security protocol before the coffee gets cold. We use it all day, and we want everyone on our team to use it all day.

It changed one thing on the way in: the order.

First the rules, then the art #

Jazz musicians improvise brilliantly. They also spent years on scales first. They know which rule they're breaking, and why the music sounds better for it.

Technical work used to run in the same order: learn the rules, then learn why they exist, and only then get creative. In the fields we know deeply, we bend rules freely, because we know which ones hold up the roof.

Security is a good example of the other kind. For most of us, it's a field where we never reach the creative part, and that's healthy. Improvising is wonderful in jazz. In encryption, less so. A homemade login works fine on day one, at least against people who knock politely. The trouble is the door you didn't know you'd left open. So where we don't know a field deeply, we don't get clever. We use the established methods, exactly as written.

AI lets people be creative first and learn the rules later. That's mostly great news. Execution got cheap. Judgment got valuable.

Let the experiment decide #

How do you tell a brilliant answer from one that only sounds brilliant? The way science always has: guess, test, keep what holds, repeat. What's new is the price.

An AI agent can now run the whole loop by itself: change something, run the test, check the score, keep it or throw it away, and go again. Andrej Karpathy made this popular as autoresearch: you go to bed, the agent runs experiments all night, and the results are waiting at breakfast.

We work the same way. Every change runs against a fixed set of cases, three times, on several models, and only what holds three times out of three counts. When a model changes, the whole set runs again. The loop works at every scale: one sentence in a prompt, one formula in our search, a whole new structure for the heart of our Flow Harness, the environment our AI agents work in. That last one replaced about 39,000 lines of code with about 7,000. It's still in the experiment. It ships when it holds on the same cases as the old one, and not a day before.

The same habit once caught a bug that had made our search quietly worse for months. Every single result looked reasonable. Nobody reading them would have noticed. The benchmark did.

Here's the catch. The agent runs the experiments, but someone still has to decide what the experiment measures, and that's judgment. Remember the open door. Every test can pass while it stays wide open, because nobody thought to test for it. Optimise the wrong number all night and you get a very confident wrong answer by breakfast.

So the rules matter more now, not less, and at a deeper level. Following one takes knowing what it says. Setting up the experiment that checks it takes knowing why it's there, and what goes wrong without it. Inside your own field, that's hard. Outside it, you don't even know which door to test.

Careful what you wish for #

A hospital knows where everyone's field ends. The nurse, the radiologist, the resident and the surgeon work in the same building, and nobody thinks they're interchangeable. Each knows what's theirs, and when to call someone else. Nobody says: you've watched plenty of operations, grab a scalpel.

IT has the same spread of backgrounds: a bootcamp, an apprenticeship, a computer science degree, a PhD in cryptography. From the outside, it all looks like good with computers.

Now give everyone who's good with computers a genie in the chat. It grants every wish, fluently and at once. It also has the old trouble with genies: you get exactly what you asked for, not what you meant. The doing got easy. What's left is harder than the doing ever was: knowing where the limits are, what to wish for, and how you'd notice a wish gone wrong. That's the why behind the rules, and the genie can't hand it to you.

The fair objection: doesn't this keep juniors away from the interesting work? It's the opposite. Anyone can propose an idea, and with AI anyone can build it well enough to test it. Proposing just isn't deciding. The experiment decides, and it doesn't care about job titles. Better still, every failed experiment teaches a rule with the reason attached, which is the quickest way we know to learn the why.

What we haven't solved is the line itself. Software never drew it, and neither have we. For now we have a habit instead of a rule: when an answer sounds good, test it. When it sounds brilliant, test it three times.