---
title: "Every idea gets an experiment"
in_short: "AI makes trying an idea almost free. Like every genie, it does exactly what you said. Saying the right thing is still up to you."
canonical: https://www.hontolab.com/en/blog/every-idea-gets-an-experiment
language: en
authors: "Marius Wehrle"
published: 2026-10-20T05:00:00Z
modified: 2026-10-20T05:00:00Z
publisher: Hontolab
---

# Every idea gets an experiment

> **In short:** AI makes trying an idea almost free. Like every genie, it does exactly what you said. Saying the right thing is still up to you.

AI is the best thing to happen to curiosity in years. Anyone can now sketch
an architecture before lunch, fit a statistical model after lunch and explain
a security protocol before the coffee gets cold. We use it all day, and we
want everyone on our team to use it all day.

It changed one thing on the way in: the order.

## First the rules, then the art

Jazz musicians improvise brilliantly. They also spent years on scales first.
They know which rule they're breaking, and why the music sounds better for it.

Technical work used to run in the same order: learn the rules, then learn why
they exist, and only then get creative. In the fields we know deeply, we bend
rules freely, because we know which ones hold up the roof.

Security is a good example of the other kind. For most of us, it's a field
where we never reach the creative part, and that's healthy. Improvising is
wonderful in jazz. In encryption, less so. A homemade login works fine on day
one, at least against people who knock politely. The trouble is the door you
didn't know you'd left open. So where we don't know a field deeply, we don't
get clever. We use the established methods, exactly as written.

AI lets people be creative first and learn the rules later. That's mostly
great news. **Execution got cheap. Judgment got valuable.**

## Let the experiment decide

How do you tell a brilliant answer from one that only sounds brilliant? The
way science always has: guess, test, keep what holds, repeat. What's new is
the price.

An AI agent can now run the whole loop by itself: change something, run the test, check the score, keep it or
throw it away, and go again. Andrej Karpathy made this popular as
*autoresearch*: you go to bed, the agent runs experiments all night, and the
results are waiting at breakfast.

We work the same way. Every change runs against a fixed set of cases, three
times, on several models, and only what holds three times out of three counts.
When a model changes, the whole set runs again.
The loop works at every scale: one sentence in a prompt, one formula in our
search, a whole new structure for the heart of our Flow Harness, the
environment our AI agents work in. That last one replaced about 39,000 lines
of code with about 7,000. It's still in the experiment. It ships
when it holds on the same cases as the old one, and not a day before.

The same habit once caught a bug that had made our search quietly worse for
months. Every single result looked reasonable. Nobody reading them would have
noticed. The benchmark did.

Here's the catch. The agent runs the experiments, but someone still has to
decide what the experiment measures, and that's judgment. Remember the open
door. Every test can pass while it stays wide open, because nobody thought to
test for it. Optimise the wrong number all night and you get a very confident
wrong answer by breakfast.

So the rules matter more now, not less, and at a deeper level. Following one
takes knowing what it says. Setting up the experiment that checks it takes
knowing why it's there, and what goes wrong without it. Inside your own
field, that's hard. Outside it, you don't even know which door to test.

## Careful what you wish for

A hospital knows where everyone's field ends. The nurse, the radiologist, the
resident and the surgeon work in the same building, and nobody thinks they're
interchangeable. Each knows what's theirs, and when to call someone else.
Nobody says: you've watched plenty of operations, grab a scalpel.

IT has the same spread of backgrounds: a bootcamp, an apprenticeship, a
computer science degree, a PhD in cryptography. From the outside, it all
looks like *good with computers*.

Now give everyone who's good with computers a genie in the chat. It grants
every wish, fluently and at once. It also has the old trouble with genies:
you get exactly what you asked for, not what you meant. The doing got easy.
What's left is harder than the doing ever was: knowing where the limits are,
what to wish for, and how you'd notice a wish gone wrong. That's the *why*
behind the rules, and the genie can't hand it to you.

*The fair objection:* doesn't this keep juniors away from the interesting
work? It's the opposite. Anyone can propose an idea, and with AI anyone can
build it well enough to test it. Proposing just isn't deciding. The experiment
decides, and it doesn't care about job titles. Better still, every failed
experiment teaches a rule with the reason attached, which is the quickest way
we know to learn the *why*.

What we haven't solved is the line itself. Software never drew it, and
neither have we. For now we have a habit instead of a rule: when an answer
sounds good, test it. When it sounds brilliant, test it three times.

**Marius Wehrle, PhD**
*A human had the idea. AI ran the drafts. The human decided which one held.*