---
title: "Genius or vending machine"
in_short: "Force an AI into a fixed form and you lose what it finds that nobody asked for. So we keep it free to think, and judge it by how often it's right, not by one lucky run."
canonical: https://www.hontolab.com/en/blog/genius-or-vending-machine
language: en
authors: "Marius Wehrle, Katarina Kostic"
published: 2026-10-13T09:00:00Z
modified: 2026-10-13T15:49:02Z
publisher: Hontolab
---

# Genius or vending machine

> **In short:** Force an AI into a fixed form and you lose what it finds that nobody asked for. So we keep it free to think, and judge it by how often it's right, not by one lucky run.

Ask an AI the same question twice and you get two answers. Everyone has done
it, and everyone has drawn the same conclusion: if it can't even repeat itself,
how can I trust it with anything real?

Fair. A single run does vary. But ask a colleague the same question on Monday
and on Thursday. Different sentences, probably a different order, maybe a
different example. Nobody calls the colleague unreliable.

AI runs on a computer, so we test it like software: same input, same output,
or it's broken. But it answers like a person: same question, different words.
We're using the machine test on something that behaves like a colleague.
That's the whole confusion, and it's a stubborn one.

## The test you lose

You want a genius that tests like a vending machine. Genius: hand it the
question, let it find what it needs, analyse it, and solve the thing. Vending
machine: press B4, get B4, every time, so you can check it with one line.

You can't have both. Put the genius behind a form and you get the vending
machine, minus the genius. Let it loose and you get the genius, minus the one
thing engineers trust most: the simple test. Same input, expected output,
green or red. Gone.

## Everyone else lives without it

People have solved this before, many times. Nobody judges a drug by one
patient. A production line doesn't ask for identical parts; it sets a
tolerance and counts how many land inside it. A weather forecast says seventy
per cent chance of rain, not "rain". Even machine learning judges a model on
thousands of examples, never on one. None of these expects the same result
twice, and none of them minds.

So it's less a problem of the technology than of the field it landed in.
Software is the one field built on a promise: same input, same output, every
time. Now there's something inside the software that doesn't keep that
promise, and a test is no longer a yes or a no. It's a whole evaluation: many
runs, a range, a rate. That's where most people's trouble starts.

And the first instinct is the wrong one: *how do I make it predictable
again?* You can, and you get a smaller product. The better question is *what
can the product do now that it couldn't before*, and then measure that.

We see it with every client. Everyone wants the report to look the same every
time, because that's what reports were always good at: same headings, same
tables, same order. Fair. But where does it go when the agent finds something
nobody asked for: a new risk, an odd number, a problem nobody knew existed? A
fixed template has no box for it. So it gets squeezed into the wrong section
or left out, and the most valuable finding of the run is the one the report
drops.

## What replaces it

Our field is data science and the interpretation of data, and two good
analysts don't write the same paragraph either. So we ask whether the answer
is inside the tolerance. Ours has three parts, and the order matters.

Facts first: every finding points at evidence the run actually found, and a
wrong fact is out, however nicely it's worded. Then completeness: everything
the question asked, answered, and every fact that bears on it, found. Short
and complete passes; long with a gap doesn't. Form last: how it's written
matters least, unless another program has to read it.

We test the same way. Many scenarios, each run
several times, often on different model setups, and every fact checked and
every gap counted. An improvement has to show up in every repeat, across
all of that. Once is a lucky run, and a lucky run doesn't ship. Do the
arithmetic: ten scenarios, three repeats, ten model setups, and one change
costs three hundred runs to judge. That's what a test is now.

*The fair objection:* three hundred runs to judge one change sounds slow and
expensive. It is slower than one demo. It's still cheaper than shipping a
change that only worked once.

Most of our time goes on the top two: teaching the agent to search wider
than the first good answer, and picking better between the answers it comes
back with. It's slower than pressing B4. It's also the only way we know to
tell a genius from a lucky run.

**Marius Wehrle, PhD** and **Katarina Kostic**
*Human thinking. AI typing. Human signing.*