Generating hypotheses is cheap. Testing them is not. Ask a model why your training run is stuck and it will give you twenty reasons before you have finished your tea, and checking any one of them can cost hours on a lab machine. research-loop is an agent that does the checking, and it is built so that it cannot quietly mark its own homework.
How it works
You give it a goal file, the North Star, with metric thresholds. Every possible hypothesis the agent thinks of becomes a node in a dependency graph, with edges for what has to be settled first.
Every hypothesis has to name its falsifier up front: the result that would prove it wrong. It gets three attempts, unless the agent writes down a reason to go past that. Once a hypothesis is falsified it stays falsified. A blocked one can only reopen if the agent records a new mechanism: a materially different way of testing it, such as four times the training budget.
I wanted a strictly structured way of working for this agent, so as much as possible is scripted in shell, to enforce standards and minimise drift from one generation to the next. None of this is a request in a prompt: the rules live in shell tools that refuse. The registry will not take a hypothesis without a falsifier, will not make an illegal state change, and will not let the agent change its goal except to raise it. The failure I was designing against is an agent that confirms itself, so the guard had to be mechanical.
The agent wakes every so often and picks the next piece of work from a ranking that scripts compute with heuristics. It does that one piece of work, writes a journal entry and books its own next wake. Cheap tests, which read results already on disk, go ahead of expensive ones, which book a machine for hours.
Four weeks
The original run was pointed at an earlier version of my Neural Audio Quantiser, and it ran unattended from 26 July to 21 August 2026. It registered 85 hypotheses. 27 were confirmed, 32 were falsified and 19 were still open when the journal ends. Most were settled by cheap probes; 14 needed a full run.
It also moved its own goal. It met the target twice and raised it each time, through a tool that only allows raises, and only once the target has held at two evaluations in a row.
Some of what it found was about itself: it found and repaired bugs in its own implementation. A falsified idea is a result here too, and the registry treats it as one.
What I’d do differently
I’d explore better ranking heuristics. The agent comes up with hypotheses far faster than it can test them, about four a day while the registry was in use, and only 14 were settled by a full run. With generation outpacing testing like that, choosing what to test next is where most of the value comes from.
Right now it ranks on four things: whether a hypothesis’s dependencies are settled, how many others it unblocks, how cheap it is to test, and spreading attempts across families. That leans towards breadth-first search and can discourage digging deep into a promising chain. The next step would be to build alternative ranking models from the original run’s history and backtest them against the hypotheses it actually settled, to see whether any weighting reaches the North Star sooner.
I’d also like to try grading hypotheses with a decision model like Jev, the way I grade CVs in another project: a large bank of a hundred or more questions, each answered for every hypothesis, turning the agent’s judgement into quantifiable metrics. Those scores could then train a selection classifier, backtested against the run’s history.
The public repository is a generalised version of the agent I used, under an MIT licence.