hiloop blog
Notes on automated research.
Search is enough
Two stock coding agents with 50 B200s ran 4,188 experiments on Karpathy's autoresearch benchmark and reached a confirmed 0.9016 val_bpb, past the published state of the art, with almost no scaffolding.
We don't know how to run autoresearch
We don't know how you'll run autoresearch, and you don't fully know either until you're in it. So we didn't ship a fixed set of eval features. We shipped one primitive: user-defined annotations you attach to any point, range, or run, and query like any other telemetry.
Decouple, decouple, decouple
What an autoresearch loop needs changes constantly. So we decouple the harness, the compute, and the data from one another, and couple exactly one thing: experiment state and observability.
Telemetry and a query engine designed for autoresearch
Agent harnesses change every week, so you can't pin a fixed schema or a fixed set of queries to a moving target. We capture low and reconstruct high: one raw record of everything an agent did, and a query engine that reshapes it into whatever the research needs.
Why we're building hiloop
Karpathy's autoresearch showed that an agent with a verifiable metric can improve a system on its own. We're building the infrastructure to do that at scale.
