We don't know how to run autoresearch

Well, we don’t know how you want to run autoresearch. But we’re building powerful primitives that let you run all sorts of experimentation.

Say you’ve run a thousand experiments, had an LLM judge score them, and want the ones it failed, worst first. With hiloop you can.

Define what a judgment looks like, once. The fields are entirely yours, and you promote the ones you’ll filter and sort on into fast typed columns:

hiloop annotation-schema register quality \
  --json-schema '{"type":"object","properties":{
    "verdict": {"enum": ["pass", "fail"]},
    "score":   {"type": "number"},
    "reason":  {"type": "string"}
  }}' \
  --promote verdict:str \
  --promote score:f64

Write one onto a run:

hiloop annotations add --run 01JZ… --schema quality \
  --data '{"verdict":"fail","score":0.2,"reason":"reward hacking on the eval set"}'

Then ask for the ones you care about, in the same SQL you query everything else with:

SELECT run_id, verdict, score
FROM events
WHERE signal = 'annotation'
  AND name = 'quality'
  AND verdict = 'fail'
ORDER BY score ASC
LIMIT 50;

One primitive

The conventional move is to build features: an evals page, a labeling tool, a feedback widget, a dashboard for each. They’re useful, but each one is a guess about what you’ll want to measure, and the data tends to end up in side tables that don’t join cleanly to the telemetry it’s about.

So we built one thing instead. An annotation is a durable, structured fact you attach to your telemetry. You register a schema, which is just a name and a JSON Schema for whatever fields you want, then attach an annotation to a single event, a time window, or a whole run. It lands in the same place as your events, and you query it with the same SQL. We don’t define what’s in it. A score, a verdict, a reviewer’s name, a free-text note: those are your fields, not ours.

One primitive is enough for three teams with three different jobs:

Scoring ML experiments. Register a quality schema with whatever a good score means to you. Have your eval harness, or an LLM judge, annotate every run. “Compute the judgment once, filter on it forever” is the point: triaging ten thousand runs becomes a WHERE clause, not ten thousand re-evaluations.

Flagging a performance regression. Register a regression schema. When a run blows the latency budget, annotate it with the offending commit in the payload. “Show me every run since Tuesday that regressed, grouped by cause” is one query against data you already have.

Auditing for security. Register a review schema. A reviewer, or an automated rule, tags the traces that touched a sensitive credential or hit an unexpected host. “Find the runs flagged in the last audit” is a predicate, and it sits right next to the raw trace it’s flagging. See the current security posture.

Making annotations powerful

A flexible primitive is only useful if the shape is right. Here’s how we implemented annotations:

• Schemas are validated and versioned. A schema is registered up front and payloads are checked against it at ingest, so a field you aggregate on is actually there and actually typed. Schemas are immutable and versioned: a breaking change is a new version, not a silent reinterpretation of last month’s data.
• It targets a point, a range, or a run. The same primitive marks one event, a window of time, or an entire experiment, which is the full range of things you actually want to say something about.
• It carries run lineage. An annotation is stamped with the same lineage-tree position as the events it annotates, so comparing two branches of an experiment includes the judgments, not just the raw events.
• Re-scoring is honest. A correction or a re-run writes a new annotation, never mutates the old one. Queries get the latest by default, and the full history when you ask for it. Nothing is overwritten.

And it stays fast

The obvious worry about “annotate everything, filter later” is that it falls over at scale. A generic key-value blob is exactly the thing that’s slow to filter across millions of rows.

So you tell us which of your fields you’ll filter and sort on, and we lift those out of the JSON blob and store them as real typed columns in the columnar store, next to your events. A query can then use column statistics and bloom filters to skip the data it doesn’t need, instead of parsing every record to test one field. The fields you don’t single out stay in flexible JSON, where flexibility costs nothing.

The difference is large. On a benchmark of 20 million rows, filtering and sorting by one of these typed fields returned in under 100 milliseconds. The same field left in a raw JSON blob took around 800. Counting results by category was starker: roughly 50 milliseconds against 1.1 seconds. A point lookup lands in single-digit milliseconds.

Because the primitive makes no assumptions about your fields, we get to pour effort into making it fast for whatever you choose, instead of guessing which fields matter and optimizing only those.

Point it at anything

Annotations are a foundation, not a finished product. A human-review workflow, a scoring dashboard, an automated audit: those are things you build on top, shaped the way you work. The questions you’ll ask of your experiments are ones we can’t enumerate, and you shouldn’t have to wait for us to ship a button for each one.

If you’re ready to leverage your experiment telemetry, we’d like to talk. Email us at founders@hiloop.ai.