docketrouter

Judging outside models on HLL: how it works, what it costs, what stays secret

Splits (per version, e.g. hll-0.1-tx)

split gold who runs it purpose
public (20% of the bank) published anyone, free, via CLI or the open-source runner self-testing, reproducibility, catching grader bugs
private (80%) secret, never leaves our box only DocketRouter, server-side the leaderboard
Private items rotate: every new version re-draws the split; retired private items become public. Item ids are salted hashes so a model can't tell which split it's in.

What is secret

  • Private item text and gold.
  • The exact judge prompt version and panel composition (we publish the axes, weights and the hallucination cap; not the wording).
  • Nothing else: methodology, grader code, calibration numbers and the public split are open.

How a submission runs

  1. Submitter signs in, gives a name, an OpenAI-compatible endpoint (base URL + key, or an OpenRouter model id) and a version string.
  2. We call the endpoint from our box for every private item (raw mode; juiced optionally), store answers with hashes, grade with the fixed panel, verify every citation.
  3. Result: overall score, per-category scores, hallucination-cap count, and per-item pass/fail (never the item text). Published on the leaderboard as community · <name> with the date and version. The submitter can keep a run private for 7 days before it publishes (embargo), but cannot cherry-pick: every completed run is listed.
  4. One official run per (submitter, model version) per 30 days. Endpoint must answer >= 95% of items or the run is void.

Pricing (behind Stripe, dormant until billing is live)

  • Public split: free.
  • Private official run: $249 per model version (covers ~$60 of judge + verification cost at 500 items, plus operations). Featured models we run ourselves: free, funded by us.
  • Enterprise: private unpublished evaluation on a custom item bank (their practice area / state), quoted.

Anti-gaming

  • Answers are graded blind (judges never see the submitter or model name).
  • Private items unseen by any public model; rotation on every version.
  • Endpoint is called with random item order and a per-run nonce in the system prompt; identical answers across runs flag replay.
  • We keep raw answers; disputes get a human re-grade of the disputed items.

Implementation (v0 shipped)

  • scripts/hll-split.ts: assigns every item to public/private deterministically (salted hash), writes data/hll/splits/<version>.json, and exports the public split with gold to data/hll/public/<version>.jsonl.
  • POST /api/v1/hll/submissions (signed-in): queue a run; GET /api/v1/hll/submissions/:id: status + results.
  • scripts/hll-runner.ts (box, cron): picks queued submissions, runs the private split against the endpoint, grades, stores results.
  • /hll shows community runs under the featured models.