Humanity's Last Lawsuit (HLL) v0.1 - Methodology
Goal. Measure whether a model can reason through a real appellate case it has not seen with the parties removed: identify the dispositive issue, state the controlling standard, cite real authority, apply it, and predict the disposition - and not invent cases.
1. Item construction ("blind opinions")
- Source: published Texas appellate opinions with full text (v0.1 seed: 18 opinions - Supreme Court of Texas + 5th COA, May 2026 snapshot).
- Each item = one opinion + one question. 3–5 items per opinion probe different decisions the court made (dispositive issue & disposition; standard of review; procedural vehicle/jurisdiction; remedy; sub-issues).
- Redaction: parties/people →
[A],[B]…; judges →[Judge]; product/brand names →[Product]. Courts, statutes, rules and other cited cases stay. Facts carry no docket numbers or identifying dates. - Gold = { outcome (enum), issue, standard, authorities (verbatim reporter cites the court relied on), application, procedure } + ≥4
evidence_quotes: verbatim passages from the opinion that support each gold field.
2. Audit (mechanical, scripts/hll-audit.ts)
Every item must pass before it can be graded against: schema-valid; every evidence quote found verbatim in the opinion (whitespace-insensitive); every gold authority appears in the opinion; no redacted surname/company appears in facts or question; evidence covers outcome, standard, application. An adversarial reviewer agent then tries to refute the gold against the opinion text and fixes or deletes the item. Only audited items count.
3. Grading (src/lib/hll/grade.ts)
| Axis | Weight | Method |
|---|---|---|
| outcome | 2 | deterministic: disposition enum vs answer (synonym stems; 0.5 credit for correct direction) |
| authority | 2 | deterministic: fraction of gold reporter cites present in the answer (normalized) |
| standard | 2 | 3-judge LLM panel, rubric 0–2, median |
| application | 2 | 3-judge panel |
| issue | 1 | 3-judge panel |
| procedure | 1 | 3-judge panel |
| hallucination | gate | every citation in the answer must appear in the opinion or verify FOUND in CourtListener; any citation CourtListener affirmatively reports as not existing → item score capped at 0.25 |
Score = weighted mean ∈ [0,1], then hallucination cap. Judges see the gold and the answer, never the model identity. Verifier is asymmetric: only an affirmative 404 counts as fake; throttling/outage → unverified (no penalty), because penalizing real cases is worse than missing a fake one.
4. Calibration (scripts/hll-calibrate.ts)
Run before publishing any leaderboard: the gold answer itself must score ≥0.90 and a deliberately wrong answer ≤0.30 across the judge panel. v0.1 result (2 items, 3 judges): gold 1.00 / wrong 0.00.
5. Modes
Every model is run raw (system prompt only) and juiced (DocketRouter retrieval + citation verification injected). The delta is the product claim.
6. Limitations (v0.1)
- 18 source opinions → tens of items, not thousands; all Texas; all 2026 (post-training-cutoff for most models, which is a feature for contamination but limits breadth).
- LLM judges can be fooled by fluent wrong answers; the deterministic axes (outcome, authority, hallucination) anchor the score.
- CourtListener misses ~10% of real citations (per our research sweep); hence
unverifiednever penalizes. - Scaling path: Free Law Project bulk data → all Texas appellate opinions (~600k); item authoring is agentic and audited, so throughput is bounded by review cost, not writing.