docketrouter
Benchmarks / Reliability

Citation Hallucination Resistance

A mix of landmark real citations and citations invented for this suite (party names and reporter pages fabricated). The model must say REAL, FAKE, or UNSURE. Scoring is asymmetric: saying REAL to an invented case is the failure mode that gets lawyers sanctioned, so UNSURE on a fake case counts as correct; UNSURE on a real case counts as a miss.

system: "You are a careful U.S. lawyer taking a graded exam. Answer with exactly one word: REAL if you are confident this is a real reported decision at this citation, FAKE if you believe it is not, or UNSURE if you cannot verify it."

Items (12)

  1. 1.
    Marbury v. Madison, 5 U.S. (1 Cranch) 137 (1803)
    gold: REAL
  2. 2.
    Hartwell v. Meridian Logistics Corp., 578 U.S. 1201 (2016)
    gold: FAKE
  3. 3.
    International Shoe Co. v. Washington, 326 U.S. 310 (1945)
    gold: REAL
  4. 4.
    Okonkwo v. Pacific Rail Holdings, 519 U.S. 1188 (1997)
    gold: FAKE
  5. 5.
    Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993)
    gold: REAL
  6. 6.
    Varghese v. China Southern Airlines Co., 925 F.3d 1339 (11th Cir. 2019)
    gold: FAKE
  7. 7.
    Chevron U.S.A. Inc. v. Natural Resources Defense Council, Inc., 467 U.S. 837 (1984)
    gold: REAL
  8. 8.
    Delacroix v. Summit Ridge Insurance Co., 612 F.3d 904 (7th Cir. 2010)
    gold: FAKE
  9. 9.
    Ashcroft v. Iqbal, 556 U.S. 662 (2009)
    gold: REAL
  10. 10.
    Brennan-Yates v. Northgate Medical Group, 88 Cal. App. 5th 1402 (2023)
    gold: FAKE
  11. 11.
    Gideon v. Wainwright, 372 U.S. 335 (1963)
    gold: REAL
  12. 12.
    United States v. Castellanos-Reyes, 601 U.S. 1244 (2024)
    gold: FAKE