Vaibhav Bajpai

Benchmark Evaluation

QEDRank is a benchmark evaluation platform for formal mathematics. It measures how well language models solve a selected corpus of formal mathematics problems, with Lean-verified, statistically analysed results, and it compares Lean formalisation benchmarks with each other.

Two workflows

Workflow 1

Evaluate Models

Run one or more language models on a selected Lean 4 benchmark corpus. Each model generates formalizations, which are checked by Lean and compared using correctness, auditability, reliability, efficiency, and difficulty-based performance.

  • Verified proof rate (pass@1) — primary metric
  • Audit-clean rate — proofs passing axiom and forbidden-import checks
  • Lean compilation success rate
  • Statement fidelity rate
  • Statement-match classification
  • Pass@k performance for k = 1–16
  • Attempt statuses and failure classes
  • Abstention, timeout, unresolved, and infrastructure-error rates
  • Latency, Lean compilation time, and token usage
  • Estimated cost per attempt and per verified proof
  • Performance by difficulty and subject area
  • Performance on the common-problem subset
  • 95% confidence intervals and exact denominators
Open Evaluate Models ↗

Workflow 2

Compare Benchmarks

Compare Lean formalization benchmarks side by side to understand their scope, mathematical coverage, difficulty, statement quality, proof complexity, and model performance.

  • Number of problems as published
  • Number of problems measured
  • Release date, source, task type, and judging method
  • Mathematical domain and MSC subject coverage
  • Largest subject area
  • Difficulty-level distribution
  • Research-level and open-problem share
  • Theorem, definition, lemma, and construction-problem types
  • Theorem signature complexity
  • Statement length, including used definitions
  • Longest 10% of statements by token length
  • Theorems per file and definitions per file
  • Reference-proof availability and quality
  • Proof length, proof-term size, tactics, and automation share
  • Statements with contradictory or vacuous hypotheses
  • Linter warnings and reference defects
  • Model verified-proof rate by benchmark
  • Coverage differences between models and benchmarks
Open Compare Benchmarks ↗

More tools

How results are reported

-- Prove that the square root of 2 is irrational.
theorem sqrt_two_irrational : Irrational (√2) := by
  sorry

An illustrative challenge file. The statement is fixed; a model has to replace the sorry with a proof that Lean accepts.

Try QEDRank

Evaluate models or compare benchmarks in the browser at qedrank.vercel.app.

Open QEDRank ↗