Benchmark Evaluation
QEDRank is a benchmark evaluation platform for formal mathematics. It measures how well language models
solve a selected corpus of formal mathematics problems, with Lean-verified, statistically analysed
results, and it compares Lean formalisation benchmarks with each other.
Two workflows
∀
Workflow 1
Evaluate Models
Run one or more language models on a selected Lean 4 benchmark corpus. Each model generates
formalizations, which are checked by Lean and compared using correctness, auditability, reliability,
efficiency, and difficulty-based performance.
- Verified proof rate (pass@1) — primary metric
- Audit-clean rate — proofs passing axiom and forbidden-import checks
- Lean compilation success rate
- Statement fidelity rate
- Statement-match classification
- Pass@k performance for k = 1–16
- Attempt statuses and failure classes
- Abstention, timeout, unresolved, and infrastructure-error rates
- Latency, Lean compilation time, and token usage
- Estimated cost per attempt and per verified proof
- Performance by difficulty and subject area
- Performance on the common-problem subset
- 95% confidence intervals and exact denominators
Open Evaluate Models ↗
∃
Workflow 2
Compare Benchmarks
Compare Lean formalization benchmarks side by side to understand their scope, mathematical coverage,
difficulty, statement quality, proof complexity, and model performance.
- Number of problems as published
- Number of problems measured
- Release date, source, task type, and judging method
- Mathematical domain and MSC subject coverage
- Largest subject area
- Difficulty-level distribution
- Research-level and open-problem share
- Theorem, definition, lemma, and construction-problem types
- Theorem signature complexity
- Statement length, including used definitions
- Longest 10% of statements by token length
- Theorems per file and definitions per file
- Reference-proof availability and quality
- Proof length, proof-term size, tactics, and automation share
- Statements with contradictory or vacuous hypotheses
- Linter warnings and reference defects
- Model verified-proof rate by benchmark
- Coverage differences between models and benchmarks
Open Compare Benchmarks ↗
More tools
-
Landscape
Benchmark Landscape
ProofBench and the Lean benchmarks on one chart: benchmark size against the best reported success rate.
Open Benchmark Landscape ↗
-
Single problem
Convert a Problem
Send one statement to every selected model and compare the Lean 4 answers side by side, with latency, tokens and your own rating.
Open Convert a Problem ↗
How results are reported
- A challenge file states the theorem and leaves its proof as
sorry. That sorry is the task, and every answer is checked against the untouched statement.
- Verified success is the strict primary endpoint: the answer compiles, uses no
sorry and no new axioms, and leaves the statement unchanged.
- Every rate is reported as numerator over eligible denominator at pass@1, with a 95% paper-cluster bootstrap interval.
- Statement fidelity, compilation and audit are reported alongside verified success and are never folded into one score. No weighted composite is computed.
-- Prove that the square root of 2 is irrational.
theorem sqrt_two_irrational : Irrational (√2) := by
sorry
An illustrative challenge file. The statement is fixed; a model has to replace the sorry with a proof that Lean accepts.