Evidence at a glance
From One Ranking to Judgments on Individual Pages
Datalab has released OmniParseBench, an open OCR benchmark for document parsers. It is not a new model but an offline evaluation harness: run a parser on a PDF page or image, then check its output against page-level yes-or-no tests. The benchmark contains 16,288 tests across 2,937 pages, 2,343 documents, and 92 languages, and both its dataset and code are public.
The design addresses a practical problem: a leaderboard score does not necessarily tell a team whether a system suits its own documents. OmniParseBench breaks parsing quality into checkable judgments and lets users inspect results by tag. That makes it closer to an auditable test suite than a ranking that merely orders systems.
How Page Tests Become a Score
Each test checks a specific fact on one page. It may ask whether a piece of text or a number appears, whether text A comes before text B, whether a line appears exactly a given number of times, whether a table cell sits under specified headings and beside expected neighbors, or whether a block is a heading, body text, or a table. The five test types feed three capability families—text, tables, and layout—and the headline score is the mean of those family scores.
A shared unit makes these capabilities easier to combine, and verdicts come from public scoring code rather than another model acting as a judge. Numbers must match digit for digit, although 1,240 and 1240 count as equivalent. Equations are compared by their KaTeX rendering, so \frac{a}{b} and \dfrac{a}{b} can be treated as equivalent. Each decision can be a pass, fail, unsupported, or error, with the latter two excluded from scoring. The headline therefore measures pass rates among tests included in the score.
Tags Turn an Average into a Diagnostic View
Results can also be filtered across dimensions such as structure, rendering, content role, writing system, and language. Tags describe what a test concerns; they do not determine its verdict or alter the headline score. Their purpose is to let teams calculate pass rates for particular kinds of material. For example, the table tag covers 6,767 tests across 1,000 pages and 875 documents. English accounts for 10,089 tests, while non-English languages account for 1,931 tests across 529 pages and 321 documents.
These slices are more useful for engineering diagnosis than a single aggregate: even when a parser looks stable overall, a team can inspect its performance on tables, a writing system, or a particular language. But slices are constrained by their sample sizes. Non-English tests are substantially fewer than English tests, so a score for a subgroup cannot automatically represent every document type in that language or remove uncertainty caused by limited coverage.
Close Scores Still Require a Clear Evidence Trail
At launch, Gemini 3.8 Flash led with 92.9, followed by Datalab's own Datalab accurate at 92.7. The top three systems were within 1.3 headline points of one another. Under this benchmark's scoring method, the leading systems have similar aggregate results. That alone does not show that they will perform similarly on a particular company's scans, forms, or low-quality images.
The benchmark records ten ways in which test answers were derived, including publisher JATS XML, PDF text layers, author-provided pages, olmOCR-bench tests, model panels, and human review. Recording provenance helps readers ask how an answer was established, while also showing that tests do not all follow one verification path. There is also a conflict of interest to account for: Datalab sells a parser and its own system appears on the leaderboard. Public code and traceable provenance make scrutiny possible, but do not erase the effects of design and coverage choices.
Useful for Screening, Not a Substitute for Acceptance Testing
For engineering teams, the route to trying OmniParseBench is fairly direct: it installs with uv on Python 3.11 or later, and scoring runs locally without API keys. Running parsers that depend on vendor services may still require the relevant credentials. A team can use the public benchmark to rule out clearly unsuitable candidates, then focus on tags and failure cases resembling its own documents instead of comparing leaderboard positions alone.
A deployment decision still requires a representative in-house document set and acceptance criteria. Teams should check whether the benchmark adequately covers their target languages, layouts, table structures, and scan quality, then consider whether the remaining score represents their needs after unsupported and error outcomes are excluded. OmniParseBench offers a more detailed and inspectable comparison tool. It narrows the gap between an aggregate score and specific failure types, but it does not decide which errors a team can accept.