Benchmark methodology

Every number published on the Performance page comes from one run of one harness, and the file it comes from is committed to the repository. This page describes how those numbers are produced and what they do and do not claim.

Reproducing a run

pip install -e ".[bench,all]"

python benchmarks/download_corpora.py                # ~257 MB, five corpora
python benchmarks/06_mega.py --legacy-timeout 5400 --publish

# then regenerate the published tables from the measurements
python tools/gen_benchmark_tables.py --write

--publish writes benchmarks/results/06_mega.<platform>.json and refuses a file that mixes runs: every corpus must have been measured by that same invocation, on a clean tree, and the result must satisfy the schema-v3 evidence contract. It never writes benchmarks/results/06_mega.json — that name holds the preserved macOS artifact, whose bytes are pinned by three separate digests.

tools/gen_benchmark_tables.py --check verifies that every published table still matches the committed measurements, and is asserted by tests/test_release_hygiene.py — so a table cannot drift from the numbers behind it without a test failing.

What is measured

Metric

How

Ingest wall

create_db() in a fresh subprocess, wall clock around the call

Peak RSS

parent polls the child’s RSS (plus its children) every 50 ms

On-disk size

recursive size of the finished database

Spatial qps

5 000 regions sampled from the corpus’s own per-seqid spans

Batched extraction

children_batched(..., format="arrow") over 5 000 anchors

Region sampling uses a fixed seed (20260501), recorded in the results file, so two runs sample the same regions.

Corpora

The five canonical human-genome annotations, downloaded from their primary sources by benchmarks/download_corpora.py: GENCODE v49 basic (both the GTF and the GFF3 release of the same biology), RefSeq GRCh38.p14, MANE v1.5 (Ensembl IDs) and CHESS 3.1.3.

Fairness

  • Both engines get merge_strategy="create_unique". This is not cosmetic: under the default "error" both refuse RefSeq and MANE outright, and timing gffbase under one duplicate-ID policy against gffutils under another would compare two different workloads on exactly the axis that decides whether the run completes.

  • The primary GENCODE GTF comparison disables parent inference for both engines, which makes the authored records directly comparable. Separate non-headline controls exercise default inference and a parent-stripped GTF; their results are never mixed into the primary namespace.

  • Each engine runs in its own subprocess, so peak RSS is attributable and neither inherits the other’s warm caches.

  • Both write to the same filesystem.

What is not comparable

  • Spatial and batched columns are gffbase-only. gffutils has no spatial index and no batched API, so there is nothing to put in the other column. The region() throughput comparison in the narrative sections is like-for-like; the per-corpus qps figures are not a head-to-head.

  • Query benchmarks run against a warm page cache, immediately after the ingest that built the database. They measure steady-state query throughput, not cold-start.

Capped runs, and why there are no extrapolated numbers

The default-inference GENCODE v49 GTF control can exceed a practical run window: GTF has no explicit parents, so every gene and transcript row must be invented through the legacy Python/SQLite path. Every comparator arm therefore has a --legacy-timeout safety valve (default 90 minutes), including arms that normally finish well below it.

A capped run is censored, not measured. In a current result its state is timed_out, cap_seconds records the safety valve, and wall_seconds is null. It produces neither a speedup nor a speedup floor. Tables render only censored at 90 min in the comparator column.

The preserved 2026-08-15 Mac artifact predates this contract and uses schema v2 names such as wall_seconds_lower_bound. Its bytes remain historical evidence, but the renderer treats those fields only as censoring metadata and never repeats the old > N× claim.

This replaced a hardcoded wall_seconds = timeout × 2.0. That factor had no measurement behind it, and it was the sole source of the previously published “≥ 2 hr 30 min” legacy wall and “≥ 32×” headline. A number produced by multiplying a timeout is not a result, and printing one beside real measurements invites a reader to distrust all of them.

The table generator refuses to render a row whose legacy run timed out but still carries a wall time, so the defect cannot come back quietly.

Uncertainty

The headline sweep is n = 1 per cell: legacy ingest of the large corpora takes hours, and repeating the whole sweep three times is not a good use of a day. Run-to-run spread is instead measured on the two cheap corpora, which share the same code paths:

python benchmarks/06_mega.py --repeats 5 --only mane --only chess

--repeats applies to the two cheap in-process measurements — the spatial sweep and the batched extraction — and not to the ingest walls, where one GENCODE job already runs over an hour: 71 minutes for GENCODE GFF3, of which the two ingests are about 21 and signature verification the rest. (The legacy ingest itself is 7–10 minutes on the published arms; it is the unpublished default-inference GTF arm that does not finish inside 90.) When it is greater than 1, the first pass is discarded as a warm-up: a smoke test measured a 6.7× max/min ratio that was entirely cold page cache, and a spread that is really a cold-start artifact is worse than no spread, because it gets published as measurement uncertainty. After the warm-up the same measurement spreads under 5%.

Treat that spread as the measurement uncertainty for every cell. Where a result file reports n = 1 it carries a value key and deliberately no median, so a renderer cannot present a single sample as a central tendency.

Equal work, or no ratio

A speedup is recorded only when both completed databases have the same strict database-signature-v3. The signature covers every logical segment, canonicalized per-segment attributes (including empty flags and value order), direct relationships, minimum-depth closure, feature counts, and the feature-type histogram. Feature counts remain a useful diagnostic but are not accepted as a correctness proof by themselves.

Attribute key order is deliberately canonicalized lexically because GFF/GTF key order is non-semantic and engines may expose inferred attributes differently. Value order is retained within each key, so reordering values changes the signature while reordering keys does not.

Two further normalizations exist because the comparison must not mistake a difference in representation for a difference in content. Both were found by whole-genome runs, and each had cost a corpus its published speedup.

The closure is derived, not read. gffutils stores relationships in a relations table, but what it records at level = 2 is not a transitive closure: it inserts, for each feature, the children of its children — one hop, at a fixed level, with no iteration to a fixed point. On a four-deep chain it records five ancestor–descendant pairs and omits the sixth. gffbase stores the real closure, so comparing the two reported RefSeq as differing by 3,218 pairs and GENCODE GFF3 by 108 — on hierarchies whose direct edges agreed exactly. The closure is a function of those edges, so the signature now computes it from them for the comparator rather than trusting the stored cache.

A comma followed by a space is not a separator. GFF3 says an unescaped comma separates values and that a literal comma must be percent-encoded. gffbase follows that; gffutils deliberately does not, keeping , `` inside the value so an unescaped ``description=kinase, subunit 1 survives as one value. Ten CHESS genes record two names as gene_name=ADAM6, RPS8P1, which reported that corpus as a 20-row divergence. The signature therefore applies one rule to both engines, and it is the comparator’s coarser one: re-splitting on every comma would shatter 2,294 correctly escaped CHESS descriptions in order to line up twenty gene names. The parse-policy difference itself is a declared API deviation, recorded in tests/parity/deviations.toml and pinned by a differential test — it belongs there, not hidden inside a benchmark digest.

What the ingest numbers do and do not say

Two properties of the ingest measurement bound how much weight a single figure carries, and both were measured rather than assumed.

Ingest cost tracks attributes. Cost follows the number of attribute values far better than the number of features. Dividing each corpus’s attribute values by its published ingest wall:

corpus

attribute values

per feature

gffbase attr/s

legacy attr/s

MANE v1.5

6,711,851

12.8

429,800

148,700

GENCODE v49 (GFF3)

108,289,882

17.9

560,900

185,200

GENCODE v49 (GTF)

96,291,901

15.9

446,700

232,400

RefSeq GRCh38.p14

55,104,379

11.2

441,200

140,900

CHESS 3.1.3

7,063,651

2.6

190,800

52,700

On the four attribute-dense corpora – 11.2 to 17.9 attributes per feature – gffbase moves 430,000 to 561,000 attributes per second, against 141,000 to 232,000 for gffutils. The rate falls to 190,800 on CHESS, where 2.6 attributes per feature leave the fixed per-feature work to dominate; the comparator’s per-feature cost is higher still there, which is why CHESS has the largest ratio. GENCODE GTF has the smallest because gffutils runs at its fastest on it: with parent inference off, its GTF path is a plain bulk insert. Before 0.3.0 gffbase held 131,000 to 168,000 attributes per second on those four, level with the comparator; the rebuilt ingest removed the Python object it built per record.

Attribute values are counted from the corpus files themselves – semicolon fields in column nine, with comma-separated GFF3 values expanded, which reproduces the per-feature ratios above – rather than from either database, so the figure does not depend on how either engine chose to store them.

More threads buy little. Measured on 0.3.0, MANE ingest took 15.8 to 16.1 s at 8, 32 and 128 DuckDB threads alike, while peak RSS rose from 1.0 GiB to 1.2 and 1.6 GiB – which is why gffbase uses at most 8 threads by default. The parse and per-record work runs on one core; the rest, DuckDB’s append and index builds, gained nothing past 8 threads here.

Before the rebuild the picture was the same. One cluster campaign run swept gffbase 0.2.0rc1 over the five corpora at 1, 2, 4, 8 and 10 threads – all twenty-five figures from that single run, since thread scaling is only meaningful within one:

corpus (0.2.0rc1)

1 thread

10 threads

speedup

MANE v1.5

41.6 s

36.0 s

1.16x

CHESS 3.1.3

99.1 s

94.6 s

1.05x

RefSeq GRCh38.p14

425.4 s

340.9 s

1.25x

GENCODE v49 (GTF)

614.4 s

498.3 s

1.23x

GENCODE v49 (GFF3)

657.2 s

534.8 s

1.23x

The comparator is single-threaded, so neither engine’s figure should be read as a parallel result.

Ingest uncertainty falls with corpus size. Repeating the ingest alone with the same wheel, on the same eight pinned cores of an otherwise idle socket:

corpus

runs

repeated gffbase ingest

spread

published figure

MANE v1.5

5

14.9 s – 15.2 s

2.3%

15.6 s

CHESS 3.1.3

5

36.0 s – 36.5 s

1.4%

37.0 s

RefSeq GRCh38.p14

3

118.8 s – 119.3 s

0.5%

124.9 s

GENCODE v49 (GTF)

3

208.2 s – 208.7 s

0.3%

215.6 s

GENCODE v49 (GFF3)

3

191.1 s – 192.5 s

0.7%

193.1 s

Every corpus repeats to within a few percent, the whole-genome ones to under one. The published single figures sit 0.7 to 4.9 percent above the probe’s medians. The canonical run shared the machine with test suites running on the other socket for its first four corpora; the fifth, GENCODE GFF3, measured alone, sits closest. Either way the published ratios understate gffbase slightly rather than flatter it. (0.2.0rc1, probed on ten pinned cores, spread 20 percent on CHESS.)

The harness measures each ingest once, while the query phases run five times and report their spread. Closing that asymmetry is future work; until it is closed, a published ingest ratio near 1.0 should be read as “comparable”, not as a ranking.

Publishing a run

A sweep writes to benchmarks/out/ (gitignored). The committed file the published tables are generated from is benchmarks/results/06_mega.json, and copying between them used to be an undocumented manual step — so a fresh run could sit on disk while the docs kept rendering the previous measurement.

python benchmarks/06_mega.py --legacy-timeout 5400 --publish
python tools/gen_benchmark_tables.py --write

--publish copies the finished run into place; the generator then rewrites every table from it. tests/test_release_hygiene.py fails if a table stops matching the numbers behind it.

Provenance

Every results file embeds the environment it was produced in — CPU model, core count, RAM, OS, Python, DuckDB, PyArrow, gffutils and gffbase versions, rustc, the git commit and whether the working tree was dirty:

import json
env = json.load(open("benchmarks/results/06_mega.json"))["environment"]

None of this was recorded before 0.2.0. The published numbers carried their hardware and versions only as hand-typed prose, which said “gffbase 0.1.0” throughout the 0.2.0 development cycle — so nothing in the repository could have detected a regression.

Disk

A full sweep is disk-bound before it is CPU-bound: the five corpus pairs total roughly 38 GiB. Each pair is purged as soon as its numbers are recorded, which holds the peak to about 16 GiB. --keep-db KEY retains one for later stages; --no-purge keeps everything and needs the full 38 GiB. Set GFFBASE_BENCH_OUT to run against another volume. The harness checks free space before each corpus and refuses to start one it cannot finish.