Troubleshooting¶
Organised by the error you are looking at. For “how does this work” questions, see the FAQ.
Ingest¶
DuplicateIDError: Duplicate ID cds-NP_...¶
The file uses the split-CDS convention: several lines sharing one ID=,
describing one discontinuous feature. RefSeq and MANE both do this.
merge_strategy defaults to "error" — the same default gffutils uses, and
it raises on these same files — because there are two defensible readings and
picking one for you would give you a whole-genome answer you did not choose:
create_db(path, "out.duckdb", merge_strategy="create_unique") # renamed, gffutils-style
create_db(path, "out.duckdb", mode="strict") # one discontinuous feature
EmptyInputError: no features in ...¶
The source yielded no feature lines at all: it is empty, holds only headers,
comments or a FASTA section, or is not GFF3/GTF text (a binary file reads this
way too). gffutils raises the same error. Check the path first. Since 0.2.1
gzip is recognized by content, so a .bgz or extensionless gzip file is
decompressed rather than read as binary; before that it silently produced an
empty database.
The same error with the transform rejected all N features means a
transform returned a false value for every record.
GFFFormatError on a file other tools read¶
You are almost certainly applying the strict rules: create_db(mode="strict"),
or the low-level gffbase.parser.parse_gff, whose default is
validation="ncbi", strict=True. Real annotations break the spec routinely;
create_db’s default mode="compat" and DataIterator (compat since
0.2.1) record violations in warnings instead of raising, and
parse_gff(path, validation="gffutils", strict=False) does the same. The
exception carries a pointer into the file:
import gffbase
try:
gffbase.create_db("annotation.gff3", "out.duckdb", mode="strict")
except gffbase.GFFFormatError as exc:
print(exc.line_no, exc.kind, exc.message)
To audit rather than abort, keep the strict rules but downgrade the action:
db = create_db(path, "out.duckdb", validation="ncbi", on_error="warn")
for w in db.warnings:
print(w["line_no"], w["kind"], w["message"])
Ingest used a lot of memory¶
Two different costs get confused here, so take them apart.
Ingest into a file is bounded by default since 0.3.0: the parser streams its input, DuckDB runs under a 512 MB budget raised only for the step that needs more, and at most 8 threads are used. Peak RSS is 1.0 GiB (MANE) to 2.8 GiB (GENCODE GFF3) on the benchmark corpora (Memory and performance tuning has the table). If yours is much higher, one of these is usually why:
The database is in memory (
":memory:"). Its tables cannot be evicted, so it holds everything, and the budget does not apply.pragmas={"memory_limit": ...}orGFFBASE_THREADSis set. Your limit replaces the budget, and each thread past eight adds buffers.
For comparison, gffutils – which only ingests, writing through to SQLite
on disk – peaks at 107–190 MB on the published corpora.
Exhaustive validation is what makes the published figures large: the numbers
in the performance tables span 2.9–61.9 GB because those runs call
validate(level="full", sample=None), which re-reads every stored attribute.
No default path does that — validate_db defaults to sample=200 and
the CLI never overrides it. If you asked for exhaustive validation on a
six-million-feature annotation, budget tens of gigabytes; otherwise you will not
see these numbers.
To set DuckDB’s threads for a whole script (the default is at most 8):
GFFBASE_THREADS=4 python your_script.py
The ingest died and left a file behind¶
It should not have. Ingest is atomic: GFFBase writes to
<dbfn>.gffbase-building.<pid> and only renames on success, so a failed run
leaves the original untouched and the scratch file is removed, together with
any spill directory DuckDB created beside it (<scratch>.tmp/; before 0.2.1
that directory was left behind). If you find a .gffbase-building.* file or
directory, the process was killed hard (SIGKILL, power loss) and it is safe
to delete.
Note that atomicity relies on os.replace being atomic, which holds within
one filesystem. If dbfn is on a different mount than its temporary
neighbour, the guarantee is weaker.
Querying¶
region() returns nothing, but the feature is there¶
Check the seqid spelling. chr1, 1 and NC_000001.11 are three different
sequences as far as the database is concerned — GENCODE uses chr1, Ensembl
uses 1, RefSeq uses NC_000001.11.
print(sorted(db.seqids())[:5])
children() returns nothing for a feature that has children¶
Either the file has no Parent= edges to build the hierarchy from, or you are
asking for the wrong level. level=1 is direct children only:
list(db.children("gene1", level=None)) # everything below, any depth
For GTF input, missing gene and transcript rows can be synthesized from
gene_id / transcript_id attributes. Many modern GTFs—including GENCODE
v49—already contain explicit parent rows; disable inference when you want to
preserve exactly that supplied hierarchy. If parent rows and identifying
attributes are both absent, there is nothing to connect or synthesize.
Iterating is slower than gffutils¶
For a loop over many IDs, yes, and this is inherent rather than a bug — DuckDB pays vectorization startup per call, which an OLTP engine like SQLite does not. Use the batched APIs:
# slow
for tid in transcript_ids:
for exon in db.children(tid, featuretype="exon"):
...
# fast: one query, no Python Feature objects
exons = db.children_batched(transcript_ids, featuretype="exon", format="arrow")
My iteration stopped early¶
Something else queried the same connection while the iteration was still streaming. A DuckDB connection holds one result set at a time. Materialize first, or use a second handle — see Connections & concurrency.
Databases and files¶
IOException: Could not set lock on file¶
Another connection holds the database. Either another process has it open, or
an earlier handle in this process was never closed. Use with, or call
close(). For concurrent readers, open with read_only=True.
ClosedDatabaseError¶
The handle was closed — usually by leaving a with block earlier than
intended. Open a new one.
SchemaVersionError¶
Either the database was written by a newer GFFBase than the one reading it
(upgrade GFFBase), or it is a schema-v1 database opened with
upgrade="error" or read_only=True. Migrate it once:
gffbase migrate old.duckdb
... has gffbase tables but no metadata at all¶
An ingest that failed part-way, or a truncated file. Rebuild it with
create_db(..., force=True). GFFBase refuses to open it rather than handing
back an empty database that reports itself as complete.
The database is larger than expected¶
A database built by 0.3.0 is 0.61× to 0.89× the size of gffutils’ SQLite file
across the benchmark corpora, although it also stores a materialized transitive
closure, a long-form attributes table and an R-tree. If yours is much larger:
It was built by 0.2, which kept multi-column indexes DuckDB never used; the same file ingested by 0.3.0 is about half the size. Rebuild it with
create_db(..., force=True).It has been updated many times. DuckDB reuses the blocks that deleted or replaced rows free, but never shrinks the file; rebuilding it returns the space.
Environment¶
this version of Cargo does not understand this lock file¶
Building from source with a Rust older than 1.78. gffbase requires Rust
≥ 1.83, but this failure happens before Cargo reads the rust-version
field, so the message names neither gffbase nor the version you need:
error: failed to parse lock file at: .../rust/Cargo.lock
Caused by: lock file version `4` was found, but this version of Cargo
does not understand this lock file
💥 maturin failed
Check what pip will actually use, which is not always what cargo
--version reports in your working directory — a rustup directory override
applies only inside the directory it names, and building an unpacked sdist
happens outside it:
cargo --version # in the directory you are building in
rustup toolchain install 1.83 # if it is older than 1.83
RUSTUP_TOOLCHAIN=1.83 pip install gffbase
Installing a wheel needs no Rust at all; this only affects source builds.
native_available() returns False¶
The Rust extension did not load, so the pure-Python fallback parser is running.
Results are identical; throughput is not. On a platform with wheels this means
the install went wrong — try pip install --force-reinstall gffbase. From a
source checkout, run maturin develop --release --manifest-path rust/Cargo.toml.
Spatial queries are slower than the published numbers¶
Check which index the database is using:
db._rtree_built # False → B-tree fallback
DuckDB downloads its spatial extension on first use. On a machine without
network egress that fails and GFFBase falls back to a multi-column B-tree —
correct answers, lower throughput. Nothing is raised, because the fallback is
not an error.
ImportError mentioning polars / pandas / pyfaidx / pybedtools¶
An optional integration whose extra is not installed. The message names the exact command:
pip install gffbase[polars]
pip install gffbase[all]
Compatibility¶
Is it really a drop-in replacement for gffutils?¶
For the large majority of code, yes — import gffbase as gffutils and
carry on. The differences that remain are declared and tested in
tests/parity/deviations.toml, and the one behavioural gotcha (row-by-row
loops) is covered in the Migration guide.
Can I hand the database to a tool that expects gffutils?¶
Yes. Export a real SQLite database that gffutils itself can open:
from gffbase import export_sqlite
export_sqlite(db.conn, "legacy.db") # a connection, not the FeatureDB
Which Python versions are supported?¶
3.10 through 3.14, one abi3 wheel per platform.