Upgrading a schema v1 database in place.
Two separate operations, deliberately not combined:
:func:migrate_v1_to_v2
Structural only -- ADD COLUMN, CREATE TABLE, CREATE VIEW. It must
never change what any query returns, so it does not attempt to recognise
v1's x_1 / x_2 rows as segments of one discontinuous feature. An
in-place upgrade that silently merged 18 features into 17 would be a data
change wearing a migration's clothes.
:func:coalesce_multipart
The opt-in second step that does exactly that, reusing the ingest resolve
pass. It changes results, which is why the caller has to ask for it.
What the migration deliberately does NOT do is backfill raw_id. NULL there
means "this database predates the column, and the id column 9 originally
carried is unknown" -- which is the truth. Setting raw_id = id would be a
lie precisely for the rows that matter: the ones create_unique renamed.
meta.raw_id_valid records it, and the resolve pass skips NULL raw_id, so a
migrated database is simply inert for fusion until coalesce_multipart runs.
MigrationResult
dataclass
MigrationResult(from_version: str, to_version: str, changed: bool = False, applied: list[str] = list())
What a migration did, for logging and for tests to assert on.
migrate_v1_to_v2
migrate_v1_to_v2(target, *, con: DuckDBPyConnection | None = None) -> MigrationResult
Upgrade a schema v1 database to v2, in place.
target may be a path or an open connection. The whole upgrade runs in one
transaction, so a database is never left half-migrated; and it is
idempotent, so running it against a database that is already v2 is a no-op
that reports changed=False rather than an error.
Every statement is additive. No feature row's data is altered, and a
caller's queries return exactly what they returned before -- which is the
property that makes this safe to run automatically at open.
Source code in python/gffbase/migrate.py
| def migrate_v1_to_v2(target, *, con: duckdb.DuckDBPyConnection | None = None) -> MigrationResult:
"""Upgrade a schema v1 database to v2, in place.
`target` may be a path or an open connection. The whole upgrade runs in one
transaction, so a database is never left half-migrated; and it is
idempotent, so running it against a database that is already v2 is a no-op
that reports ``changed=False`` rather than an error.
Every statement is additive. No feature row's data is altered, and a
caller's queries return exactly what they returned before -- which is the
property that makes this safe to run automatically at open.
"""
owned = False
if con is None:
if isinstance(target, duckdb.DuckDBPyConnection):
con = target
else:
con = duckdb.connect(str(target))
owned = True
try:
found = _read_version(con)
current = found if found is not None else "1"
if current == SCHEMA_VERSION:
return MigrationResult(from_version=current, to_version=current, changed=False)
if current != "1":
raise SchemaVersionError(
f"cannot migrate schema version {current!r}: this gffbase upgrades "
f"v1 to v{SCHEMA_VERSION} only"
)
applied: list[str] = []
# One transaction: a database is never left half-upgraded, so a failure
# mid-way leaves a still-valid v1 database rather than something no
# version of gffbase can read.
con.execute("BEGIN TRANSACTION")
try:
_add_columns(con, "features", _FEATURE_COLUMNS, applied)
_add_columns(con, "attributes", _ATTRIBUTE_COLUMNS, applied)
con.execute(_NEW_TABLES)
applied.extend(["segments", "id_conflicts"])
con.execute("CREATE INDEX IF NOT EXISTS segments_fid ON segments(feature_id, seg_idx)")
con.execute(SEGMENTS_ALL_VIEW)
con.execute(_COMPAT_VIEW)
applied.extend(["segments_all", "features_compat"])
con.executemany(
"INSERT OR REPLACE INTO meta(key, value) VALUES (?, ?)",
[
("schema_version", SCHEMA_VERSION),
("n_multipart", "0"),
# `raw_id` is left NULL rather than backfilled to `id`; see
# the module docstring. This records that, so
# `coalesce_multipart` and the validator can tell a
# migrated database from a natively-built one.
("raw_id_valid", "false"),
# v1 relied on physical insertion order to reproduce
# attribute key order, which no SQL engine guarantees, so
# `ord` cannot be recovered. The raw blob is intact, and
# that is where key order should be read from.
("attributes_ord_valid", "false"),
],
)
con.execute("COMMIT")
except Exception:
con.execute("ROLLBACK")
raise
_log.info("migrated %s from schema v1 to v%s", target, SCHEMA_VERSION)
return MigrationResult(
from_version="1", to_version=SCHEMA_VERSION, changed=True, applied=applied
)
finally:
if owned:
con.close()
|
coalesce_multipart
coalesce_multipart(db, *, on_multipart_conflict: str = 'error') -> int
Re-fuse rows a v1 create_unique split apart. Returns the count fused.
Deliberately NOT part of :func:migrate_v1_to_v2: it changes query
results, merging what were N features into one, and an in-place upgrade
must never do that behind a caller's back.
Reuses the ingest resolve pass, so the GFF3 predicate applies unchanged --
segments must share seqid, source, featuretype and strand, and a run that
does not is reported rather than fused.
Source code in python/gffbase/migrate.py
| def coalesce_multipart(db, *, on_multipart_conflict: str = "error") -> int:
"""Re-fuse rows a v1 `create_unique` split apart. Returns the count fused.
Deliberately NOT part of :func:`migrate_v1_to_v2`: it changes query
results, merging what were N features into one, and an in-place upgrade
must never do that behind a caller's back.
Reuses the ingest resolve pass, so the GFF3 predicate applies unchanged --
segments must share seqid, source, featuretype and strand, and a run that
does not is reported rather than fused.
"""
from gffbase._options import IngestOptions
from gffbase.ingest import resolve_multipart
con = db.conn if hasattr(db, "conn") else db
_reconstruct_raw_ids(con)
options = IngestOptions(mode="strict", on_multipart_conflict=on_multipart_conflict)
n = resolve_multipart(con, options, {}, has_spatial=_has_bbox(con))
con.execute(
"INSERT OR REPLACE INTO meta(key, value) SELECT 'n_multipart', "
"CAST(COUNT(*) AS VARCHAR) FROM features WHERE n_segments > 1"
)
con.execute("INSERT OR REPLACE INTO meta(key, value) VALUES ('raw_id_valid', 'true')")
if hasattr(db, "_n_multipart"):
db._n_multipart = int(
scalar_or(con, "SELECT COUNT(*) FROM features WHERE n_segments > 1", 0)
)
return n
|