validate

Post-ingest invariants. Each is a single set-based query, so the whole set is cheap enough to run in CI — gffbase validate --strict is the command-line form.

The one that matters most is INV-5: a fused feature whose envelope is narrower than its segments simply stops being returned by region(), with nothing raised anywhere. Errors and warnings are separate, because an error is a broken invariant while a warning is something legal but suspect.

Post-ingest invariants.

Schema v2 spreads one logical feature across two tables, and the interesting failure mode is not a crash -- it is a database that answers every query plausibly and wrongly. The worst case is INV-5: if a fused feature's envelope is narrower than its segments, region() silently stops returning it. Nothing raises, no count looks odd, and the feature is simply gone from results.

So these run automatically at the end of a strict-mode ingest, and can be run by hand at any time. Every check is a single set-based query returning a count plus a few examples, so validating a GENCODE-scale database is a handful of aggregate scans rather than a row-by-row walk.

Severity is not decoration. An error means the database will give wrong answers; a warning means something is unusual but defensible in real data -- abutting segments, for instance, occur in files people actually ship.

Violation dataclass

Violation(invariant: str, name: str, severity: str, count: int, detail: str, examples: tuple = ())

One invariant that did not hold.

ValidationReport dataclass

ValidationReport(level: str, checked: list[str] = list(), violations: list[Violation] = list(), skipped: list[str] = list())

What validation found. Falsy when anything failed at error severity.

ok property

ok: bool

True when nothing failed at error severity. Warnings do not count: they flag data that is unusual, not a database that answers wrongly.

raise_for_status

raise_for_status() -> None

Raise if any invariant failed at error severity.

Source code in python/gffbase/validate.py

def raise_for_status(self) -> None:
    """Raise if any invariant failed at `error` severity."""
    if self.errors:
        raise ValidationError(
            f"{len(self.errors)} invariant(s) violated:\n"
            + "\n".join(str(v) for v in self.errors)
        )

ValidationError

Bases: AssertionError

A database violates an invariant that makes its answers wrong.

AssertionError rather than ValueError: this is never bad input from the caller, it is gffbase having produced or been handed a database whose internal structure contradicts itself.

validate_db

validate_db(db, level: str = 'fast', *, raise_on_error: bool = False, sample: int = 200) -> ValidationReport

Check a database's structural invariants.

level="fast" runs every check that is a fixed number of aggregate scans. level="full" adds INV-12, which re-parses stored attribute blobs for sample features and is the only check whose cost grows with the corpus.

Checks that depend on structures a database does not have -- a v1 shim has no segments -- are recorded as skipped rather than silently passing, so a green report cannot mean "nothing was looked at".

Source code in python/gffbase/validate.py

def validate_db(
    db,
    level: str = "fast",
    *,
    raise_on_error: bool = False,
    sample: int = 200,
) -> ValidationReport:
    """Check a database's structural invariants.

    `level="fast"` runs every check that is a fixed number of aggregate scans.
    `level="full"` adds INV-12, which re-parses stored attribute blobs for
    `sample` features and is the only check whose cost grows with the corpus.

    Checks that depend on structures a database does not have -- a v1 shim has
    no `segments` -- are recorded as skipped rather than silently passing, so a
    green report cannot mean "nothing was looked at".
    """
    if level not in LEVELS:
        raise ValueError(f"level must be one of {LEVELS}; got {level!r}")

    con = db.conn if hasattr(db, "conn") else db
    report = ValidationReport(level=level)

    has_segments = _has_table(con, "segments")
    has_bbox = _has_column(con, "features", "bbox")

    for number, name, severity, fn, needs_segments in _CHECKS:
        if needs_segments and not has_segments:
            report.skipped.append(f"{number} ({name}): this database has no `segments` table")
            continue
        if name == "bbox_matches" and not has_bbox:
            report.skipped.append(f"{number} ({name}): no R-tree was built for this database")
            continue
        try:
            count, examples = fn(con)
        except duckdb.Error as exc:  # pragma: no cover - a malformed database
            report.violations.append(
                Violation(number, name, severity, 1, f"check could not run: {exc}")
            )
            continue
        report.checked.append(f"{number} ({name})")
        if count:
            report.violations.append(
                Violation(number, name, severity, count, _DETAILS[name], examples)
            )

    if level == "full":
        count, examples = _inv12_attributes_reparse(con, sample)
        report.checked.append("INV-12 (attributes_reparse)")
        if count:
            report.violations.append(
                Violation(
                    "INV-12",
                    "attributes_reparse",
                    ERROR,
                    count,
                    _DETAILS["attributes_reparse"],
                    examples,
                )
            )

    for violation in report.violations:
        (_log.error if violation.severity == ERROR else _log.warning)("%s", violation)

    if raise_on_error:
        report.raise_for_status()
    return report