Shorkie_LM

The problem. A masked language model only predicts where it is masked. To get a genuine prediction at every one of the 16,384 positions you must mask every position at some point — but masking all of them at once leaves the model nothing to condition on.

The method. Partition the positions into K disjoint strided sets — position i joins set i mod K — mask each set in turn, and read every position back only from the pass that masked it. With K = 7 that puts 14.3% of positions under mask, matching the checkpoint's own mask_rate: 0.15, and the stride guarantees every masked base keeps unmasked immediate neighbours, exactly as in pretraining. Seven forward passes, about two seconds a locus.

Why the stride and not a random 15%. A partition means each position is predicted exactly once, with no position predicted twice and none missed — so the result is a function of the model, not of a seed.

The check that it is not an artefact of K. On TDH3, K = 10 gives 43.93% argmax and 1.7528 bits against K = 7's 43.00% and 1.7571 — so the stride is not what sets the number. Those two figures are TDH3's, not the page's: argmax runs 41.3% (GAL1) to 46.3% (FUN12) across the nineteen windows, and the bar above always reports the window you are looking at.

What it is not. Not the same as masking one base alone: the iterative pass also has 1/7 of the rest of the window hidden, so it is the same question asked slightly harder. And not a generative sample — each position is scored independently, so reading the argmax across a span is not "the sequence the model would write".

The formula is the paper's verbatim: conservation = 2 − entropy, with H(p) = −Σ p log₂ p over the four bases and each letter drawn at p × IC. Letters stack ascending by probability, the PWM convention, over a fixed 0–2 bit axis — not descending by magnitude with a zero rule, which is the attribution convention on the other page.

Glyphs are the paper's: real DejaVu Sans Bold outlines at its hand-tuned per-letter offsets, scaled 1.35× on both axes, in its saturated X11 colours (A green, C blue, G orange, T red), fixed across every theme because a figure's base colours are part of the figure.

A tall column is not a functional site. It says the model can predict the base from its context. Homopolymer runs, repeats and strongly biased composition all produce high information content with no regulatory meaning whatever — which is exactly why the enrichment panel below tests constraint against annotation rather than asserting the connection.

Quantisation. Probabilities ship as uint8 in log space, not linear: the displayed quantity is the entropy and −p log₂ p is steepest where p is smallest, which is where a linear grid is coarsest. Measured across all nineteen loci the worst entropy error is under 0.02 bits on a 2-bit axis.

The method. Zero the four DNA channels across the entire site — the model's own masking, indistinguishable from a run of N — run one forward pass, and read the argmax across the masked span.

The mismatch. Pretraining masks 15% of positions scattered, so every masked base keeps unmasked immediate neighbours. A contiguous 7–11 bp hole removes exactly that local context, and the model has never been asked to fill one. The same positions, masked in the scattered pattern instead, are recovered far more often — the two numbers are on the same page above.

Read identity against the floor, never against zero. A promoter that is 60% A/T rewards a constant-A guess, so an unanchored "recalled 6 of 11 bases" has no scale. The floor is the largest single-base frequency in that window.

What a success would and would not mean. Recovering a site's sequence from its surroundings is evidence the model has learned that this sequence is likely in this context. It is not evidence the model represents the transcription factor, and it says nothing about whether the site is used.

The released checkpoint's params.json carries four loss weights that are not mentioned in the paper text: exon_loss_scale: 0.1 and repeat_loss_scale: 0.1, against 1.0 for non-exon and non-repeat. The model was deliberately trained to care ten times less about exons and repeats.

That is a prediction, not a description: if the loss weights did what they were meant to, the LM should be measurably less constrained over coding sequence than over the intergenic sequence around it. The CDS row in the table above is that test.

Measured across all nineteen windows, the exon half of the prediction fails and the repeat half holds. Coding sequence scores above 1.0 in 19 of 19 loci — mean 1.120×, range 1.041–1.266 — so the model is more certain over exons, not less, despite their loss being weighted at a tenth. Repeats go the other way, but on far thinner evidence: the curated annotation for these windows contains no transposon features at all and just three solo LTRs — delta elements, one each in the TDH3, PGK1 and RPL26A windows — scoring 0.68–0.80×. All three fall on the same side, which is suggestive, but three features in three windows is not the fourteen-window result the CDS row is. The table in Constraint across all nineteen windows shows a — for the sixteen windows that contain no LTR, so the support is visible rather than implied.

The likely reason is that the two classes are not comparable targets. A repeat is only predictable because it recurs, and down-weighting it removes the incentive to memorise it. Coding sequence is predictable because the genetic code constrains it — reading frame, codon bias, amino-acid composition — and that structure is there to be learned from the 90% of the loss that remains. A loss weight can discourage memorisation; it cannot make a compositionally constrained sequence look random.

The null is a circular shift, the same one the expression page uses: rotating the annotation preserves the feature count, every length and every gap, and destroys only the alignment. Offsets are deterministic and zero is excluded, so a published ratio is reproducible rather than a draw.

What it cannot tell you. Constraint and function are different things. Repeats and homopolymers are highly predictable and carry no regulatory meaning; a class can be enriched here because it is compositionally simple rather than because it matters.

This is PCA, computed in the browser by power iteration from a fixed seed. The paper's Figure 2E uses t-SNE. The swap is deliberate: a projection whose axes move between page loads cannot be written about, and t-SNE distances are not distances — its cluster separations and gaps are famously not interpretable as magnitudes. PCA's axes are the directions of greatest variance and its distances are real, at the cost of showing less separation.

So a lack of visible clustering here is weaker evidence than a lack in the paper's figure would be, and the reverse is stronger: anything PCA does separate is separated along genuine high-variance directions.

One point is 128 bp, so a position's label is whatever annotation covers most of that stretch — at this resolution a 7 bp binding site cannot have its own point, and the classes shown are the ones large enough to dominate a cell.