DNA Attention & Receptive Field Studio

Layer-by-layer receptive field expansion & self-attention flow · Shorkie 16,384 bp window← Shorkie Lab Expression Language Model Genome Paper Docs

Interactive Enhancer-to-TSS Dependency Tracer

Drag either probe along the 16,384 bp sequence track. Probe A (Cyan) simulates a distal regulatory element (enhancer / UAS), while Probe B (Orange) marks the promoter / transcription start site (TSS). Curved bezier arcs visualize real multi-head attention flow connecting distal regions to the promoter.

Genomic Distance0 bpSpan between Probe A & Probe B

Active Layer ConnectivityConnectedWhether distance ≤ theoretical RF

Attention Weight (TSS → Distal)0.00%Direct attention in active layer/rollout

Layer-by-Layer Receptive Field Ladder

Track how receptive field expands stage by stage through Shorkie's 19 architectural layers. Click any layer to inspect its receptive field diameter and test communication feasibility between Probe A and Probe B.

Stage 1: Conv Stem

Stage 2: 7 Residual Blocks (Local Context)

Stage 3: 8 Transformer Layers (Global Attention)

Stage 4: U-Net Decoders & Output Head

Transformer Layer 1

Receptive Field: Global (16,384 bp / 100%)

First multi-head self-attention layer operating on 128 bottleneck tokens. Instant O(1) communication across entire 16 kb locus.

Theoretical Receptive Cone (Input Sequence Projection)Centered on Probe B (TSS)

128 × 128 Multi-Head Attention Heatmap

Precomputed attention matrices decoded from Shorkie's bottleneck weights. Each token spans exactly 128 bp.

Layer:

Empirical Context Convergence Plot

Altschul-Erikson dinucleotide shuffle experiments: flanking context outside radius $r$ is randomized to destroy regulatory motifs while preserving dinucleotide distribution. The curve measures how prediction stabilizes towards the unperturbed baseline.

Shuffled Context Prediction 5% Full-Context Tolerance Zone Effective Context Convergence Point

Cross-Architecture Signal Transmission Sandbox

Simulate how different deep learning paradigms propagate regulatory signal across genomic distance. Adjust the slider to see how long-range enhancer-promoter communication behaves.

Enhancer-to-Promoter Distance:6,000 bp

100 bp (Local)2.5 kb (ResNet Limit)10 kb20 kb (Ultra Distal)

Hybrid Model (Shorkie)—

ResNet Stem + Transformer Bottleneck

Routing Hops:1 hop (O(1))

Memory:O(N^2) on 128 tokens

Convolution extracts high-resolution 128 bp motif tokens; multi-head self-attention enables direct O(1) pairwise routing between any two positions up to 16,384 bp.

Dilated ConvNet (BPNet / Basenji)—

Exponentially Dilated Convolutions

Routing Hops:O(log D) layers

Memory:O(L) linear

Information cascades through exponentially dilated residual layers. Signal decays as (0.88)^hops, creating an effective receptive ceiling.

State Space Model (Mamba / Caduceus)—

Selective Structured State Space

Routing Hops:O(D) recurrent steps

Memory:O(L) linear

Linear-time sequence model compressing history into hidden states. Excellent continuous decay, but requires bidirectional scans to prevent upstream bias.

Mathematical Framework & Architectural Principles

1. Convolutional Receptive Field Expansion

For a sequential cascade of convolutional and residual layers, the theoretical receptive field $RF_l$ at layer $l$ expands according to the recurrence relation:

RF_l = RF_{l-1} + (k_l - 1) \cdot S_{l-1}

where $k_l$ is the kernel size (5 bp in Shorkie's residual blocks) and $S_{l-1} = \prod_{i=1}^{l-1} s_i$ is the cumulative stride up to layer $l-1$ . Because Shorkie performs pooling ( $s_i = 2$ ) at each residual block, the effective stride doubles at every step, causing the receptive field to expand from 11 bp in the stem to 582 bp by Block 7 — the widest reach any purely convolutional stage has before the bottleneck. Both ends are derived from the checkpoint's own kernels rather than the paper's, which gives the stem as 15 bp.

2. Attention Rollout Across Layers

To track how information flows across multiple Transformer layers with residual skip connections, we use Attention Rollout (Abnar & Zuidema, 2020). The effective attention $\tilde{A}^{(l)}$ accounting for identity shortcuts is:

\tilde{A}^{(l)} = 0.5 A^{(l)} + 0.5 I

Recursive matrix multiplication yields the total end-to-end information flow from input tokens to layer $L$ :

R^{(L)} = \prod_{l=1}^L \tilde{A}^{(l)} = \tilde{A}^{(L)} \tilde{A}^{(L-1)} \cdots \tilde{A}^{(1)}

3. The Biological Pareto Optimum

Why not use pure Transformers or pure ConvNets?

Pure Transformers: An attention matrix on raw 16,384 bp sequence requires $16,384^2 \approx 2.68 \times 10^8$ operations per head per layer, consuming enormous GPU VRAM with poor inductive bias for local DNA motifs.
Pure Dilated ConvNets: Exponential dilation achieves large receptive fields but suffers from the receptive field lottery and exponential signal attenuation over 5+ kb.
Shorkie's Hybrid Pareto: Residual CNNs pool 16,384 bp into 128 compact 128 bp tokens. The Transformer then executes attention over a lightweight $128 \times 128 = 16,384$ matrix—providing instant $O(1)$ global connectivity with negligible memory overhead.