2D Spatial Convolution & Feature Extraction

Interact with the discrete 2D cross-correlation kernel sliding across an input feature map. Click any input cell to cycle activation values ( $0 \to 4$ ), or select weight presets below to inspect edge detection, sharpening, and smoothing.

(I * K)(i, j) = \sum_{m} \sum_{n} I(i \cdot s + m \cdot d, \; j \cdot s + n \cdot d) \cdot K(m, n) + b

Kernel Presets:

Input Shape: 5 × 5

Kernel (

K \times K

)

Stride (

S

)

Padding (

P

)

Dilation (

d

)

Channels (

C_{\text{in}} \to C_{\text{out}}

)

→

Groups / Topology

Depthwise

Learnable Bias (

b

)

Enabled

Padded Input Matrix

X

Click cell to cycle 0..4

Kernel Filter

W

Click to edit weight

Output Feature Map

Y

Click cell to jump

Speed:

Step 1 / 25

🔬

Real-Time X-Ray Dot Product & Accumulation Inspector

Output Spatial ShapeDimension

5 × 5

O = \left\lfloor \frac{W - K + 2P - (K-1)(d-1)}{S} \right\rfloor + 1

Learnable ParametersWeights + Bias

\text{Params} = \frac{C_{\text{out}}}{G} \cdot C_{\text{in}} \cdot K^2 + C_{\text{out}}

Computational CostComplexity

\text{MACs} = H_{\text{out}} W_{\text{out}} \cdot \frac{C_{\text{in}} C_{\text{out}} K^2}{G}

Activation MemoryFP32 Tensor

\text{VRAM} = H_{\text{out}} \cdot W_{\text{out}} \cdot C_{\text{out}} \times 4\text{ B}

1D Genomic Sequence Scanner & Transcription Factor Motif Convolutions

In genomic deep learning architectures (e.g., DeepBind, SpliceAI, Borzoi, Enformer), 1D convolutional layers scan one-hot encoded DNA sequences. Each 1D filter represents a learned Position Weight Matrix (PWM) calculating log-odds affinities for transcription factor binding sites (TFBS), splice junctions, or core promoter elements.

S_t = \sum_{j=0}^{k-1} \sum_{c \in \{A,C,G,T\}} X_{t + j, c} \cdot W_{c, j}

🧬

One-Hot Encoded Nucleotide Track & Sliding Window

📊

1D Convolution Activation Profile / Affinity Score Profile

🔬

1D Filter Window Position Weight Matrix (PWM) Dot Product

Receptive Field (RF) Expansion Ladder

Understand how input spatial context accumulates through successive convolutional operations. For each layer $l$ , the effective receptive field expands according to the canonical recurrence relation:

RF_l = RF_{l-1} + (k_l - 1) \cdot S_{l-1}, \quad \text{where } S_{l-1} = \prod_{i=1}^{l-1} s_i

Production-Ready PyTorch Module Generator

Copy fully synchronized, production-grade PyTorch code implementing the currently configured convolution layer, complete with explicit tensor dimension asserts and test execution harness.

# Initializing PyTorch module generator...

Foundational Architectural Deep Dives

Rigorous mathematical formulations and systems engineering principles underpinning modern deep convolutional networks.

In virtually all state-of-the-art architectures (ResNet, ConvNeXt, EfficientNet), any nn.Conv2d immediately preceding an nn.BatchNorm2d sets bias=False. Here is the complete algebraic cancellation proof demonstrating why a convolutional bias is mathematically redundant.

Let $x_i$ denote the $i$ -th spatial activation patch in mini-batch $\mathcal{B} = \{x_1, \dots, x_m\}$ , $W$ the convolutional filter weights, and $b$ an arbitrary additive bias scalar:

\hat{y}_i = W * x_i + b

During training, Batch Normalization computes the mini-batch sample mean $\mu_{\mathcal{B}}$ :

\mu_{\mathcal{B}} = \frac{1}{m} \sum_{i=1}^{m} \hat{y}_i = \frac{1}{m} \sum_{i=1}^{m} (W * x_i + b) = \left( \frac{1}{m} \sum_{i=1}^{m} W * x_i \right) + b

The centering step of Batch Normalization subtracts this mini-batch mean $\mu_{\mathcal{B}}$ from each activation $\hat{y}_i$ :

\hat{y}_i - \mu_{\mathcal{B}} = (W * x_i + b) - \left[ \left( \frac{1}{m} \sum_{i=1}^{m} W * x_i \right) + b \right] = W * x_i - \frac{1}{m} \sum_{i=1}^{m} W * x_i

Notice that the constant bias term $b$ is subtracted out identically: $b - b = 0$ . Batch Normalization then applies its own learnable affine transformation parameters $\gamma$ (scale) and $\beta$ (shift):

y_i = \text{BN}_{\gamma, \beta}(\hat{y}_i) = \gamma \cdot \frac{\hat{y}_i - \mu_{\mathcal{B}}}{\sqrt{\sigma^2_{\mathcal{B}} + \epsilon}} + \beta

Key Takeaway: The learnable parameter $\beta$ in BatchNorm already serves as the channel bias. Including bias=True in the convolution wastes GPU memory, increases parameter count, and calculates gradient updates for a parameter that is cancelled out at every step.

The physical memory layout of 4D tensors dramatically impacts cache hit rates, memory bandwidth utilization, and Tensor Core throughput on modern NVIDIA GPUs (Ampere, Ada Lovelace, Hopper, Blackwell).

1. Default PyTorch: NCHW (`torch.contiguous_format`)

Values along spatial width $W$ are contiguous in memory. However, accessing all channels $C$ for a given spatial location $(h, w)$ requires strided memory loads with step size $H \times W$ .

Because GEMM convolution algorithms (via im2col or Winograd) assemble channel vectors into matrix rows, NCHW necessitates expensive transpose/repack operations in VRAM.

2. Optimized PyTorch: NHWC (`torch.channels_last`)

All channels $C$ at coordinate $(n, h, w)$ reside in contiguous memory bytes.

NVIDIA Tensor Cores perform matrix multiply-accumulate ( $D = A \cdot B + C$ ) on sub-matrices using 8-element or 16-element channel vectors directly. Under NHWC, 128-bit memory instructions (LDG.128) stream full channel vectors straight into Tensor Core registers without transposition overhead.

\text{Speedup: } \text{Throughput}_{\text{NHWC}} \approx 1.20\times \text{ to } 2.40\times \text{ vs. } \text{NCHW on FP16/BF16 AMP}

Activating channels-last in PyTorch requires just two lines of code before training:

model = model.to(memory_format=torch.channels_last)
inputs = inputs.to(memory_format=torch.channels_last)

Standard convolutions jointly map spatial relationships and cross-channel correlations in a single compute-heavy operation. Depthwise Separable Convolutions (pioneered in MobileNet and Xception) factorize this computation into two separate steps:

Depthwise Convolution: A spatial filter applied independently to each input channel ( $G = C_{\text{in}}$ ).
Pointwise Convolution: A standard $1 \times 1$ convolution that computes linear combinations across all channels.

Complexity Derivation:

For output dimensions $H_{\text{out}} \times W_{\text{out}}$ , kernel size $K \times K$ , and channels $C_{\text{in}} \to C_{\text{out}}$ :

\text{FLOPs}_{\text{standard}} = 2 \cdot H_{\text{out}} \cdot W_{\text{out}} \cdot C_{\text{in}} \cdot C_{\text{out}} \cdot K^2

In depthwise separable convolution:

\text{FLOPs}_{\text{depthwise}} = 2 \cdot H_{\text{out}} \cdot W_{\text{out}} \cdot C_{\text{in}} \cdot K^2

\text{FLOPs}_{\text{pointwise}} = 2 \cdot H_{\text{out}} \cdot W_{\text{out}} \cdot C_{\text{in}} \cdot C_{\text{out}} \cdot 1^2

The theoretical computational reduction ratio is:

\frac{\text{FLOPs}_{\text{separable}}}{\text{FLOPs}_{\text{standard}}} = \frac{C_{\text{in}} \cdot K^2 + C_{\text{in}} \cdot C_{\text{out}}}{C_{\text{in}} \cdot C_{\text{out}} \cdot K^2} = \frac{1}{C_{\text{out}}} + \frac{1}{K^2}

For standard $3 \times 3$ convolutions ( $K=3$ ) with $C_{\text{out}} \ge 64$ , this represents an $\approx 8\times$ to $9\times$ reduction in compute and parameters with only marginal drops in top-1 classification accuracy!

Dilated (atrous) convolutions expand the effective receptive field exponentially without downsampling (pooling) or introducing extra parameters:

K_{\text{eff}} = K + (K - 1)(d - 1) = d \cdot (K - 1) + 1

For instance, with $K=3$ and dilation $d=4$ , the kernel spans a $9 \times 9$ spatial window while executing only 9 multiply-accumulate operations.

The Gridding (Checkerboard) Problem:

When multiple layers with identical dilation $d > 1$ are stacked sequentially (e.g., $d=2, 4, 8$ ), the kernel samples from a disjoint, checkerboard-like grid. Intermediate pixels in the receptive field are never sampled by any filter tap, resulting in high-frequency checkerboard artifacts and complete loss of local spatial consistency.

The Solution: Hybrid Dilated Convolution (HDC):

To eliminate gridding, dilation rates across consecutive layers must be relatively prime or follow a sawtooth sequence (e.g., $d = [1, 2, 5, 1, 2, 5]$ ) such that the maximum distance between two sampled pixels satisfies:

M_i = \max(M_{i+1} - 2d_i, \; M_{i+1} - 2(M_{i+1} - d_i), \; d_i) \le K

This ensures dense, continuous sampling across the entire receptive field without holes or aliasing artifacts.