One small CNN, four questions. This lab builds the three ideas you need — what a convolution computes, why filters have depth, and how output sizes shrink — then walks the network end to end to the number 48,770.
A convolutional layer slides a small grid of weights — the kernel — across the input. At every position it multiplies each weight with the input value under it and adds everything up. That single sum (plus a bias) becomes one cell of the output map.
The exercise states it directly: “each kernel has the same number of channels as the input.” On an RGB input a 5×5 filter is really 5×5×3 — 75 weights — and its dot product with a 5×5×3 patch produces one number. One filter → one 2D map. Twenty filters → a stack of 20 maps.
Count the positions where the kernel still fits. Start at the left edge; the last valid start is W−F; stepping by S:
W input width · F filter size · P padding per side (valid ⇒ P=0) · S stride. Same formula for height. Max pooling obeys it too — with F = pool size.
F·F weights per channel slice, through Cin channels, +1 bias — that’s one filter. Times N filters.
Parameters don’t depend on the image size — the same filter slides everywhere (weight sharing).
Pooling takes the max of each window. Nothing is learned — no weights, no biases. It only shrinks the map (same size formula, F = pool size).
It also never changes the channel count: 40 in → 40 out.
Every input connects to every unit — a weight per pair, plus one bias per unit. The flattened vector length is Nin.
Flatten itself is free: reshaping moves numbers, it doesn’t learn any.
At every stage track two things: the shape flowing out (you need it as the next layer’s Cin, and for the flatten), and the parameters added. Numbers animate in as you scroll.
| stage | output shape | parameters |
|---|---|---|
| A conv 5×5 | 22×22×20 | 1,520 |
| B conv 3×3 | 20×20×40 | 7,240 |
| C maxpool + flatten | 10×10×40 → 4000 | 0 |
| D fully connected | 10 | 40,010 |
| total | 0 |
The convolutional layers do the seeing; the little FC head holds 82% of the parameters. Keep that in mind for part iv.
Set every weight and bias to 0 and push any image through. Follow one output cell of layer A:
Layer B then convolves an all-zero input with all-zero weights — zero twice over. The input image never matters: multiplying by 0 erases it at the first layer.
With every weight equal to 1, a filter’s “weighted sum” is just a plain sum of its patch — an unnormalised box (averaging) filter that also adds the RGB channels together. Nothing is negative, so ReLU changes nothing. And since all 20 filters of A are identical, all 20 maps are the same map. Same again for B’s 40.
Maps are shown brightness-normalised (the raw values are huge — see below). The stacked outline behind each map is the point: every channel is this identical picture, 20× for A and 40× for B.
One A cell = Σ over 5×5×3 patch = up to 255·75 = 19,125.
One B cell = Σ over 3×3×20 window of A = 20 × (3×3 sum of the A map) — up to 3,442,500.
Sums of non-negatives are non-negative, so ReLU never fires — it passes everything through.
Two box filters in a row are one bigger, gentler filter: 5×5 ∘ 3×3 has 7×7 support and a pyramid profile — the weights taper from the middle out (this is the separable-Gaussian story from the filtering workshop, in reverse):
The constraint: keep the number of filters (20, 40) and their effective kernel size (5×5, 3×3 receptive fields). The two official moves both restructure the convolutions — and on this tiny network the arithmetic has surprises worth knowing.
Spatially separable — constrain each 5×5 kernel slice to be an outer product of a 5×1 column and a 1×5 row (exactly how Gaussian and Sobel filters factor). Per slice: 5+5 = 10 numbers instead of 25.
Depthwise separable (the MobileNet pattern) — first one F×F filter per channel (depthwise), then 1×1 filters to mix channels (pointwise). Receptive field still F×F; filter count unchanged.
Two 3×3 convs see a 5×5 receptive field (each output cell of the second conv reaches 3+2 input cells) — and the output size even matches: 26→24→22. The VGG argument: for C channels in and out, . Bonus: an extra ReLU in between.
Caveat for layer A: its input depth is only 3, so the 5×5 is cheap (1,520) while the naïve stack pays twice for channel mixing. It only wins if the middle conv stays narrow:
| i | 1,520 + 7,240 + 0 + 40,010 = 48,770 parameters |
| ii | 20×20×40, all values exactly 0 |
| iii | 20×20×40; 40 identical blurry copies of the input (conv layers = averaging filters) |
| iv | separable convolutions · stacked 3×3 in place of the 5×5 |
Pooling, flatten, ReLU: 0 parameters. And always ask “what is Cin here?” before counting.