Four questions about one small CNN. By the end, every number on these slides can be reproduced from the network diagram alone — including the outputs when all weights are 0, or all 1.
iTotal number of parameters — show your work.
iiAll parameters initialised to 0 — what does layer B output?
iiiAll weights 1, biases 0 — what does layer B output look like?
ivReduce parameters without reducing filter count or effective kernel size.
What one filter computes · why filters have depth · how sizes shrink
A kernel of all ones just sums its patch — an unnormalised box filter. Hold that thought for part iii.
First valid start: 0. Last: W−F. Step by the stride:
A | = 22 | |
B | = 20 | |
C | = 10 |
Same formula for height — everything here is square.
2×2 windows land at columns 0, 2, 4, 6 — each row of 20 becomes 10, and channels are untouched: 40 in, 40 out.
The exercise states it directly: “each kernel has the same number of channels as the input.” A “5×5” filter on RGB is really 5×5×3 = 75 weights, and its dot product with a patch gives one number.
F·F weights per slice · through Cin channels · +1 bias · × N filters. Independent of image size — the filter is shared across positions.
Taking a max, reshaping, clamping at zero — none of it is learned. Zero parameters, every time.
A weight for every input–unit pair, plus one bias per unit. Nin is the flattened length.
That is the whole toolkit. Everything in part i is these two formulas plus the size formula.
Both wrong answers survive a quick glance because they look like “kernel × filters”. The check that catches them: one filter must match its input in depth, and every filter carries a bias.
Before counting any conv layer, ask:
“What is Cin here?” — for layer B it is 20, the number of filters in A. The RGB 3 is long gone.
Same discipline at the flatten: it flattens the pooled shape (10×10×40), not the conv output before pooling.
Two ledgers through the network: the shape flowing out, and the parameters added
out 22×22×20 — one 22×22 map per filter.
ReLU changes values, never shapes or parameter counts.
75 weights and 1 bias per filter, twenty filters. Running total: 1,520.
out 20×20×40.
Cin = 20 — layer A’s filter count. Each 3×3 filter is a 3×3×20 brick: 180 weights, not 9.
Running total: 1,520 + 7,240 = 8,760.
out 10×10×40 — channels untouched.
One long vector for the FC layer. Flatten the pooled shape — 20×20×40 here would silently quadruple layer D.
Flatten adds 0 too. Running total: still 8,760.
Every one of the 4,000 flattened values connects to every one of the 10 units — no sharing, unlike a conv filter. That is why this little layer dwarfs everything.
The habit that catches every miscount: print model.summary() and reconcile it line by line with your hand count.
A | conv 5×5, 20 f | 22×22×20 | 1,520 |
B | conv 3×3, 40 f | 20×20×40 | 7,240 |
C | pool + flatten | 10×10×40 → 4,000 | 0 |
D | fully connected | 10 | 40,010 |
| total | 48,770 |
The layers that see hold 18% of the parameters. The little classifier head holds 82% — because it shares nothing.
File that away: it is the whole motivation for part iv, and for architectures that replace flatten+FC with pooling.
Initialise every weight and bias to 0 — run an image through
Layer B outputs 20×20×40, every value exactly 0 — whatever the input image was.
All 20 filters are identical, so they receive identical gradients — they can never become different detectors. Random init exists to break this tie.
Weight gradients scale with the activations flowing in — which are all 0. The conv layers cannot even start updating.
Weights 1, biases 0 — what does layer B look like?
A cellΣ over 5×5×3 ≤ 255·75 = 19,125
B cell20 × (3×3 sum of A) ≤ 3,442,500
Cascading the 5×5 and 3×3 box filters is one 7×7 filter whose weights taper from the centre — a gentler, wider blur (the separable-Gaussian story from the filtering week, run in reverse):
So layer B’s map is the input, channel-summed, blurred with this 7×7 pyramid, ×20.
Layer B outputs 20×20×40: a stack of 40 identical, blurry, brightness-summed copies of the input — both conv layers act as (unnormalised) averaging filters. Values are large positive sums; ReLU is a no-op. (Maps above are shown normalised for display.)
Same number of filters · same effective kernel size — where can parameters go?
Constrain each kernel slice to a rank-1 outer product — exactly how Gaussian and Sobel kernels factor. 10 numbers per slice instead of 25.
Cost: only rank-1 kernels are expressible — a real constraint, usually a mild one.
One F×F filter per channel (space only), then 1×1 filters to mix channels. Receptive field and filter count unchanged.
Layer B’s 7,200 weights become 180 + 800 = 980 — same 3×3 receptive field, same 40 maps out.
Here Cin is only 3, so the 5×5 is cheap (1,520) while a stack pays for channel mixing twice. It only wins if the middle conv stays narrow.
Break-even at m ≤ 7. The claim “stacked is smaller” is an architecture-dependent fact, not a law — check it with the counting rules you now own.
(The interactive lab page has this as a slider.)
Both keep 20 / 40 filters and the 5×5 / 3×3 effective kernels — the constraint is satisfied.
The constraint never mentioned layer D — where 82% of the parameters live. Swap flatten for global average pooling (mean of each 10×10 map → 40 values): D becomes 40·10+10 = 410, total 9,170. Filters and kernels untouched.
| original | 48,770 |
| spatially separable convs | 45,470 |
| depthwise separable convs | 41,208 |
| + global average pooling head | ≈ 1,600 |
The conv moves shave the 18%; touching the head removes the mountain. Constraint-shaped answers and best answers are not always the same thing.
1,520 + 7,240 + 0 + 40,010 = 48,770. Shapes: 26→22→20→10, flatten 4,000.
20×20×40 of exact zeros — 0·x kills the image, +0 and ReLU(0) keep it dead.
40 identical blurry copies of the (channel-summed) input — all-ones filters are unnormalised averaging filters; ReLU is a no-op.
Separable convolutions · stacked 3×3s — filters and effective kernels intact.
Pool / flatten / ReLU: 0. And always ask: “what is Cin here?”
→ / spaceadvance fragments & slides
Sspeaker notes view
Oslide overview
?print-pdfappend to the URL, then print to PDF
Exercise © COMP90086 Computer Vision (University of Melbourne) workshop materials. Deck follows the Workshop 2 slide system; built with reveal.js (MIT). All demo computation runs in-slide and is verified against a NumPy reference.