COMP90086 · Computer Vision · Workshop 4

Colour, light
and learned filters

How a scene becomes three numbers per pixel —
and what happens when we stop choosing the kernels ourselves.
colour · light & surfaces · learned filters

QR code linking to the Workshop 4 slides

Scan to open the interactive slides on your device hesamasad.github.io/week-4-colour-and-cnns-workshop

Where we're going

1 — COLOUR

Why three numbers
RGB · HSV · XYZ · Lab
Manipulating colour
OpenCV's traps

2 — LIGHT & SURFACES

Diffuse reflectance
What sets a pixel value
Why vision is hard
Why edges survive

3 — LEARNED FILTERS

MNIST & preprocessing
MLP vs CNN
Counting parameters
A network, running live

The thread: parts 1 and 2 are about what a pixel value means. Part 3 is about letting a machine discover which combinations of pixels are worth measuring — the same convolution as last week, with the numbers learned instead of chosen.

Colour

An entire spectrum, collapsed into three numbers.

Three numbers from a whole spectrum

Each number is the area of the overlap between the light and one sensor's sensitivity.

\( I_X = \int I(\lambda)\,S_X(\lambda)\,d\lambda \) — the integral sign just says "multiply the two curves and measure the area under the result". Do it three times and the entire spectrum is gone, replaced by three numbers.

Walk the three numbered steps in order and do not rush: ① is the light, ② is the three overlaps, ③ is the colour. The dashed line in each small panel is that sensor's sensitivity; the solid shaded region is the light multiplied by it. The number underneath is the area of the shaded region — that IS the integral, and saying so out loud is the whole point of the slide. Now drag the peak slider slowly from blue to red and narrate what the areas do: as the hump slides right, the blue overlap collapses and the red one grows. Students who find integrals abstract can still see an area shrink. The three curves are the CIE 1931 2-degree standard observer functions — the same figure that is in the worksheet, and a standardised stand-in for cone sensitivity. The peak of y-bar at 555 nm is why we are most sensitive to green. If asked "why three and not four?" — because we have three cone types. Some people have four (tetrachromacy); most animals differ. The number 3 is biology, not physics.

Two different lights, one colour

If only the three areas survive, then any two spectra with the same three areas are indistinguishable.

Spectra that differ but look identical are metamers. This is not a curiosity — it is the only reason screens work: a projector has just three narrow primaries, yet it can forge almost any colour by matching the three areas.

Point at the two swatches on the right before you say anything. They are drawn edge to edge on purpose — ask the room whether they can see the join. They cannot. Then point at the two plots on the left, which could hardly be more different: one smooth hump versus three isolated spikes. Different light, identical numbers. Say plainly that this is happening right now: the projector showing this slide has three primaries, and every colour on it is a metamer of the real thing. Drag the slider to the far red or far blue and the spike mixture eventually needs a negative weight, which is impossible — the swatch goes hatched. That is a gamut boundary, and it is why a monitor cannot show every colour you can see. It is also the same boundary they will see drawn as an actual shape on the Lab plane a few slides later. Worth one sentence: metamerism is why two garments can match in the shop and clash outside. The shop's lights and daylight have different spectra.

One photo, four ways to split it into three numbers

Same information, different axes — each is an invertible function of the others. What changes is which question becomes easy: RGB smears brightness through all three channels; HSV and Lab put it in one and leave the other two for colour alone.

Read the picture row by row rather than reading the text out. RGB — cover the labels and ask which channel is which. They cannot reliably tell, because all three are the same photo at slightly different contrasts. That is the point: brightness is smeared through every channel, so you cannot change colour without changing brightness. HSV — V is the whole picture again, but now H and S carry colour on their own. The red bird and the blue-and-gold bird separate cleanly in H. XYZ — Y looks like a sensible greyscale because Y *is* luminance, by definition. X and Z look strange alone; they are sensor responses, not pictures. Lab — L* is the greyscale, a* has the red bird at one extreme and the leaves at the other, b* picks out the gold breast. This is the row to linger on, because the a* axis is what the Lab-plane slide, the colour-swap slide and worksheet exercise 2 all act on. Say why the display maps differ: H, a* and b* are shown through colour maps rather than grey ramps because H is an angle and a*/b* are signed opponent axes — a grey ramp would put "very green" and "very red" at opposite ends of a scale that looks like brightness, which is exactly the wrong intuition. L*, S, V, X, Y, Z are genuine intensities, so they stay grey. The equivalence claim in the takeaway is worth defending if challenged: the transforms are bijective on real numbers. In 8-bit storage they are NOT lossless, which is the round trip on the next slide and the gotchas slide after it.

The same colours, three shapes

All three are built from the same set of colours — only the axes they are plotted against change.

RGB and Lab are both rectangular — three perpendicular axes; only the meaning of the axes changes. HSV is the polar one: hue is an angle, so it wraps. Lab has a polar form too, LCh, where chroma is the radius and hue the angle — that is Lab's cylinder, and it is what "how colourful, and which colour" really means.

Drag one of the three and all three rotate together, because they are literally the same point set — the surface of the RGB cube, replotted. That correspondence is exact, not approximate, and it is worth saying: the cube's three faces where a coordinate equals 1 become the lid of the HSV cylinder (V = 1), and the three faces where a coordinate equals 0 become its outer wall (S = 1). HSV is not a different set of colours, it is the cube in polar coordinates. A CORRECTION worth making explicitly, because it is a common and sticky misconception: Lab is NOT a spherical system. It is rectangular, exactly like RGB — L* up, a* and b* perpendicular. What makes it different from RGB is not its geometry but what the axes measure. If someone wants the "round" version of Lab, that is LCh: same space, polar coordinates in the a*b* plane, C = sqrt(a*^2 + b*^2) and h = atan2(b*, a*). Then point at the Lab picture and ask why it is lumpy while the other two are neat. Because the neatness of the cube and the cylinder is an artefact of the coordinates, not of colour: RGB draws a cube because we DEFINED the axes to be the three lights. Lab's axes were fitted to human perception instead, so the set of colours a screen can make lands in it as an awkward blob. The gamut slide coming up is a horizontal slice through exactly this solid — say so, and the earlier faded region stops being mysterious.

So why are there so many?

Space	What it is best at	What it costs you
RGB	what sensors capture and screens emit — universal, and free	channels are correlated, and distance does not match perceived difference
HSV	picking and thresholding by colour; cheap to convert	hue wraps at 360° and is undefined at grey; still not perceptual
YCbCr	compression and transmission — JPEG keeps Y sharp and throws away most of Cb and Cr, because your eye barely notices	a transmission format, not a working space
Lab*	perceptual distance, and edits that leave lightness alone	needs a white point; the most arithmetic of the four

Pick the space the task needs: RGB to store it, HSV to select it, YCbCr to ship it, Lab to measure or edit it.

This table follows the comparison in Justin Johnson's EECS 442 lecture 4 at Michigan (slide credit there to James Hays), which is a good one to send students who want a second telling — it is the same four spaces with the same verdicts. YCbCr is on the slide because it is the one they have all used without knowing it. Every JPEG and essentially every video codec converts to it and then stores Cb and Cr at half or quarter resolution — "4:2:0 chroma subsampling" — while keeping Y at full resolution. That only works because luminance carries the detail your eye cares about. It is worth 30 seconds because it is the most consequential use of a colour space in everyday computing. The two swatch pairs make the RGB row concrete, so do not skip past them. Both pairs are 60 apart in RGB. The left pair is a ΔE of about 3, which is near the threshold people can see at all; the right pair is about 67, which is unmistakable. Same number in RGB, twenty times the perceived difference. That is why "nearest colour in RGB" is a bad way to match, segment or cluster colours — and why the fix is one conversion away. Honest caveat if a student pushes: Lab is only APPROXIMATELY uniform. ΔE76, the plain Euclidean distance we use here, is the simplest measure; CIE later published ΔE94 and ΔE2000 precisely because ΔE76 is still wrong in places, particularly for saturated colours. The direction of the argument holds regardless.

There and back again

Pick a space, read off the real channel ranges, then convert straight back to RGB.

The picture on the right is the round trip, and it looks perfect. It is not bit-identical: each hop re-quantises to 8 bits. Note the ranges under each channel too — H is 0–360° here, but OpenCV has to squeeze it into a byte.

Pick a pixel, read it four ways

Drag from the red macaw to the blue-and-gold one: a* swings from strongly positive to negative while L* barely moves. That single number is "how red versus how green" — which is why the exercise inverts it.

What the a* and b* axes actually are

Neutral grey sits at a* = b* = 0, in the centre. "Invert the a* axis" means reflect this plane left-to-right: every red becomes its matching green, and greys are left untouched.

Drag L* up and down and watch the solid region shrink as you approach white — this is a horizontal slice through the lumpy Lab solid from the geometry slide. Be careful how you describe the faded area, because students reasonably ask "are there colours RGB cannot show?" The answer is YES, but the faded region here is two different things mixed together: 1. Real colours that sRGB cannot reach. Any three primaries span a TRIANGLE in the chromaticity diagram, while the set of visible colours is a curved horseshoe — and a triangle cannot cover a curve. Concretely, NO pure single-wavelength colour is inside sRGB: a 520 nm laser green, a deep spectral cyan and a saturated violet are all outside it. Wider gamuts (Display P3, Rec.2020) reach more of them; printers reach a different set again, which is why print proofing exists. 2. Coordinates that are not colours at all. Lab is a mathematical space, so a triple like L* = 65, a* = 105, b* = −105 is a perfectly good set of numbers that no physical light can produce. A good chunk of the corners of this square are in that category. So: "faded = your screen cannot show it" is the safe statement, and it is what the caption says. Do not say "faded = colours you can see but your screen can't" — that is only true of part of it. The percentage is of THIS square, which is an arbitrary ±110 window, so do not let it be quoted as "sRGB covers N% of colours". It is there to show the gamut shrinking with L*. The centre-is-grey fact is the key to the next slide. Because grey is the fixed point, inverting a* leaves the lighting and the neutral parts of the scene alone, and only re-colours what was coloured.

Two ways to "swap the colours" — and why they differ

Swapping R and G also moves brightness around, because both channels carry luminance. Inverting a* changes only the green–red opponent axis: L* is untouched, so the shading and the shadows survive intact.

Four things OpenCV will do to you

1 · IMREAD GIVES YOU BGR

plt.imshow expects RGB, so a raw imshow(img) shows the birds with red and blue traded.

rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

2 · uint8 RANGES ARE SQUASHED

Hue is 0–360° but a byte only holds 0–255, so OpenCV stores H/2 → 0–179. Lab stores L×255/100 and a*, b* offset by +128, so neutral grey is 128, not 0.

3 · BGR2XYZ SKIPS THE GAMMA

It applies the XYZ matrix to the gamma-encoded values. BGR2Lab linearises first; BGR2XYZ does not — so the two are not consistent with each other.

For mid-red (128,0,0): OpenCV gives X = 53, the gamma-correct answer is X ≈ 23.

4 · ROUND TRIPS ARE NOT LOSSLESS

Every conversion re-quantises to 8 bits. On kodim23.png, converting there and back changes ~75–80% of pixels:

max |error| — HSV 5, Lab 8, XYZ 24. Convert to float first if you plan several hops.

EMPHASISE: point 4 is the one that gets marks. The worksheet titles a figure "hsv converted image" and it looks identical — it is not. Students should form the habit of measuring: np.max(np.abs(a.astype(int) - b.astype(int))) — the same instrument as last week. The numbers on this slide were measured on kodim23.png, not remembered — you can rerun them in one cell if a student challenges you. Point 3 is genuinely an inconsistency inside OpenCV, not a student mistake. If someone compares cv2's XYZ against a colour-science library and gets different answers, this is why. Do not let them conclude their code is broken. Point 2 explains a very common bug: students write mask = (hsv[:,:,0] > 300) to select magenta and get an empty mask, because hue never exceeds 179. A fifth one worth mentioning verbally: the solution notebook inverts a* with -img_lab[:,:,1] on a uint8 array. That is modular arithmetic — it computes 256 − a, which happens to be exactly right, because stored = a* + 128. It is correct but accidental-looking; 255 − a is off by one, and doing it in int then clipping is what you would actually write.

Part 1 in one line each

Colour is three integrals of a spectrum, not three properties of a surface.
Because it is three integrals, different spectra can look identical — metamerism is what makes displays possible.
Colour spaces are re-parameterisations: same information, different axes.
Pick the space where your edit is one operation on one axis.
Every conversion is lossy in 8 bits. Measure, don't eyeball.

Next: that spectrum did not come from nowhere. It is a light source, bounced off a surface, at an angle — and all three of those are baked into every pixel you have.

Light and surfaces

Where a pixel value actually comes from.

Diffuse (Lambertian) reflectance

\[ {\color{#7fd6a8} I_D(x)} \;=\; {\color{#f7d774} I_L} \;\cdot\; {\color{#f2a65a} R} \;\cdot\; \bigl(\mathbf{N}(x) \cdot \mathbf{L}\bigr) \]

A matte surface scatters light equally in all directions, so where you stand doesn't matter — only how the surface is tilted relative to the light. That single dot product is the entire shading model.

Three panels, read left to right: the scene, the geometry, the falloff. The middle panel is the one to teach from. N and L are both unit vectors — that is what the dashed semicircle is for — so the dot product is exactly the length of the blue bar: the shadow L casts onto N. That is what a dot product IS, and students who have only ever seen it as "multiply and add" often have not been told. Drag the direction slider and watch the bar shorten as L swings away. Then push past 90°. The bar disappears, the panel says N·L ≤ 0, and the right-hand curve goes flat at zero. That flat region is the max(0, ·) in the model — a surface facing away receives nothing, and there is no such thing as negative light. It is the only non-linear thing on this slide, and it is what makes the sharp terminator on the sphere. Note the readout says max(0, cos θ), not cos θ. At 105° the cosine really is −0.26; the zero is the clamp, not the arithmetic. Worth saying if anyone is checking on a calculator. Then make the point that the camera never moves in this demo and would not matter if it did — that is what "diffuse" means, and it is why Lambertian is the easy case. Turn R down and I_L up so the sphere returns to the same brightness. Ask: has anything about the object changed? Yes — but not in the image. Hold that thought for two slides. The warm/blue light buttons matter for part 1: the light's own colour multiplies the surface's reflectance, which is why white paper looks warm under a tungsten bulb, and why "what colour is this surface" is not a question the image can answer directly. Note N·L is clamped at zero — a surface facing away receives nothing. That clamp is the only non-linear thing here, and it is what creates the sharp terminator.

So what sets the R, G, B of one pixel?

\( I_L \) — THE LIGHT

Its intensity and its colour. A red surface under blue light is not red in the image.

\( R \) — THE SURFACE

Reflectance, or albedo: the fraction of each wavelength the material sends back. This is the thing we usually mean by "the object's colour".

\( \mathbf{N}\cdot\mathbf{L} \) — THE GEOMETRY

The angle between the surface normal and the direction to the light. Pure shape, no colour.

This is part (i) of the worksheet exercise. Three things — light, material, orientation — and the image only ever shows you their product.

The problem: only the product reaches the camera

A bright surface in dim light and a dark surface in bright light produce the identical image. Recovering the world from one picture is underconstrained — there are more unknowns than measurements.

Slide it slowly and let the middle figure stay visibly constant. The point lands harder if nobody says anything for a few seconds. This is the lecture's "goal of vision" slide restated as a problem: we want R and N, we measure only I_L · R · (N·L). Three unknowns, one number per channel. Every classical solution adds an assumption to close the gap: constant albedo (shape from shading), smooth normals, "lighting is smooth so sharp edges must be reflectance", known boundary normals. None works in general — that is the honest summary in the lecture. Modern answer: a network learns a prior over scenes from data instead of you writing one down. Same underconstrained problem, different source of the extra assumption. That is a nice one-sentence bridge to part 3 if you want it early.

Your visual system solves it anyway — by guessing well

\( \text{Luminance} \;=\; {\color{#f2a65a}\text{Reflectance}} \;\times\; {\color{#f7d774}\text{Illumination}} \)

WHAT YOU SEE

In Adelson's checker-shadow figure, two squares with identical pixel values look obviously different — because your visual system has already decided one of them is in shadow and is reporting reflectance, not brightness.

WHAT THAT COSTS

The split is a guess built on assumptions: illumination varies smoothly, shadows have soft edges, surfaces are locally uniform. Break the assumptions and the guess breaks — which is exactly what a visual illusion is.

"Lightness constancy" is the visual system doing an inverse problem with a prior. The illusion is not a bug — it is the price of the prior being usually right.

If you have a projector and 20 spare seconds, pull up persci.mit.edu/gallery/checkershadow live and cover the surround with two fingers so the two squares abut — they visibly become the same grey. It is the single most convincing demo in this part of the subject, and it is Adelson's own page (he is the source cited in the lecture slides). Link it back: this is the same decomposition as the lecture's "recovering surface reflectance" figure — an image factored into a reflectance layer and an illumination layer. Intrinsic image decomposition is still an active research area. Do not let this become a psychology tangent. The point for this subject is: the problem is underconstrained, so any solution — biological or learned — is importing assumptions.

Your turn: which parameter is changing?

All surfaces Lambertian, one light source. At the circle — what is changing, and what is constant?

Do NOT press reveal until they have committed to an answer. Take a show of hands per image. Walk them through the three questions on screen IN THAT ORDER — same surface? facing the same way? getting the same light? — because the order is what stops them from stopping at the first plausible answer. Answering out of order is how image 3 gets scored as "reflectance only". Image 1 — illumination changes. The arrow crosses the boundary of the QUEEN'S CAST SHADOW, staying on one continuous light square the whole way. Same wood and a flat board, so R and N never move; only the amount of light landing on the surface does. Point at the shadow and at the queen so the cause is named, not just the category. Image 2 — reflectance changes. Crossing from a light square to a dark one on the same flat, evenly lit board. Image 3 — reflectance AND the surface normal change; the light source is constant. The arrow leaves the flat board and climbs the curved base of the piece. The common wrong answer on image 3 is "reflectance only" — students see the colour change and stop. Push them: is the base of that piece facing the same way as the board? No. So N changed too, and you can see it in the gradient up the curve. The point of the whole exercise: a change in the image tells you something in the world changed, but NOT which of the three it was. That ambiguity is why the next slide matters.

Which is why vision leans so hard on edges

THE ARGUMENT

We cannot recover \(R\) or \(\mathbf{N}\) from one image. But a change in \(I_D\) reliably signals a change in something real: an occlusion boundary, a corner, a change of material, or a shadow.

Which is why last week's derivative filters are not a side topic — they are the most robust measurement we have.

AND THEY ARE INVARIANT

∂I/∂x is invariant to an intensity shift \(I + b\) and tolerant to a contrast change \(aI\). Turn the room lights up and the edges stay put.

Invariant to X = does not vary with X. Tolerant to X = mostly insensitive to X. The lecture's distinction, and it is examinable.

Almost every vision system starts by measuring local change — and, as we are about to see, a CNN rediscovers exactly that in its first layer without being told to.

Learned filters

The same convolution — with the numbers chosen by data.

Last week we chose the numbers. This week we don't.

Workshop 3	Workshop 4
you write np.array([[1,0,−1],[2,0,−2],[1,0,−1]])	you write layers.Conv2D(8, (5, 5))
one kernel, chosen because you know what it detects	8 kernels, initialised at random
cv2.filter2D slides it over the image	the layer slides all 8 over the image — identically
the output is "the vertical edges"	the output is "8 activation maps" — you find out later what they mean
you tune it by thinking	gradient descent tunes it by measuring the loss

The convolution operation does not change at all. What changes is where the 25 numbers in each kernel come from.

The dataset: MNIST

60,000 training and 10,000 test images, 28×28, one 8-bit channel, labels 0–9 — and roughly 6,000 examples per class, so accuracy is a fair metric here. On an unbalanced dataset it would not be.

Why divide by 255?

Gradient descent takes one step size in every direction at once. If features live on wildly different scales, the loss surface is a stretched valley and no single learning rate suits it.

Press the button. The contours go round and the path stops zig-zagging. Be precise about what the picture is: axes are two weights, contours are the loss, the orange line is the descent path. The stretch factor is the ratio of the input scales. For MNIST specifically the effect is milder than the picture suggests — all 784 inputs share the same 0–255 range, so they are already commensurate with each other. The real problems with unscaled inputs here are that the initial activations and gradients are ~255× larger than the initialisation assumes, so Adam's first steps are badly mis-scaled. The lesson generalises to any model where features have genuinely different units. Do not oversell this. astype(float)/255 is one line and always safe; that is the whole practical point.

An MLP starts by throwing the geometry away

The shuffled digit is unreadable to you — but that is not the point. Train an MLP on shuffled images and it scores exactly the same, because Flatten() already threw away which pixels were neighbours. The CNN is the one that breaks. Its advantage is a restriction, not extra power.

The shuffle argument is the cleanest way to make this concrete, and it is a real result: permute every image with the same fixed permutation and an MLP's achievable accuracy is unchanged, because a permutation of the inputs can be undone by permuting the first layer's weights. A conv net collapses, because its whole design assumes neighbours mean something. So a CNN's advantage is not extra capacity — it is a *constraint*, a prior that says nearby pixels are related and the same pattern is worth looking for everywhere. Constraints that are true about your data are free accuracy. If someone asks whether the MLP could learn the spatial structure from data: in principle yes, with enough data; in practice it wastes capacity rediscovering something you already knew.

The MLP, counted

Input28×28×1

→

Flatten7840

→

Dense 161612,560

→

Dense 1010170

→

softmaxp(0…9)

\( \underbrace{784 \times 16 + 16}_{12{,}560} \;+\; \underbrace{16 \times 10 + 10}_{170} \;=\; \mathbf{12{,}730} \)

A dense layer's weights are (inputs × outputs), plus one bias per output. Nearly every parameter is in the first layer — because it has to look at all 784 pixels at once.

The two non-linearities in this model

ReLU sits after each hidden layer: without something non-linear between them, stacked linear layers collapse into a single linear layer. Softmax sits at the very end, turning 10 unconstrained scores into a distribution.

The "collapse" argument is worth stating as algebra: W2(W1 x) = (W2 W1) x, which is just another matrix. Depth without non-linearity buys nothing. That single line justifies every activation function in the subject. Drag the scale slider: multiplying the scores by a larger number makes softmax more confident, by a smaller one more uniform. Nothing about the *ranking* changes — argmax is unaffected. So a confident-looking probability is not evidence of a correct answer. Connect to the loss: because the output is a probability distribution and the labels are integers, the loss is SparseCategoricalCrossentropy with from_logits=False — "sparse" because labels are 3, not [0,0,0,1,0,...]. If they one-hot the labels themselves, they must switch to CategoricalCrossentropy.

A conv layer is last week's sliding window

One kernel sweeps the whole image and produces one activation map (lectures 6–8's term; worksheet 5 calls the same thing a feature map). The same 25 weights are reused at every position — that reuse is called parameter sharing, and it is where the savings come from.

Where every shape comes from

\[ O \;=\; \left\lfloor \frac{W - F + 2P}{S} \right\rfloor + 1 \]

W input size F filter size P padding S stride

Layer	W	F	S	Output
Conv2D(8, 5×5)	28	5	1	(28−5)/1 + 1 = 24
MaxPooling2D(2×2)	24	2	2	(24−2)/2 + 1 = 12
Conv2D(16, 5×5)	12	5	1	(12−5)/1 + 1 = 8
MaxPooling2D(2×2)	8	2	2	(8−2)/2 + 1 = 4

Keras defaults to padding='valid' (P = 0), so every 5×5 conv costs you 4 pixels. Use padding='same' if you want the size preserved.

Lecture 7 writes the no-padding case as \( \lceil (W - F + 1)/S \rceil \) and the padded case as \( \lceil W/S \rceil \). The first is algebraically the same formula as the one above — use whichever you find easier to remember.

Make them do the arithmetic for at least one row before revealing it — this is exactly the kind of question that appears on the exam and in next week's worksheet. On the two forms: lecture 7 (CNNs II) gives ceil((W − F + 1)/S) for the no-padding case, which is the same number as floor((W − F)/S) + 1 for every W, F and S — write n = W − F as qS + r and both come to q + 1. Students sometimes think they have two competing rules; they have one. The CS231n form generalises more readably to non-zero padding, which is why it is on the slide. The floor only matters when the stride does not divide evenly; here it never does anything. Say so, so they do not think it is decorative. Common student error: forgetting that pooling shrinks by exactly a factor of 2 only because the pool size equals the stride. Keras's MaxPooling2D defaults strides=pool_size, which is why (2,2) gives non-overlapping blocks — the worksheet's table spells this out.

Max pooling: keep the strongest response, forget where exactly

It has no parameters. It buys a little tolerance to small shifts, and it shrinks every later layer's work by 4×.

Emphasise "no parameters" — students often expect pooling to be learned. It is a fixed reduction, which is why it appears with 0 in the summary output. Lecture 7 puts max pooling alongside two alternatives, worth naming here so the worksheet connects: strided convolution (cheapest, but the window skips over parts of the image) and average pooling (blurs over features, so strong responses get diluted). Max pooling is the one most likely to preserve the strongest feature, at the cost of being the slowest of the three. Note also that "tolerance to small shifts" is the lecture's careful wording — pooling does not buy true invariance. The trade: throwing away three quarters of the positions is exactly downsampling, and last week we spent a whole session on why downsampling without pre-filtering causes aliasing. Max pooling is not a low-pass filter. If you have 30 seconds, mention Zhang (ICML 2019), "Making Convolutional Networks Shift-Invariant Again" — blur before the stride and CNNs get measurably more shift-stable. Same argument as Assignment 1, applied inside a network. That link back to Week 3 is the most valuable thing on this slide.

The worksheet's CNN, layer by layer

Input28×28×1

→

Conv 8@5×524×24×8208

→

MaxPool 2×212×12×80

→

Conv 16@5×58×8×163,216

→

MaxPool 2×24×4×160

→

Flatten2560

→

Dense 10102,570

blue = output shape orange = trainable parameters

\( \underbrace{(5\!\times\!5\!\times\!1)\!\times\!8 + 8}_{208} \;+\; \underbrace{(5\!\times\!5\!\times\!8)\!\times\!16 + 16}_{3{,}216} \;+\; \underbrace{256\!\times\!10 + 10}_{2{,}570} \;=\; \mathbf{5{,}994} \)

A conv layer's cost is (filter area × input channels) × filters, plus one bias per filter — and crucially it does not depend on the image size.

What it actually learned

Several are clear oriented light-to-dark ramps — last week's derivative filters, discovered rather than designed. But look again: unlike Sobel, most of these sum well above zero, so they measure "how much ink is here" and "which way it runs" at the same time.

This is the payoff for the "why edges survive" slide in part 2. Say it explicitly: we argued from physics that local change is the robust thing to measure, and a network given nothing but pixels and labels partly rediscovered it. But do NOT oversell the resemblance — the honest reading is the interesting one. Measured on these actual weights: filters 0, 3 and 7 have strong directional gradients and really do look like oriented derivative operators. The other five are weaker and more diffuse. And every one of the eight has a positive weight sum, between +0.34 and +2.39, where a Sobel kernel sums to exactly zero. That difference is the lesson. A hand-designed edge filter is built to ignore absolute brightness. This network was never asked to; on MNIST, "is there ink in this patch" is a genuinely useful feature, so it learned filters that report both. Learned features answer the task you set, not the task in the textbook. All eight are active — none is dead — so do not claim otherwise if a student asks. Also worth flagging: at 5×5 with only 8 filters on MNIST, these are much less clean than the famous Gabor-like filters in AlexNet's first layer on ImageNet. Next week's worksheet visualises VGG16's, which are far more striking. If asked how these were obtained: this exact architecture was trained offline in numpy, and these are its real weights; the next slide runs the real forward pass in the browser.

The whole network, running right now

Sharp, digit-shaped responses early; small, abstract, hard-to-read maps late. That progression — from where the strokes are, to what the strokes mean — is the whole idea of a deep network.

MLP vs CNN — the answer to the worksheet's last question

	MLP	CNN
Trainable parameters	12,730	5,994
Test accuracy	≈ 95.4%	≈ 98.0%
Errors per 10,000 test digits	≈ 460	≈ 200
Where the parameters sit	99% in the first dense layer	54% in conv 2, 43% in the final dense
If the image doubled to 56×56	≈ 4× the parameters	conv layers unchanged; only the layer after Flatten grows
Training time (CPU)	seconds	minutes

Fewer than half the parameters, and less than half the errors. Not because the CNN is bigger, but because parameter sharing and locality are true of images — so the constraint costs nothing and saves a great deal.

The error-count row is there because "95% vs 98%" sounds like a small gap and "460 errors vs 200" does not. Same numbers, honest framing. Use it. Be precise on the last two rows, because the tempting slogan is wrong: - The CNN's parameters are NOT mostly in the dense layer here — conv 2 is the biggest single block at 3,216 of 5,994 (54%), because each of its 16 filters spans all 8 incoming channels. - "A CNN doesn't care about image size" is only true of the CONV layers. Double the input to 56×56 and the conv layers still hold 208 + 3,216, but the flatten goes from 256 to 1,936, so the final dense jumps from 2,570 to 19,370 — a 3.8× total, barely better than the MLP's 3.96×. That second point is worth the extra thirty seconds: it is exactly why real architectures replace flatten-then-dense with global average pooling, and it is the reason "where do the parameters actually live" is a question worth asking of every model. Note also the CNN is slower per epoch despite having fewer parameters — parameter count and compute are different resources. Students conflate these constantly. Caveat to state plainly: these are single-run numbers on an easy dataset, with different epoch counts (20 for the MLP, 10 for the CNN). The direction of the result is robust; the exact figures are not, and their own runs will differ by a few tenths of a percent.

What to carry out of this room

Habits that outlast the worksheet.

Take-homes — colour and light

1 · CHOOSE THE SPACE, THEN THE OPERATION

Before writing a colour manipulation, ask which axis of which space your intent is a single move along. If the answer is "three coupled changes in RGB", you are in the wrong space.

"Keep the lighting, change the hue" is one subtraction in Lab and a mess in RGB.

2 · A CONVERSION IS A COMPUTATION, NOT A RELABELLING

Every hop through cvtColor quantises, clips, and may or may not linearise. Convert to float once, do all your work, convert back once.

And check the ranges — the one that bites is hue at 0–179.

3 · A PIXEL IS A PRODUCT, NOT A PROPERTY

\(I_D = I_L \cdot R \cdot (\mathbf{N}\cdot\mathbf{L})\). When something in an image changes, the useful question is not "what colour is that" but which of the three moved?

Light, material, orientation. Getting into the habit of listing all three is most of the marks on this kind of question.

4 · UNDERCONSTRAINED MEANS YOU MUST ADD SOMETHING

One image, three unknowns. Every method that recovers shape or reflectance is smuggling in an assumption — smooth lighting, constant albedo, a learned prior.

Name the assumption out loud and you will predict where the method fails.

Take-homes — learning

5 · COUNT THE PARAMETERS BEFORE YOU TRAIN

(F·F·C_in)·C_out + C_out for conv, in·out + out for dense. It takes ten seconds and tells you where your model's capacity actually is.

Do the sum rather than guessing: in today's CNN it is the second conv layer that dominates, not the dense one — and that only flips as the input grows.

6 · AN ARCHITECTURE IS A SET OF ASSUMPTIONS

Convolution assumes nearby pixels are related and that a useful pattern is worth finding everywhere. Those are claims about images, and they are why the CNN wins with fewer parameters.

When an architecture underperforms, ask which of its assumptions your data violates.

7 · ACCURACY IS NOT A RESULT ON ITS OWN

Report it against a baseline, on a balanced set, from more than one run — and prefer error counts when the numbers are close to 100%.

95% vs 98% sounds marginal. 460 errors vs 200 does not. Same data.

8 · TRAIN, VALIDATION AND TEST ARE THREE DIFFERENT THINGS

validation_split=0.2 carves the validation set out of the training data. It guides your choices; it does not measure them.

The moment you tune anything against the test set, it stops being a test set.

EMPHASISE card 8. It is the one with real consequences for the assignment and the project. Say plainly: if you pick your epoch count by looking at test accuracy, your reported test accuracy is optimistically biased and no longer means what you claim it means. Card 5 connects directly to next week's worksheet, which opens with a parameter-counting exercise on a larger network. If they can do the two counts on today's slides, that question is free marks. Card 6 is the conceptual bridge to everything after this: transformers relax the locality assumption, and pay for it with data. Deliberately NOT covered today, because the worksheet does not use them, but flag that they are coming so nobody thinks this model is complete: regularisation (L1/L2, dropout, early stopping) and data augmentation, both in lecture 7, and transfer learning in lecture 8. Our CNN has no regularisation at all — it does not need it on MNIST, and that is itself worth saying, because the lectures introduce regularisation as the response to overfitting on harder datasets.

Your turn

worksheet04.ipynb

The five exercises

1 · RGB → XYZ AND RGB → Lab

Convert, show each channel, convert back.

Show channels with cmap='gray' — they are single-channel, not colour.

2 · INVERT THE a* AXIS

Convert to Lab, flip a, convert back.

Neutral is stored at 128, not 0 — so the flip is 255 − a, not −a on a raw uint8. Compare against the R↔G swap.

3 · WHICH PARAMETER IS CHANGING?

The three chessboard images: name the parameters, then say which move and which hold.

Part (i) wants all three named. Part (ii) wants a decision per image — and image 3 has two answers.

4 · RESHAPE FOR KERAS

Make the input 4-D: (N, 28, 28, 1).

np.expand_dims(x, axis=-1). Do this before the visualisation cell — it indexes [s,:,:,0] and will fail otherwise.

5 · BUILD THE CNN

Fill in the four missing layers, train it, then answer: how does it compare with the MLP on parameter count and test accuracy?

Read the table in the worksheet literally — 8 then 16 filters, 5×5, stride 1, ReLU, with 2×2 pooling after each. Then check your hand-computed parameter counts against summary().

Heads-up on the notebook: the exercises restart their numbering halfway through (there are two "Exercise 1"s and two "Exercise 2"s), and the shading question is labelled Exercise 5 in your copy but Exercise 3 in the solutions. Go by the section heading, not the number.

Walk the room in this order of likely trouble: Exercise 2 — the uint8 inversion. Many will write -img_lab[:,:,1] because the solution does. That works by modular arithmetic (it computes 256 − a) but it is not what anyone means to write, and it silently misbehaves at a = 0. Get them to do it in int and clip. Exercise 4 — the ordering trap. If they run the visualisation cell before doing the expand_dims, they get an IndexError, and it is not obvious why. Exercise 5 — the layer-2 parameter count. They will write 5*5*16+16 and get 416 instead of 3,216, forgetting the 8 input channels. Also warn them: summary() after training shows a bigger "Total params" than the trainable count, because Keras 3 counts Adam's optimiser state. The number the exercise wants is "Trainable params".

Cheat sheet


# --- colour ----------------------------------------------------------
img  = cv2.imread(path)                      # BGR, not RGB
rgb  = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)  # ...before plt.imshow
lab  = cv2.cvtColor(img, cv2.COLOR_BGR2Lab)  # uint8: L*255/100, a/b +128
lab[:, :, 1] = 255 - lab[:, :, 1]            # invert a*: reflect through grey
out  = cv2.cvtColor(lab, cv2.COLOR_Lab2BGR)
hsv  = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)  # uint8: H is 0-179, NOT 0-359

# --- networks --------------------------------------------------------
x = x.astype(float) / 255                    # scale before anything else
x = np.expand_dims(x, axis=-1)               # (N, 28, 28) -> (N, 28, 28, 1)

model = keras.Sequential([
    layers.Input((28, 28, 1)),
    layers.Conv2D(8, (5, 5), activation='relu'),   # -> 24x24x8,   208 params
    layers.MaxPooling2D((2, 2)),                   # -> 12x12x8,     0 params
    layers.Conv2D(16, (5, 5), activation='relu'),  # ->  8x8x16,  3216 params
    layers.MaxPooling2D((2, 2)),                   # ->  4x4x16,     0 params
    layers.Flatten(),                              # ->     256
    layers.Dense(10, activation='softmax')])       # ->      10,  2570 params
model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
              metrics=['accuracy'])                # "sparse" = labels are ints
model.summary()                                    # call it BEFORE fit()

Output size (W − F + 2P)/S + 1 · conv params (F·F·C_in)·C_out + C_out · dense params in·out + out.

Play with these afterwards

CNN Explainer (Georgia Tech) — click any neuron, see the convolution that made it poloclub.github.io/cnn-explainer/
Adelson's checker-shadow illusion (MIT) — the original, with the proof image persci.mit.edu/gallery/checkershadow
Foundations of Computer Vision (MIT Press) — ch. 8 Colour, ch. 24 ConvNets visionbook.mit.edu/color.html
CS231n notes: Convolutional Networks (Stanford) — shapes, parameter sharing, the conv demo cs231n.github.io/convolutional-networks/
EECS 442 lecture 4, Light + Color (Michigan) — the same colour-space comparison, told again web.eecs.umich.edu/~justincj/slides/eecs442/winter2020/
A guide to convolution arithmetic for deep learning — Dumoulin & Visin's animations github.com/vdumoulin/conv_arithmetic
Metamers demo (Brown) — sketch two spectra, match their colour cs.brown.edu/courses/cs123/archive/2020/demos/metamers/
OpenCV colour conversion reference — the exact formulas, including the ranges docs.opencv.org/4.x/de/d25/imgproc_color_conversions.html

Next week: what those learned filters look like deeper in a real network (VGG16), and how to make convolution cheaper.

End of workshop 4

Questions?

Press S for speaker notes · O for the slide overview · F for fullscreen