Two views.
A third dimension.
A pixel tells you a direction. A second view lets you recover depth. Discover how camera geometry and careful matching turn two photographs into a 3D measurement.
Start the workshop ↗By the end, you should be able to explain each arrow: what calibration estimates, where a correspondence can lie, why stereo images must be rectified, and when disparity can become depth in metres.
left.jpeg / right.jpeg demonstrate epipolar geometry; Tsukuba and Aloe demonstrate rectified stereo. The checkerboard calibration is not the calibration of the other pairs. Their supplied files contain no metric stereo calibration or depth ground truth.Live calculated in your browser Measured precomputed from the supplied images with OpenCV 4.13.0 Illustration controlled geometry, not a measurement of the images.
See the cameras in space.
A photograph is a 2D slice of a 3D world. Drag the scene below. Each camera is a pinhole plus an image plane; each pixel is a ray. Depth appears only when two rays from different viewpoints meet.
A stereo rig looking at a desk
Live 3D illustrationThis is the physical situation behind Tsukuba: two cameras, a baseline, and objects at different depths. Choose an object. Both cameras send a ray through it. The left and right image points pL and pR are simply where those rays punch the image planes.
Drag to orbit · scroll to zoom · arrows also work
Illustration, not a reconstruction of Tsukuba. Y is up. Image planes are drawn in front of the pinholes (the usual computer-vision convention). OpenCV’s camera frame is X right, Y down, Z forward along the optical axis — the same pinhole, a different axis label.
What to notice. One pixel only names a direction. The lamp and the far shelf can project to nearby pixels if you pick the wrong depth. The second camera’s ray is what pins P down. That is the whole course in one picture: calibration describes each camera; epipolar geometry says where the second projection may lie; matching picks the point; Z = fB/d converts the leftover shift into metres.
Straight lines, curved images.
Calibration estimates the camera’s internal geometry and distortion. It does not require two images to contain identical features; stereo needs sufficient shared, visible scene content.
The pinhole, the image plane, and a known board
Live 3D illustrationCalibration is a 3D fitting problem. Every inner corner is a known point on a rigid plane. The camera is a pinhole a distance f in front of an image plane. Drag the board’s tilt and distance; the projected corners on the gold image plane are what calibrateCamera observes.
Drag to orbit · scroll to zoom
Increasing f pulls the image plane farther from C and magnifies the board. Increasing distance shrinks the image. A single view can trade those two effects; several tilted views cannot, which is why Exercise 1 uses more than one photograph.
Extra bonus: calibrate for your own camera. This page uses OpenCV’s own sample chessboard photos, but the same steps work for any camera:
- Get a chessboard pattern and print it.
- Put the printed pattern on something flat and rigid.
- Take at least 10 photos of it, varying angle and distance.
Where does the distortion formula come from?
ProblemProblem. A pinhole camera maps a 3D ray to a pixel by straight-line projection. A real lens does not: it bends light more the further a ray sits from the optical axis, so straight scene lines land on curved image lines. A correction has to be fitted from photographs alone.
Approach. Model the deviation as a function of radial distance from the principal point, in normalized (unitless) camera coordinates — the same coordinates an ideal pinhole model would predict before any pixel scaling.
Remove focal length and principal point from the observed pixel to get an ideal, undistorted coordinate:
Distortion is radially symmetric around the principal point, so it can only depend on distance from the centre:
A smooth, radially symmetric deviation is a polynomial in . Reflection symmetry rules out odd powers of , so the standard model keeps only even ones, truncated at the sixth:
Real lenses also decentre slightly, adding two tangential terms (p₁, p₂). The live illustration sets them to zero so the k₁-only mapping stays easy to read; OpenCV still fits all five together.
Solving it is the hard direction. This equation distorts an ideal point outward or inward. Undoing it — given a distorted pixel, recovering the ideal one — has no closed form, because on the right is measured in the coordinate you don’t have yet. OpenCV’s undistort routines solve this iteratively, or by building a dense forward map and resampling — never by dividing by the same factor.
What does radial distortion do?
Live illustrationDashed grey: ideal grid. Solid teal: forward-distorted grid. Other coefficients are zero here; using the measured k₁ alone does not reproduce the full camera model.
At k₁ = 0, the two grids coincide.
x and y are ideal normalized camera coordinates, before distortion: for an ideal pixel, and . The origin is the principal point, which need not be the image centre.
Undistortion reverses this model. Multiplying an observed coordinate by the same radial factor is not an inverse correction. OpenCV creates a mapping between corrected output pixels and samples in the distorted input.
The grid uses a restricted radius so the k₁-only mapping stays monotonic across the slider range. Negative k₁ gives barrel distortion; positive k₁ gives pincushion distortion.
Where do fₓ, fᵧ, cₓ, cᵧ come from?
ProblemProblem. A pinhole camera’s intrinsics — focal length and principal point — are fixed properties of one physical camera, but nothing prints them on the lens. They have to be estimated from images of something whose real-world geometry is already known.
Approach. Photograph a flat checkerboard, whose corner spacing is known exactly, from several angles. Every corner supplies one known 3D point and one observed pixel; fit the camera model that explains all of them at once.
The pinhole model projects a 3D point to a pixel, up to an unknown scale s:
K holds the intrinsics shared by every photograph; [R∣t] is that photograph’s own board pose, and changes view to view.
For a candidate K, distortion, and every view’s R, t, project each known corner and compare it against where
cornerSubPixactually found it. Summarise the disagreement over all N observed corners, in every view, in one number:calibrateCameraadjusts K, the five distortion coefficients, and every view’s pose together to make that number as small as possible, starting from a closed-form estimate. Nine views converged to the 0.156 px figure below.
Why one photo isn’t enough. A single planar view fixes only a lower-dimensional slice of K — a board at one distance and tilt can trade focal length against board distance and still reproject almost perfectly. Multiple views at different angles are what pins the parameters apart, which is exactly what Exercise 1 asks you to confirm.
Recover K from the actual checkerboards
Measured · 13 images(i) Set up
(ii) Calibration
| Parameter | All nine views |
|---|---|
| fₓ, fᵧ | 534.16, 534.25 px |
| cₓ, cᵧ | 341.71, 232.05 px |
| k₁, k₂, k₃ | −0.29427, 0.12325, 0.01021 |
| p₁, p₂ (tangential) | 0.001139, −0.000138 |
OpenCV distortion-vector order is [k₁, k₂, p₁, p₂, k₃]. The 7 × 6 pattern counts inner corners, not squares. Four failed detections are retained in the viewer; a failure alone does not identify its cause.
More information on the functions used above, see: cv2.findChessboardCorners, cv2.cornerSubPix, cv2.drawChessboardCorners (set up), and cv2.calibrateCamera (calibration).
How to judge calibration, beyond one RMS number
Use sharp, complete boards at varied tilts, distances and image locations. Repeated nearly identical views add little information. Reprojection RMS measures agreement with the fitted corner observations; it is not independent evidence of correct intrinsics.
Inspect per-view residuals, their spatial pattern, plausible parameter values, stability across subsets and performance on held-out views. A new view still needs its own fitted pose to evaluate reprojection. Change of resolution, crop, focus, zoom or camera settings can require transformed intrinsics or recalibration.
| Detected view | RMS (px) |
|---|
(iii) UndistortionAfter undistortion
Measured · left12.jpegOnce K and the distortion coefficients are known, every photo from this camera can be corrected the same way. Edges that the lens bent now come out straight.
More information on the function used here, see: cv2.getOptimalNewCameraMatrix.
Can one photo fool the error metric?
Predict: will one-image calibration always have a larger reprojection error? Choose a training view, then compare both camera parameters and the undistortion of the same test photo.
Both outputs use getOptimalNewCameraMatrix with α = 1 and the full 640 × 480 canvas, without cropping. Different new camera matrices change field of view as well as the visible warp. The all-view fit includes left12; this is a sensitivity demonstration, not held-out validation.
| Quantity | Nine views | Selected single view |
|---|
From worksheet08_solution.ipynb, Exercise 1 Solution: “The result varies depending on which image you use, but the calibration (and the undistortion result) generally will not be correct. It may fail to compensate for the radial distortion of the lens, or it may assume a wrong focal length so the ‘undistorted’ image will look stretched.”
Reveal the reasoning
The match lies on a line.
A point in the left image defines a ray through the left camera. Seen by the right camera, that ray projects to an epipolar line. Geometry narrows the search; appearance still has to identify the match.
Background reading from the source notebook: the Wikipedia article on epipolar geometry illustrates baseline, epipoles and epipolar planes for a general camera pair; CVonline’s explanation works through the same concepts from the ground up.
One left pixel, many possible scene depths
Live geometric illustrationMove P along the ray from the left camera. Its left-image projection stays fixed. Its right-image projection moves along one epipolar line, because both camera centres and the entire ray lie in one plane.
Oblique schematic of a parallel, horizontal pinhole pair; image planes are drawn in front of the cameras. Teal shaded triangle: part of the epipolar plane. Blue rectangles: image planes. Dashed gold lines: intersections with that plane. This special case has parallel epilines and epipoles at infinity; a general camera pose can give slanted epilines and finite epipoles.
The epipolar plane is a real triangle in 3D
Live 3D illustrationThe 2D schematic above is the same geometry, flattened. Here you can walk around it. The teal triangle is the epipolar plane: it always contains both camera centres and the whole left ray. The right camera therefore sees that ray as a single line on its image plane.
Drag to orbit · scroll to zoom
Slide P farther away: pL stays put (same left ray) while pR travels along the gold epipolar line. Add toe-in and the cameras converge — the line tilts, and the epipoles become finite. Parallel stereo is the special case used for Tsukuba and Aloe.
(i) Find the Fundamental Matrix
Where does the epipolar line come from?
ProblemProblem. A pixel in the left photo could be the image of a 3D point at any depth along one ray. Searching the whole right photo for its match would be slow, and mostly spent on candidates geometry already rules out.
Approach. The two camera centres and any 3D point on that ray all lie in one plane — the epipolar plane. Turning that single fact into algebra gives a matrix equation every true match must satisfy.
The same 3D point has coordinates in the left camera’s frame and in the right’s, related by the rigid motion between the two cameras:
Each camera’s image ray points along that camera’s own coordinate vector to the point (, are normalized directions — pixels with the intrinsics already divided out):
Substitute = and = into step 1 and rearrange: the baseline becomes a combination of and alone — the unknown depths cancel:
A vector that is a combination of two others is coplanar with them, and three coplanar vectors have zero scalar triple product:
Writing the cross product as multiplication by the skew-symmetric matrix leaves a plain bilinear equation in the two image rays — the essential matrix :
Real pixels also carry each camera’s own intrinsics. Substituting the normalized-to-pixel relation for both cameras and absorbing , into one matrix gives the pixel-level form used throughout this page:
Where rank two comes from. is skew-symmetric, so it always has rank two ( lies in its own null space); multiplying by the full-rank rotation and the invertible , cannot raise that. That is the rank-two structure the panel on the right relies on — and because the defining equation is homogeneous, scaling by any nonzero constant changes nothing, which is why it is only meaningful up to scale.
Explore the fundamental matrix
Measured F · live projectionClick either photograph to draw the corresponding line in the other. Or use the stored-inlier control to inspect a matched pair and its point-to-line residual.
The equality is exact for ideal corresponding pinhole projections. Estimated features, residual lens distortion and an estimated F produce nonzero residuals. A point being close to an epiline is a necessary geometric consistency check, not proof that it is the correct match.
SIFT and FLANN results are precomputed, not rerun in JavaScript. Each threshold has its own RANSAC-estimated F: 1 px threshold, confidence 0.999, at most 10,000 iterations, RNG seed 8. These example F matrices fit the supplied raw photographs; unknown lens distortion is not corrected using the unrelated checkerboard K. Counts vary by OpenCV version and matching settings.
Inspect F and understand its scale
F has rank two in the nondegenerate model and is only defined up to a nonzero scalar. Compare epilines or scale-invariant residuals, not raw matrix entries. cv2.findFundamentalMat computed F above; cv2.computeCorrespondEpilines, cv2.line and cv2.circle draw the lines you clicked into existence above. Point-to-line distance is , in pixels.
Filter appearance first, geometry second.
- Create SIFT descriptors and two-nearest-neighbour FLANN matches.
- Keep a match when distance₁ < τ × distance₂. Increase τ from 0.60 to 0.90 and record counts.
- Fit F robustly. Inspect inlier residuals and whether features cover the image.
The ratio test removes ambiguous descriptor matches; RANSAC rejects pairs inconsistent with one epipolar geometry. A larger match count is not itself better.
Reveal robust implementation advice
Undistorted is not rectified.
Undistortion corrects the lens. For the horizontal stereo setup used here, rectification warps a pair so corresponding epipolar lines share image rows, enabling a horizontal disparity search.
Rectification is a virtual rotation
Live 3D illustrationNothing in the room moves. Rectification rotates each camera so the image planes become coplanar and the baseline is parallel to the image x-axis. Correspondences that used to lie on tilted lines then share a row — which is why stereo matching can search horizontally.
Drag to orbit · use the slider to rectify
Four scene points stay fixed. Watch their image-plane projections: before rectification the dashed joins are skewed; after, they become horizontal. Undistortion (Section 1) is a different warp — it straightens the lens, not the stereo geometry.
What has to be true for rows to line up?
ProblemProblem. The fundamental matrix only promises a match lies on some line. In general that line is tilted, and a different tilt for every point — expensive and awkward to search directly.
Approach. Warp each image with its own homography so every epipolar line becomes horizontal and lands on the same row in both images. “Search the epiline” then becomes “search this one row.”
Each rectifying warp is a homography — a linear map on homogeneous pixel coordinates:
Every epipolar line in an image passes through that image’s epipole . For every epipolar line to become horizontal, itself has to map to the point at infinity in the horizontal direction — the one direction with no finite image:
Choosing L and R this way, while keeping the two warps consistent with each other, is exactly what leaves matching points on the same row:
What this does and doesn’t fix. stereoRectifyUncalibrated only has and the matched pixels to work with, so it can straighten and align rows — it has no way to recover focal length, baseline or scale. Do that first with paired calibration and stereoRectify if metric depth is the goal, not just an aligned search direction.
From general epilines to shared scanlines
Measured projective warpsThe black regions are areas without source-image support after warping. They must not be treated as ordinary texture. Compare the same scene feature across the images, not the borders of the two canvases.
stereoRectify and initUndistortRectifyMap; retain the returned projection matrices or Q.How does the physical camera setup matter?
Why did this pixel choose that shift?
Compare a left patch with candidate right patches along the same row. The sum of absolute intensity differences (SAD) gives a cost; the smallest cost wins.
From the source notebook: the goal is a greyscale heat map where lighter greys mean objects close to the lens and darker greys mean objects further away. Two of its practical notes carry over directly — calibrate and undistort your own images first, and keep both cameras at the same height, as your eyes are; this blog post covers building your own stereo rig.
More information on the functions used below, see: cv2.StereoBM_create, cv2.StereoSGBM_create.
Where does the SAD cost come from?
ProblemProblem. Given a left pixel and a candidate shift , we need a single number that says how well the two neighbourhoods agree — low for the same surface point, high otherwise.
Approach. Comparing single pixel intensities is too noisy on its own — sensor noise or a slight exposure difference can make a correct match look wrong. Compare small windows instead, and sum the disagreement.
Take a × window around the left pixel, and the same-shaped window shifted by pixels in the right image. Sum the absolute intensity difference over the whole window:
Repeat for every candidate shift from 0 to D−1, and keep whichever produced the lowest cost — a winner-take-all rule:
Why this can still be wrong. Arg-min always returns some disparity, even over a flat wall where every shift scores about the same. A low cost is necessary for a good match; it is not sufficient — exactly what the ambiguity filter in the lab below is checking for.
The live SAD laboratory
Computed in your browserChoose a textured point or a flat region. Move the candidate disparity and watch both the patch location and its cost. Then change the window or search range to recompute the whole map.
Preparing the stereo pair…
Exactly what is this browser implementation doing?
Tsukuba is area-averaged to 192 × 144 pixels. All disparity and patch coordinates here refer to that half-resolution grid; a shift of 10 px here corresponds to 20 px at the original size.
The matcher evaluates integer disparities d = 0,…,D−1 using a full b × b window and integral-image SAD sums. Every valid pixel uses the full search range: the common ROI excludes h = (b−1)/2 pixels at top, bottom and right, and h+D−1 pixels at left. Missing image support is masked instead of being filled with invented intensity values.
The optional educational filter rejects patches with intensity standard deviation < 2 (on 0–255 intensity) or a best-vs-second-best cost gap ≤ 5% of the second-best cost. Ties are therefore rejected. This is not OpenCV’s uniqueness implementation, and neither this score gap nor the retained fraction measures accuracy.
This transparent SAD matcher does not reproduce OpenCV StereoBM: it omits prefiltering, subpixel refinement, speckle cleanup and left-right consistency checks. The next section shows actual OpenCV outputs.
Change one parameter at a time.
- Compare 3 × 3 and 21 × 21 at the same disparity range. Inspect thin structures and depth boundaries.
- Reduce D from 32 to 16. If the true shift lies outside the search, the matcher must reject or choose a wrong candidate; it need not return the maximum shift.
- Compare raw winners with the ambiguity filter on. Explain why a filled-in map is not automatically better.
Local matching meets spatial regularity.
StereoBM uses local block evidence plus filtering. StereoSGBM aggregates matching costs along multiple paths and penalizes disparity changes. Neither guarantees a correct or complete map.
Lift the disparity map into a point cloud
Live 3D from the SAD mapA disparity map is already 3D information, flattened into colours. Here every valid SAD match is back-projected with an illustrative pinhole: X = (u − cₓ) Z / f, Z = fB / d, and coloured by the left Tsukuba pixel. Orbit it: the lamp, statue and desk separate in depth.
Drag to orbit · rebuilds when the SAD settings change
Pedagogical geometry, not Tsukuba’s missing metric calibration. Rejected (magenta) pixels never become points — that is why a “complete-looking” filled map is not automatically a better reconstruction. Change block size or the disparity range in the SAD laboratory and watch thin structures appear or collapse here.
OpenCV results explorer
36 precomputed mapsKeep the color scale fixed. Each dataset uses one fixed disparity scale across all settings, so colors remain comparable. Magenta is invalid, not far away. The retained-pixel fraction is coverage after filtering, not ground-truth accuracy.
Where do the smoothness penalties come from?
ProblemProblem. Block matching scores every pixel on its own. Nothing stops a neighbourhood of nearly tied costs from picking wildly different winners pixel to pixel — the noisy, speckled BM column on the left.
Approach. Add a second term to the cost that penalizes a pixel’s disparity for disagreeing with its neighbours, then minimize both together.
The energy SGBM approximately minimizes, over the whole map D, combines the per-pixel matching cost with a penalty on how much neighbouring disparities differ:
The two penalties are deliberately unequal:
A one-step change (a sloped surface) costs little; a bigger jump (a real depth edge) costs more but is still finite — so genuine discontinuities survive instead of being smoothed away.
Exact minimization over a full 2-D image is intractable, so SGM aggregates cost along several 1-D paths through each pixel — exact along any single path by dynamic programming — and sums the results. Semi-global, not global.
This explorer’s fixed rule scales both penalties with window area and channel count, so a bigger window does not automatically make disparity jumps look proportionally cheaper:
Parameters, matching penalties and what these results establish
SGBM’s P₁ penalizes a one-pixel disparity change and P₂ penalizes larger changes along a path. P₂ > P₁. Here , with C the channel count. As b changes, these penalties also change following that rule; this explorer is a practical configuration comparison, not an isolated causal test of window size.
Aloe uses color for SGBM and grayscale for BM, as in the worksheet. Thus differences there include both matcher choice and color information. Tsukuba is grayscale for both. No provided disparity ground truth means we can discuss apparent artifacts, but cannot certify one setting as most accurate.
Make Aloe’s leaves survive the reconstruction.
Select Aloe. Compare D = 64, 96 and 128, then b = 5, 9 and 15. Inspect thin leaves, the background, textureless surfaces and occlusion boundaries. Explain the trade-offs rather than chasing a visually smooth target.
raw.astype(np.float32) / 16). Preserve it before making any display image.For minDisparity = 0, invalid output uses raw = −16. Keep an explicit validity mask. An accepted zero disparity still cannot produce a finite Z via fB/d. Do not apply min–max image normalization before reconstruction.
Reveal a practical answer outline
Exercise 4 Solution
worksheet08_solution.ipynbThe worksheet’s own solution tunes StereoSGBM specifically for this pair: minDisparity=16, numDisparities=96 (112 − 16, divisible by 16), blockSize=16, with P₁/P₂ scaled from a window size of 3 across 3 colour channels (P1 = 8×3×3² = 216, P2 = 32×3×3² = 864), disp12MaxDiff=1, uniquenessRatio=10, speckleWindowSize=100, speckleRange=32.
Computed with the parameters above — compare it to your own settings in the explorerReference source for these tuned parameters (cited in the worksheet solution): OpenCV’s own stereo_match.py sample.
A small shift can mean a long way.
For a calibrated horizontal stereo pair with aligned principal points, disparity and depth . The focal length and disparity must use the same pixel scale.
Where does Z = fB/d come from?
ProblemProblem. Rectification turns matching into a 1-D search and returns a pixel shift, . A pixel shift is not a distance — it has to become metres.
Approach. Similar triangles. Two rectified cameras share a focal length and image row, offset by baseline along one axis; project the same 3D point into both and compare.
Place the left camera at the origin and the right camera at (B, 0, 0), both looking down +Z. A point at (X, Y, Z) projects in the left image at:
In the right camera’s own frame the same point’s X-coordinate is shifted by −B:
Disparity is the difference between the two projections. Z and X cancel out of everything except the shift:
Solve for Z:
To see how a small error in moves Z, differentiate:
For a disparity uncertain by ± (with much smaller than ), that slope gives the error bound used in the calculator:
Why this gets sensitive at range. The 1/² in that last line is the whole story: as objects get farther away, shrinks, and the same one-pixel matching error swings Z by much more. Try it in the calculator below — halve the disparity and watch the same ε open a much wider interval.
Similar triangles, standing in the room
Live 3D illustrationThe formula Z = fB / d is two similar triangles: the baseline B at the cameras, and the leftover shift d on the image planes. This view is wired to the sliders in the calculator below. Move disparity and watch P fly away; the gold segment on the image planes is d.
Drag to orbit · uses the f, B and d sliders below
When d is small, P is far and a 1 px matching error becomes a large error in Z. That is why distant, textureless regions on Aloe and Tsukuba are the first to look geometrically implausible.
A stereo depth calculator
Live · synthetic camera“Halve resolution” halves f, d and ε together; B stays fixed. It demonstrates the coordinate scaling law, not the noise characteristics of a resampling algorithm.
Axial depth Z
Z is distance along the rectified optical axis, not Euclidean distance from the camera centre. For metric XYZ use the calibrated projection model or Q. If rectified principal points differ, use the disparity offset or Q rather than blindly applying fB/d.
What can the images really tell you?
These questions separate plausible visual impressions from the geometry needed to measure distances.
Two cues per scene. What does each recover?
Use the original room and lake photographs from the worksheet. Choose two cues in each, then separate what you can infer from what would need additional assumptions.
Images embedded from the image collection linked by the source notebook.
Reveal a worked answer
From worksheet08_solution.ipynb: Room — Perspective (lines on floor, walls) gives relative depth on the surface; Occlusion (chairs occlude the table) and Height in image (chairs) both give rank order, not actual depth; Real world size of known objects (chairs) could give distance if the camera is calibrated. Lake — Height in image (boats) gives rank order, not actual depth; Texture (water) could give relative distance or surface normal; Real world size of known objects (boats) could give distance if calibrated.
| Scene / cue | What it supports | What it does not give by itself |
|---|---|---|
| Room: parallel wall edges and floorboards | A vanishing point and perspective structure; with calibration and known geometry, stronger reconstruction constraints. | Absolute distances without scale and camera/scene assumptions. |
| Room: chairs occlude the table or wall | Which visible surface is in front at the overlap. | The numerical separation between the overlapping surfaces. |
| Lake: boats have different apparent sizes | Relative distance under an assumption of comparable true boat/sail dimensions. | A reliable depth ordering if the boats actually have very different sizes, or absolute distance without calibration/scale. |
| Lake: boat–water contact points approach the horizon | On a shared horizontal water plane, points closer to the horizon are farther away, given a suitable camera model. | Metric distance without camera geometry and a scale such as camera height. |
The changing apparent scale of wave texture is another possible cue, provided the physical texture is sufficiently comparable. These images do not justify claiming that a reflection or apparent size alone gives a calibrated depth measurement.
Explain epipoles. Then recover the missing scale.
6a. Without formulas, explain why all epipolar lines in one image meet at the projection of the other camera centre.
Reveal the geometric explanation
6b. Two calibrated cameras are 10 m apart. Their relative rotation and translation direction are unknown. Can corresponding points give you metric distances?
Reveal the full recovery route and its conditions
From worksheet08_solution.ipynb: “Compute the essential matrix from the matched points, decompose into rotation and translation between cameras. Since the baseline is known (10 m) and the cameras are calibrated, the distances to matched points can be triangulated from their view ray in each camera. Constraints: requires a minimum number of matches (5–8 depending on algorithm) and the matches cannot be coplanar.”
Yes, generically. Undistort and normalize matched points using each camera’s own calibration. Estimate an essential matrix robustly (or transform a compatible F using E = Kᵣᵀ F Kₗ), recover the relative rotation and translation direction, and choose a physically valid pose using positive depths in both cameras.
Translation from essential-matrix recovery has unknown magnitude. Scale its unit direction to 10 m, triangulate with the calibrated projection matrices and that translation, then compute either axial depth Z or Euclidean distance √(X²+Y²+Z²), as required.
This needs a rigid/static scene, reliable and well-distributed matches, correct calibration and sufficient translational parallax. Degenerate or ambiguous arrangements, particularly planar configurations for unconstrained F estimation, need care or additional constraints; tiny parallax and distant points produce poor numerical accuracy. Knowing the baseline does not solve wrong correspondences.
A virtual object, anchored to the board.
Known 3D board corners and their 2D detections let solvePnP estimate a board-to-camera pose. Project a new object through that pose and the lens model to overlay it on the original photograph.
The camera, the board, and the cube — in the world
Live 3D illustrationThe photograph overlay is a projection. This is the same pose in world coordinates: the checkerboard sits on the ground, solvePnP places the camera, and the cube is specified in board units then drawn with the same rigid motion. Change the photo or the cube height and both views stay consistent.
Drag to orbit · follows the pose controls below
Board frame: X along the squares, Y along the other side, Z out of the board (mapped to height here). The white dot is the first inner corner — the same origin used by the 2D overlay.
How does a 3D point become a drawn pixel?
ProblemProblem. solvePnP gives a pose — where the camera sits relative to the board. To draw a cube “sitting on” the board, every one of its 3D corners has to be pushed through that same camera and lens to find its pixel.
Approach. Run each corner through exactly the pipeline calibration fitted, in reverse: rigid motion, perspective division, lens distortion, then intrinsics.
Move the board-frame point into the camera’s own frame, using the rigid motion
solvePnPestimated:Perspective projection: divide by depth to get a normalized image coordinate:
Apply the same distortion model calibration fitted — forward this time, point to pixel, so it is the direct formula rather than its inverse:
Apply the intrinsics to land on an actual pixel:
Same code, real numbers. This is exactly the computation the live projection below runs for every cube corner and axis tip, using this camera’s own K and five distortion coefficients — not a simplified stand-in.
Project a cube and coordinate axes
Live projection · measured poseFor this board coordinate convention, negative board Z extends outward toward the viewer. It is not negative camera depth. The cube footprint is three squares wide.
The pose was estimated with solvePnP using the nine-view calibration. Shape height and projection are computed live, including all five distortion coefficients. Overlays do not infer occlusion or lighting from the real scene.
The corrected implementation.
A runnable Python reference, a compact troubleshooting guide and a record of the technical corrections.
Run the workshop yourself
- Extract the supplied
week8.zip. Keep the image names and thechessboardfolder. - Download the reference script below into the extracted
week8directory. It expects that working directory. - Use Python with NumPy and OpenCV 4.x; this tutorial’s saved results were computed with OpenCV 4.13.0.
python -m pip install numpy "opencv-python==4.13.0.92" python week8-reference.py
Use one OpenCV Python package per environment. If already using a compatible OpenCV 4.x installation, do not install a second package variant. SIFT is available in the main package for these versions.
The script writes calibration parameters, rectified photographs, numerical disparity arrays with validity masks, display images and a cube overlay. It does not invent metric calibration for Tsukuba or Aloe. The worked notebook packages the same reference functions with experiment prompts.
Read or copy the complete reference script
When a result looks wrong
| Symptom | Check first | Why |
|---|---|---|
| Calibration looks good only on one view | View diversity, corner order, residuals and subset stability | Low training error can hide uncertain intrinsics and distortion. |
| Epilines miss most matches | Left/right convention, F transpose, image resizing, outliers | F and point coordinates must belong to the same coordinate systems. |
| Horizontal search fails across the image | Rectification and vertical alignment | Lens correction alone does not align corresponding rows. |
| A band at the left is invalid | Search range and valid stereo ROI | Some candidate patches would be outside the right image. |
| Depth is wrong by a factor of 16 | Fixed-point conversion | BM/SGBM disparities must be divided by 16 before geometry. |
| Depth changes after resizing | Scale f and d consistently; update K/Q | Geometry uses pixel coordinates at the actual working resolution. |
| Bad results near mirrors, leaves or plain walls | Occlusion, non-Lambertian appearance, ambiguity and window mixing | Geometry cannot create a visible, distinctive correspondence where none exists. |
| Projected cube floats or flips | Board coordinates, pose convention, distortion and image crop | The image and camera model must describe the same pixel frame. |
Validation record and changes from the supplied tutorial
- Recomputed all thirteen checkerboard detections and nine-view calibration. Reproduced the reported approximately 0.156 px RMS and k₁ ≈ −0.294.
- Corrected the forward distortion equation and normalized-coordinate definition; bounded the illustrative grid to avoid non-monotonic radial folding.
- Added measured single-view experiments, with explicit differences between training fit, parameter stability and held-out validation.
- Retained subpixel matches, used guarded RANSAC estimation and replaced fixed target counts with measured, version-specific values. Added scale-invariant line residuals and safe border clipping.
- Added the missing rectification stage and separated uncalibrated projective alignment from metric calibration.
- Corrected disparity units, the search interval, invalid masks, depth scaling and zero-disparity handling. Kept display scales fixed across parameter choices.
- Replaced the original live matcher’s truncated edge windows and artificial out-of-image costs with full valid windows and an explicit common ROI.
- Added matching-cost curves, ambiguity inspection and 36 genuine OpenCV BM/SGBM outputs. No output is labelled ground truth or an accuracy target.
- Expanded all six worksheet exercises with worked reasoning; made AR projection continuous, with consistent frame and unit conventions.
- Added keyboard-accessible equivalents for image interactions, responsive layout, optional local notes, answer feedback, printing and downloadable code/results. All demonstrations work without external libraries or image requests.
Measured outputs may differ slightly across builds and platforms. Synthetic illustrations and live browser SAD results are explicitly labelled separately from OpenCV measurements. The original ZIP is unchanged.
Source material and API references
Primary teaching source: the supplied worksheet08.ipynb and its images. Technical definitions and implementation constraints were checked against the OpenCV documentation. The diagrams and interactive explanations are newly authored. External reference links require an internet connection; the tutorial itself does not.
Your workshop record
Complete the eight checkpoints and record evidence for the six exercises.
Notes and successful checkpoints are saved only in this browser when supported. They are not sent anywhere or treated as an assessment submission. Export a copy to keep or share your work.