The world is three-dimensional, but a camera flattens it into a 2D image. 3D vision is about recovering what was lost: how far away things are, what shape they have, and how a scene looks from a new viewpoint. It drives robotics, self-driving, AR and VR, mapping, and 3D content creation.
Monocular depth estimation predicts a depth map from a single image, and models such as MiDaS and Depth Anything do it remarkably well. Point clouds, sets of 3D points from sensors such as LiDAR, need networks like PointNet that can handle unordered data. With many photos of a scene, structure from motion recovers camera positions and a sparse 3D model.
Neural scene representations then build on that. NeRFs train a network to render a scene from any viewpoint, and Gaussian splatting represents the scene as many 3D Gaussians for much faster rendering. Watch the lessons in order, then continue on to Evaluation & Deployment.
How networks infer depth from a single image, covering MiDaS, Depth Anything V1 and V2, disparity space, and the training tricks behind them.
How PointNet learns directly from unordered point clouds, using symmetric functions and max pooling to extract global features.
A quick introduction to NeRF: representing a 3D scene with a neural network so it can be rendered from a small set of 2D photos.
A comparison of NeRFs and Gaussian splatting for novel view rendering and 3D reconstruction.