Lens Introduction

In this case you will learn about the general lens model used in rendering pipeline. It will be explained where
lens are taken into account during calculations, how it affects ray generation process, its geometrical
properties and common parameters for all lenses in SkyRenderer.

Agenda:

• Lens overview
• Lens types in SkyRenderer

Lens overview

In our system, the camera and its lens are responsible for the transformation from the 3D space into 2D rendered
image. The lens itself is a part of system that describes what properties camera has, hence describes the
3D->2D transformation characteristics.

Lens is described by following factors:

1. Projection part - it can be treated as a zoom. Mathematically it corresponds to multiplicative part of a linear
   3D-2D transformation (f_x, f_y in OpenCV camera matrix).
2. Center of projection - it can be treated as a lens shift. Mathematically it corresponds to additive part of a
   linear 3D-2D transformation (c_x, c_x in OpenCV camera matrix).
3. Distortion - distortion coefficients associated with lens distortion model. Mathematically it corresponds to
   nonlinear 3D-3D transformation.

Each lens has multiple ways of defining its projection part, such as:

1. Horizontal field of view (hfov) definition - It is a horizontal opening angle of the view (in degrees).
   Object will appear to be much closer if we set this angle to smaller value. Default hfov is 70 degrees.
2. f_x, f_y (px) - OpenCV style of defining projection part.
3. f (px) - focal length of a lens [mm] divided by pixel size in mm.

Projection and generation pipeline

../res/ray-generation-model.png

During point-to-camera projection following steps are done:

1. Projecting point P onto projection unit sphere S making P'.
   This step is done via simple line projection:
   \begin{align}
   P'_x &= P_x / P_z \
   P'_y &= P_y / P_z
   \end{align}
2. Transforming P' along radial segment (r) so new radius follows projection equation (perspective, equidistant,
   etc.) - P''.
   This is done differently for all lens types. For example, in pinhole perspective projection:
   \begin{equation}
   r = f \cdot \tan(\theta)
   \end{equation}
3. Distorting P'' in projection plane (according to distortion model) - P'''.
   This is lens model dependent, see specific lens cases for further information.
4. Projecting P''' onto image space with intrinsic matrix - (u,v).

In this stage, linear transformation using intrinsic matrix is done:

\begin{equation}
\begin{bmatrix}
u \
v \
1
\end{bmatrix} = \begin{bmatrix}
f_x & 0 & c_x \
0 & f_y & c_y \
0 & 0 & 1
\end{bmatrix} \cdot
\begin{bmatrix}
P''_x \
P''_y \
1
\end{bmatrix}
\end{equation}

When generating ray whole process is reversed:

1. (u,v) are reprojected onto projection plane (z=1).
   \begin{align}
   P''_x &= \dfrac{u - c_x}{f_x} \
   P''_y &= \dfrac{v - c_y}{f_x} \
   P''_z &= 1
   \end{align}
2. Points are being undistorted, to match original ray directions (using distortion back mapping and
   precalculated distortion map).
   This step is different in each lens model, see lens cases for more information.
3. Based on re-projection radial distance and projection equation ray angle (theta) is calculated.
   This step is different in each lens model, see lens cases for more information.
4. Projection sphere coordinates are reconstructed (unit vector with angle theta between incident plane pi and
   sensor optical axis).
   \begin{align}
   P'_x &= P''_x \cdot sin(\theta) \
   P'_y &= P''_y \cdot sin(\theta) \
   P'_z &= cos(\theta)
   \end{align}
5. Ray made from O and P' is generated.

In ray generation pipeline, steps 1,4,5 are the same for each projection model. Distortion maps are calculated
differently and projection reverse is calculated differently in each type of lenses.

Lens types

In SkyRenderer we distinguish 3 types of lenses:

1. Pinhole lens - most basic type of lens, based on the natural phenomena. The same lens is usually used in most
   common cameras.
2. Fisheye lens - ultra wide-angle lens. Produces strong visual distortion. This lens is used in peephole cameras
   and in cars for in-cabin cameras.
3. Telecentric lens - lens that have a constant magnification regardless of the object's distance or location in
   the field of view. Commonly seen as the orthographic view in 3D software.

Summary

In this section you have learnt:

• Why do we need lens in the system.
• How zoom, lens shift and distortion work like in SkyRenderer.
• Common parameters in all types of lens.
• How lens are affecting ray generation pipeline.
• SkyRenderer has implemented 3 types of lenses: PinholeLens, FisheyeLens, TelecentricLens.