Topics

Shape-from-Shading plus Lena – Math

2 December 2017

Shape-from-Shading (SfS) is a well-known problem in computer vision.

It is a difficult inverse problem which consists in reconstructing a 3-dimensional surface from a single image. Normally, such a problem is also ill-posed as a unique solution in not guaranteed.

The model

Suppose that $\Omega \subset \mathbb{R}^2$ is a bounded domain (the reconstruction domain) and let $u\colon \Omega \to \mathbb{R}$ be the 3-dimensional surface that we want to reconstruct.

In other words, the domain $\Omega$ is the picture, the photo, or the image that we want to reconstruct the surface from.

The SfS problem can be modeled by the image irradiance equation \[ R(\mathbf{n}(x_1,x_2)) = I(x_1,x_2), \quad (x_1,x_2) \in \Omega, \qquad\qquad (1) \] where $I(x_1,x_2)$ is the brightness measured in the image at all points $(x_1,x_2)$ ($I(x_1,x_2)$ represents the irradiance at $(x_1, x_2)$) and $R(\mathbf{n}(x_1, x_2))$ is the reflectance function which gives the value of the light reemitted by the surface as a function of its orientation, i.e., of the unit normal $\mathbf{n}(x_1, x_2)$ to the surface at point $(x_1, x_2, u(x_1, x_2))$.

Usually, the irradiance function $I$ is the datum in the model since it is measured at each pixel of the image, for example, in terms of a grey level from 0 to 255. We assume that $I$ is a continuos function taking real values in $[0,1]$ ($0$ means white while $1$ means black).

The normal $\mathbf{n}$ can be easily expressed as \[ \mathbf{n}(x_1,x_2) = \frac{(-\nabla u(x_1,x_2),1)}{\sqrt{1 + |\nabla u(x_1,x_2)|^2}}, \qquad \qquad (2) \] where $\nabla u(x_1,x_2) = (\partial u /\partial x_1, \partial u/\partial x_2)$.

Assumptions

In order to make the problem manageable, a set of assumptions about the surface and the source of light are usually considered:

  • The material (of the reconstructed surface) is Lambertian, that is, its brightness does not depend on the observation point, and the albedo, that is, the ratio between the energy reflected and the energy captured is constant;
  • The light source is unique;
  • The light source is at infinity, that is, ray lights are parallel;
  • Multiple reflections are negligible;
  • The surface is completely visible by the camera, that is, there are not hidden regions (undercuts).

Equation

Under the above assumptions, we have $R(\mathbf{n}(x_1, x_2)) = \omega\, \mathbf{n}(x_1,x_2)$, where $\omega = (\omega_1,\omega_2,\omega_3)\in \mathbb{R}^3$ is a unit vector which indicates the direction of the light source. Therefore, using $(2)$, the image irradiance equation $(1)$ can be written as

\[ I(x_1,x_1) \sqrt{ 1 + |\nabla u(x_1,x_2)|^2 } + (\omega_1,\omega_2) \cdot \nabla u(x_1,x_2) - \omega_3 = 0 \qquad \forall \; (x_1,x_2) \in \Omega, \qquad (3) \]

that is a first-order nonlinear partial differential equation of Hamilton-Jacobi type. If the light is completely vertical, that is $\omega = (0,0,1)$, the equation above becomes an eikonal equation

\[ |\nabla u(x_1,x_2)| = \sqrt{\frac{1}{I(x_1,x_2)^2} - 1} \qquad \forall \; (x_1,x_2) \in \Omega. \qquad \qquad (4) \]

A 3-dimensional Lena

Here, I show you an application of SfS to 3D printing. This is indeed a work made by Emiliano Cristiani.

Lena (or Lenna) is the name given to the most celebrated image for benchmark in computer vision and image processing:

Responsive image

Using the PDE model based on equation (4), i.e., assuming vertical illumination $\omega = (0,0,1)$, a 3D virtual representation of Lena is reconstructed starting from the single photograph above.

The domain $\Omega$, i.e., our photograph, is discretised by a structured grid of $401 \times 401$ nodes corresponding to the pixels of the image.

The equation (4) is solved using a semi-Lagrangian scheme and assuming homogeneous boundary conditions. The figure below, shows the reconstructed surface (the solution of equation (4)) viewed from the front (left) and from the rear (right).

Responsive image

You may think: what the hell is this?

Well, the corresponding "solid" object has been created using a triangular mesh of the surface saved into a STL file (I spoke about the STL format here).

You can download the Lena STL file from Emiliano's Publications page.

If you open the STL file (here I used the open-source StL viewer for Mac OS) you will see something like this:

Responsive image

On the left you can see the object as it is as soon as you open the STL file while on the right the object has been moved so that you are watching it from above.

Now it comes the magic. You can move around the light source. Move it in order to have vertical illumination and voilĂ :

Responsive image

3D printed Lena

If you have a desktop 3D printer at home, you can 3D-print the object by yourself. Otherwise, just bring the STL file to any 3D printers shop (they are becoming more and more popular).

Emiliano 3D-printed the STL file above. Physical dimensions are $120\times 120\times 109 \text{mm}$. Here you can see the result:

Responsive image

The picture on the right is a photograph of the object taken from above and illuminated by a flash.

...want to see the entire photograph of Lena? Here it is!