Shadow Carving using the sun
Rob Fergus, EE148 Final Project
The aim is to discover the 3-D structure of an object by looking at its shadows, as cast by the sun. The motivation behind this it requires the absolute minimum of equipment, just a camera (on a tripod), the object you want to render in 3-D and sunlight.
The theory of shadow carving is based on work by Silvio Savarese et al. [1]. In their experiments, they use a moveable light source (i.e. a lamp). In this project the lamp is replaced by the sun, which simplifies the theory somewhat.
The overall idea is this: A concave surface can be estimated using the
shadow cast by a light source - the sun in our case. The movement of the sun allows us to carve out regions in 3-D space, so converging upon the true 3-D structure of the object.
The diagram below gives a 2-D slice of the 3-D shadow carving process. The real object Vobj is in brown, whereas its upper bound estimate Vest (Vest Ê Vobj) is represented by the outside yellow area and is delimited by Sest. The light source, through the ray l L, casts a shadow in the concave part of the object. The shadow, in the 2D cross section depicted in figure, is delimited by the points Xp and Xc. The corresponding points in the image plane of the camera are Xp3 and xc3 respectively. Oc, Xp3 and Xp3 define the visual cone Cv (in pink). The intersection between Cv and the estimated surface Sest, yields Xp2 and Xc2. The light source L, Xp2 and Xc2 define the shadow cone Cs. The intersection between the visual cone, the shadow cone and Vest is the rejected area (dashed green area), i.e. the area we can carve out.
Now by repeating this process may times, we can reduce the yellow volume,
Now, given a voxel in
Sest, we can only remove it, if it is in the carvable volume. The calculation of this can be simplified by introducing a virtual plane, which is perpendicular to the rays from the sun. If a point lies in shadows in both the real and virtual images then the point lies in the carvable region.However, the virtual shadow image does not exist by default – it has to be created. We do this by running along the top surface of
Sest and projecting a ray up into the real image plane to record the status of shadow/non-shadow. Then we project along a vector towards the sun (the Sun vector) and mark a point on our virtual plane as being shadow/non-shadow. By moving over the whole top surface, we build up our virtual iimage. This surface is flat for the first image, but for subsequent images, the surface will have chunks carved out of it from previous carving operations, so a ray-tracing check must be done to ensure that the point on the surface is visible from the virtual plane.The figure below shows the virtual image plane and a point in the carvable region.
Set initial upper bound of surface (Height, Max and min X,Y)
Obtain sun vector
For each image
Identify and label shadow regions
Form virtual image:
For each point on surface:
Project upto image plane and record shadow label
If visible, project upto virtual plane and plot label
End
Perform shadow carving:
For each point on surface:
If point is in same shadow label in both real and virtual
images, then reduce depth by one. Repeat until no longer in
both shadow regions.
End
End
On the roof of the Moore building, from 8.30am to 7pm, an image was taken every 30 mins of the scene which we hoped to perform shadow carving. Below we can see the setup at three different times of day. Notice the difference in shadows on the objects:
The object we are attempting to carve is the terracotta pot on top of the white box. The gray brick gives us a shadow to use for establishing the sun vector. The post is an immovable part of the table.
Attempts were made to automatically segment the shadows from the image, but it proved too difficult, particularly on the curved surface of the pot. Therefore it was manually segmented. Above is a typical shadow mask and its corresponding image.
The we must establish the direction of the sun in each frame. This is done by clicking on the base of an object of known height and the tip its shadow. From this we can deduce the direction of the sun. Both the brick and the white box were used for this purpose, since at times one of them did not have a clear shadow.
The figure below shows this happening on the white box. The user also clicks to set the extents of the surface, as indicated by the blue wire-frame box.
Then, the virtual image can be formed and the shadow carving run:
So in the above image (white is high, black is low), we can see that a scoop has been taken out of the top of the pot and a section of the outside of pot, but most of the image remains unaltered. This is because shadow falls over a relatively small portion of the surface, as illustrated by looking at the virtual image.
Continuing with all 22 images, we finally get the following result:
Here we can see that the inside of the pot has been removed as has the exterior. However, the rim of the pot has, correctly, not been carved.
It took approximately 4 hours to generate the above image. While slow, the shadow carving was not actually the limiting factor – the ray tracing routine was the bottleneck. So unless the routines are rewritten in C, the images cannot be processed in real time.
Although the above results are encouraging for a first rough attempt, it is clear that a large portion of the object has not been carved. This is mainly due to the fact that the sun moves in a plane, so there is a distinct limitation in the range of shadows that will be cast over the image, hence it will not be possible to recover all the structure. This is a considerable disadvantage of using the sun as compared with a movable light source such as a lamp.
It is also surprisingly difficult to get a setup in which the sun can effectively be used, since the object needs unobstructed sunlight from a wide range of angles. In addition the weather needs to be reliable – not a problem in California, but a real issue in somewhere like England!
The routines were written in Matlab, which made it simple to implement, but the routines were slow, especially the ray-tracing. Some clever tricks were used to speed things up to make the processing time reasonable, but it still took a considerable time to process each image.
So, in summary, the sun can be used to perform shadow carving, but it has some serious limitations. The results achieved here are somewhat crude but the implementation was distinctly time-limited.
S. Savarese, H. Rushmeier, F. Bernardini and P. Perona, "Shadow Carving" to appear in Proc. Of Ninth IEEE International Conference of Computer Vision.