Shadow Detection and Removal from RGB Images

Suprotim Majumdar and Harsh Patel

Virginia Tech

Abstract

Shadows greatly attenuate the performance of many computer vision algorithms such as object recognition. Removing shadows from video is easy using background subtraction. On the other hand, removing shadows from stationary images is difficult. A significant amount of work is done in this area using 1D invariant images. We took a different approach where in, we first detect the shadow present in the image and then select a small window from the same image to replace the shadow with.

Teaser figure





Introduction

When there are shadows in the images, it interferes with the intended output. For example, an object recognition program may detect false objects due to the shadows present in the image. As, for stationary images, there is no information about what is background and foreground, we cannot easily decide where the shadow is, unlike video frames, where in we can subtract previous frames from the current one to find background.

Recently, a method for removing shadows from color images was developed [Finlayson, Hordley, Lu, and Drew, PAMI2006] that relies upon finding a special direction in a 2D chromaticity feature space. This “Invariant Direction” is that for which particular color features, when projected into 1D, produce a grayscale image which is approximately invariant to intensity and color of scene illumination. Thus shadows, which are in essence a particular type of lighting, are greatly attenuated.

The main approach to finding this special angle is a camera calibration: a color target is imaged under many different lights, and the direction that best makes color patch images equal across illuminants is the invariant direction.

Using a different approach, producing a 1D projection in the correct invariant direction will result in a 1D distribution of pixel values that have smaller entropy than projecting in the wrong direction. The reason is that the correct projection results in a probability distribution spike, for pixels all the same, except differing by the lighting that produced their observed RGB values and therefore lying along a line with orientation equal to the invariant direction.

The approach we have taken involves detecting shadows in the image first and then change the R,G,B values of the pixels such that it is as illuminated as the rest of the image.

Approach

  1. Convert the image from RGB space to LAB space, where L is lightness component and, A and B are color-opponent dimensions, based on nonlinearly compressed coordinates.
  2. Then we create a shadow mask by thresholding. If the Lightness Component is within a third of the standard deviation of the mean, then it belongs to the shadow mask.
  3. We take the input from the user regarding how many shadows the user want to remove.
  4. Based on the input i.e. number of shadows to be removed, the user is asked to select a shadow region.
  5. Region Growing is used to get the shadow from the image.
  6. Then the user is asked to select a small window from the image itself other than the shadow.


Experiments and results

Provide details about the experimental set up (number of images/videos, number of datasets you experimented with, train/test split if you used machine learning algorithms, etc.). Describe the evaluation metrics you used to evaluate how well your approach is working. Include clear figures and tables, as well as illustrative qualitative examples if appropriate. Be sure to include obvious baselines to see if your approach is doing better than a naive approach (e.g. for classification accuracy, how well would a classifier do that made random decisions?). Also discuss any parameters of your algorithms, and tell us how you set the values of those parameters. You can also show us how the performance varies as you change those parameter values. Be sure to discuss any trends you see in your results, and explain why these trends make sense. Are the results as expected? Why?



Qualitative results

Show several visual examples of inputs/outputs of your system (success cases and failures) that help us better understand your approach.



Conclusion and Future Work

The whole idea of removing shadows is based on the premise that the lightness component in the image is separate and not associated with the chromaticity component. The proportion of R:G:B is same in the whole image irrespective of the lighting conditions. If we can find what is the ratio of the R,G,B values where there is no shadow and the ration of the same inside the shadow, we can change the ratio of R:G:B inside the shadow region to that of the R:G:B outside the shadow region.

Also, this argument is based on the assumption that there are some chromaticity values captured in the shadow region. If the shadow region is so dark that there are no chromaticity values captured by the camera for that region, there is no way for us to determine what colors are there in the shadow region. In which case the algorithm fails.

References