Joint 2D-3D Reasoning for Image Segmentation

Gordon Christie
Spring 2014 ECE 6504 Probabilistic Graphical Models: Class Project
Virginia Tech

---

Goal

The goal of this project is to segment 2D images taken in an urban environment into several different semantic classes. The dataset used contains both 2D images and 3D data from a laser scanner. This project aims to show that the sparse points from the laser scanner can be used to improve the segmentation accuracy beyond a 2D only baseline. An additional goal of the project was to show that using a graphical model to reason about pairwise terms of adjacent superpixels improves accuracy beyond a method using only unary terms.

Approach

The main steps involved in segmenting the 2D images into different semantic classes are (in order): generating superpixels, calculating 2D and 3D features for each region, training a model to predict probabilities given features at test time, and finally forming a graphical model and performing inference on test images.

Instead of training and testing at the pixel level, which can be computationally expensive, I am using VLFeat's' quickshift algorithm [1] to obtain superpixels. Once these superpixels have been found, 2D and 3D features are calculated for each superpixel. These 2D features include mean RGB and Luv color values, gabor filter responses, GLCM features, and gradient information. The 2D location of the superpixel within the image can also be used, but this was not included to avoid overfitting to the training set. The 3D features are based on projected 3D points within the superpixel. Using the current point and a local neighborhood of 3D points, features of the region such as shape, size, orientation, and height within the point cloud are calculated. Note that not all superpixels will contain 3D points projected inside of them. To handle this, I use the features of the nearest projected point outside of the superpixel, as well as the distance to that projected point (in pixels).

Once the features are calculated for all images, a model can then be trained so that the probabilities for each class can be estimated given a feature vector at test time. The model used for this project was the logistic regression implementation from LIBLINEAR [2]. By taking the negative log of these probabilities, I obtain the energy for a superpixel belonging to each class (the unary term) for the graphical model. The pairwise terms in the graphical model used were based only on differences in RGB values. A constant, set with cross-validation, is multiplied by the pairwise term to control the smoothness. To perform belief propagation on this graph structure, I used Oliver Woodford's MATLAB wrapper for Vladimir Kolmogorov’s implementation [3].

The dataset that was used for this project is the CMU/VMR Urban Image + Laser Dataset [4]. The dataset contains 372 labeled images, and also 3D laser points that have been projected onto the image. There are a total of 30 class labels, as well as a "void" class for regions such as the sky. I present results with and without the "void" class. The other classes include general parts of an urban environment, such as road, sidewalk, vehicle, person, etc. The full list can be found just above the references section at the bottom of this page.

Results

The following results were found using cross-validation with 297 training images and 75 test images. For each fold, cross-validation was also run on the training set to find the smoothness term that controls the effect of the pairwise term in the model.

The following image shows the F1 scores (combination of precision and recall) for classes where at least one of the methods had an F1 score above 0.01. It is believed that the model presented for this project does not model rare classes well. The proposed approach (3D + 2D + graph) shows superior performance for road, building, and small-vehicle. The baseline methods include using only the unary term with and without 3D data, and only 2D data with the graphical model. For all of the classes except for the "void" class, 3D data is shows increased performance. The majority of the "void" class belongs to the sky region where no 3D data is available, and therefore improved performance is not expected.



The following figures show the overall accuracy for the different methods. As seen, 3D information shows improved performance, but when classifying all classes (including "void"), the graphical model performed slightly worse than using only the unary. However, when reporting the accuracy without the "void" class, we see that the graphical model actually shows improved accuracy over the baselines.



The following images show example segmentations. The image on the left shows the segmentation for 3D + 2D + graph, and the image on the right shows a segmentation for 2D + graph. As seen, the 3D data is able to help identify the sidewalk and both vehicles, while the 2D only approach classifies both the sidewalk and one of the vehicles as building.