Chromaticity Segmentation — Image Processing

Image Segmentation | OpenCV

As exemplified in my previous article, we segmented the objects in an image using Color Image Segmentation through the HSV color scale. But that is not the only way we can segment our images. For this blog, we look at the image segmentation process under the lens of chromaticity segmentation.

Why should one prefer Chromaticity over RGB/HSV Segmentation?

• We can get the ratios between RGB/HSV channels.
• We can use target patches consisting of secondary colors and a mixture of other colors.

We formally define chromaticity segmentation as the process of extracting objects from images using the chromaticity space of the RG channels. The latter make up a 2-dimensional color representation that neglects image information with respect to its intensity values. We do this by looking at the proportion of the different color channels and map it using the normalized RGB space. Hence, to compute for the RG Chromaticity of an image, we use the following equations:

We primarily just look at the r and g equations since from there we can intuitively compute for the b channel.

Let us put chromaticity segmentation into action using our good old friend — Python.

Install the following libraries before we proceed with the implementation:

from skimage.io import imread, imshow
import matplotlib.pyplot as plt
import numpy as np

The discussion will focus on this image:

Mixed Fruits | Drive (Click here to download the image)

Let us get started! Before going through the code, we outline the steps we shall undertake.

Step 1: Compute for RG Chromaticity of the image

• This is done by using our equations defined in the introduction.

Step 2: Compute for the 2D histogram of the color values (original image)

• This is by flattening both R and Chromaticity values and feeding it in the hist2d function.
• Through this, it can be noticed what color or group of colors comprises our image.

Step 3: Select a reference image patch

• Generate a patch from the object of interest. In this case, we are segmenting strawberries. So, the patch that will be selected will be of that of a strawberry.

Step 4: Compute for the RG Chromaticity of the patch

• Repeat Step 1 but using the image patch in Step 3

Step 5: Compute for the 2D histogram of the color values (patch)

• Repeat Step 2 but using the image in Step 3

So far, we have gotten the RG chromaticity values of the concerned images. Now, we proceed to the next step the — Parametric Segmentation.

Step 6: Parametric Segmentation

• This step requires us to fit a gaussian distribution which will determine the pixels that belong to the color of interest. Before implementing this, the mean and standard deviations are calculated from the object of interest (reference patch). These are then fed to the gaussian distribution function.

The code implementation of Steps 1–6 are implemented as follows:

 #image chromaticity values
 fruits_R = fruits[:,:,0]*1.0/fruits.sum(axis=2)
 fruits_G = fruits[:,:,1]*1.0/fruits.sum(axis=2)
 #patch
 patch_strw = fruits[60:90,50:90,:]
 
 #patch chromaticity values
 patch_R = patch_strw[:,:,0]*1.0/patch_strw.sum(axis=2)
 patch_G = patch_strw[:,:,1]*1.0/patch_strw.sum(axis=2)
 #mean and stdev calculation of patch 
 std_patch_R = np.std(patch_R.flatten())
 mean_patch_R = np.mean(patch_R.flatten())
 std_patch_G = np.std(patch_G.flatten())
 mean_patch_G = np.mean(patch_G.flatten())
 #gaussian function
 def gaussian(p,mean,std):
     return np.exp(-(p-mean)**2/(2*std**2))*(1/(std*((2*np.pi)**0.5)))
 x = np.linspace(0,1)
 y = gaussian(x,mean_patch_R,std_patch_R)
 #plotting
 fig, ax = plt.subplots(2, 3, figsize=(20, 7))
 ax[0,0].scatter(fruits_R.flatten(),fruits_G.flatten())
 ax[0,0].set_title('RG Chromaticity', size=14)
 
 ax[0,1].hist2d(fruits_R.flatten(), fruits_G.flatten(),   
         bins=100,cmap='binary');
 ax[0,1].set_title('Color Values 2D Histogram', size=14)
 ax[0,2].imshow(patch_strw)
 ax[0,2].set_title('Strawberry Patch', size=14)
 ax[1,0].scatter(patch_R.flatten(),patch_G.flatten())
 ax[1,0].set_title('Patch RG Chromaticity', size=14)
 ax[1,1].hist2d(patch_R.flatten(), patch_G.flatten(), 
         bins=100,cmap='binary')
 ax[1,1].set_title('Patch Color Values 2D Histogram', size=14)
 ax[1,2].plot(x,y)
 ax[1,2].set_title('Gaussian Plot', size=14)
 fig.suptitle('Strawberry', size = 30)

Brilliant! Let us proceed to segmenting our strawberries from the image.

Using the gaussian distribution function we have defined earlier, let us obtain the probability of a color being part of our image through both the R and G coordinates. We implement this using the code below:

 prob_R = gaussian(fruits_R,mean_patch_R,std_patch_R)
 prob_G = gaussian(fruits_G,mean_patch_G,std_patch_G)
 prob=prob_R * prob_G
 fig, ax = plt.subplots(2, 2, figsize=(8, 7.5))
 
 ax[0, 0].imshow(prob_R)
 ax[0, 0].set_title('prob_R')
 ax[0, 0].axis('off')
 ax[0, 1].imshow(prob_G)
 ax[0, 1].set_title('prob_G')
 ax[0, 1].axis('off')
 ax[1, 0].imshow(prob)
 ax[1, 0].set_title('prob_B x prob_G')
 ax[1, 0].axis('off')
 threshold = 3
 ax[1, 1].imshow(prob>threshold)
 ax[1, 1].set_title(f'Prob threshold > {threshold}')
 ax[1, 1].axis('off')
 
 fig.tight_layout()
 fig.suptitle('Parametric Chromaticity Segmentation', size=20)

As you have observed, similar to the color image segmentation method, there is a threshold that is being determined arbitrarily. Experiment and play around with the values and choose the value that will return the most desirable output. Also, in the final graph, although strawberries were clearly segmented, irrelevant information were still captured. This is where morphological operations come in handy to further clean the image so that the segmentation is as accurate as ever. To know more about morphological operations do check out my previous articles.

Great job! Little by little you are becoming an Image Processing expert! Keep pushing, you got this.