Could you please provide the pre-processing files for the dataset?

[mwy2167]

Can you provide pre-processing files for the following datasets?
Shi's Re-processing of Gehler's Raw Dataset ： https://www.cs.sfu.ca/~colour/data/shi_gehler/
NUS-8 Camera Dataset ： https://yorkucvil.github.io/projects/public_html/illuminant/illuminant.html
I really don't know what to do. Thanks.

[mahmoudnafifi]

Hi,

I don’t have a copy of the processed files for those datasets. However, you can generate them yourself. If PNG raw (16-bit) files are available, download those; otherwise, download the DNG files and extract the raw images using a library like rawpy and use any demosaicing method (after black-level normalization) to get the RGB raw (e.g., color-demosaicing).

To apply black level normalization, one approach is min-max normalization, i.e.,
processed_image = (image - black_level) / (white_level - black_level),
or you can use a simpler form:
processed_image = (image - black_level) / white_level.

The black and white levels (sometimes referred to as saturation level) are usually provided on the dataset website (e.g., NUS includes them in a .mat file) or can be extracted directly from the DNG files using tools like rawpy.

Hope that helps; good luck!

[mwy2167]

Thanks for your reply.
I have another question: I have the image001_metadata.json file, is there anything else I need to do with it?
This is the true value of the first image derived from the real_illum_568..mat of the file Shi's Re-processing of Gehler's Raw Dataset ： https://www.cs.sfu.ca/~colour/data/shi_gehler/

[mahmoudnafifi]

I don't know the source of this json file. However, you can find the ground-truth illuminant colors in the .mat files of the Gehler-Shi and NUS datasets. Also, I forgot to mention in my previous reply that you should mask out the color chart in each image. The coordinates of the color chart can be found on the website of each dataset.

[mwy2167]

Do you want to normalize or standardize the real value data of the json file? What formula to use? Thank you for your help.

[mahmoudnafifi]

It doesn’t hurt to normalize using the L2 norm, but you typically use angular error as the loss/metric to train and evaluate the model. The angular error between the predicted and ground truth illuminant vectors inherently factors out the vector magnitude.