I have been looking into bug #325564 the past few days and I know how this should be solved, and I'm looking for advice on how best to implement this in GIMP.

The problem is this:

The layer mode "Color" in GIMP works differently from Photoshop's "Color" blend mode. If you load a Photoshop PSD file with layers that are set to "Color", they will look different in GIMP than in Photoshop.

The cause of the bug is the following:

The "Color" layer mode is implemented in the method gimp_composite_color_only_any_any_any_generic in app/composite/gimp- composite-generic.c (lines 533-591). This method converts the RGB data of both source layers to HSL and then determines the output by taking the Hue and Saturation components from source 1 and Lightness from source 2 and converting this back to RGB.

Photoshop works differently: it converts the RGB data of both source layers to CIE Lab and then determines the output by taking the a and b (chromaticity, color) components from source 1 and Luminance from source 2 and converting this back to RGB.

I looked up some documentation about color models and it turns out that Lightness from the HSL model is something entirely different than Luminance from the CIE Lab model, and therefore you get different results. In fact, according to the Poynton Color FAQ [1], HSL is not very useful at all for accurately specifying color information.

The solution is this:

Instead of doing the calculations by converting to HSL and back, they should be done by converting to CIE Lab and back, taking ab from src1 and L from src2, just like Photoshop does. I've hacked a bit in the GIMP code and doing this indeed returns the expected result.

Problem #1:

As already suggested in the bug report, I should probably not just repair GIMP's "Color" mode, because people may have used the current "Color" mode and are relying on its effect. So for compatibility I should leave the current "Color" mode and add a new mode. The old mode should be renamed to "Hue and Saturation" or "Color (HSL)", and the new mode should be called "Color" to not confuse people who are familiar with Photoshop. The enum value of the old "Color" mode should stay the same for compatibility. Is this OK?

Problem #2:

The solution described above seems simple enough, but it becomes more complicated because there is no single, fixed formula for converting RGB to Lab and back. The conversion depends on the ICC color profile of the RGB data. I noticed that GIMP's decompose plug-in does it wrong: it uses some fixed formulas that are only valid for sRGB (added a new bug report for this: bug #499794).

The conversion can easily be done using the lcms library, but using lcms in app/composite/gimp-composite-generic.c means adding a dependency to the GIMP application itself (I had to add lcms to two Makefile.am libraries to make it compile).

Is adding this dependency a problem? If it is, where should I put the RGB Lab conversion code that uses lcms?

[1]: http://www.poynton.com/ColorFAQ.html