Greetings fellow Gimp Users,

I make images using Gimp, but I assume that this question is not really Gimp specific.

I have tens of thousands of images (postage stamps) on my site. Every now and then when I am looking at a page I discover that the image (a JPEG) has had is colors "sort of inverted". The JPEGs were created in large batches by a script from UNcompressed TIFF images. When I go back and look at the the original TIFF, I discover that its colors are "sort of inverted" -- thus the JPEG is a correct rendition of the appearance of its TIFF source.

Thus the problem is in the TIFF. But, the problem happens now and then, over the course of years. The TIFFs are _not_ being intentionally manipulated in that time. The images was originally okay, now its not. It seems to be completely random, just one image here and there.

Somehow the TIFF is getting corrupted. I am assuming by a memory error or a disk/RAID controller error, or such. The images are still openable in Gimp.

This is only happening to one out perhaps one out of five thousand images, every five years. (I am just *guessing* at the error rate because I only find out about them by randomly coming across them.) But, if I have 40,000 images, that is eight images destroyed every five years. (And often I am not able to replace the image because I no longer have the item.)

This example image was originally created in 2006. I suspect (mostly guessing) that it was corrupted sometime since 2010. There is no reason that it would have been edited since that time and file modification information shows nothing since 2006.

On Ubuntu Linux, using "identify -verbose filename.tif" I can read the header information. The only odd thing (to my eye) is that the create date is 2011 and the modification date is 2006:

Properties: date:create: 2011-09-13T11:30:24-04:00 date:modify: 2006-12-21T00:53:03-05:00

I am guessing that means the corruption may have happened in 2011, even though the filesystems own file datestamp is 2006 and the lsattr command shows nothing unusual.

Here is example of a) the resulting JPEG (just to illustrate the nature of the corruption); b) a similar JPEG to show generally what it is supposed to look like; c) the corrupted TIFF.

Corrupted: http://jsa.viewimage.net/jsa/web/Lists/Denmark/AdPairs/Spec/re02-pair_used-vf-b_136468_r_l.jpg

Correct image of a similar, but different item: http://jsa.viewimage.net/jsa/web/Lists/Denmark/AdPairs/Spec/re02-pair_used-vf-a_136467_r_l.jpg

This is the TIFF file (corrupted, but viewable in Gimp; colors are sort-of-inverted) Size 496 KB:
http://jsa.viewimage.net/temp/gimp/re02-pair_used-vf-b_136468.tif

My primary question is whether there is a "particular bit that is getting flipped" that could be "unflipped" by some sort of non-visual editing of the source TIFF file?

My secondary question is whether or not other people have seen this type of problem crop up in large image libraries and what the causes have been?

Any thoughts appreciated.

Jay

On Sun, May 31, 2015 at 4:25 PM, Jay Smith > wrote:

Greetings fellow Gimp Users,

I make images using Gimp, but I assume that this question is not really Gimp specific.

I have tens of thousands of images (postage stamps) on my site. Every now and then when I am looking at a page I discover that the image (a JPEG) has had is colors "sort of inverted". The JPEGs were created in large batches by a script from UNcompressed TIFF images. When I go back and look at the the original TIFF, I discover that its colors are "sort of inverted" -- thus the JPEG is a correct rendition of the appearance of its TIFF source.

Thus the problem is in the TIFF. But, the problem happens now and then, over the course of years. The TIFFs are _not_ being intentionally manipulated in that time. The images was originally okay, now its not. It seems to be completely random, just one image here and there.

Somehow the TIFF is getting corrupted. I am assuming by a memory error or a disk/RAID controller error, or such. The images are still openable in Gimp.

This is only happening to one out perhaps one out of five thousand images, every five years. (I am just *guessing* at the error rate because I only find out about them by randomly coming across them.) But, if I have 40,000 images, that is eight images destroyed every five years. (And often I am not able to replace the image because I no longer have the item.)

This example image was originally created in 2006. I suspect (mostly guessing) that it was corrupted sometime since 2010. There is no reason that it would have been edited since that time and file modification information shows nothing since 2006.

On Ubuntu Linux, using "identify -verbose filename.tif" I can read the header information. The only odd thing (to my eye) is that the create date is 2011 and the modification date is 2006:

Properties: date:create: 2011-09-13T11:30:24-04:00 date:modify: 2006-12-21T00:53:03-05:00

I am guessing that means the corruption may have happened in 2011, even though the filesystems own file datestamp is 2006 and the lsattr command shows nothing unusual.

Here is example of a) the resulting JPEG (just to illustrate the nature of the corruption); b) a similar JPEG to show generally what it is supposed to look like; c) the corrupted TIFF.

Corrupted: http://jsa.viewimage.net/jsa/web/Lists/Denmark/AdPairs/Spec/re02-pair_used-vf-b_136468_r_l.jpg

Correct image of a similar, but different item: http://jsa.viewimage.net/jsa/web/Lists/Denmark/AdPairs/Spec/re02-pair_used-vf-a_136467_r_l.jpg

This is the TIFF file (corrupted, but viewable in Gimp; colors are sort-of-inverted) Size 496 KB:
http://jsa.viewimage.net/temp/gimp/re02-pair_used-vf-b_136468.tif

My primary question is whether there is a "particular bit that is getting flipped" that could be "unflipped" by some sort of non-visual editing of the source TIFF file?

My secondary question is whether or not other people have seen this type of problem crop up in large image libraries and what the causes have been?

Any thoughts appreciated.

Jay

On 05/31/2015 08:46 PM, Daniel Smith wrote: > when you say youre seeing thison a web page, > can i ask what type of system yure serving the pages with? > wordpress etc? or is it an image generation with php or something? > thanks
> dan

Hi Dan,

When I say I saw it on a web page, all I meant is that was where I was looking when I noticed the corrupted file (JPEG version thereof).

All viewing (of both the JPEG and the original TIFF that the JPEG was made from) after that time has been done using Gimp to view the corrupted files.

The fact that I first noticed the corrupted JPEG on a web page was not relevant and I should not have included that detail. I only mentioned it that way because that is the only way I actually would notice that one of tens of thousands of files had been corrupted. I don't systematically go through and view the library of files just for fun.

I am sorry for the misleading detail.

Jay

On 05/31/2015 07:09 PM, Liam R. E. Quin wrote:

On Sun, 2015-05-31 at 17:25 -0400, Jay Smith wrote:

My primary question is whether there is a "particular bit that is

getting flipped" that could be "unflipped" by some sort of non-visual editing of the source TIFF file?

My secondary question is whether or not other people have seen this

type of problem crop up in large image libraries and what the causes have been?

The nearest I have seen to this that i can remember involves software changes - e.g. a different version of libtiff or whatever. For example, recently some images on http://www.fromoldbooks.org/appeared to break, and it turned out to be a version of the jpeg library that had stopped supporting arithmetic encoding (for software patent reasons).

TIFF is one of the more complex graphics formats in widespread use; for my own part I prefer to use PNG because at least the core image part is relatively simple and well-specified and widely supported.

If by any chance you have both an "uncorrupted" and a "corrupted" TIFF file, and you know for sure what programs were used to create them and on what platform (e.g. Linux on SPARC, Linux on Intel-64, 32-bit Windows on Intel, etc) I'm willing to take a look, althogh I don't think I have TIFF debugging stuff around any more.

It seems unlikely that the same single-bit error would happen on multiple images because of a hard drive problem, especially if a RAID 3 or higher storage system didn't detect it. Not impossible - an infinite number of monkeys typing for all eternity might all type nothing except page 54 of the January 1936 Great Western Railway timetable through Crewe - but I'd look for more likely causes first, the most likely of which might be a software change.

Liam

Liam,

Thanks for your thoughts.

I am not understanding how a different version of "libtiff or whatever" could result in the corruption of a very few random TIFF image files (each just one of tens of thousands), every now and then, that are just sitting there, not actively being used by any process (that I am aware of).

If the corruption happened to many files that were being processed in the same manner, from time to time, then I would understand.

As for my idea about a memory or controller error on the server causing corruption, I certainly agree with you that it does not seem likely that the corruption would be of exactly the same type on the files that are affected, and that I have _not_ detected corruption of any other types of files or programs.

So, that leaves back to not having any explanation at all. :-)

Thanks again,

Jay