Is there a single bit that can accidentally flip an image to negative? (not Gimp specific)
This discussion is connected to the gimp-user-list.gnome.org mailing list which is provided by the GIMP developers and not related to gimpusers.com.
This is a read-only list on gimpusers.com so this discussion thread is read-only, too.
| Is there a single bit that can accidentally flip an image to negative? (not Gimp specific) | Jay Smith | 28 Jul 00:41 |
| Is there a single bit that can accidentally flip an image to negative? (not Gimp specific) | Ofnuts | 28 Jul 01:28 |
| Is there a single bit that can accidentally flip an image to negative? (not Gimp specific) | Ofnuts | 28 Jul 01:55 |
| Is there a single bit that can accidentally flip an image to negative? (not Gimp specific) | Elle Stone | 28 Jul 11:00 |
| Is there a single bit that can accidentally flip an image to negative? (not Gimp specific) | Elle Stone | 28 Jul 11:21 |
| Is there a single bit that can accidentally flip an image to negative? (not Gimp specific) | Jay Smith | 28 Jul 18:01 |
| Is there a single bit that can accidentally flip an image to negative? (not Gimp specific) | Michael Schumacher | 28 Jul 18:08 |
| Is there a single bit that can accidentally flip an image to negative? (not Gimp specific) | Ofnuts | 29 Jul 00:48 |
Is there a single bit that can accidentally flip an image to negative? (not Gimp specific)
Hi,
I am sorry that this query is not Gimp specific, but since I use Gimp to create the images and I don't subscribe to any "images" list, I hope it will be allowed here. I won't make a habit of it.
From time to time I notice an image in our library or on our website has "become" negative. This is very unsettling as I have used such images on our website for years and then all of a sudden I notice in the middle of a web page an image that is in negative.
Yes, it is theoretically possible that I simply never noticed this, but I have been finding one every few months now for three or four years. I know everybody says "I did not do it", but these are images which have not intentionally been manipulated, processed, or uploaded to the hosting service for for years. The differences in image appearance are so dramatic that I just can't believe that they have been like that all along and I just did not see them.
Also, the images don't seem to suffer any other type of damage. They just become negative.
In the most recent example case (see links below), only the file residing on the hosting service has "become" negative. The original JPEG that I uploaded is still correct! (However, over the last couple years, I have found an occasional "negative-ized" original JPEG on our server and also a few "negative-ized" TIFFs on our server. It seems that the alteration to negative can happen anywhere along the line.
Since the source JPEG version residing on my server is still okay, the damage happened either during transfer to the hosting service or while on the hosting service.
Here is an example (the two are images of slightly different objects, but the overall appearance is supposed to be extremely similar -- the person who can determine the difference is a keen-eyed philatelist):
My process: All on Linux. After scanning to TIFF, the images are edited in Gimp as TIFFs. Then they are processed using a script that runs various ImageMagick actions to create various sizes (copies) as JPEGs. The source TIFF is preserved unchanged (and the original TIFF for the negative example above is CORRECT!). The JPEGs are then uploaded to a hosting service (in this case all the 'original' JPEGs still on my server are CORRECT!!).
So... The question is..... is there a bit in an image file that can be hit by a "stray neutrino" (or whatever happens) that can cause the image file to "become" negative". If yes for JPEG, is it also possible for TIFF?
Or, it seems more likely to me, would such a change from positive to negative require a very large number of changes to the image file?
If the type of occasional damage I am observing in a "small" library of about 100,000 image files is happening everywhere, to everybody, to all types of computer files, we are in deep do-do in the long term. As they say.... "Houston, we have a problem."
Thanks,
Jay
Is there a single bit that can accidentally flip an image to negative? (not Gimp specific)
On 07/28/2013 02:41 AM, Jay Smith wrote:
Hi,
I am sorry that this query is not Gimp specific, but since I use Gimp to create the images and I don't subscribe to any "images" list, I hope it will be allowed here. I won't make a habit of it.
From time to time I notice an image in our library or on our website has "become" negative. This is very unsettling as I have used such images on our website for years and then all of a sudden I notice in the middle of a web page an image that is in negative.
Yes, it is theoretically possible that I simply never noticed this, but I have been finding one every few months now for three or four years. I know everybody says "I did not do it", but these are images which have not intentionally been manipulated, processed, or uploaded to the hosting service for for years. The differences in image appearance are so dramatic that I just can't believe that they have been like that all along and I just did not see them.
Also, the images don't seem to suffer any other type of damage. They just become negative.
In the most recent example case (see links below), only the file residing on the hosting service has "become" negative. The original JPEG that I uploaded is still correct! (However, over the last couple years, I have found an occasional "negative-ized" original JPEG on our server and also a few "negative-ized" TIFFs on our server. It seems that the alteration to negative can happen anywhere along the line.
Since the source JPEG version residing on my server is still okay, the damage happened either during transfer to the hosting service or while on the hosting service.
Here is an example (the two are images of slightly different objects, but the overall appearance is supposed to be extremely similar -- the person who can determine the difference is a keen-eyed philatelist):
My process: All on Linux. After scanning to TIFF, the images are edited in Gimp as TIFFs. Then they are processed using a script that runs various ImageMagick actions to create various sizes (copies) as JPEGs. The source TIFF is preserved unchanged (and the original TIFF for the negative example above is CORRECT!). The JPEGs are then uploaded to a hosting service (in this case all the 'original' JPEGs still on my server are CORRECT!!).
So... The question is..... is there a bit in an image file that can be hit by a "stray neutrino" (or whatever happens) that can cause the image file to "become" negative". If yes for JPEG, is it also possible for TIFF?
Or, it seems more likely to me, would such a change from positive to negative require a very large number of changes to the image file?
If the type of occasional damage I am observing in a "small" library of about 100,000 image files is happening everywhere, to everybody, to all types of computer files, we are in deep do-do in the long term. As they say.... "Houston, we have a problem."
Actually it's not a negative... and the changes would be on more than one bit since the bad image file is about 50% bigger than the good one.
But the images aren't even the same size (314x431 vs 312x426), and if you look closely in the corners the perforations aren't the same (you can invert the colors of the bad ones to make it more visible). So these are two images of different stamps.
The 'bad' image has a color profile, and this one may be incorrect.
Is there a single bit that can accidentally flip an image to negative? (not Gimp specific)
On 07/28/2013 03:28 AM, Ofnuts wrote:
On 07/28/2013 02:41 AM, Jay Smith wrote:
Hi,
I am sorry that this query is not Gimp specific, but since I use Gimp to create the images and I don't subscribe to any "images" list, I hope it will be allowed here. I won't make a habit of it.
From time to time I notice an image in our library or on our website has "become" negative. This is very unsettling as I have used such images on our website for years and then all of a sudden I notice in the middle of a web page an image that is in negative.
Yes, it is theoretically possible that I simply never noticed this, but I have been finding one every few months now for three or four years. I know everybody says "I did not do it", but these are images which have not intentionally been manipulated, processed, or uploaded to the hosting service for for years. The differences in image appearance are so dramatic that I just can't believe that they have been like that all along and I just did not see them.
Also, the images don't seem to suffer any other type of damage. They just become negative.
In the most recent example case (see links below), only the file residing on the hosting service has "become" negative. The original JPEG that I uploaded is still correct! (However, over the last couple years, I have found an occasional "negative-ized" original JPEG on our server and also a few "negative-ized" TIFFs on our server. It seems that the alteration to negative can happen anywhere along the line.
Since the source JPEG version residing on my server is still okay, the damage happened either during transfer to the hosting service or while on the hosting service.
Here is an example (the two are images of slightly different objects, but the overall appearance is supposed to be extremely similar -- the person who can determine the difference is a keen-eyed philatelist):
My process: All on Linux. After scanning to TIFF, the images are edited in Gimp as TIFFs. Then they are processed using a script that runs various ImageMagick actions to create various sizes (copies) as JPEGs. The source TIFF is preserved unchanged (and the original TIFF for the negative example above is CORRECT!). The JPEGs are then uploaded to a hosting service (in this case all the 'original' JPEGs still on my server are CORRECT!!).
So... The question is..... is there a bit in an image file that can be hit by a "stray neutrino" (or whatever happens) that can cause the image file to "become" negative". If yes for JPEG, is it also possible for TIFF?
Or, it seems more likely to me, would such a change from positive to negative require a very large number of changes to the image file?
If the type of occasional damage I am observing in a "small" library of about 100,000 image files is happening everywhere, to everybody, to all types of computer files, we are in deep do-do in the long term. As they say.... "Houston, we have a problem."
Actually it's not a negative... and the changes would be on more than one bit since the bad image file is about 50% bigger than the good one.
But the images aren't even the same size (314x431 vs 312x426), and if you look closely in the corners the perforations aren't the same (you can invert the colors of the bad ones to make it more visible). So these are two images of different stamps.
The 'bad' image has a color profile, and this one may be incorrect.
Hmmm. Answered without reading the end of the post... So these are indeed different subjects... But if you want anyone to determine what happens when an image is corrupted you have better provide the before/after versions of the same image.
A bad hard disk or file system problems can cause A JPEG to become corrupt but the look is different.
Image hosts often have a cavalier attitude wrt to the images deposited on them (arbitrary format changes, quality reduction...) but I have never heard of any adding a color profile.
If that can help the profile seems to be added by some Hewlett-Packard software, but I've never seen any HP software running on Linux and your whole process is on Linux.
Is there a single bit that can accidentally flip an image to negative? (not Gimp specific)
I'm going to call the image that isn't inverted "right.jpg" and the image that is inverted (not really inverted, but certainly it's not right) "wrong.jpg", to avoid typing really long file names.
"Right.jpg" doesn't have an embedded ICC profile. Right.jpg was created using Gimp, according to the metadata. Upon selecting it to open it, Gimp displays a normal-looking thumbnail that resembles "right.jpg". Upon opening it, Gimp automatically assigns the Gimp built-in sRGB profile and right.jpg looks like you'd expect.
"Wrong.jpg" does have an embedded ICC profile with description "sRGB IEC61966-2.1". From the embedded profile metadata it appears to be some version or another of the original sRGB profile created by Hewlett-Packard and Microsoft back in 1998. That profile or versions thereof has been used by almost all imaging software and operating systems, until V4 profiles started creeping in.
Upon selecting wrong.jpg with Gimp to open it, the thumbnail looks more or less like the thumbnail for right.jpg, but not exactly (the colors are washed out, the blue looks purple, the green looks brown). But upon actually opening it, the colors go all wrong. However, assign the Gimp built-in sRGB profile in place of the embedded profile and wrong.jpg now looks very similar to the right.jpg, slightly washed out, purple instead of blue, brown instead of green, but not "inverted".
According to the metadata, wrong.jpg is a Photoshop thumbnail. Was the original image created by Photoshop? Was the thumbnail really created using Photoshop and extracted using some software?
I've attached a spreadsheet with the embedded metadata for right.jpg, wrong,jpg, and the argyllcms version of the original MS/HP sRGB profile, lined up so the metadata fields match. There are very small differences in the metadata, not enough to explain why wrong.jpg looks wrong until the Gimp built-in sRGB profile is assigned.Attached is also a copy of "wrong.jpg" with the argyllcms version of sRGB embedded so you can see the color differences.
I'm going to try to extract the embedded ICC profile in wrong.jpg to see what the tone curves look like. Ofnuts is right, the problem (or at least one problem) is the embedded profile. And the two images aren't the same to begin with.
Elle Stone
http://ninedegreesbelow.com - articles on open source digital photography
Is there a single bit that can accidentally flip an image to negative? (not Gimp specific)
It turns out that the embedded profile in "wrong.jpg" does have an incorrect tone response curve. See the attached screenshot comparing the incorrect TRC to what it should look like. The embedded profile compresses the original sRGB 1024-point TRC range from 0 to 65535 down to about half the original range, then drops abrubtly to 0. Very odd. Never seen that before.
So is there some software in your toolchain of softwares that handles your images, that is somehow embedding a corrupt version of the original sRGB ICC profile? Is there a timestamp that might let you know when the "inverted" images were last handled, which might let you know what software embedded the corrupt profile?
Elle Stone
http://ninedegreesbelow.com - articles on open source digital photography
Is there a single bit that can accidentally flip an image to negative? (not Gimp specific)
On 07/28/2013 07:00 AM, Elle Stone wrote:
I'm going to call the image that isn't inverted "right.jpg" and the image that is inverted (not really inverted, but certainly it's not right) "wrong.jpg", to avoid typing really long file names.
"Right.jpg" doesn't have an embedded ICC profile. Right.jpg was created using Gimp, according to the metadata. Upon selecting it to open it, Gimp displays a normal-looking thumbnail that resembles "right.jpg". Upon opening it, Gimp automatically assigns the Gimp built-in sRGB profile and right.jpg looks like you'd expect.
"Wrong.jpg" does have an embedded ICC profile with description "sRGB IEC61966-2.1". From the embedded profile metadata it appears to be some version or another of the original sRGB profile created by Hewlett-Packard and Microsoft back in 1998. That profile or versions thereof has been used by almost all imaging software and operating systems, until V4 profiles started creeping in.
Upon selecting wrong.jpg with Gimp to open it, the thumbnail looks more or less like the thumbnail for right.jpg, but not exactly (the colors are washed out, the blue looks purple, the green looks brown). But upon actually opening it, the colors go all wrong. However, assign the Gimp built-in sRGB profile in place of the embedded profile and wrong.jpg now looks very similar to the right.jpg, slightly washed out, purple instead of blue, brown instead of green, but not "inverted".
According to the metadata, wrong.jpg is a Photoshop thumbnail. Was the original image created by Photoshop? Was the thumbnail really created using Photoshop and extracted using some software?
I've attached a spreadsheet with the embedded metadata for right.jpg, wrong,jpg, and the argyllcms version of the original MS/HP sRGB profile, lined up so the metadata fields match. There are very small differences in the metadata, not enough to explain why wrong.jpg looks wrong until the Gimp built-in sRGB profile is assigned.Attached is also a copy of "wrong.jpg" with the argyllcms version of sRGB embedded so you can see the color differences.
I'm going to try to extract the embedded ICC profile in wrong.jpg to see what the tone curves look like. Ofnuts is right, the problem (or at least one problem) is the embedded profile. And the two images aren't the same to begin with.
Elle Stone
On 07/28/2013 07:21 AM, Elle Stone wrote:
It turns out that the embedded profile in "wrong.jpg" does have an incorrect tone response curve. See the attached screenshot comparing the incorrect TRC to what it should look like. The embedded profile compresses the original sRGB 1024-point TRC range from 0 to 65535 down to about half the original range, then drops abrubtly to 0. Very odd. Never seen that before.
So is there some software in your toolchain of softwares that handles your images, that is somehow embedding a corrupt version of the original sRGB ICC profile? Is there a timestamp that might let you know when the "inverted" images were last handled, which might let you know what software embedded the corrupt profile?
Elle Stone
Hi Elle & Ofnuts,
Thank you for your replies. However, I think I have wasted (at least some of) your time because my original information was incorrect. But, it has been a terrific learning process for me, so thank you!
a) Yes, the two example images are different; they were not supposed to be the same. The "correct/right" one was just to generally show what the colors were generally supposed to look like. I should have posted copies of the exact same file (on my server vs on the hosting site), but it turns out that would not have helped (or actually would have shown up the real problem), because...
b) I was INCORRECT (damn, my perfect record for the year has been ruined -- ha!) about the "wrong/negative" original TIFF being okay and the original (on my server) JPEG being okay. They are NOT okay. However, when I look at them using Linux file management and file viewer programs they look okay ... that must be because those programs are only looking at the preview and are not parsing the whole image file and/or looking at the color profile in the image file. The preview is okay, but the image (actually the color profile in the image file) is mucked up and looks "wrong/negative". It did not occur to me that there there were _three_ components involved (preview component, profile component, and image itself). LESSON: Look at the precursor image files in Gimp, not in file managers or file viewers.
c) Ofnuts: The "wrong/negative" image on the hosting provider is identical to the image on my server, when viewed by the exact same tools (and also filesize, etc.). I understand some "image hosting sites" mess with images. However, I am using a web hosting service that just gives me space and does not seem to mess with anything (thank goodness). Thanks for that idea, however.
Okay... the rest of the story.
The "wrong/negative" image dates from 2004 OR EARLIER. I think it really dates from the early-to-mid 1990s. However, all versions of it have a 2004 timestamp due to an incorrectly handled migration between hardware (three times between 1990 and 2004). The timestamps were not properly preserved.
The "wrong/negative" image was indeed created originally in the 1990s using Photoshop running on Windows 95. The color profile embedded is whatever Photoshop stuck on it.
I have _many thousands_ of such 1990s Photoshop-created images, but only a handful have shown this problem.
So, if I am understanding correctly, all I have to do is remove the profile and the image should be fine.
I _have_ successfully removed the profile and the original TIFF now looks perfect in Gimp.
To do this on Linux, using commands from the ImageMagick package, I did:
TO SEE THE PROFILE INFO: identify -verbose inputfilename.tif
TO REMOVE ALL PROFILES (and comments) convert inputfilename.tif -strip outputfilename.tif
(Not sure if the convert syntax is legacy or modern, but it worked for me).
(I could have used mogrify instead of convert. They are similar, but mogrify does the changes in-place thus destroying the old state of the image and with NO UNDO unless you have a backup original file. Much more dangerous for somebody like me who is experimenting!)
So.... the "damaged image file" was actually a damaged color profile within the image file.
In a way, that still leaves the original question; but perhaps it is a silly question. Whether or not flipping one bit (in the profile section of the image file) would cause this kind of damage? Or whether there would have to be extensive changes to the profile to cause it?
My best guess about the underlying source of the cause is either defective memory (RAM) or defective individual disk controller or defective RAID disk controller sometime in the last 15+ years. Unfortunately, all three problems have occurred at least once and I am just lucky that only a few random images have been found to be damaged. Still, the stray neutrino theory sounds better.
Jay
Is there a single bit that can accidentally flip an image to negative? (not Gimp specific)
On 28.07.2013 20:01, Jay Smith wrote:
(I could have used mogrify instead of convert. They are similar, but mogrify does the changes in-place thus destroying the old state of the image and with NO UNDO unless you have a backup original file. Much more dangerous for somebody like me who is experimenting!)
http://www.imagemagick.org/script/command-line-options.php#path
HTH, Michael
Is there a single bit that can accidentally flip an image to negative? (not Gimp specific)
On 07/28/2013 08:01 PM, Jay Smith wrote:
So, if I am understanding correctly, all I have to do is remove the profile and the image should be fine.
Technically you should better replace the profile by a correct one... or convert the images to sRGB (not using the imbedded profile, profile)
In a way, that still leaves the original question; but perhaps it is a silly question. Whether or not flipping one bit (in the profile section of the image file) would cause this kind of damage? Or whether there would have to be extensive changes to the profile to cause it?
My best guess about the underlying source of the cause is either defective memory (RAM) or defective individual disk controller or defective RAID disk controller sometime in the last 15+ years. Unfortunately, all three problems have occurred at least once and I am just lucky that only a few random images have been found to be damaged. Still, the stray neutrino theory sounds better.
I don't believe that all images would be subject to the same random change due to cosmic rays or else... My hypothesis is that someone once played with a color profile on one image, and never noticed that this changed the profile data file... Then all images subsequently scanned on that machine received a bad profile... As to why did it went unnoticed.... until recently most software (including web browsers) would not handle color profiles by default (or not at all) and thus would display an almost-correct profile-less rendering.
