From: Leonard Rosenthol
To: Nathan Carl Summers
CC: The Gimp Developers' list
Subject: Re: [Gimp-developer] GimpCon RFC: Portable XCF Date: Wed, 20 Aug 2003 10:40:51 -0400

At 8:45 AM -0700 8/12/03, Nathan Carl Summers wrote:

Not necessarily. You should be able to do it with any format with a good catalog system, but some will be easier than others.

How would you handle resizes? Either we could do immediate compaction or garbage collection. Both have their disadvantages.

Correct.

How about a TIFF-like directory chunk at the beginning (except

>hierarchical)?

That would be one solution - sure.

Can you think of a better one?

Well, it needs to be a directory of some sort - whether it is TIFF-like, XML-based, ZIP-like, whatever..

I think the goal of the XCF redesign is to become the de-facto standard for interchange of layered images.

Unless you get Adobe to adopt support for it in their applications - that simply won't happen! Whether you like it or not, Adobe is the standards bearer in this regard, followed by the other major commercial players - Corel, Jasc, etc.

well, it'd be interesting to see if Adobe added XCF to Photo$hop, after all, GIMP is the competition, it wouldn't be in their interests to support a multilayered image format that it controlled by someone else(although they might support PSP, i don't know haven't checked.)

it would be good if Jasc and Corel were to pick it up as a standard interchange format, that would put Adobe under some pressure at least.

And that is also part of my suggestion for using a pre-existing format like TIFF or PSD. There is always wide support for them...

but that means you'd have to leave out any new improvements to GIMP layer handling that are made, otherwise you'd be breaking compatibility with the "standard"(yes, that is a joke), PSD is only fully supported really by adobe, if we're going to have a standard file format that's simply a broken version of PSD5 i don't see much point.

as for TIFF, you wouldn't be able to do it in a standard readable TIFF, you'd need to update the format entirely, or simply break it, which, again, defeats the object.

In other words, any XCF
reader should be able to read any XCF writer's output.

A reasonable requirement, to an extent. Do we expect that every XCF reader support ALL features of XCF?

i wouldn't expect them to, some would only want to produce a thumbnail, as with Nautilus, but i guess the standard decoder could provide a way of doing that anyway.

A layered TIFF by that name wouldn't cut it, because most tiff readers don't support layered images.

Sure they do! Well, at least for any program that supports multiple layers/pages to begin with. And this goes to the question above...

can you post a multilayer TIFF somewhere for me to try out, i've never encountered one before, and it would be interesting to see how it's handled.

If my application doesn't support a particular feature of XCF, am I not compliant? Should I not bother doing XCF?

it'd be nice if your app could read and render a flat version of the image if you don't support layers i supose, this is an interesting one since all these different target apps will handle things like layermodes differently, and some wouldn't even be supported.

Of course, we could always use TIFF internally but call it XCF.

We could do that.

Adobe does that with .ai, which is really .pdf...

i thought it was a kind of encapsulated post script

We might want to change the magic number as well.

Wouldn't do that, since the whole idea is to maintain compatibility...

no, that simply wouldn't be flexible enough, surely, i mean you could have extra data about how do use the layers in the TIFF but if those aren't recognised by other readers you just get a strange result and a confused decoder.

I have no problem with basing Portable XCF on TIFF. It seems to be well designed without really being too overdesigned. On the other hand, I think there are a few improvements that we could make to make it better for the purposes of GIMP.

I agree, though I think we can add all of these through additional tags and not having to redesign...

i'm sure it's fine for a basis, but not to keep compatible with the existing TIFF format.

/me wonders if the CinePaint people have any thoughts...

Definitely!

hmmm, yes, would be interesting, but they're sticking with their current tree aren't they, they wont make the eventual move to GEGL, and i thought this discussion was about the XCF version designed for it.

Phil.

Leonard
--
--------------------------------------------------------------------------- Leonard Rosenthol

Phil Harper wrote:

From: Leonard Rosenthol
To: Nathan Carl Summers
CC: The Gimp Developers' list
Subject: Re: [Gimp-developer] GimpCon RFC: Portable XCF Date: Wed, 20 Aug 2003 10:40:51 -0400

At 8:45 AM -0700 8/12/03, Nathan Carl Summers wrote:

A layered TIFF by that name wouldn't cut it, because most tiff readers don't support layered images.

Sure they do! Well, at least for any program that supports multiple layers/pages to begin with. And this goes to the question above...

can you post a multilayer TIFF somewhere for me to try out, i've never encountered one before, and it would be interesting to see how it's handled.

If my application doesn't support a particular feature of XCF, am I not compliant? Should I not bother doing XCF?

it'd be nice if your app could read and render a flat version of the image if you don't support layers i supose, this is an interesting one since all these different target apps will handle things like layermodes differently, and some wouldn't even be supported.

Of course, we could always use TIFF internally but call it XCF.

We could do that.

Adobe does that with .ai, which is really .pdf...

i thought it was a kind of encapsulated post script

What about mng? It contains png and has layers and comments. Seems to be basically unmaintained. I suggested it at another not so important software project and the idea went over real big!! I would like to know the GAP authors idea about mng, actually.

carol

Leonard Rosenthol wrote:

At 8:26 PM -0400 8/13/03, Carol Spears wrote:

What about mng? It contains png and has layers and comments.

Yes, but still has the limitations of no-CMYK (Lab, ICC, etc.) colorspaces (among other things) out of the box...

The last time I got the mng libraries, they came along with liblcms. Are you sure that liblcms does not do all of this?

Seems to be basically unmaintained.

PNG/MNG/JNG is fully supported and maintained by the libpng group, which is headed by Glenn Randers-Pehrson who is also one of the maintainers (along with myself) of ImageMagick and GraphicsMagick.

Sorry, my confusion. It is the plugin at that other ill-maintained godforsaken software package that is having maintenance reviews or something.

Rightly or wrongly, I consider ImageMagick to be the gimps command line until it gets too ugly and you need to start scripting. Probably this is wrong also?

tiff is so old. it seems like many old ideas have a lousy way of handling some of the new ideas. Sort of like comparing oggs to mp3's, the way I understand it, the framing idea is sort of a miracle to work in mp3's the way it does while with oggs this idea was built in from the beginning. This explanation explains the sound quality differences to me. I am wondering if this same line of thinking can be applied to tiffs?

carol

On Wed, Aug 20, 2003 at 08:54:55PM -0400, Leonard Rosenthol wrote:

At 11:42 PM +0000 8/13/03, Phil Harper wrote:

as for TIFF, you wouldn't be able to do it in a standard readable TIFF,

This, however, is wrong! We can represent EVERYTHING in GIMP today, and EVERYTHING for GEGL (etc.) in the future with TIFF. And other programs simply will ignore them as they do with other features of TIFF they don't support.

Does TIFF support, for example, float16 data, or a CIE XYZ colorspace?

I'm somewhat concerned with going with an externally standardized format, then running into a wall with it at some later point, and not being able to add a feature without breaking standards compliance.

-Yosh

From: Leonard Rosenthol
To: "Phil Harper" ,gimp-developer@xcf.berkeley.edu Subject: Re: [Gimp-developer] GimpCon RFC: Portable XCF Date: Wed, 20 Aug 2003 20:54:55 -0400 MIME-Version: 1.0

At 11:42 PM +0000 8/13/03, Phil Harper wrote:

well, it'd be interesting to see if Adobe added XCF to Photo$hop, after all, GIMP is the competition, it wouldn't be in their interests to support a multilayered image format that it controlled by someone else(although they might support PSP, i don't know haven't checked.)

They support a number of formats that they don't control - because they are standard formats that their customers are requesting. But today XCF isn't one of them, and probably won't be for a while.

the last thing Adobe wants to do is support XCF, it's a competing format belonging to a competing(and competatively priced) app.

it would be good if Jasc and Corel were to pick it up as a standard interchange format, that would put Adobe under some pressure at least.

It might, but again, I can't see them doing it. What's the ROI for them to invest in this? They already support PSD and TIFF as the two richest formats for layered raster images. (not to mention PS and PDF).

there's very little, but eveything else seems rather speculative at this stage, so why not.

but that means you'd have to leave out any new improvements to GIMP layer handling that are made, otherwise you'd be breaking compatibility with the "standard"(yes, that is a joke),

That is true, and a big reason to not use PSD...

as for TIFF, you wouldn't be able to do it in a standard readable TIFF,

This, however, is wrong! We can represent EVERYTHING in GIMP today, and EVERYTHING for GEGL (etc.) in the future with TIFF. And other programs simply will ignore them as they do with other features of TIFF they don't support.

why would i want to save to a file format that would render my image that's built up of layer masks and vector text layers really badly if opened in a standard viewer rather than in a format engineered from the ground up for all of the requirements i could have, and that is distinctive as being a GIMP image format, rather than just a really ugly TIFF?

can you post a multilayer TIFF somewhere for me to try out, i've never encountered one before, and it would be interesting to see how it's handled.

Easy enough to create one with ImageMagick using a bunch of other files.

convert a.png b.jpg c.gif +adjoin output.tiff

thanks i might try that.

it'd be nice if your app could read and render a flat version of the image if you don't support layers i supose, this is an interesting one since all these different target apps will handle things like layermodes differently, and some wouldn't even be supported.

EXACTLY my point!

Whatever file format we end up with, we need to accept that not all consumers of that file format will be able to support 100% of the features (perhaps not even GIMP itself).

so why use a format that all consumers would expect to be able to utilise 100%, it would surely confuse the hell out of your average photo$hop users to use TIFF in this way, especially if we're looking at cross compatibility.

Adobe does that with .ai, which is really .pdf...

i thought it was a kind of encapsulated post script

It was through version 8, but since version 9, it has been PDF...

ah, i see

no, that simply wouldn't be flexible enough, surely, i mean you could have extra data about how do use the layers in the TIFF but if those aren't recognised by other readers you just get a strange result and a confused decoder.

You could get that just as easily with XCF when a given consumer/reader doesn't support 100% of the features of the format...

in which case you'd have to do something about a workaround, like maybe have an optional pre-rendered version of the image stored in the XCF for viewers that don't support it properly, but, at that point it's questionable that you'd want to view an XCFin something other than GIMP, remember, GIMP has this handy thing called export, if your target audience wont be able to read an XCF then don't give them one, give then a PNG instead.

Phil.

Leonard
--
--------------------------------------------------------------------------- Leonard Rosenthol

__________________

On Wed, Aug 20, 2003 at 10:53:14PM -0400, Leonard Rosenthol wrote:

At 6:51 PM -0700 8/13/03, Manish Singh wrote:

Does TIFF support, for example, float16 data, or a CIE XYZ colorspace?

Yes to both...

Hmm, got a reference to that? It wasn't immediately apparent in my reading of the spec.

I'm somewhat concerned with going with an externally standardized format, then running into a wall with it at some later point, and not being able to add a feature without breaking standards compliance.

Worse case is that we add new tags (that we've registered with Adobe) and other readers don't support that information...

What's the turnaround time for that? Taking weeks or months isn't really acceptable...

Another thing I'm worried about is simply adding to the confusion of "what is a tiff file". There are few tiff readers/writers that support the entire feature set of tiff, and many broken implementations out there. There's an advantage of starting from a clean slate, and not having to worry about existing baggage.

-Yosh

On Wed, 13 Aug 2003 23:42:36 -0700, Manish Singh wrote:

On Thu, Aug 21, 2003 at 01:38:01AM -0400, Leonard Rosenthol wrote:

At 8:12 PM -0700 8/13/03, Manish Singh wrote:

What's the turnaround time for that? Taking weeks or months isn't really acceptable...

It's there to make sure that people don't duplicate tags - so as long as we pick pretty unique tags related to the GIMP, it's not a problem.

It's not just the tags, but extending value ranges for tags (needed for the two cases above).

If I am not mistaken, one cannot extend the value range of tags that are already registered. So the only solution is to define a new tag based on the old one and extending its capabilities. This also means that most TIFF readers will just discard the new tags because they are not able to deal with them. One solution (kludge) would be to encode as much of the data as possible in the "standard" tags (i.e., for CIE LAB) and then encoding the differences in a new tag (whatever parts of XYZ do not fit in LAB). That would allow more software to read at least a part of the data, but do we really want to do that? Probably not.

For GIMP, we're better off to have a native file format we have the last word on rather than trying to co-opt something else and twisting things to work. Certainly, there should be a libxcf for other programs to use, and include thumbnails and other convenience ancillary data for simpler programs to work with.

From my point of view, this is the best solution. However, there is

one thing that has not been mentioned in this discussion until now: we have to state very clearly that the new XCF is meant to be an open format (or not!). There has always been some confusion about whether the current XCF could be used by other applications (e.g. file managers, indexing programs or other image editing software). I think that we have to choose one of the following two solution and not something in between:

- XCF is an open format that can be used by other applications and can be used as an interchange format. In this case, every single feature of the file format has to be documented very clearly. The documentation should not lag behind the implementation (even during development). Also, the file format should use version numbers or tag names for each part of the file and there should be a way for other applications to know if understanding a given part of the file is mandatory or if it is optional (like PNG does with the naming of its tags). Ideally, the code for loading and/or saving XCF files should be available as one or two libaries distributed under the LGPL or maybe a BSD license. If XCF files can be produced by other applications than the GIMP (and other libraries than the XCF library included in the GIMP), then we should be prepared to handle files that are broken in various ways and fail gracefully.

- XCF is mostly a GIMP internal file format and XCF files are not intended to be distributed widely and read or written by other applications. In this case, we can have some documentation but we make no promises to other applications. The file format can change at any time (as the GIMP developers add new features or fix some bugs in the file format) and the others have to deal with it. This also means that the file format is not a standard so nothing would prevents another application from extending XCF in some incompatible way: if XCF files are not intended for distribution, any application can do whatever it wants to do with its private version of XCF. Although I think it could have been done in a better way (more communication and more care about XCF version numbers), there is nothing really wrong in creating an incompatible version of XCF for FilmGimp/CinePaint as long as XCF files are intended to be private files for the application that created them.

We are leaning towards the first solution for the new XCF file format. Previously, XCF was usually defined as belonging to the second category. If we want XCF to be an open standard that can also be used for distribution of images and exchange of files between several applications, we have to do it in the right way. We also have to be prepared to deal with the consequences: one of them will be that we cannot make any assumptions about the correctness of the files that we try to read. From now on, the GIMP will have to be (more) careful when reading the XCF files and implement some (more) error detection and recovery.

Also, if we define a new version of the XCF file format to be used in GIMP 2.2 or 3.0, this means that the old versions used in GIMP 1.x and 2.0 and the versions used in FilmGimp/CinePaint will become frozen. This may not be true for the CinePaint version if the CinePaint developers do not want to adopt the new XCF file format, but I hope that we can define a new format that will suit everybody. If the old versions are frozen, this would be a good opportunity to document them so that the applications that are interested in backwards compatibility can read these files.

-Raphaël

On Thu, Aug 21, 2003 at 01:33:50AM -0400, Leonard Rosenthol wrote:

why would i want to save to a file format that would render my image that's built up of layer masks and vector text layers really badly if opened in a standard viewer

Because at least you COULD open it up in a standard viewer.

Is it better to be able to get at the data in SOME format, but not perfect, using a 3rd party tool - OR not get ANY of your data?!?!

I see no point in being able to open a GIMP-file called .tif with CIE XYZ data in it - most of the viewer would simply say: "this is broken".

I suppose that most of the time, the GIMP-TIFFs are so special that they cannot be viewed with a standard TIFF viewer. As already noted, if your audience cannot read GIMP-files, you can always export the image.

IMO we gain nothing by using TIFF (apart from (ab?)using an existing file format). I'm still for the archive+XML+image data as PNG (or TIF?) approach - it allows the image to be manipulated externally. A thumbnail could be embedded such that it's easy to extract, so viewers have a chance to display something.

so why use a format that all consumers would expect to be able to utilise 100%, it would surely confuse the hell out of your average photo$hop users to use TIFF in this way, especially if we're looking at cross compatibility.

Actually, many users already DO use Photoshop and TIFF this way! If you have a multi-layered PSD file, including text layers, layer effects, etc and you save as TIFF, Photoshop writes out all the information necessary for it to coime back into Photoshop with full fidelity. BUT if you open it up in some simple TIFF viewer - of course, you don't get the same effect.

GIMP's use of TIFF would be EXACTLY the same...

I don't see the point of being able to get a rough approximation (or total garbage) of the image when opening it in a "simple viewer".

in which case you'd have to do something about a workaround, like maybe have an optional pre-rendered version of the image stored in the XCF for viewers that don't support it properly,

Which is what Photoshop does in PSD...

For applications that support layers, you can read them. If you don't, there is an already rendered/flattend version available.

I don't like the idea of having my A3/300dpi poster stored prerendered in the file. Of course, this could be an option. But I had to work with such beasts and even on kick-ass machines, you need some patience and the files tend to get huge.

GIMP has this handy thing called export, if your target audience wont be able to read an XCF then don't give them one, give then a PNG instead.

Sure, and lose all the extra information that might be useful to them in other authoring environments...

And what about posting things online or to archives??

I think, this could be implemented as an extra: If you export an MNG, the XML description could be embedded into the file. Then you have the archive+XML+imagedata approach but a bit reversed. It would also work for TIFF.

The biggest problem I see is that users will start using weird image formats if GEGL becomes available. Maybe, I want my images to be 16.8 fixed point in HLS colorspace? There are probably only a few readers out there which are able to display this... but I may overestimate this.

Also, being able to get at the layer data does not mean that you can represent the image appropiately. You'd need to implement lots of layer blending modes etc. Of course, a feature-rich libxcf could solve part of that problem.

Bye, Tino.

leonardr@lazerware.com (2003-08-21 at 1016.13 -0400):

At 11:42 PM -0700 8/13/03, Manish Singh wrote:

Supports IEEE floats, but not float16 (a 32-bit float cut in half). R&H added this to filmgimp since they had established this format in their workflow with other tools already.

Why would you only use half of a 32bit float?? That reduces your accuracy/precision and makes you incompatible with the rest of the world doing floating point imaging.

It is a trade off, ints vs floats. Better ranges than ints (in a rough sense) with less bandwith than floats. Currently supporting float is is the contrary of incompatible: it is a format for video cards (that explains why it was created and the tradeoff). People working in games want it, people working in using hw to render 3d anims want it to.

GEGL uses XYZ as a native format.

Why? Lab is a richer model esp. for handling chromanicity and is also a standard in the print world natively. Why limit to XYZ??

Does using XYZ imply that LAB is not supported?

It's not just the tags, but extending value ranges for tags (needed for the two cases above). And a lag time means either waiting for an updated spec, which is a holdup, or going ahead and running the risk it not being granted because someone else tried to get their conflicting values in first.

The spec is only updated every 18-24 months when Adobe releases a new version of Photoshop - so you definitely don't wait for that! As for the other, yes, that is true you could wait, but nobody does...

Where are those updates? Is it some kind of errata or addon? The PDF I have says 1992 (TIFF v6.0).

Never implemented a file format, have you ;). Reading/Writing the 'ar' archive, and reading/writing the XML is the easy part - because you can leverage existing libraries to

[...]

Worth the work, sure! Trivial - no way!

That is the reason other ideas are being examined.

GSR

leonardr@lazerware.com (2003-08-21 at 1016.13 -0400):

At 11:42 PM -0700 8/13/03, Manish Singh wrote:

Supports IEEE floats, but not float16 (a 32-bit float cut in half). R&H added this to filmgimp since they had established this format in their workflow with other tools already.

Why would you only use half of a 32bit float?? That reduces your accuracy/precision and makes you incompatible with the rest of the world doing floating point imaging.

nVidia graphics hardware uses half-float - it's useful where bandwidth is a premium - such as downloading high dynamic range images into graphics hardware at realtime rates - and in a lot of cases, a half precision float is still very useful.

---- Steve Baker (817)619-2657 (Vox/Vox-Mail) L3Com/Link Simulation & Training (817)619-2466 (Fax) Work: sjbaker@link.com http://www.link.com Home: sjbaker1@airmail.net http://www.sjbaker.org

Leonard Rosenthol wrote:

At 8:26 PM -0400 8/13/03, Carol Spears wrote:

What about mng? It contains png and has layers and comments.

Yes, but still has the limitations of no-CMYK (Lab, ICC, etc.) colorspaces (among other things) out of the box...

Has anyone considered going to the PNG maintainers and asking for these things to be included?

The GIMP project is not without influence in the OpenSource world.

---- Steve Baker (817)619-2657 (Vox/Vox-Mail) L3Com/Link Simulation & Training (817)619-2466 (Fax) Work: sjbaker@link.com http://www.link.com Home: sjbaker1@airmail.net http://www.sjbaker.org

Leonard Rosenthol wrote:

At 11:42 PM -0700 8/13/03, Manish Singh wrote:

GEGL uses XYZ as a native format.

Why? Lab is a richer model esp. for handling chromanicity and is also a standard in the print world natively. Why limit to XYZ??

I am not sure what you mean by a "richer model." Lab is a one-to-one mapping of XYZ. Every color in Lab is also in XYZ and visa versa. The transforms to/from XYZ from most other colorspaces is faster. Besides, Lab is _defined_ in terms of the XYZ colorspace. (well actually Lab is defined in terms of the xyY colorspace, which in turn is defined by the XYZ color space). And XYZ is not a "limit." XYZ is, to the best of the worlds scientific ability, contains every other 3 components human perception colorspace. You can convert from any colorspace to XYZ without a loss of information. And as far as it being gegl's "native" format, what he really means is that XYZ is used precisly in this fation. As a connection space between different colorspaces (just like most generic ICC color profiles are designed to do).

-- Dan

Leonard Rosenthol wrote:

At 6:51 PM -0700 8/13/03, Manish Singh wrote:

Does TIFF support, for example, float16 data, or a CIE XYZ colorspace?

Yes to both...

I'm somewhat concerned with going with an externally standardized format,
then running into a wall with it at some later point, and not being able to add a feature without breaking standards compliance.

Worse case is that we add new tags (that we've registered with Adobe) and other readers don't support that information...

Adobe needs to register with us first.

carol

On Thu, 14 Aug 2003, Sven Neumann wrote:

Hi,

I never understood the reasoning for this discussion anyway. IMHO the format that Nathan suggested seems like something from the dark ages of file formats (where TIFF and the like originated from).

PNG is something from the dark ages?

I haven't heard a single good argument for it except that it can do most of the things that the XML/archive approach can do.

s/most/all, and many other good things besides.

There was however nothing mentioned that it can do better. Or did I miss something?

XML is a text markup language. If the designers thought of using it for raster graphics, it was an afterthought at best. XML is simply the wrong tool for the job. The XML/archive idea is the software equivalent of making a motorcycle by strapping a go-cart engine to the back of a bicycle. It will work, of course, but it's an inelegant hack that will never be as nice as something designed for the job.

But to answer your question:

1. Putting metadata right next to the data it describes is a Good Thing. The XML "solution" arbitrarily separates human readable data from binary data. No one has yet considered what is to be done about non-human readable metadata, but I imagine it will be crammed into the archive file some way, or Base64ed or whatever. Either way is total lossage.

2. Imagine a very large image with a sizeable amount of metadata. If this seems unlikely, imagine you have some useful information stored in parasites. The user in our example only needs to manipulate a handfull of layers. A good way of handling this case is to not load everything into memory. Say that it just parses out the layer list at the start, and then once a layer is selected and the metadata is requested, it is read in. With the XML proposal, the parser would have to parse through every byte until it gets to the part it is interested in, which is inefficient. Frankly, this wouldn't be feasable. Only two crappy ways would be possible to get around this: store everything in memory (hope you have plenty of virtual memory!) or write out a temp file with the metadata in it, for later use, and in a random-accessable format. If you're going to do that, why not do it right the first time and save yourself the trouble?

3. None of the current suggestions for archive formats do a good job with in-place editing. AR can't even do random access. Zip can do an ok job with in-place editing, but it's messy and often no better than writing a whole new file from scratch. This means that a program that makes a small change to a file, such as adding a comment, needs to read in and write a ton of crap.

4. Implementing a reader for the XML/archive combo is unnecessarily complex. It involves writing a parser for the semantics and structure of XML, a parser for the semantics and structure of the archive format, and a parser for the semantics and structure of the combination. It is true that libraries might be found that are suitable for some of the work, but developers of small apps will shun the extra bloat, and such libraries might involve licensing fun. The semantics and structure of the combination is not a trivial aspect -- with a corrupt or buggy file, the XML may not reflect the contents of the archive. With an integrated approach, this is not a concern.

5. Either the individual layers will be stored as valid files in some format, or they will be stored as raw data. If they are stored as true files, they will be needlessly redundant and we will be limited to whatever limitations the data format we choose uses. If we just store raw data in the archive, then it's obvious that this is just a kludge around the crappiness of binary data in XML.

Rockwalrus

leonardr@lazerware.com (2003-08-14 at 1440.34 -0400):

The updates were originally done as technical notes, now they are incorporated into the main TIFF v7 spec which is part of the Photoshop SDK.

They seem to be very friendly and open about it:

From http://partners.adobe.com/asn/photoshop/index.jsp

"Q: Why is Adobe changing the policy on how the Photoshop SDK is distributed?

A: The Photoshop SDK contains Adobe-owned intellectual property and for that reason, Adobe needs to capture and verify contact information for every party that desires to use this developer kit for business or personal use."

By bundling TIFF into it they are doing a "nice" service to spread the format and make everyone have compatible readers and writers. At least it seems clear, it is about lawyers not about technology.
GSR

PS: Sorry, I forgot, quotation from a document "Copyright © 2003 Adobe Systems Incorporated. All rights reserved." Cos copyright laws still allow quotation, no? :]

Guillermo S. Romero / Familia Romero wrote:

leonardr@lazerware.com (2003-08-14 at 1440.34 -0400):

PS: Sorry, I forgot, quotation from a document "Copyright © 2003 Adobe Systems Incorporated. All rights reserved." Cos copyright laws still allow quotation, no? :]

That would depend on which side of the DMCA you stand.

Hi,

On Thu, 2003-08-14 at 22:58, Nathan Carl Summers wrote:

I haven't heard a single good argument for it except that it can do most of the things that the XML/archive approach can do.

s/most/all, and many other good things besides.

Which are?

There was however nothing mentioned that it can do better. Or did I miss something?

XML is a text markup language. If the designers thought of using it for raster graphics, it was an afterthought at best. XML is simply the wrong tool for the job. The XML/archive idea is the software equivalent of making a motorcycle by strapping a go-cart engine to the back of a bicycle. It will work, of course, but it's an inelegant hack that will never be as nice as something designed for the job.

I think it is an elegant solution to the problem of designing a file format w/o knowing beforehand what will have to go into it. I don't think that binary chunks are feasible for a format that will have to extend a lot while it is already being used. None of the file formats mentioned provide this functionality and I think it is essential here.

But to answer your question:

1. Putting metadata right next to the data it describes is a Good Thing. The XML "solution" arbitrarily separates human readable data from binary data. No one has yet considered what is to be done about non-human readable metadata, but I imagine it will be crammed into the archive file some way, or Base64ed or whatever. Either way is total lossage.

How is metadata in the archive total lossage? If the metadata is binary it should of course be treated just like image data.

2. Imagine a very large image with a sizeable amount of metadata. If this seems unlikely, imagine you have some useful information stored in parasites. The user in our example only needs to manipulate a handfull of layers. A good way of handling this case is to not load everything into memory. Say that it just parses out the layer list at the start, and then once a layer is selected and the metadata is requested, it is read in. With the XML proposal, the parser would have to parse through every byte until it gets to the part it is interested in, which is inefficient.

The XML parser would only have to read in the image structure which tells it where to locate the actual data in the archive, nothing else.

4. Implementing a reader for the XML/archive combo is unnecessarily complex. It involves writing a parser for the semantics and structure of XML, a parser for the semantics and structure of the archive format, and a parser for the semantics and structure of the combination. It is true that libraries might be found that are suitable for some of the work, but developers of small apps will shun the extra bloat, and such libraries might involve licensing fun.

We are already depending on an XML parser right now. I don't see any problem here. I do know however that the code that reads stuff like TIFF or PNG is ugly and almost unreadable. SAX-based XML parsers tend to be darn simple however.

The semantics and structure of the combination is not a trivial aspect -- with a corrupt or buggy file, the XML may not reflect the contents of the archive. With an integrated approach, this is not a concern.

I don't see how an integrated approach avoids this problem any better.

5. Either the individual layers will be stored as valid files in some format, or they will be stored as raw data. If they are stored as true files, they will be needlessly redundant and we will be limited to whatever limitations the data format we choose uses. If we just store raw data in the archive, then it's obvious that this is just a kludge around the crappiness of binary data in XML.

I don't understand you. If you think that raw data is a good idea, we can have have raw data in the XML archive. Allowing a set of existing file formats to be embedded makes the definition of our format a lot simpler however and allows for various established compression techniques to be used.

Sven

Leonard Rosenthol wrote:

At 3:33 PM -0400 8/14/03, Carol Spears wrote:

So this combination would answer your LAB & CMYK issues and possibly my need to use a greater than 256 color palette then?

No, it would not.

ICC profiling is a VERY different thing that actual "raw" CMYK or Lab data...

Paletizing of an image is also different...

Well, I don't understand the color issues that well. Merely my own limitations with TheGIMP so far.

Complaints I remember reading from more "technically inclined" people about tiff were mostly about the lwz compression. I guess while it was not free it was also not the best way to go about doing such a thing.

Yes, that was a legal issue, not a truly technical one. (LZW, not lwz).

Here is an example of my lazy brain working for me. As soon as I read something that makes me think "expensive, selfish and substandard" (as this compression and those three letters make me think) my brain stops giving time and space to the idea.

My worst fear is that TheGIMP will settle for something that came from this sort of thought process and development cycle.

Eh, something like "spiritually unsound" is fine if we are getting the best thing. I don't think we would be if we took this tiff route.

Does tiff have a comments area? I use jpeg comment often and am anxious to start using comments in pngs. Rumor has it that the capability is there ....

However, I read recently about artifacts appearing in compressed pngs, so this might not be the miracle fix I had hoped for.

PNG won't artifact images unless you are palettizing them, which is NOT the default.

This was someone bitching on the irc. I don't know all of the details and I did not see the image.

I was without power for more than a day, I am hoping to read the rest of the mail and see that we will be using mng as a base and redesigning it some. :)

carol

Nathan Carl Summers wrote:

On Thu, 14 Aug 2003, Sven Neumann wrote:

Hi,

I never understood the reasoning for this discussion anyway. IMHO the format that Nathan suggested seems like something from the dark ages of file formats (where TIFF and the like originated from).

PNG is something from the dark ages?

I haven't heard a single good argument for it except that it can do most of the things that the XML/archive approach can do.

s/most/all, and many other good things besides.

There was however nothing mentioned that it can do better. Or did I miss something?

XML is a text markup language. If the designers thought of using it for raster graphics, it was an afterthought at best. XML is simply the wrong tool for the job. The XML/archive idea is the software equivalent of making a motorcycle by strapping a go-cart engine to the back of a bicycle. It will work, of course, but it's an inelegant hack that will never be as nice as something designed for the job.

But to answer your question:

1. Putting metadata right next to the data it describes is a Good Thing. The XML "solution" arbitrarily separates human readable data from binary data. No one has yet considered what is to be done about non-human readable metadata, but I imagine it will be crammed into the archive file some way, or Base64ed or whatever. Either way is total lossage.

binary cludge is lossage? The
recent time I spent communing with dselect, I saw a couple of binary editors. The existence of such software makes me think that binary can be easily xmlized also.

Working with software my APT wrote to suit my needs on the new outdated and unloved web site (http://mmmaybe.gimp.org), it makes me want more of this same sort of editing ability with gimp stuff.

I have proven myself to be very very human though. The machines perhaps will not like the xml as much as I did.

2. Imagine a very large image with a sizeable amount of metadata. If this seems unlikely, imagine you have some useful information stored in parasites. The user in our example only needs to manipulate a handfull of layers. A good way of handling this case is to not load everything into memory. Say that it just parses out the layer list at the start, and then once a layer is selected and the metadata is requested, it is read in. With the XML proposal, the parser would have to parse through every byte until it gets to the part it is interested in, which is inefficient. Frankly, this wouldn't be feasable. Only two crappy ways would be possible to get around this: store everything in memory (hope you have plenty of virtual memory!) or write out a temp file with the metadata in it, for later use, and in a random-accessable format. If you're going to do that, why not do it right the first time and save yourself the trouble?

When someone asks me to imagine a large image file I naturally think of the biggest image files I ever worked with. This would be psd. It seems like the GIMP developers should be able to make something smaller than this.

Sorry, I got stuck on the first line of this item. Imagining a very large image file, and the previous mail about how psd is a personalized tiff makes me even less want to use it ever.

3. None of the current suggestions for archive formats do a good job with in-place editing. AR can't even do random access. Zip can do an ok job with in-place editing, but it's messy and often no better than writing a whole new file from scratch. This means that a program that makes a small change to a file, such as adding a comment, needs to read in and write a ton of crap.

4. Implementing a reader for the XML/archive combo is unnecessarily complex. It involves writing a parser for the semantics and structure of XML, a parser for the semantics and structure of the archive format, and a parser for the semantics and structure of the combination. It is true that libraries might be found that are suitable for some of the work, but developers of small apps will shun the extra bloat, and such libraries might involve licensing fun. The semantics and structure of the combination is not a trivial aspect -- with a corrupt or buggy file, the XML may not reflect the contents of the archive. With an integrated approach, this is not a concern.

mmmaybe has an xml reader. It is small and nice. An attribute rewritter that skips a lot of the crap that the old tools brought along with it. Or is this mention out of line here....

5. Either the individual layers will be stored as valid files in some format, or they will be stored as raw data. If they are stored as true files, they will be needlessly redundant and we will be limited to whatever limitations the data format we choose uses. If we just store raw data in the archive, then it's obvious that this is just a kludge around the crappiness of binary data in XML.

How much binary data is in images. This is very confusing to me. For example, which image formats contain binary data ....

carol