Sven,

Hi. Thanks for writing.

This is your first post to our list cinepaint-developers, and people here may wonder who you are and why you are introducing yourself by making demands. You haven't identified your role in GIMP and aren't mentioned in the authors list on gimp.org (http://gimp.org/the_gimp_about.html). Just so everyone knows, the GIMP has a different management style than CinePaint and doesn't identify its team -- except for Yosh, the GIMP release manager. Sven is perceived as the project manager of GIMP.

You are right that our gbr brushes are different from GIMP's. Although I appreciate you considering it, they can't be made fully compatible with GIMP because of bit depth. The best you could do would be to crush down to 8-bit when opening our 16-bit brushes.

The situation is worse than you think. Not only has the brush format been modified, the XCF format has been broken, too. Again, for higher bit depths than GIMP supports. Our users are surprised that they can't read 8-bit GIMP XCF files even though XCF is listed as our native format. While researching how to make our files more compatible with GIMP, I found your for-the-record statement that GIMP XCF is deliberately undocumented:

http://lists.xcf.berkeley.edu/lists/gimp-developer/2003-February/008106.html

I agree with you that our file formats are a mess, but none of this happened on my watch. I can't answer your question who made the change to the brush format because it predates my participation. A person who may know is Yosh, the GIMP release manager. In any case, it seems clear that the person who made the change would have been one of the original GIMP developers who worked on Film Gimp. I'm unaware of any implementation docs on what they did. By the way, where would I find your docs for the GIMP gbr format?

You are probably unaware that we have already done a quite bit of work eliminating file collisions with GIMP. Because the original plan for the Film Gimp branch was for it to be GIMP, the original developers didn't anticipate any name change.

The plan is for CinePaint to move away from this undocumented and incompatible XCF format we inherited -- and can't properly support -- to a new XML-ish file format that will be called CPX. I am still designing that, but have already documented the basic format. It should replace gbr too.

Cheers,

Robin --------------------------------------------------------------------------- Robin.Rowe@MovieEditor.com Hollywood, California www.CinePaint.org Free motion picture and still image editing software ----- Original Message -----
From: "Sven Neumann"
To:
Cc:
Sent: Wednesday, July 09, 2003 5:31 AM Subject: [CinePaint-dev] GIMP brush format version 3

Hi,

yesterday someone from the GIMP developers stumpled across this page:

http://cinepaint.sourceforge.net/dev/brushes.html

After examing the brushes that can be downloaded there, we found that GIMP can not read them although they have the .gbr extension and the magic file header for GIMP brushes. Actually it's even the very same format but version 3. But, version 3 of the GIMP brush format does not exist. What happened here?

I don't know exactly when this change to the brush format was made nor who made it. I just want to state that this is an inacceptible procedure. If FilmGIMP or CinePaint needs a new brush format for whatever reasons, they are of course free to design the new format closely to the GIMP brush format. But it can not be that you simply take the GIMP brush format and increase the version number. This is something that only the GIMP developers can decide to do. If for some reason you wanted to stick with the GIMP brush format, you should have at least asked. We could then have decided on a format change and include support for the new file format version in both applications.

From reading your web-page I understood that CinePaint does not even

support the GBR versions 1 and 2 that are used by The GIMP. This seems to indicate that you just needed a new and different format. You should have at least changed the magic file header then so that utilities such as file(1) are able to differentiate between GIMP brushes and CinePaint brushes.

There is not much we can do to change this situation now that it has happened. I am not going to yell at anyone since I don't even know how far this change dates back. But I want you to know that we are upset about what happened and I ask you to assure that similar things don't happen in the future. File formats are crucial and we can not accept any incompatibilities caused by third-party developers changing our file formats.

Well, there is one thing we can do: If someone would provide us with the details about the version 3 brush format, we might decide to include support for it in The GIMP. This would at least reduce confusion among our users.

Sven

------------------------------------------------------- This SF.Net email sponsored by: Parasoft Error proof Web apps, automate testing & more. Download & eval WebKing and get a free book. www.parasoft.com/bulletproofapps

Hi,

Robin Rowe wrote:

Just so everyone knows, the GIMP has a different management style than CinePaint and doesn't identify its team -- except for Yosh, the GIMP release manager. Sven is perceived as the project manager of GIMP.

I would like to clarify this - the GIMP is now 8 years old, 8 or 9 years old, and the original authors are long gone. What has happened since then is something of a co-operative effort, whereby a group of the current developers just kind of decide stuff.

This is haphasard at times, but it would be incorrect to say that we don't identify our team. But the team changes very regularly. That is not to say that certain characters who have been around for a while do not have more credence than others. It might be worthwhile to keep an up-to-date list of current developers, along with their domains of interest/responsibility, but the GIMP's authors are publicly acknowledged in the AUTHORS file.

I'm not sure how Sven feels about the title "Program manager" - if it helps you to label him as such, then so be it. He is one of the main developers around at the moment, and works more or less full time on the GIMP, so he has more say in contentious issues than blow-ins like me, but everyone is heard (often leading to bikeshed arguments, but there we go), and decisions get made more or less by concensus (which is a partly a factor in certain things taking longer than they might).

You are right that our gbr brushes are different from GIMP's. Although I appreciate you considering it, they can't be made fully compatible with GIMP because of bit depth. The best you could do would be to crush down to 8-bit when opening our 16-bit brushes.

As a proposal for a modification which would bring back compatibility, we could expand the header by 4 bytes to include bit depth (8 or 16), which could then be factored into the load routines of both our apps (we would crush 16 bit nbrushes down to 8 bits, and you would expand 8 bit brushes to 16 bits). As a file format change, it would allow is backward compatibility, since the format changes nothing in the other header fields.

By the way, where would I find your docs for the GIMP gbr format?

After reading this mail, I drew up a specification of the gbr format. It's fairly straightformward, and is obvious from the header files & source, but a text file documenting the format is no harm.

I will put it in the devel-docs directory of the GIMP CVS, and attach it inline here (I believe that our list software, and probably yours, doesn't like attachments).

Cheers, Dave.

The GIMP Paintbrush File Format Version 2 (.gbr) ------------------------------------------------

HEADER ------

Bytes 0 - 3: header_size: Type: 32 bit unsigned int
Value: size of brush header (28) + length of brush name

Bytes 4 - 7: version Type: 32 bit unsigned int
Value: The file format version. Currently

Bytes 8 - 11: width Type: 32 bit unsigned int
Value: Brush width

Bytes 12 - 15: height Type: 32 bit unsigned int
Value: Brush height

Bytes 16 - 19: bytes Type: 32 bit unsigned int
Value: Colour depth of brush.
1 = greyscale, 4 = RGBA

Bytes 20 - 23: magic_number Type: 32 bit unsigned int
Value: GIMP brush magic number.
('G' << 24) + ('I' << 16) + ('M' << 8) + 'P'

Bytes 24 - 27: spacing Type: 32 bit unsigned int
Value: Default spacing to be used for brush. Percentage of brush width.

Bytes 28 - (header_size - 28): Type: char *
Value: UTF-8 string - name of brush

BODY ----
Size: width * height * bytes
Type: uchar *
Value: Pixel values (row-first) for brush

On Wed, 09 Jul 2003 18:42:08 +0200 Sven Neumann wrote:

Him

"Robin Rowe" writes:

The plan is for CinePaint to move away from this undocumented and incompatible XCF format we inherited -- and can't properly support -- to a new XML-ish file format that will be called CPX. I am still designing that, but have already documented the basic format. It should replace gbr too.

You might then be interested to hear that we started to discuss a new file format for GIMP more than 3 years ago and came up with an XML based design as well. More details will probably be made up at the GIMP developers conference next month.

Couldn't both teams try to find a common format? After all XML is quite flexible and it should be easy to flag the extensions that are not supported by the other program. Both programs have a lot in common. Users will really appreciate it when they can easily exchange data between the two programs.

(I know this is probably not quite the right moment to suggest this)

greetings,

Ernst Lippe

(PS. Robin I would apreciate it if you could forward this message to the cinepaint-developers list. I think that this is an important issue for both groups of developers.)

David,

As a proposal for a modification which would bring back compatibility, we could expand the header by 4 bytes to include bit depth (8 or 16), which could then be factored into the load routines of both our apps (we would crush 16 bit nbrushes down to 8 bits, and you would expand 8 bit brushes to 16 bits). As a file format change, it would allow is backward compatibility, since the format changes nothing in the other header fields.

We're open to that. However, until someone volunteers and sends us a patch, we have no one ready to take that effort when our format is about to be replaced anyway.

The GIMP Paintbrush File Format Version 2 (.gbr) ------------------------------------------------

HEADER ------

Bytes 0 - 3: header_size: Type: 32 bit unsigned int
Value: size of brush header (28) + length of brush name

Bytes 4 - 7: version Type: 32 bit unsigned int
Value: The file format version. Currently

Bytes 8 - 11: width Type: 32 bit unsigned int
Value: Brush width

Bytes 12 - 15: height Type: 32 bit unsigned int
Value: Brush height

Bytes 16 - 19: bytes Type: 32 bit unsigned int
Value: Colour depth of brush.
1 = greyscale, 4 = RGBA

Bytes 20 - 23: magic_number Type: 32 bit unsigned int
Value: GIMP brush magic number.
('G' << 24) + ('I' << 16) + ('M' << 8) + 'P'

Bytes 24 - 27: spacing Type: 32 bit unsigned int
Value: Default spacing to be used for brush. Percentage of brush width.

Bytes 28 - (header_size - 28): Type: char *
Value: UTF-8 string - name of brush

BODY ----
Size: width * height * bytes
Type: uchar *
Value: Pixel values (row-first) for brush

Thanks, appreciate the docs!

To return the favor, enclosed is the spec for CPX that I posted in June. It is so new it hasn't made it onto our web page docs yet. For clarification, since my preliminary spec was rather terse, the data blob in CPX is a raw write of the raster like the PPM P6 format. The CPX format was inspired by the simplicity of PPM, but with the enhancement of XML tags. Like PPM, all CPX tags are text but raster data is binary. See at the bottom below.

....it would be incorrect to say that we don't identify our team.

I can clarify my meaning, although it seems a moot point. Some GIMP developers are identified in internal documents within the source code, as you correctly point out. However,when I asked for a list of developers on the GIMP list I was told those documents were too inaccurate to be trusted. It was suggested I could get the names of the GIMP developers by grepping the check-in names from GIMP CVS -- which seemed a bit of bother -- but I took the effort to do so anyway. I offered to give that list back to GIMP so they could post a better public list of developer acknowledgements. The response was that no new list should be posted by GIMP until it was perfect, and that my list was inaccurate because Sven and others often check in patches on behalf of others. I was also told my help wasn't needed.

FYI, using the best information I could gather, we have a list on our web site recognizing the GIMP developers.

http://cinepaint.sourceforge.net/people/gimp.html

Cheers,

Robin --------------------------------------------------------------------------- Robin.Rowe@MovieEditor.com Hollywood, California www.CinePaint.org Free motion picture and still image editing software

----- Original Message ----- From: "Robin Rowe"
To:
Sent: Saturday, June 07, 2003 3:11 PM Subject: [CinePaint-dev] cpx file type

Hi. Just FYI, creating a new image format called CPX (CinePaint XML).

This format will be similar in simplicity to PPM but more flexible using

XML

tags. Both spec and code will be made available.

A 640x480 8-bit image file example.cpx would be laid out something like this:

'depth' may be 8u|16u|16f-ilm|16f-rnh|32f 'raster' may be RGB|RGBA|RGBAZ or alternatively 'planes' with the same choices

A lot more features could be added later. This is just to let everyone

know

this is in the works and coming maybe in July.

Cheers,

Robin --------------------------------------------------------------------------

-

Robin.Rowe@MovieEditor.com Hollywood, California www.CinePaint.org Free motion picture and still image editing software

Ernst,

Couldn't both teams try to find a common format?

In theory yes, but it seems unlikely.

Sven said today that GIMP has been (privately) discussing a new XML-based file format for GIMP for more than three years -- which is the first I've heard of it. However, no spec has been made available. Without any knowledge of what GIMP was up to, I've been working on the design of a new XML-based file format for CinePaint for two months. I posted the preliminary spec for comment a month ago, and it is about to go into implementation. It could appear in CinePaint in as little as a month. We're not going to wait for GIMP.

Cheers,

Robin
--------------------------------------------------------------------------- Robin.Rowe@MovieEditor.com Hollywood, California www.CinePaint.org Free motion picture and still image editing software

Sven Neumann wrote:

We actually had something else in mind but since you don't seem to be interested I won't waste my time explaining our ideas.

It is very sad to see that Sven thinks that Robin Rowe is the only person to whom his ideas should be told. Pity the rest of the GIMP developers (current and future) who might like to comment on it.

Christopher

ccurtis@aet-usa.com (2003-07-10 at 1140.21 -0400):

It is very sad to see that Sven thinks that Robin Rowe is the only person to whom his ideas should be told. Pity the rest of the GIMP developers (current and future) who might like to comment on it.

Use the list archives. Search engines let you restrict to servers. And IIRC basically it was about packing XML and some common image format like PNG into a some common archive format like TAR, so while not all soft will be able to open it, at least the user will be able to dissasamble it and load things by hand.

GSR

At 02:34 PM 7/11/2003 +0200, Sven Neumann wrote:

XML is very well suited to describe the structure of a multi-layered, multi-framed image/animation and it can be used perfectly to embed meta information as well as vector layers, paths and the like. XML namespaces make it easy to add application-specific extensions later.

You bet!

We discussed to bundle one XML file with a set of files that store the image data. These files would preferably be known image formats or perhaps even strict subsets of known image formats.

Agreed.

I would suggest that these files are then put together in an ar archive. Such an uncompressed archive has the advantage that the XML metadata can easily be extracted.

You just described a JAR file, which is not unlike the OpenOffice file format ;).

A JAR is a special type of ZIP archive, which contains one or more "data files" along with an XML "manifest" about the contents. I've worked on a number of projects (both commercial and "open") that have used such a format - it works quite nicely and is compatible with existing tools and technologies. Always better than reinventing the wheel!!

I think the approach of a JAR-like file for the future GIMP (and possibly CinePaint) file format is an excellent choice and allows for many avenues of expansion.

Leonard

On Fri, Jul 11, 2003 at 10:08:55AM -0400, Leonard Rosenthol wrote:

A JAR is a special type of ZIP archive, which contains one or more "data files" along with an XML "manifest" about the contents. I've worked on a number of projects (both commercial and "open") that have used such a format - it works quite nicely and is compatible with existing tools and technologies. Always better than reinventing the wheel!!

the ar file format is much better established then jar, quick to access (unlike jar), and very very very much simpler. a complex container format like jar is not very helpful, as the added complexity just gets in the way (we just need to bundle files...)

I think the approach of a JAR-like file for the future GIMP (and possibly CinePaint) file format is an excellent choice and allows for many avenues of expansion.

i think jar-like is backwards. tar-like is much better (but tar itself is unsuitable due to the lack of rigid definitions and a multitude of different formats, the same is true for cpio).

Sven Neumann wrote:

Excuse me?!?! JAR is used by every Java implementation in existence, and since it is 100% compatible with ZIP, means you have all of those implementations as well.

Java is not exactly what I would call well established, but that's not a relevant argument here.

One issue we should at least think about with JAR is that since it *is* the JAVA library mechanism, there is perhaps a risk of allowing virus writers to attach bits of JAVA executable in what *appears* to be a GIMP image.

This is the kind of thing that email virus writers *love*.

Whilst I can't quite see how this would be a problem, it might be better to just avoid the risk.

----
Steve Baker (817)619-2657 (Vox/Vox-Mail) L3Com/Link Simulation & Training (817)619-2466 (Fax) Work: sjbaker@link.com http://www.link.com Home: sjbaker1@airmail.net http://www.sjbaker.org

* Guillermo S. Romero / Familia Romero [030714 13:13]:

ccurtis@aet-usa.com (2003-07-10 at 1140.21 -0400):

It is very sad to see that Sven thinks that Robin Rowe is the only person to whom his ideas should be told. Pity the rest of the GIMP developers (current and future) who might like to comment on it.

Use the list archives. Search engines let you restrict to servers. And IIRC basically it was about packing XML and some common image format like PNG into a some common archive format like TAR, so while not all soft will be able to open it, at least the user will be able to dissasamble it and load things by hand.

I am currently working on some code related to how GEGL works/is going to work in cooperation with gimp, bundling complete projects in a folder/tarball could be a good idea for this as well, my current format

http://phpweb.hig.no/~oey_kola/bauxite/bauxite.png

contains a screenshot where a similar processing graph is being edited.

/pippin

Date: Mon, 14 Jul 2003 08:30:45 -0400 From: Leonard Rosenthol

At 7:16 AM -0500 7/14/03, Stephen J Baker wrote:

>One issue we should at least think about with JAR is that since it >*is* the JAVA library mechanism, there is perhaps a risk of >allowing virus writers to attach bits of JAVA executable in what >*appears* to be a GIMP image.

If you don't open up the JAR file with a Java-based tool - you can't have Java executing.

And even if you DO use Java to open up the JAR, nothing "auto-executes" - you'd have to manually kick it off.

SO even if someone were to put Java bytecodes into a GIMP image file, it would never get executed...

What happens if in the future someone writes a gimp-java interface (like gimp-perl)? Would there be any security issues there?

At 08:38 AM 7/14/2003 -0400, Robert L Krawitz wrote:

What happens if in the future someone writes a gimp-java interface (like gimp-perl)? Would there be any security issues there?

No.

Leonard

On Sat, Jul 12, 2003 at 11:57:09PM -0400, Leonard Rosenthol wrote:

the ar file format is much better established then jar, quick to access (unlike jar), and very very very much simpler.

Excuse me?!?! JAR is used by every Java implementation in existence, and since it is 100% compatible with ZIP, means you have all of those implementations as well.

Ahem, still what I said above is true, isn't it? ar certainly is better establsihed, is faster to acesss (no compression), and is also extremely simple (it's fully human readable!)

Second, because of ar's simplicity it isn't suitable for this task for a number of technical reasons including name size limitations, incomplete hierarchy support, etc.

Well, ar's limitations with respect to name length etc. are in practise not every severe (nobody will have 10000 character file names (the ar limit), and even antique ar implemnentations will support up to 255 characters).

a complex container format like jar is not very helpful,

JAR/Zip isn't very complex and there is LOTS of implementations to work from...

Same is true for ar, except that it is even simpler and faster :)

I am not opposed to uncompressed jar files, but compression is certainly a bad idea (the jar compression algorithm is rather ineffective), so rathet than to support half of a complex file format a very simple one that simply stores files makes sense to me.

Yes, you need to bundle files, BUT in a hierarchical organization, esp. for layers...

well, nobody keeps you from doing that in ar.

(will abstain from any further jar/ar discussion... if you are inzterested, check the file formats itself).

On Mon, Jul 14, 2003 at 02:34:23PM +0200, Øyvind Kolås wrote:

I am currently working on some code related to how GEGL works/is going

Looks cool, however, one note:

You use id1, ref1 etc... attributes (and you are perfectly fine to use them), however, XML itself uses the id and idref attributes, and requires a "Name" as value. I'd think it would be a good measure to use the same syntax for your id1 and ref1 fields, so that people can transfer their knowledge about ids and references in xml to your application.

In other words: I'd replace the "#" by a letter.

:)

pcg@goof.com ( Marc) (A.) (Lehmann ) wrote:

Well, ar's limitations with respect to name length etc. are in practise not every severe (nobody will have 10000 character file names (the ar limit), and even antique ar implemnentations will support up to 255 characters).

Technically I don't know if that's true. From my ar man page: GNU ar can maintain archives whose members have names of any length; however, depending on how ar is configured on your system, a limit on member-name length may be imposed (for compatibility with archive formats maintained with other tools). If it exists, the limit is often 15 charac- ters (typical of formats related to a.out) or 16 charac- ters (typical of formats related to coff).

But, that is of no consequence -- we wouldn't need to keep meaningful names, we just need unique resource ids that the XML structure can refer to... these could quite simply be numbers counting up from zero, or hash strings. In either case just 15 characters could be fine.

I am not opposed to uncompressed jar files, but compression is certainly a bad idea (the jar compression algorithm is rather ineffective)

I like the idea of ar files. Compression then happens by [un]{b,g}zipping the ar via a compression plugin in the usual GIMP style.

HOWEVER, this might be a good time to think about whether we'd prefer a compressed format that we can random-access de/compress on the fly instead of going via a huge (and with image data we can easily be talking HUGE) temporary intermediate file. In this case something like a ZIP (or okay, equivilently, a compressed jar, whatever you want to call it) is a better (and still exceedingly standard in its most basic form) choice of wrapper for the hierarchy-file-plus-linked-resources.

--Adam

On Wed, Jul 16, 2003 at 04:28:18PM +0100, "Adam D. Moss" wrote:

Technically I don't know if that's true. From my ar man page: GNU ar can maintain archives whose members have names of any length; however, depending on how ar is configured on your system, a limit on member-name length may be imposed (for compatibility with archive formats maintained with other tools). If it exists, the limit is often 15 charac- ters (typical of formats related to a.out) or 16 charac- ters (typical of formats related to coff).

But, that is of no consequence -- we wouldn't need to keep

The "archive formats" that the ar manpage above refers to are what mortla people call "object files". Since ar is part of binutils it tries to be compatible to other object formats which often have severe limitations.

ar also handles hierarchical structures within ar files, but for compatibility it doesn't create them.

(and there are common extensions that allow unlimited name lengths, these are also handled by ar).

can easily be talking HUGE) temporary intermediate file. In this case something like a ZIP (or okay, equivilently, a compressed jar, whatever you want to call it) is a better (and still exceedingly standard in its most basic form) choice of wrapper for the hierarchy-file-plus-linked-resources.

"something like zip" is the key. zip won't do, of course, since it's lousy at compression and also doesn't support random access like we want. Also, I think compression at the file level (whole file) is not a good idea anyway, so we could keep a simple wrapper format (like ar) and implement a more intelligent block structure within the constituent files.

However, the reason why people propose ar, jar, cpio etc... as formats is that they cna be handled by users with ease, being well established.

If we would design our own extremely simple wrapper format there would be no need to work with the compatibility mess existing formats are (all of them :). If we say that access by other tools than gimp is not important we could get away with an extremely simple format, say, cat files*, index at end which could be just offset and length, indexed by number, meta-info is all handled by an xml "sub"-file.

The question is simply wether it should be a standard format or not.

If we want to implement a kind-of tile cache within the image to speed up loadign and operations, using a format like ar/jar/etc. would just be a hindrance, as users wouldn't be able to deal with the "files" within them anyways.

pcg@goof.com (2003-07-16 at 2223.29 +0200):

If we would design our own extremely simple wrapper format there would be no need to work with the compatibility mess existing formats are (all of them :). If we say that access by other tools than gimp is not important

Brainstorming, a dir named .xcf2 with the proposed contents inside? :]

GSR

On Wed, Jul 16, 2003 at 10:43:13PM +0200, Sven Neumann wrote:

designed as a GIMP-only format. People already use XCF in other apps simply because there is no reasonable format for layered images.

That is not true. MIFF was around for years, for example, was always able to store layers, animations, and an unlimited amount of meta information. There are probably others, too.

XCF isn't the first (nor the most "open") format for this task (and it wasn't intended as one).

there's seems to be a need for such a format which is why I would like to make access to it as simple as possible.

This, of course, is a good goal in it's own. However:

- maybe there is a need to have a gimp-specific file format, especially when you want to store the image data in a non-trivial way to speed up access.
- maybe there is a need to have a nicely defined exchange format for complex images (xml + data is nicer than the ad-hoc design of miff).

These are conflicting goals.

Realisticelly, however, it'll be a long time till gimp wants a special, optimized on-disk format.

(as for a format, miff is basically line-based header + image data, iterated, which is very nice... it'S not nifty XML, but the fatc that image data and metadata are grouped so near to each other is also nice...)

On 07/16/03 20:26, pcg@goof.com ( Marc) (A.) (Lehmann ) wrote:

- maybe there is a need to have a gimp-specific file format, especially when you want to store the image data in a non-trivial way to speed up access.
- maybe there is a need to have a nicely defined exchange format for complex images (xml + data is nicer than the ad-hoc design of miff).

If we really are in brainstorming mode here, following the suggestions listed above, how about a format something like the following, which is essentially just an XML preamble, followed by raw binary data:

My Example Image Christopher W. Curtis
A big, purple, E
2003/07/18
2003 FSF, All Rights Reserved





[...]

\000\000\000 [...] \000 until file offset=2000

The nice thing about this is that it should be fully parseable by XML parsers (up until the first NULL [1 is required, the rest are optional buffer space for those too lazy to do math]), it is completely human readable and very descriptive, highly extensible, and fully functional. You can add an "encoding=" or "compression=" option, specifying "none", "bzip2", or whatever, or even "format=" and "jpg" (at the layer mode, making the dimension, etc. tags optional; you'd still need data offset, etc.) The other nice thing is that you can just read the header, and load the rest of the layers on demand (for other 'preview' tools or what have you), and it can be used for other items as well. Instead of "type=image" you can have "type=brush", etc. Maybe even add an attribute like .

Chris

On Wed, Jul 16, 2003 at 06:19:27PM +0200, Sven Neumann wrote:

HOWEVER, this might be a good time to think about whether we'd prefer a compressed format that we can random-access de/compress on the fly instead of going via a huge (and with image data we can easily be talking HUGE) temporary intermediate file.

If we would compress the image data in the archive there would be no need for compression of the archive. Sure you could gain a few bytes by compressing the XML but since the already compressed image data doesn't compress well and in the worst case even gets larger, I don't see why anyone would want to compress the archive.

Agreed.

I have a more concrete suggestion (I'm still favoring ZIP files because they can somehow be handled without a GIMP which is useful, e.g. if they are broken): Make an uncompressed ZIP (or maybe compress the XML part only). The XML describes the structure and all attributes.

GIMP-Image.ZIP |
+- image.xml
|
+- comment.{txt,html,xml}
|
+- layer1.png (stored uncompressed)
|
+- layer1.1.png
|
+- layer2.png
...
|
+- image-thumbnail.png (optional)
|
+- whole-image.png (flattened image, optional)

IMO this is a sane approach. It has lots of benefits: - image contents are accessible without a GIMP - thumbnails can be extracted easily - a user can mess with the image
- images could be generated with external tools (just create the XML, add some images as layers, here you are) - it could be scanned for Java-viruses :->

Bye, Tino.

On Wed, Jul 16, 2003 at 11:40:15PM +0200, Guillermo S. Romero / Familia Romero wrote:

Brainstorming, a dir named .xcf2 with the proposed contents inside? :]

Would probably cause problems (ie. be cumbersome) copying, moving around on the web, etc. :-) We aren't quite at reiser4 yet. :-)

/* Steinar */

Sven Neumann wrote:

"Christopher W. Curtis" writes:

The nice thing about this is that it should be fully parseable by XML parsers (up until the first NULL [1 is required, the rest are optional

I don't think the format you proposed is valid XML. There might be XML

Rule #1 in brainstorming: don't criticise any idea, no matter how silly.

parsers that are able to read it but it violates the spec. If I am wrong here, please show me where the spec defines the special role of the NULL character.

I would reiterate what I said, but you quoted it. "fully parseable by XML parsers up until the first NULL". I'm not trying to jam image data into an XML format - simply prepending an XML header and using NULL as a separator. This means you can cat the file and know exactly what's there. Or you can open it in notepad (maybe make it NULL, ^Z for Windows people...) "file" will be able to readily identify the individual formats, and people could even use dd to extract the layers.

Is there a special reason you dislike the idea of using an archive instead of a single XML file? I thought we would have been past this point already...

I just don't see using another archive format as giving you anything. So say you use ZIP or JAR or TAR or AR: you still have to unpack (and possibly decompress) the thing just to find out what's in it. OTOH, any program that can open a file can read the XML header here, even if they don't parse it, it's still human readable. And this lets you do your fancy compression-based-on-data-type instead of generic-text-compression over each layer or the whole archive. If you want something fast, you can leave compression off and modify the data directly on disk without having to unpack/pack, as long as you stay within limits (ie: you can paint on a layer or move the layer around, but not add a new layer or change the bit-depth, and you just have to munmap() that section). This makes it easy to have specialized external tools that manipulate layer data without all the GIMP overhead, easily scriptable, etc...

This would also be a much simpler format, and I like simple.

If you want to talk about downsides, I can think of only one: the first data offset is not predictable. But I assert that that is irrelevant because you can specify it to be anywhere.

Chris

On Thu, Jul 17, 2003 at 12:06:12PM -0400, Christopher Curtis wrote:

Sven Neumann wrote:

"Christopher W. Curtis" writes:

The nice thing about this is that it should be fully parseable by XML parsers (up until the first NULL [1 is required, the rest are optional

I don't think the format you proposed is valid XML. There might be XML

Rule #1 in brainstorming: don't criticise any idea, no matter how silly.

parsers that are able to read it but it violates the spec. If I am wrong here, please show me where the spec defines the special role of the NULL character.

I would reiterate what I said, but you quoted it. "fully parseable by XML parsers up until the first NULL". I'm not trying to jam image data into an XML format - simply prepending an XML header and using NULL as a separator. This means you can cat the file and know exactly what's there. Or you can open it in notepad (maybe make it NULL, ^Z for Windows people...) "file" will be able to readily identify the individual formats, and people could even use dd to extract the layers.

Is there a special reason you dislike the idea of using an archive instead of a single XML file? I thought we would have been past this point already...

I just don't see using another archive format as giving you anything. So say you use ZIP or JAR or TAR or AR: you still have to unpack (and possibly decompress) the thing just to find out what's in it. OTOH, any program that can open a file can read the XML header here, even if they don't parse it, it's still human readable. And this lets you do your fancy compression-based-on-data-type instead of generic-text-compression over each layer or the whole archive. If you want something fast, you can leave compression off and modify the data directly on disk without having to unpack/pack, as long as you stay within limits (ie: you can paint on a layer or move the layer around, but not add a new layer or change the bit-depth, and you just have to munmap() that section). This makes it easy to have specialized external tools that manipulate layer data without all the GIMP overhead, easily scriptable, etc...

This would also be a much simpler format, and I like simple.

If you want to talk about downsides, I can think of only one: the first data offset is not predictable. But I assert that that is irrelevant because you can specify it to be anywhere.

Another downside: needing a special tool to manipulate it.

Consider the case of a corrupted xcf file. Maybe only 1 layer out of 20 is corrupted. With this proposal, a user needs either a special tool to extract out the good layers, or do a lot of work by hand to figure out how to use dd to grab it.

With say ar, the user can extract individual layers by some tag, referenced in the xml metadata. Then can edit the xml to stub out the broken layer, and repack it and have a valid xcf file. This could be the difference between losing 10% of the work vs. all of it.

So while a user can open a text file header in an editor, they are going to need a tool anyway to manipulate it effectively.

That's the advantage of using a standard format. Using standard tools to manipulate it. More likelihood of a machine having a tool installed, and less work for the GIMP team in maintaining special tools.

You're right about simplicity though, and ar is simpler than tar or zip/jar, which is why I prefer it. zip/jar is especially crappy since the file index is at the end, which means it's harder to recover from a partial file.

There's also no reason the GIMP can't have a native ar parser to load and map in a file directly without unpacking it somewhere on disk.

-Yosh

Manish Singh wrote:

On Thu, Jul 17, 2003 at 12:06:12PM -0400, Christopher Curtis wrote:

Rule #1 in brainstorming: don't criticise any idea, no matter how silly.

So much for rules...

Another downside: needing a special tool to manipulate it.

Well, now, I want to end this silliness right here. Here's what you need to manipulate files in my propsed format: (cat and dd), or (vi), or (type and wordpad). There are no tools more standard than these.

Consider the case of a corrupted xcf file. Maybe only 1 layer out of 20 is corrupted. With this proposal, a user needs either a special tool to extract out the good layers, or do a lot of work by hand to figure out how to use dd to grab it.

With this format, a tool like the gimp can say "layer X is corrupt" and the user would have to do something crazy like edit the file with "vi" and delete the `` ... '' section from the XML header.

With say ar, the user can extract individual layers by some tag, referenced in the xml metadata. Then can edit the xml to stub out the broken layer, and repack it and have a valid xcf file. This could be the difference between losing 10% of the work vs. all of it.

Ahh yes, this way all the user has to do is:

ar x file look for the right xml and edit it out or something based on some tag (I'm so glad this is so well thought out...) ar a file everythingelse1,2,3

Won't the Windows people be happy that they can do this instead of just opening a friggin binary-safe text editor.

So while a user can open a text file header in an editor, they are going to need a tool anyway to manipulate it effectively.

Yeah - it's called the same text editor.

That's the advantage of using a standard format. Using standard tools to manipulate it. More likelihood of a machine having a tool installed, and less work for the GIMP team in maintaining special tools.

... is this thing on?

You're right about simplicity though

-Yosh

Sven Neumann wrote:

Hi,

But you couldn't use any generic XML tools like validators or XSL transformations. If we go for XML we should really make sure that we make it proper XML, otherwise XML doesn't make sense at all. We could then as well go for a sexp syntax. Actually the latter would be a lot easier to implement since we could build on the established GimpConfig framework.

I'm not going to argue it. I disagree that an XML preamble is useless becuase you can't XSLT the entire file just as much as I'd disagree if you said that saving a file to disk is useless because it's not a full core image.

You don't have to unpack the whole archive and you can use existing widely-available tools to extract the parts you need.

Fair enough. When I save a file, I typically need the whole thing.

Chris

Ok ..

I want to apologise to everyone for overreacting. After a brief moment of reflection, a simple container has notable advantages over a single file, not to mention that a one of my assumptions didn't make sense.

Where is there documentation on the ar format? I can't seem to find any. I'd like to read up on it some before commenting further...

Thanks, Chris

sgunderson@bigfoot.com (2003-07-17 at 0959.32 +0200):

Brainstorming, a dir named .xcf2 with the proposed contents inside? :]

Would probably cause problems (ie. be cumbersome) copying, moving around on the web, etc. :-) We aren't quite at reiser4 yet. :-)

Well, I think people can copy dirs like they copy files (the small nitpick is shell users, and those should know how to solve it). And pack them for web too. Only problem I see is users complaining about what was a single files now being a single dir, and "oh, damn, I can look inside". In the end, both are project contaniers. No need of any future FS, NeXT did it long time ago, it is more about the upper level tools than the FS.

GSR

On Thu, Jul 17, 2003 at 01:31:42PM -0400, Christopher Curtis wrote:

Another downside: needing a special tool to manipulate it.

Well, now, I want to end this silliness right here. Here's what you need to manipulate files in my propsed format: (cat and dd), or (vi), or (type and wordpad). There are no tools more standard than these.

Using dd means the user has to tally the offsets and length by hand, versus ar just does it for you.

Consider the case of a corrupted xcf file. Maybe only 1 layer out of 20 is corrupted. With this proposal, a user needs either a special tool to extract out the good layers, or do a lot of work by hand to figure out how to use dd to grab it.

With this format, a tool like the gimp can say "layer X is corrupt" and the user would have to do something crazy like edit the file with "vi" and delete the `` ... '' section from the XML header.

Your original proposal had offsets, which would need to be recalculated when removing a layer.

With say ar, the user can extract individual layers by some tag, referenced in the xml metadata. Then can edit the xml to stub out the broken layer, and repack it and have a valid xcf file. This could be the difference between losing 10% of the work vs. all of it.

Ahh yes, this way all the user has to do is:

ar x file look for the right xml and edit it out or something based on some tag (I'm so glad this is so well thought out...) ar a file everythingelse1,2,3

Your proposal:

vi file
look for the right xml and edit it out or something based on some tag also need to change all the offsets of the layers in the file

Won't the Windows people be happy that they can do this instead of just opening a friggin binary-safe text editor.

Won't the Unix people be happy that recovery and manipulation is less convenient for them because Windows people are afraid to install ar?

Using ar makes it easy for scripts and the like to change stuff, which would be good if this format goes beyond the scope of gimp.

So while a user can open a text file header in an editor, they are going to need a tool anyway to manipulate it effectively.

Yeah - it's called the same text editor.

And do a lot of work by hand, as above. You did mention having tools to manipulate it yourself in your previous mail. What I'm saying is why create more maintainence work with making tools when someone else has it done for us already?

-Yosh

On Thu, Jul 17, 2003 at 09:45:51AM -0700, Manish Singh wrote:

Consider the case of a corrupted xcf file. Maybe only 1 layer out of 20 is corrupted. With this proposal, a user needs either a special tool to

(in this case, tar and zip would be preferable over ar, as ar tools are not well-versed at repairing/ignoring corruption, tar/zip tools often are better prepared).

which is why I prefer it. zip/jar is especially crappy since the file index is at the end, which means it's harder to recover from a partial file.

Actually, zip has all file headers twice: once before the file within the stream, and another time at the end.

On Thu, Jul 17, 2003 at 01:56:00PM -0400, Christopher Curtis wrote:

Where is there documentation on the ar format? I can't seem to find

man 5 ar
google ar file format

etc.. easy to find.. like tar and cpio (and to some extent zip), there is no "the" ar format. susv3 has to say this about the format:

If an archive file consists entirely of printable files, the entire archive file is printable. When ar creates an archive file, it creates administrative information in a format that is portable across all machines.

Yupp, that's all. Another good resource apart from manpages and google is the binutils source, which contains a lot of real-world info.