On Mon, May 11, 2009 at 9:48 PM, Jerson Michael Perpetua wrote:

I'm back and I've changed. I don't want to be verbose anymore. So, without further ado... Wait, let me just say this: GeglBuffers are awesome! Now that that's out of the way, let me tell you exactly why GeglBuffers rock. :)

This is a good introduction to the architecture of GeglBuffers and shows a good understanding of what they currently are capable of. It could probably be integrated in the current documentation [1]

What follows is a blue-sky wishlist of features that should be possible to add to the architecture making it and GEGL a powerful base.

- Add GPU based means of setting and retrieving pixel data with a unified CPU/GPU storage (an important initial part of making GEGL GPU accelerated.)
- mmap the tile data in the backend instead of reading/writing (port away from GIO).
- re-enable shared access across processes using mmap (this can currently be made to work with patches to GIO). - Make the background/empty color configurable (by making the empty tile's contents configurable).
- Add abyss policies to be used by resamplers, allowing configuring the behavior of affine transforms and warping image operations. - Detect when a tile is set to be a solid block of the background color, and make the tiled shared with the empty tile. - Implement backends directly fetching tile data with random access from image file formats. (tiled EXR and TIFF), some formats with scanlines or groups of scanlines as tiles.
- Implement backends relaying tiles from OpenStreetMap or other http based tiled mipmap servers.
- Add a webserver that serves tiles, perhaps from a different processes reading the tiles shared by a different process. Allowing web hosted javascript front ends.
- Use webserver / web backend to implement multi-host read/writeable image buffers for collaborative painting and networked processing. - Detect uniformly colored and perhaps other types of tiles and store them more compactly, perhaps even just compressing tile contents and share duplicate tiles detected through hashing of their compressed data.

/Øyvind K.

1: http://gegl.org/api.html#GeglBuffer

- Add GPU based means of setting and retrieving pixel data with a

unified CPU/GPU storage (an important initial part of making GEGL GPU accelerated.)
- mmap the tile data in the backend instead of reading/writing (port away from GIO).
- re-enable shared access across processes using mmap (this can currently be made to work with patches to GIO).

That is only what I meant with the minimaps.

Utkarsh Shukla, Btech, Material Science and Metallurgy, IIT KANPUR.
Mob: 9936339580

I expect the following items from Øyvind's list to be tackled by Adam Turcotte and Eric Daoust (both GSoC students which I mentor). Consequently, if someone is planning to tackle these issues in the foreseable future, please let everyone know.

Øyvind Kolås writes: > ...
>
> What follows is a blue-sky wishlist of features that should be possible > to add to the architecture making it and GEGL a powerful base. >
> - Make the background/empty color configurable (by making the empty > tile's contents configurable).
> - Add abyss policies to be used by resamplers, allowing configuring the > behavior of affine transforms and warping image operations.

I'm not making promises on the following (given that they are not part of the original GSoC) but they are related to Adam and Eric GSoCs, but it would make sense to have them have a look at the following (I may give more programming help with the resampler parts to free them for what's below):

> - Detect when a tile is set to be a solid block of the background color, > and make the tiled shared with the empty tile. > - Detect uniformly colored and perhaps other types of tiles and store > them more compactly, perhaps even just compressing tile contents and > share duplicate tiles detected through hashing of their compressed data.

Nicolas Robidoux Universite Laurentienne

On Tue, May 12, 2009 at 3:50 PM, Nicolas Robidoux wrote:

I'm not making promises on the following (given that they are not part of the original GSoC) but they are related to Adam and Eric GSoCs, but it would make sense to have them have a look at the following (I may give more programming help with the resampler parts to free them for what's below):

 > - Detect when a tile is set to be a solid block of the background color,  >   and make the tiled shared with the empty tile.  > - Detect uniformly colored and perhaps other types of tiles and store  >   them more compactly, perhaps even just compressing tile contents and  >   share duplicate tiles detected through hashing of their compressed data.

I forgot to add one more item to my list of potential/desired enhancements of GeglBuffer and that is to make the extent (width and heights) of the buffer be updated on demand when tiles are changed, allowing automatically growing buffers that are "auto-clipped" to the bounding box of content that differs from the configured background color.

/Øyvind K.

Øyvind Kolås writes:
> I forgot to add one more item to my list of potential/desired > enhancements of GeglBuffer and that is to make the extent (width and > heights) of the buffer be updated on demand when tiles are changed, > allowing automatically growing buffers that are "auto-clipped" to the > bounding box of content that differs from the configured background > color.

Do I gather correctly that this is related to the possible solution you gave for

http://bugzilla.gnome.org/show_bug.cgi?id=549999

namely,

The bug is manifest because the different interpolation methods have different sampling footprints. The resulting image size is computed based on the size of the kernel, this is similar in behaviour to the gaussian blur which increases the size of the input image as well by smearing it out. The upper left corner of the bounding box is stored to be in negative coordinats so sampling from 0,0 would yield the upper left sample anyways. One way to work around this would be to crop the image to the original bounding box.

Not completely sure if this is a bug or not, I guess it depends on what types of behavior we want to support for the edges of transformed buffers.

Please let me know if I am completely off track, but there are some issues which I want to understand if possible (even though you may have explained it to me already) in order to guide Eric and Adam:

Question 1:

If my memory is good, samplers in GIMP use the "center" image size convention, which means that the centers of the boundary pixels "creep in" when downsampling, but "creep out" when upsampling, because what is kept fixed is the position of the corners of the areas implied by the boundary pixels, which are half a pixel width out from their centers.

The most common alternative is the "corner" convention, which anchors the positions of the centers of the boundary pixels.

I am omitting discussion the pros and cons for each. (Generally, "center" is better for downsampling, and "corner" is better for upsampling, so no single convention works best for all situations.) However, it is my opinion that "corner" convention is generally better.

Do you have a preferred "implied" convention for GEGL? Should it comply with what is generally the "implied" convention in GIMP?

This matters, because if the abyss policy is "no blending: image goes to the boundary ignoring the abyss, and the abyss is constant whatever (0, say, but one of your requested features is that this be configurable), we need to know where the boundary is.

Question 2:

Do you want resampled image sizes to be enlarged by the size of the footprint (more specifically, by the size of smallest rectangle which contains the footprint, since, for example, the footprint of nohalo is not a rectangle), or do you want the resampled image size to be cropped, in accordance with the image size convention (in the corner convention, the image ends right at the center of the boundary pixels, in the center convention, it ends one half pixel past it).

Prior to our conversations last week, I was sure that cropping should be done. however, now that I have been exposed to the "wonderfulness" of GEGL buffers, I have the impression that the buffer should be extended by this footprint so that the "blending" implicit in most abyss policies be applied not only inside the extent of the original image, but also outside. A consequence is that the bounding box will depend on the abyss policy, which is the case now. In other words, this would be a feature, not a bug.

Question 3:

Because the footprint of the samplers is often asymetrical (for example, for bicubic, the "anchor" is the second pixel from the left, out of the four positions which make up the footprint in the horizontal direction; this breaks left/right symmetry) the number of pixels by which the image is extended at the top and left is often not the same as the number by which it is extended at the bottom and right. Of course, symmetry is resolved in that "extra" ones are set to the background abyss colour (0).

Again, I previously saw this as a bug, which in principle could be avoided using some logic or cropping. However, at this point, I am enclined to treat it as a quirky feature which is a side effect of a sound design decision.

Is this agreed?

----------------------------------------------------------------------

I hope that I am not making my confusion too manifest here.

Cheers,

Nicolas Robidoux Universite Laurentienne

Christopher Montgomery writes:
> ...
> I suspect you are conflating center/corner with 'are we point sampling > or area sampling the pixel?'.
> ...

I am not.

Although area sampling generally works better with the center convention, and point sampling works better with the corner convention, one can use either with either convention. In my mind, these are related, but separate, issues. I have programmed and used area sampling methods with either convention, for example.

More about this later.

-----

(Thanks for joining the discussion.)

Nicolas Robidoux Laurentian University

hi,
Actually I am not a GSOC student but I was very much interested in writing some code or some patch for gimp. Please guide me in case you feel I can be of bit help.

On Wed, May 13, 2009 at 2:13 AM, Nicolas Robidoux < nrobidoux@cs.laurentian.ca> wrote:

Christopher Montgomery writes:
> ...
> I suspect you are conflating center/corner with 'are we point sampling > or area sampling the pixel?'.
> ...

I am not.

Although area sampling generally works better with the center convention, and point sampling works better with the corner convention, one can use either with either convention. In my mind, these are related, but separate, issues. I have programmed and used area sampling methods with either convention, for example.

More about this later.

-----

(Thanks for joining the discussion.)

Nicolas Robidoux Laurentian University

_______________________________________________ Gegl-developer mailing list
Gegl-developer@lists.XCF.Berkeley.EDU https://lists.XCF.Berkeley.EDU/mailman/listinfo/gegl-developer

On Tue, May 12, 2009 at 9:55 PM, utkarsh shukla wrote:

hi,
Actually I am not a GSOC student but I was very much interested in writing some code or some patch for gimp. Please guide me in case you feel I can be of bit help.

I am not a mentor in the summer of code but I am interested in GEGL becoming a powerful framework for both using and developing image processing algorithms. I welcome you to discuss and help implement and improve any part of GEGL. You've got knowledge of relevant technologies like OpenGL, GLSL as well as development in general and can be valuable in guidance on prototype code and ideas as that project gets off the ground.

/Øyvind K.

Let me try to be a bit more specific, one issue at a time.

----------------------------------------------------------------------

I agree that, at least at low order (which is all we're doing for now), abyss policies are independent of whether we are using the "center" or "corner" image conventions, and are also independent of whether we are implementing an exact area method (like most implementations of box filtering), or a point resampler (like standard bicubic).

However, Øyvind expressed a desire to have something implemented in or for the resamplers which cannot really be done with the usual abyss policy implementations.

Here is:

Associate a "default" constant colour with a buffer. What Øyvind would like (if I understand correctly) is that instead of sampling through the implied boundary using the resampler as if there was no boundary (filling in the missing footprint values using the abyss policy), the resampled values "drop dead" at the boundary and from then on are filled (filled, not blended) with the default colour. For example, the sampler could use the nearest neighbour abyss policy (a.k.a. clamp) for requested values up to the boundary line, and return the default colour past it.

Comment: Thinking about this some more, I am starting to think that it is a bad idea to implement this. (Later.)

Irregardless of whether this is a good or bad idea, the main point is:

Where should the "drop dead" transition occur:

Aligned with the (center of) the boundary pixels? Or half a pixel width out (at the boundary of the pixel area associated with boundary pixels)?

One of my worries is that if one when doing something else than resizing this will essentially behave like nearest neighbour (from the outside in), which will be ugly. So, >>I<< am actually starting to think that "drop dead" is not such a good idea.

But I'd love to hear opinions on this.

nicolas

Christopher Montgomery writes:
> ...
> Ah, you mean that the implementation is easier, not that the output > results differ?
> ...

Christopher:

Can we start from scratch? I'm afraid that we may take a long time explaining to each other exactly what we mean, with the most likely end result that we agree with each other on the essential (code-wise consequences). I started the ball rolling being vague, let me see if I can stop it.

It's not that I don't welcome your input (I actually sought it, and it is important to me that we align the GEGL resampler team with their GIMP counterparts), it is just that I am too tired/busy to get into a "I actually mean this, which really is the same although it is different from what you mean, is that right? Oh! you were talking about this limited situation but I was talking about the more general one and things are not the same but you can emulate the other if you are willing to blah blah blah" debate.

It's more important to me to be useful than to be right.

nicolas

2009/5/12 Øyvind Kolås :

- Add abyss policies to be used by resamplers, allowing configuring the  behavior of affine transforms and warping image operations.

An efficient way to do this is to remove the rule that buffers need to have fixed-size, non-overlapping tiles.

If you remove this restriction, then an image, rather than having to have it's own set of tiles, can instead just have a pointer to the tiles on another image. This lets you efficiently implement operations that keep most pixels unchanged but which make some small change to an edge (for example).

You can then implement abyss policies by writing an operation that enlarges an image in a certain way: for example by extending the edges out by 10 pixels and filling with a solid color. For most of the image, the operation can just use a tile from the original image at effectively no cost. It's only tiles which include part of an edge that need a copy and paint.

The resampler is then given the enlarged image to work on, and need only generate pixels where the interpolation stencil is completely filled. In effect, the abyss code runs per-edge-tile rather than per-image-pixel and the cost almost vanishes.

Anyway: you get a very flexible abyss system which has almost no memory or CPU cost. You can use the same system to implement zero-copy crop and paste operators, of course.

John

On Wed, May 13, 2009 at 2:06 AM, wrote:

2009/5/12 Øyvind Kolås :

- Add abyss policies to be used by resamplers, allowing configuring the  behavior of affine transforms and warping image operations.

An efficient way to do this is to remove the rule that buffers need to have fixed-size, non-overlapping tiles.

If you remove this restriction, then an image, rather than having to have it's own set of tiles, can instead just have a pointer to the tiles on another image. This lets you efficiently implement operations that keep most pixels unchanged but which make some small change to an edge (for example).

GeglBuffers already allows shared tiles, but they are not fully used in the locations where they should (gegl_buffer_copy, which is also used by for instance gegl_buffer_dup should be duplicating the tile data by hooking into the circular list of shared tiles.)

/Øyvind K.

Hello Utkarsh,

Maybe implementing multi-threading in GEGL would be a good, not yet taken, project?

(Note that I do not have enough of a broad/detailed view on GEGL to make this suggestion from a position of authority.)

Nicolas Robidoux Universite Laurentienne

Thanks.
Are you sure that Multi threading has not been taken. I feel ( I may surely be wrong) that the project of the adding of Gpu support should deal with that. Basically What I was planning for the GPU support project was to create a type of multiple parts or divide the GEGLBuffer tiles or kind of break up the buffer into different parts. and to store them into either CPU or GPU or over the network(where they can be accessed by HTTP) or diferent core of the CPU or RAM, and to consider them as a distinct identity an to apply the GEGL operatoins on them. There should be a memory map for it and so that you can know where the tile is residing and then you can apply the corresponding operation. This i believe could or infact be implemeted in various threads as parallelization is obvious here. ( I am sure I am wrong somewhere as I havent yet gone into the understanding of the GEGL (as I had no time as I was looking for something to get some money) ) (Please correct me where I am wrong)

I hope you wished to say about the multi threading at a higher level. (If I am again not wrong)
with different threads at the same time.I am not exactly sure of it. I also feel that multithreading at that higher level should been already implemented. ( I am not at all sure of the above line, I havent looked into the structure of GEGL much). It would be better if you help me out.

Utkarsh Shukla

On 5/13/09, Nicolas Robidoux wrote:

Hello Utkarsh,

Maybe implementing multi-threading in GEGL would be a good, not yet taken, project?

(Note that I do not have enough of a broad/detailed view on GEGL to make this suggestion from a position of authority.)

Nicolas Robidoux Universite Laurentienne

Dear Utkarsh:

I was talking about CPU multi-threading, that is, using multi-cores, e.g.

http://www.vips.ecs.soton.ac.uk/index.php?title=Benchmarks#Results_in_detail

And unfortunately, I am swamped, and consequently, I basically can't help.

I may be wrong, but I am pretty sure that GEGL does not multi-thread as of now.

Nicolas Robidoux

On Wed, May 13, 2009 at 5:35 PM, Nicolas Robidoux wrote:

Dear Utkarsh:

I was talking about CPU multi-threading, that is, using multi-cores, e.g.

http://www.vips.ecs.soton.ac.uk/index.php?title=Benchmarks#Results_in_detail

And unfortunately, I am swamped, and consequently, I basically can't help.

I may be wrong, but I am pretty sure that GEGL does not multi-thread as of now.

This is correct. GEGL has been designed to paralellize the rendering work-load and most of this architecture is in place. At an earlier stage when I initially tried this I had some locking issues and decided to discontinue that development at the time instead of destabilizing GEGL.

The first attempt was to split out the actual rendering from the graph manipulation, this would already be a great benefit for instance for GIMP since it means GEGL processing would happen in the background and be non-blocking.

After that is working, multiple GeglProcessors should be used concurrently (instead of just one working serially through chunks of the output image like it is now). Exchange of data should happen as extensions to the GeglBuffer architecture.

/Øyvind K.

Utkarsh:

Although clearly the two projects probably should not proceed without some mutual consultation, I feel that GPU and multi-core involve fairly seperate programming. For this reason, I would not be overly concerned that your fixing multi-core (say) would duplicate the GPU effort.

Clearly, some coordination would be desirable. However, in the nitty gritty, these are separate projects.

Nicolas Robidoux Laurentian University

So you are suggesting to parallelize the Gimp image operation means that multiple GEGL processes should run at the same time. So currently the GEGL has graph manipulation and then the rendering. I am not sure how you tried to split the rendering from graph manipulation. Dou you mean to say that curretnly Gimp first processes the graph and then renders the whole of the image afterwards. So if I get a clear overview then what I feel that this thing is much different than the GPU thing as here the parallelization is needed at much higher level and just very below the Gimp interface infact on the interface the things should be parallel threaded. Am I correct??

On Wed, May 13, 2009 at 10:10 PM, Øyvind Kolås wrote:

On Wed, May 13, 2009 at 5:35 PM, Nicolas Robidoux wrote:

Dear Utkarsh:

I was talking about CPU multi-threading, that is, using multi-cores, e.g.

http://www.vips.ecs.soton.ac.uk/index.php?title=Benchmarks#Results_in_detail

And unfortunately, I am swamped, and consequently, I basically can't help.

I may be wrong, but I am pretty sure that GEGL does not multi-thread as of now.

This is correct. GEGL has been designed to paralellize the rendering work-load and most of this architecture is in place. At an earlier stage when I initially tried this I had some locking issues and decided to discontinue that development at the time instead of destabilizing GEGL.

The first attempt was to split out the actual rendering from the graph manipulation, this would already be a great benefit for instance for GIMP since it means GEGL processing would happen in the background and be non-blocking.

After that is working, multiple GeglProcessors should be used concurrently (instead of just one working serially through chunks of the output image like it is now). Exchange of data should happen as extensions to the GeglBuffer architecture.

/Øyvind K. --
«The future is already here. It's just not very evenly distributed» -- William Gibson http://pippin.gimp.org/ http://ffii.org/

On Wed, May 13, 2009 at 6:08 PM, utkarsh shukla wrote:

So you are suggesting to parallelize the Gimp image operation means that multiple GEGL processes should run at the same time. So currently the GEGL has graph manipulation and then the rendering. I am not sure how you tried to split the rendering from graph manipulation.

By having a dedicated thread that does the actual processing work, and reporting that results have finished rendering in the main thread (which is with the application/GUI/graph manipulation code).

Dou you mean to say that
curretnly Gimp first processes the graph and then renders the whole of the image afterwards. So if I get a clear overview then what I feel that this thing is much different than the GPU thing as here the parallelization is needed at much higher level and just very below the Gimp interface infact on the interface the things should be parallel threaded. Am I correct??

Not quite sure I understand what you mean. The paralellization in GEGL should occur within the existing API and be splitting up the workload into paralell rendering tasks for different subregions of the output image.

GEGL is a demand driven caching data-flow based rendering system. First the graph is set up, nodes connected and properties set. Then either a sink node is processed or more normally like in a GIMP use case; a GeglProcessor is set up with a full region to be computed. GEGL doesn't compute all data at once but splits the requested region into rectangular chunks and processes these chunks in paralell. As chunks are computed GEGL signals will be emitted on the node reporting which areas have been computed. Behind the scenes a GeglProcessor could be doing it's real work in a thread, or even create multiple GeglProcessors that operate on separate parts of the output in paralell. Going beyond this, with buffers shared between processes and hosts it should be possible to extend the processing even further. After this GSOC I hope to have the means to have a unified memory architecture for GeglBuffers, at that stage GEGL could perhaps even do portions of the processing and|or compositing on the GPU (IO will be a concern in this case).

/Øyvind K.

There are two forms of parallelization that we can tackle above.

This i quite correct..

I could be wrong, but my intuition tells me that implementing the latter parallelization is difficult to tackle in my GSoC's timeframe.

Let us start the we will see

The reason why a certain level of communication is required to

implement both projects is that we need to agree on where tiles and rectangular pixel data are stored/cached (GPU memory vs. main memory) and when.

I am not sure about this. Your work woulbe like to access the memory map and find out where the tile is residing. Then Correspondingly you have to apply the operation if it is CPU or GPU.
We need to talk upon the structure of the memory map.I also (not quiet sure) was thinking that since the we are implementing the two level parallelization so why not justcreate the memory map at the lowest level. I mean if I put a rectangle into a GPU then I would write information like all the tiles are in GPU instead of the rectangle in GPU. they can be grouped but i feelIn the unified memory the tiles must be written. It would be better if someone looks into this logic.

A node whose associated operation publishes GPU-support will create GPU-based buffers only, CPU/RAM-based buffers otherwise.

I would say let them flow freely. you can track them by memory map

Cached GeglTiles in my current plans are always stored in GPU memory

for GPU-based buffers

Are You sure?

how and when do we store tiles in GPU memory then?

If Gpu memory is free we try to use it, but we will have to find a function to optimize the storage and data Transfer.

I dont know about the locking issue and would prefer if someone explains it bettter.

Here is a link to the

Nona-GPU: Image Remapping on the Graphics Processor

talk:

http://river-valley.tv/nona-gpu-image-remapping-on-the-graphics-processor/

Nicolas Robidoux Universite Laurentienne

J. Perpetua writes:

...
OpenGL can only process 1-4 tuples of vector data in the shader level, so I can't imagine how we can provide support for Babl formats with channels exceeding 4. Also, current generation GPUs don't support double-precision floating point data. Providing support for RGBA doubles, for example, is difficult.

My impression is that currently everything is done with float[4] RGBA pixels (converted from/to whatever by BABL) within GEGL operations.

Nicolas Robidoux Laurentian University

+-----+-----+-----+
| +-+-----+--+ |
| |R|RRRRR|RR| |
+---+-+-----+--+--+
| |R|RRRRR|RR| |
| |R|RRRRR|RR| |
+---+-+-----+--+--+
| |R|RRRRR|RR| |
| +-+-----+--+ |
+-----+-----+-----+

(the rectangle enclosing the Rs is the requested rectangle of pixels)

I was really very very dumb when i said about the tiles, actually I did not looked into the structure of GEGL properly. I just was proposing that the we should find out a way so that the map is not two level based. (I hope I am correct). I am very very sorry for not being able to spend much time in this as I am frantically searching for some place to work and some income. My work at hotel takes more time and I dont get much money but I have no option so I am just a bit busy looking for a job in coading, buts its been hard luck for me. :P

I hope to discuss upon the final structure to be implemented on both the level.