Ok.. so I am working on a large image and want to see if I can increase performance speed any. I knew things would be slow, but was hoping I could get a bit better than what I currently have. The image will be a map(as in fantasy world map) I want to print(will scale down for web version to .jpeg), 36x24 inches @300 DPI, so 10800x7200 resolution(ie, poster print size).

I have a fresh install of Linux Mint 11 on a laptop which is about two years old. I don't remember the full PC specs off the top of my head though but is was a mid-high end range gaming Laptop(so probably in the top 70%-80% of "best" available laptop hardware specs at the time of purchase).

What I do know off the top of my head:

Machine: multiple partitions
20GB Linux swap
Mint installed to single partition 190GB out of the entire HD's 500GB(both /home(location of the xcf file) and /tmp are on the same partition. 6GB RAM (8GB max)
multiple USB 2 ports
1 eSATA port
Wacom bamboo tablet
video card is (I BELIEVE) a GTX 200M

Gimp: built from source (git) as of umm... Friday night(or so) Currently, I have my tile-cache size set to 5GB

I typically have 3-6 chrome browser windows open and perhaps 1-2 open directory folders, but other than that, there are few applications running other than those(occasionally Thunderbird).

Currently, the file opens up around 4.5 GB in memory, with spikes up to 10GB so far that I have seen. I have about 15 layers so far, with about half of those using a layer mask. Most of the layers are transparent at this point, with a few being full color with layer masks to define geological features(grassy plains, desert, etc).

I have yet to add the additional layers needed to represent mountains and forests(at least 4 layers each, line-work, color, highlights, and lowlights, so min 8 layers for that) as well as several layers for some type of desert texture, labels(4-5 layers), and likely a compass rose and a Cartouche of some type(likely 4-6 layers for shape/color as well as 2-4 text sections). So all together, this will likely encompass around 35-40 layers when completed(if I can get that far!!!).

So, are there any suggestions you guys might make? As slow as it is currently, I don't know if I will even be able to get anywhere near to the number of layers I expect to need. Should I just give up on such a large image and reduce the scale or is there any additional tweaks I can make? Will adding a eSATA or USB2 drive to hold the the /tmp help at all? I this was a desktop, I could easily just slap another Harddrive(or two) in and have different read/writes working in parallel(ie, /tmp and swap on a separate physical drive), but I don't have that luxury with a Laptop... Would adding an additional 2GB make a noticeable(as in very noticeable) difference?

I appreciate any suggestions you guys might have. Please let me know if there is any additional information(ie, L2 Cache, Processor, etc) and I can get that info later tonight.... though of course those are things I cannot change...

Joe

On 11/15/2011 04:21 PM, jfrazierjr@nc.rr.com wrote:

Ok.. so I am working on a large image and want to see if I can increase performance speed any. I knew things would be slow, but was hoping I could get a bit better than what I currently have. The image will be a map(as in fantasy world map) I want to print(will scale down for web version to .jpeg), 36x24 inches @300 DPI, so 10800x7200 resolution(ie, poster print size).

I have a fresh install of Linux Mint 11 on a laptop which is about two years old. I don't remember the full PC specs off the top of my head though but is was a mid-high end range gaming Laptop(so probably in the top 70%-80% of "best" available laptop hardware specs at the time of purchase).

What I do know off the top of my head:

Machine: multiple partitions
20GB Linux swap
Mint installed to single partition 190GB out of the entire HD's 500GB(both /home(location of the xcf file) and /tmp are on the same partition. 6GB RAM (8GB max)
multiple USB 2 ports
1 eSATA port
Wacom bamboo tablet
video card is (I BELIEVE) a GTX 200M

Gimp: built from source (git) as of umm... Friday night(or so) Currently, I have my tile-cache size set to 5GB

I typically have 3-6 chrome browser windows open and perhaps 1-2 open directory folders, but other than that, there are few applications running other than those(occasionally Thunderbird).

Currently, the file opens up around 4.5 GB in memory, with spikes up to 10GB so far that I have seen. I have about 15 layers so far, with about half of those using a layer mask. Most of the layers are transparent at this point, with a few being full color with layer masks to define geological features(grassy plains, desert, etc).

I have yet to add the additional layers needed to represent mountains and forests(at least 4 layers each, line-work, color, highlights, and lowlights, so min 8 layers for that) as well as several layers for some type of desert texture, labels(4-5 layers), and likely a compass rose and a Cartouche of some type(likely 4-6 layers for shape/color as well as 2-4 text sections). So all together, this will likely encompass around 35-40 layers when completed(if I can get that far!!!).

So, are there any suggestions you guys might make? As slow as it is currently, I don't know if I will even be able to get anywhere near to the number of layers I expect to need. Should I just give up on such a large image and reduce the scale or is there any additional tweaks I can make? Will adding a eSATA or USB2 drive to hold the the /tmp help at all? I this was a desktop, I could easily just slap another Harddrive(or two) in and have different read/writes working in parallel(ie, /tmp and swap on a separate physical drive), but I don't have that luxury with a Laptop... Would adding an additional 2GB make a noticeable(as in very noticeable) difference?

I appreciate any suggestions you guys might have. Please let me know if there is any additional information(ie, L2 Cache, Processor, etc) and I can get that info later tonight.... though of course those are things I cannot change...

Joe

Hm...,
this is going to be tough!
So much layers, especially with masks and with that resolution is way beyond the specs of even a new mid-level desktop computer, as far as I can tell.

In any case I will suggest that you utilize the full RAM capacity of your motherboard. Then the machine will swap 2GB less, which will make some difference at least until you reach certain level.

I think USB2 will be horribly slow, so no point of adding such drive. And, I hope that somebody will correct me if I am wrong, but I don't think that GIMP uses /tmp that much if at all. What you really need is more RAM and a fast drive for the swap partition.

Apart from this you could split the file in several separate files, that will hold several logically grouped layers each (I presume that you don't have to work simultaneously on all of the layers). This way you can speed things drastically. In each of these files you could include one (or few layers) that represent all of the other layers (from the other files), but merged, with layer masks applied, etc. - just as a preview of the other parts of the whole image... i hope you get the idea despite my explanation. This way you will work with only say 5 - 10 layers + one or few 'preview' layers, that represent the rest 30 layers.

Good luck, Petar Kostov

I think USB2 will be horribly slow, so no point of adding such drive. And, I hope that somebody will correct me if I am wrong, but I don't think that GIMP uses /tmp that much if at all. What you really need is more RAM and a fast drive for the swap partition.

Throwing out another thought - can gimp temp and gimp swap be set up to use a ram based filesystem?

-Rob A>

Hi,
rob.antonishen@gmail.com (2011-11-15 at 1105.59 -0500):

I think USB2 will be horribly slow, so no point of adding such drive. And, I hope that somebody will correct me if I am wrong, but I don't think that GIMP uses /tmp that much if at all. What you really need is more RAM and a fast drive for the swap partition.

eSATA would do. If some kind of RAID0 can be done, even better. Multiple swap partitions with equal priority should balance the operations, and for data partitions he would have to use MD, LVM or similar systems.

Throwing out another thought - can gimp temp and gimp swap be set up to use a ram based filesystem?

Yes, look in preferences, you can configure where you want the files to be written; so point to a RAM based FS there. Another trick would be using symlinks to redirect the directory structure (useful for apps that do not allow configuration). Linux tmpfs is backed by swap, so it could hit the disks anyway.

A different matter would be if tmpfs would compress contents (not supported AFAIK), using RAM more efficiently; otherwise it would just copy data in memory and not improve anything. So it would be worth looking into the rather new zram system, maybe it can be used instead of tmpfs for this.

GSR

On 11/15/2011 06:05 PM, Rob Antonishen wrote:

I think USB2 will be horribly slow, so no point of adding such drive. And, I hope that somebody will correct me if I am wrong, but I don't think that GIMP uses /tmp that much if at all. What you really need is more RAM and a fast drive for the swap partition.

Throwing out another thought - can gimp temp and gimp swap be set up to use a ram based filesystem?

-Rob A>

Yes, through using /dev/shm, but the OP doesn't have enough RAM to do this.

On Tue, Nov 15, 2011 at 8:21 AM, wrote:

The image will be a map(as in fantasy world map) I want to print(will scale down for web version to .jpeg), 36x24 inches @300 DPI, so 10800x7200 resolution(ie, poster print size).

Another thought - if you can divide the image up into quadrants for some of the multi-layer work, then combine them later into one image that may help some (but of course that may be a pain as well).

But yes - get as much RAM installed as your machine can handle.

Chris

On 2011-11-15 17:43, GSR - FR wrote:

Yes, look in preferences, you can configure where you want the files to be written; so point to a RAM based FS there. Another trick would be using symlinks to redirect the directory structure (useful for apps that do not allow configuration). Linux tmpfs is backed by swap, so it could hit the disks anyway.

BTW, why does GIMP have this swap folder? Why doesn't it just allocate all memory it needs and let the OS do the swapping?

Is it to support multi-GB image data on a 32 bit system? Then it would be unnecessary on a 64 bit system.

Hi,
mikael@staldal.nu (2011-11-15 at 2032.43 +0100):

On 2011-11-15 17:43, GSR - FR wrote:

Yes, look in preferences, you can configure where you want the files to be written; so point to a RAM based FS there. Another trick would be using symlinks to redirect the directory structure (useful for apps that do not allow configuration). Linux tmpfs is backed by swap, so it could hit the disks anyway.

BTW, why does GIMP have this swap folder? Why doesn't it just allocate all memory it needs and let the OS do the swapping?

OS swap is, normally, fixed size, and in some cases even zero (at OS level it is a cushion, not a fix, and sometimes really bad if the system becomes unresponsive). With own swapping files, it can swap more, without affecting other processes, until the data partition fills up.

GSR

How about installing an SSD?
Best regards
Chris

Hm...,
this is going to be tough!
So much layers, especially with masks and with that resolution is way beyond the specs of even a new mid-level desktop computer, as far as I can tell.

In any case I will suggest that you utilize the full RAM capacity of your motherboard. Then the machine will swap 2GB less, which will make some difference at least until you reach certain level.

I think USB2 will be horribly slow, so no point of adding such drive. And, I hope that somebody will correct me if I am wrong, but I don't think that GIMP uses /tmp that much if at all. What you really need is more RAM and a fast drive for the swap partition.

Apart from this you could split the file in several separate files, that will hold several logically grouped layers each (I presume that you don't have to work simultaneously on all of the layers). This way you can speed things drastically. In each of these files you could include one (or few layers) that represent all of the other layers (from the other files), but merged, with layer masks applied, etc. - just as a preview of the other parts of the whole image... i hope you get the idea despite my explanation. This way you will work with only say 5 - 10 layers + one or few 'preview' layers, that represent the rest 30 layers.

Thanks to everyone who made suggestions!! A quick update, I found out that my machine "IS" maxed out on memory(at 6GB), so no joy spending $50... bummer...

Anyway, I have seen several suggestions that I can try(can't buy new computer at at this time):

1) eSATA drive(ie, external) 2) Solid State Drive internal(likely not a cost effective solution to replace my 500GB internal HD) 3) Groups of content xcf files saved and flattened and then merged into a final image 4) Other?

I have already started on 3 by breaking out my background and "frame" into a separate file, but even then, with only 10 layers, running something like the Layer Effects Python filter 30+ minutes to complete. Still, it's something that something that "can" be done with no cost if it is very annoying to have to do so.

Now, on to the technical details.. If I were to purchase a solid state drive(say 50GB for so for roughly $150 with eSATA enclosure added), how would I configure Linux and/or GIMP to get the most efficiency out of the set up? While I have been using Linux for the past 2 years or so, I am by no means an expert. Note that while I may not know tons of shell commands, I am comfortable with using the shell and understanding output of shell commands (used command prompt in Windows 95-7 FAR more than Windows Explorer for navigation, delete, directory creation, etc), including the more common stuff such as sudo, etc.

As a mid term solution(3-6 months), I will likely save up to replace my desktop's MB, CPU and RAM, followed by a long term(12-18 months) plan of replacing my Laptop with something that will support 12-16 GB RAM.

Again, I really appreciate all of the help and any additional advice. I am also looking into Inkscape to see if I can learn enough to possibly offload at least some of this to a Vector editor, but my brain just keeps thinking in GIMP terms, so the transition is a bit difficult for me.

Joe

On Wed, Nov 16, 2011 at 10:44 AM, wrote:

2) Solid State Drive internal(likely not a cost effective solution to replace my 500GB internal HD)

I could be wrong - but this one you'd likely add in addition to your main drive and mount/use it as a RAM filesystem. Basically a faster swap drive, or a "scratch disk" dedicated to GIMP.

Chris

Hi,
jfrazierjr@nc.rr.com (2011-11-16 at 1144.40 -0500):

4) Other?

You forgot: reduce tile cache (so you have free RAM) and use zram (Linux 2.6.37 or newer) to create a compressed memory based block device, to be used as partition where gimp writes own cache. The problem is the partition will be fixed size and not huge, so it could fill up and cause a mess.

Or do not reduce the tile cache, but still use zram as high priority swap device so gimp use lots of "memory": part real, part swapped to the compressed zone, and as last resort, part disk based. This way lets you increase swap size manually, as needed, by adding new partitions or files to cope with overflow of memory + zram-swap + current disk-swap. The important detail is setting the priorities (and maybe the kernel swap settings) right so data is first put in the fastest place avaliable.

BTW, I checked your original mail again. As much as possible, crop layers to the real contents (leave some work margin and you can always add more later). This is one reason why having user controllable and visible layer edges is useful: you can discard zones as you need, and no matter what you do, they will never pop back (no silly "paint by error, notice an hour later so no undo possible, delete with eraser tool to not destroy smooth gradients around" and have crap wasting resources).

So the compass rose, all the labels or separate mountains or forest zones that can be described as non overlapping rectangles (per continent, for example) can be small layers.

GSR

On 11/18/2011 08:13 AM, paynekj wrote:

Script-fu question:
Is there an equivalent to DIR-SEPARATOR for search paths? i.e. the return from (gimp-gimprc-query "script-fu-path") As the path separator is different on Windows and Linux (and who knows what on Mac)

I don't think so. The gimprc file contains:

# This path will be searched for scripts when the Script-Fu plug-in is run.
# This is a colon-separated list of folders to search. #
# (script-fu-path "${gimp_dir}/scripts:${gimp_data_dir}/scripts")

So this means that Gimp will use a colon on all platforms. That makes sense, since this is a Gimp-only setting.

Kevin

As no-one gave me an answer, and this thread got rudely hijacked, I've raised a bugzilla: https://bugzilla.gnome.org/show_bug.cgi?id=664316

On Fri, 2011-11-18 at 08:37 +0100, Ofnuts wrote:

On 11/18/2011 08:13 AM, paynekj wrote:

Script-fu question:
Is there an equivalent to DIR-SEPARATOR for search paths? i.e. the return from (gimp-gimprc-query "script-fu-path") As the path separator is different on Windows and Linux (and who knows what on Mac)

I don't think so. The gimprc file contains:

# This path will be searched for scripts when the Script-Fu plug-in is run.
# This is a colon-separated list of folders to search. #
# (script-fu-path "${gimp_dir}/scripts:${gimp_data_dir}/scripts")

So this means that Gimp will use a colon on all platforms. That makes sense, since this is a Gimp-only setting.

No, this means that you are looking at the file on a UNIX platform.

I have added SEARCHPATH-SEPARATOR as constant to script-fu now.

Regards, --mitch

Redux:

Ok, so I have ordered a new MB and 16GB ram(MB supports up to 32GB) for my desktop machine and am waiting for it to arrive. However, I don't currently have a processor, and would like to get an idea of which would be better, specifically for GIMP's use:

More Cores, Slower clock speed vs
Fewer Core's, higher Clock speed

Given that GIMP seems to allow setting your number of CPUs it can utilize, my initial thought here would be the more cores, the better. Also, since I will likely be running at least 1-2 other programs at the same time, I expect this would likely be the best approach. At this point, I don't have any plan's to add additional components in the near future, but long term once prices drop, I will likely buy 1 or more SSD's.

Any thoughts on the CPU debate?