I don't have any Groovy experience but I can say that the Docs on python.orgare great for learning Python. See their tutorial which includes best
practices for the language and other tid bits.

http://docs.python.org/2/tutorial/index.html

On Mon, Feb 24, 2014 at 12:05 AM, scl wrote:

Hi,

I'm currently considering using Python, Perl or a JVM based scripting language (Groovy etc.) for the buildjobs. The goal are platform independence, performance and easy maintainability.

Without wanting to start a silly flamewar about which tool is the greatest:
- Where are the experienced benefits and downsides of these languages for the desired task?
- Which readings would you recommend for Python and the JVM based language?

Thank you in advance,

Sven

On Mon, Feb 24, 2014 at 2:09 PM, scl wrote:

Forwarded with Sam Gleske's permission:

On 24.2.2014 at 3:21 PM Sam Gleske wrote:

I'm not familiar with the shorthand "sth". I can start running tests against www.gimp.org and begin to start ironing

out bugs which would need to be fixed in my program. I've run into bugs here and there running my crawler against large websites. Right now I've crawled and run against Drexel University IRT website using the crawler which is ~70k links. I'll do it for www.gimp.org too and then contribute a Job when I have it ready

(and for the other GIMP websites as well).

I don't know whether this could cause technical problems for www.gimp.org. The reason why I wrote 'something stable for production use' is that I want to avoid just them ;-)
Thus it would be good to hear our website administrators opinion to that.

Both the crawler and the tests following it support a --request-delay. Requests can be arbitrarily delayed by 0.3 seconds (smaller or larger if chosen) so as to mitigate impact of the website. I won't run any tests against the GIMP website until I get permission to do so. Also, currently my testing suite support saving crawl data in a JSON format. The advantage of this would be separating the crawl from the actual testing (if that's desired later on). Currently I mostly use it to debug my test suite because crawling can take a bit of time so saving that data allows me to avoid that step.

On Mon, Feb 24, 2014 at 2:09 PM, scl wrote:

Also, I'd like to note that

if you're not using git hooks now would be a good time to use them. Rather than using Jenkins to constantly poll the SCM simply implement a hook to launch a Jenkins job when a certain branch is committed to.

Currently we're polling SCM every few minutes. Adding a git hook to the repository would surely need involvement of the GNOME administrators. Is there a good reason to switch over from polling to hooking?

The advantage of hooks vs polling is that hooks are on demand. i.e. a job or process is only launched if there is an actual commit. For the sake of covering all topics I'll briefly describe polling. Polling simply keeps checking the state of the repository periodically (e.g. every 5 minutes). If the state hasn't changed (no commits) then it does nothing. If the state has changed (i.e. commits have been made) then it executes based on what was defined for the Job.

Hooks simply take less processing time. On a smaller scale they save energy because they're not constantly polling. It's also lowering the request load of the server serving the repository being polled. It's not a big deal if a server is serving a single repository or a few repositories. However, when you have e.g. 10000 repositories on a server the polling request load of each repository every few minutes starts to eat up the server's ability to serve. For smaller servers it's not the end of the world to have polling because it is easier to implement. I realize the GIMP project may be only a small part of that ecosystem being served but it is something worth considering.

SAM

On 24.02.2014 22:39, Sam Gleske wrote:

Both the crawler and the tests following it support a --request-delay. Requests can be arbitrarily delayed by 0.3 seconds (smaller or larger if chosen) so as to mitigate impact of the website

I guess the website can handle this, especially if the crawling only happens when...

The advantage of hooks vs polling is that hooks are on demand.

...a commit hook triggers it. With some suitable delay - right now this would better be days, until we finally overcome the laziness and set up a commit-triggered website build on cube again :)

Regards,
Michael
GPG: 96A8 B38A 728A 577D 724D 60E5 F855 53EC B36D 4CDD

On February 27, 2014 6:03:59 AM CET, scl wrote:

how about building the website on Jenkins, too? This would unfetter us from asking regularly in IRC whether 'somebody is around who can update the website', i.e. streamline our website maintenance process.

On IRC, I've asked for hints about how to get this back to the old commit triggered workflow on cube, but don't remember any replies. I'm pretty sure this can't be much work.

The build time itself is less than 30 seconds, abigger tipic is to discuss what we want on the site.

The newly generated website could then be fetched the same way like the we currently do with the API docs.

Manually, unless this has changed since the last time I updated it?

To still be able to review the commits we could agree to either use a separate branch (a review branch or feature branch)

This is why I recently asked mitch about resurrecting/creating at least one other virtual host for creative testing.

or commit patches only via Bugzilla,

We need more discussion about some things, at least. Getting a 'Help, what is this going to achieve?' feeling for a new tutorial like thr most recent one isn't optimal, especially if it makes us postpone a site update.

or we later use a review tool like [Gerrit] (in co-ordination with GNOME).

I think we should start to have semi-regular meetings on IRC, logged and those logs published.

Regards, 
Michael

On Thu, Feb 27, 2014 at 12:03 AM, scl wrote:

how about building the website on Jenkins, too?

Testing the website can be part of the build process in the Jenkins job (or calling another job which does the testing upon a successful build of the website job).

Also, when do IRC meetings happen and can anyone participate? What server and channel? http://www.gimp.org/irc

SAM

On 27.02.2014 20:10, scl wrote:

To still be able to review the commits we could agree to either use a separate branch (a review branch or feature branch)

This is why I recently asked mitch about resurrecting/creating at least one other virtual host for creative testing.

... for the website or more stuff?

Website. There are some changes, in particular to the site creation code itself, that I don't want to do to a live site.

Regards,
Michael
GPG: 96A8 B38A 728A 577D 724D 60E5 F855 53EC B36D 4CDD

On Thu, Feb 27, 2014 at 11:28 PM, scl wrote:

We're in #gimp at irc.gimp.org, usually in evening hours (UTC).

GIMP users' discussions are in #gimp-users and GEGL development discussions in #gegl.

Cool, when I do join I'll be user sag47. I say that because my user will be different than my email on the list.

SAM

On Sat, Mar 1, 2014 at 12:49 PM, scl wrote:On Mon, Feb 24, 2014 at 2:09 PM, scl wrote:

together with Andrea Veri I've set up the hook based build trigger. Currently it is active for testing purposes on the branches babl/hooktest and GEGL/master. If we encounter no trouble the other branches will follow in one week or two.

That's how it works: 1. You push a commit to babl/hooktest or GEGL/master in our public repository.
2. Git's Post-Receive hook notifies Jenkins about changes in the affected repository and branch.
3. Jenkins' Git Plugin will scan all the jobs which check out the specified URL and if they are configured with polling, they'll poll. 4. The polling will usually find something that's worth a build, so a build will be triggered.
5. You get an answer about success or failure of your commit via IRC and RSS (quicker than before)

For more information see: https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin http://kohsuke.org/2011/12/01/polling-must-die-triggering- jenkins-builds-from-a-git-hook/

Thanks to Sam Gleske for the advice and Andrea Veri for setting up the Git side.

+1 Awesome.

@gimp-dev, @gimp-web My crawler has a stable release now. I'd like to request permission to crawl www.gimp.org. It will be restricted to that domain and have request delays implemented to minimize impact.

SAM

On Sat, Mar 1, 2014 at 12:49 PM, scl wrote:On Mon, Feb 24, 2014 at 2:09 PM, scl wrote:

together with Andrea Veri I've set up the hook based build trigger. Currently it is active for testing purposes on the branches babl/hooktest and GEGL/master. If we encounter no trouble the other branches will follow in one week or two.

That's how it works: 1. You push a commit to babl/hooktest or GEGL/master in our public repository.
2. Git's Post-Receive hook notifies Jenkins about changes in the affected repository and branch.
3. Jenkins' Git Plugin will scan all the jobs which check out the specified URL and if they are configured with polling, they'll poll. 4. The polling will usually find something that's worth a build, so a build will be triggered.
5. You get an answer about success or failure of your commit via IRC and RSS (quicker than before)

For more information see: https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin http://kohsuke.org/2011/12/01/polling-must-die-triggering- jenkins-builds-from-a-git-hook/

Thanks to Sam Gleske for the advice and Andrea Veri for setting up the Git side.

+1 Awesome.

@gimp-dev, @gimp-web My crawler has a stable release now. I'd like to request permission to crawl www.gimp.org. It will be restricted to that domain and have request delays implemented to minimize impact.

SAM

On Sat, Mar 1, 2014 at 10:33 PM, Sam Gleske wrote:

@gimp-dev, @gimp-web
My crawler has a stable release now. I'd like to request permission to crawl www.gimp.org. It will be restricted to that domain and have request delays implemented to minimize impact.

FYI most of the requests will only be HTTP method HEAD - http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.4

It only requests headers and not the response body so it would actually be less of an impact than most crawlers when doing the actual link testing. For the crawling itself the response body is received so that it can find other links.

SAM

On Sat, Mar 1, 2014 at 10:33 PM, Sam Gleske wrote:

@gimp-dev, @gimp-web
My crawler has a stable release now. I'd like to request permission to crawl www.gimp.org. It will be restricted to that domain and have request delays implemented to minimize impact.

FYI most of the requests will only be HTTP method HEAD - http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.4

It only requests headers and not the response body so it would actually be less of an impact than most crawlers when doing the actual link testing. For the crawling itself the response body is received so that it can find other links.

SAM