Wednesday, 1 June 2011

Parallel Frameworks

I have been looking into workflow systems that could be used to tie together tasks. The system needs to
  • be able to handle dependencies
  • work on a workstation/laptop and a cluster using some middleware (I guess Oracle Grid Engine)
  • remember the tasks it has already computed
  • workflows/scripts should be nestable and multiple instances should be runable
My initial thought was to use meandre and its scripting language zigzag. I am no longer convinced that this is the way forward: In particular I think it fails in that it does not remember what tasks it has already computed. I am also not sure how easy it is to setup on a stand-alone machine.

Anyway, Mike suggested to look at ganga which is a job creation framework written in python. It supports different backends such as local hosts and SGE. It maintains state between invocations. Interestingly, it supports job trees which would nicely map onto the problem at hand. So, I think ganga needs to be seriously considered. It is licensed under the GPL which might be problematic.

Another project which looks interesting is jug. Another python framework for tying together collections of tasks. Tasks are coordinated via files in a particular directory. This works over NFS and can therefore be used by SGE. Workers can be added dynamically. I wonder if they can also be removed.

Finally, I also came across the wonderful GNU parallel program. It works similarly to xargs but will execute commands in parallel depending on the number of available cores (it also works with remote machines). This is brilliant for generating animations.