XGrid --- The Grid of the Future / Past

XGrid!

XGrid is a cluster computing technology developed by Apple Inc. and up until Mac Os 10.7 (Lion) It comes shipped and supported on every Mac. This default installation is what has helped XGrid become so popular, unfortuantely as of 10.8 (Mountain Lion) Apple have stopped supporting XGrid and is no longer in active development, but don't let that stop you (More to come on that soon)! XGrid is possibly one of the easiest grid technologies to set up, maintain and put to use within minutes. As said earlier, XGrid is available on all Macs prior to Mac OS 10.8 and only requires a Mac OS X Server to host the controller. Once the controller has been set up, there is a simple XML file that needs to be enabled on each node (Agent in XGrid terminology) and that is it. Once they have been enabled and the controller correctly and easily configured, then a user (client) can submit jobs to the grid via the GUI, command-line, third-party GUIs (GridStuffer is excellent) or via many of the API's available (Objective-C, Java and Python to name a few).

So, what do I need to get the ball rolling? 

Macs, and lots of them! Actually, that isn't quite true, the only thing you really need is a Mac OS Server with the XGrid controller set up on it, the XGrid Agents (nodes) can actually be Linux boxes also (although this isn't technically supported, more of a handy hack). There are heaps of guides on the internet for all things XGrid related so never fare, even the newest of IT techs can get it working in no time. There are a few pitfalls to look out for however (see below). 

Some Quirks

Each core is a separate machine in the eyes of XGrid. This can be handy as precious cores arent going to waste just for a single threaded process, can be a pain when you forget to take RAM requirements into account for your job! Unfortunately XGrid doesn't seem to want walltimes to help limit issues which can be annoying!

Apple and security huh (I feel so bad for you iPhone users out there) and no, XGrid is no different. XGrid has a nice little security feature called sandboxing which pretty much traps XGrid into the /tmp directory. And no, you can't get out of it (not easily at least, it is possible to hack the sandboxing feature though). And as /tmp has plenty of quirks of its own this can prove to be a problem, such as using XGrid and Galaxy mounted over NFS in which Galaxy hardcodes all paths to where it is mounted on the NFS share (and theres no way thats going in /tmp!). But thats neither here nor there.

One major quirk of XGrid is the way it deals with absolute and relative paths when specifying jobs to run. Absolute paths will cause it to read the data/program off the agent (most of the time) and relative paths will cause the files / programs to be staged in (most of the time..). There are a few situations in which this isn't the case and the user will need to be aware of this, again, Googling usually provides a reasonable explanation of this behaviour.

Grid congestion (Aucklanders know all about that) can prove to be an issue when stagine in large issues. XGrid unfortunately doesn't handle staging in files ~1GB in size and can cause data to be lost or jobs to fail. For large data jobs such as those we reccommend using an NFS share (if you can get it to work nicely with /tmp!). 

Isn't it dead?

No! Well, not without a fight by the looks of it. Universities and institutions have a lot of Macs (sorry Bill, but its true) so its a shame to see your nice new XGrid just dissapear in a puff of smoke when you finally get round to installing Mountain Lion, but never fear, there is a solution! It turns out that in a fit of genius, the XGrid developers made the binaries for XGrid completely self-contained, this means that it is possible to just copy the binaries from your 10.7 to your 10.8 machines and with a bit of luck keep enjoying the Grid!  

Should we turn it on?

Don't see why not! For embarassingly parallel problems Xgrid works brilliantly, it lets you submit jobs asynchronously, do batch jobs and will queue them when all the cores at maxed out. It won't do fancy priority queuing or anything like that, it is more or less FIFO (First in, first out) but for simple jobs, there is no greater friend!

Google?

Google it! There is heaps out there on XGrid for all your gridding needs. 

Submitted by ehills on

Comments

Any other easy-to-use systems?

I've always wondered why XGrid is so easy to use, but other (open source) tools seem really clunky. I love the idea of just plugging lots of devices together and telling them to get along!

From what I've seen of BOINC, you need your own server infrastructure for command and control.  Condor seems to be quite prominent too. But the configuration is pretty daunting.