BigJob Generic (with plugable communication & coordination backends)

Installation:

$ curl https://svn.cct.lsu.edu/repos/saga-projects/applications/bigjob/trunk/generic/bootstrap/bigjob-bootstrap.py -o bigjob-bootstrap.py
$ python bigjob-bootstrap.py $HOME/.bigjob/python/
$ . $HOME/.bigjob/python/bin/activate

see http://faust.cct.lsu.edu/trac/bigjob for further information

Currently supported backends:

A) SAGA Advert Service
B) REDIS, http://redis.io/
C) ZMQ (ZeroMQ), http://www.zeromq.org/

***************************************************************************
A) SAGA Advert Service

1) Utilise coordination url parameter of BigJob to utilize the Advert Service:

advert://advert.cct.lsu.edu:8080/ (Postgresql)
advert://localhost/ (SQLite)


***************************************************************************
B) REDIS

1.) Install and run Redis (http://redis.io/)

Mac Homebrew:
brew install redis redis-tools hiredis

Linux:
Download Redis http://redis.io/download
Currently only tested with Redis 2.2.12
$ tar -xzvf redis-2.2.12.tar.gz
$ cd redis-2.2.12 
$ make
$ make install (if root)


Start Redis (redis-server executable is located in src/ dir):

cd src
./redis-server


2) Utilise coordination URL parameter of BigJob to connect to Redis instance:

redis://cyder.cct.lsu.edu:6379

Standard settings are:
REDIS_SERVER="localhost"
REDIS_SERVER_PORT=6379



***************************************************************************
C) ZMQ

1.) Install ZeroMQ and pyzmq

http://www.zeromq.org/intro:get-the-software
http://zeromq.github.com/pyzmq/index.html

1.1 Install ZMQ:

$ wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
$ tar -xzvf zeromq-2.1.7.tar.gz
$ cd zeromq-2.1.7
$ ./configure --prefix=$HOME/sw/zeromq-2.1.7
$ make install


1.2 Activate BigJob virtualenv:

. $HOME/.bigjob/python/bin/activate

1.3 Install Python bindings:

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/sw/zeromq-2.1.7/lib/
$ export ZMQ_DIR=$HOME/sw/zeromq-2.1.9/
$ pip install pyzmq


2) Utilise coordination url parameter of BigJob to utilize ZMQ:

tcp://*

will start a local tcp server.