Poster Abstract: HPC User Symposium 2012

Presenters: Ole Weidner, Shantenu Jha, Andre Merzky, Andre Luckow

An increasing number of computational scienctists require the concurrent use of
multiple distributed compute and storage resources in order to scale their
applications along both the data and compute axis.  However, inhomogeneous
access methods, software stacks and middleware services still present a major
barrier to cross-resource scalability and interoperability.

To address these fundamental and persistent challenges, we have developed
SAGA-Bliss and BigJob. SAGA-Bliss is a distributed computing package for
Python; BigJob is a pilot-job framework implemented using SAGA-Bliss. We show
how SAGA-Bliss and BigJob can be used in two different modes: (A) to implement
novel application architectures, abstractions and frameworks and (B) to
distribute and run large number of jobs and their associated data concurrently
across multiple HPC resources. 

SAGA-Bliss and BigJob are both production-grade tools for scalable science, as
well as a platform for research into applied distributed computing in
heterogeneous environments.

SAGA-Bliss (https://github.com/saga-project/bliss):

SAGA-Bliss is a light-weight Python package that implements parts of the OGF
GFD.90 SAGA interface specification and provides plug-ins for different
distributed middleware systems and services. SAGA-Bliss implements the most
commonly used features of GFD.90 based upon extensive use-case analysis,
usability and simple deployment in real-world heterogeneous distributed
computing environments and application scenarios.

Currently, SAGA-Bliss implements the job and the file management core APIs as
well as the resource management API extension.  SAGA-Bliss provides a plug-in
mechanism to connect with different middleware systems and back-ends.  The
latest version of SAGA-Bliss provides job and file management plug-ins for SSH,
SFTP, the HPC scheduling systems PBS, Torque and SGE (locally and remotely -
tunneled via SSH), as well as a resource management plug-in for Eucalyptus
(EC2) clouds. SAGA-Bliss can be used to access distributed cyberinfrastructure
like XSEDE, LONI and FutureGrid, as well as clouds and local clusters.

BigJob (https://github.com/saga-project/BigJob):

BigJob is a Python pilot-job framework implemented on top of SAGA.  Pilot-jobs
provide a flexible and dynamic execution model by decoupling workload
submission from resource assignment, which allows the distributed scale-out of
applications on multiple, potentially heterogeneous resources. Another
important advantage of this approach is that aggregated queue waiting times,
which often contribute significantly to the overall time-to-completion of
many-job workloads, can be severely reduced.

BigJob supports a wide range of application types and can be used over a broad
range of infrastructures, including XSEDE, OSG, FutureGrid, EGI and LONI.
BigJob supports multiple types and granularity of concurrency, e.g., large MPI
jobs as well as single node/core jobs equally well. BigJob has been used to
support Molecular Biophysics, Computational Chemistry, Statistical Physics and
Computer Science research, on XSEDE, FutureGrid, OSG and other distributed
infrastructures