2. Installation

RADICAL-Pilot is a Python module which can be installed either via the python package installed pip or as part of the Anaconda distribution. RADICAL-Pilot must be installed in a virtual environment and cannot be installed as a system-wide Python package.

2.1. Requirements

RADICAL-Pilot requires: software packages, a properly configured ssh login, and a MongoDB server.

2.1.1. Software

RADICAL-Pilot requires the following packages:

  • Python >= 3.7

  • virtualenv >= 16

  • pip >= 20

  • setuptools >= 45.2.0

or

  • Anaconda Python >= 3.7

All other dependencies are installed automatically by RADICAL-Pilot installer.

2.1.2. SSH

RADICAL-Pilot requires to setup a password-less ssh login between the machine where your application is executed and the machine where the tasks of your application will be executed. For example, if you execute your application written with the RADICAL-Pilot API on a virtual machine hosted on a public or private cloud provider, and you want to execute your application’s tasks on an XSEDE HPC machine, you will have to configure a password-less (gsi)ssh login between the virtual and the HPC machine. (Here one of the many guides available about How to Setup Passwordless SSH Login).

2.1.3. MongoDB

The MongoDB server is used to store and retrieve operational data during the execution of an application which uses RADICAL-Pilot. The MongoDB server must be reachable from both, the host that runs the RADICAL-Pilot application and the target resource which runs the pilots.

Warning

If you want to run your application on a workstation that does not have a public IP address, but run your tasks on a remote HPC cluster, installing MongoDB on your workstation won’t work. Without a public IP and a port publicly reachable, the components of RADICAL-Pilot running on the HPC cluster will not be able to access the MongoDB server.

The MongoDB location is communicated to RADICAL-Pilot via the environment variable RADICAL_PILOT_DBURL. The value will have the form:

export RADICAL_PILOT_DBURL="mongodb://user:pass@host:port/dbname"

Many MongoDB instances are by default unsecured: we strongly recommend the usage of a secured MongoDB instance!

The dbname component in the database URL can be any valid MongoDB database name (i.e., it must not contain dots). RADICAL-Pilot will not create that DB on the fly and requires the DB to be setup prior to creating the session object. But RADICAL-Pilot will create collections in that DB on its own, named after RADICAL-Pilot session IDs.

Once you have identified a host that can serve as the new home for MongoDB, installation is straight forward. You can either install the MongoDB server package that is provided by most Linux distributions, or follow the installation instructions on the MongoDB website.

If you don’t have access to a suitable machine with a public IP, you can open a ticket with RADICAL lab and we will provide a testing MongoDB instance for you.

2.2. Installation

RADICAL-Pilot is distributed via PyPi and Conda-Forge. To install RADICAL-Pilot to a virtual environment do:

2.2.1. via PyPi

virtualenv --system-site-packages $HOME/ve
source $HOME/ve/bin/activate
pip install radical.pilot

2.2.2. via Conda-Forge

conda create -n ve -y python=3.7
conda activate ve
conda install radical.pilot -c conda-forge

For a quick sanity check, to make sure that the packages have been installed properly, run:

$ radical-pilot-version
1.0.2

The exact output will obviously depend on the exact version of RADICAL-Pilot which got installed.

Installation is complete!

2.3. Support

RADICAL-Pilot undergoes constant evolution, implementing new capabilities, supporting new resources and keeping up with the progressing of its dependencies. If you encounter any error, please do not hesitate to contact the RADICAL lab team by opening an issue.