[email protected]
Mar 11, 2021
1 Logging in 3
2 PEARL Storage Areas 5
3 The Lmod Modules System 7
4 Available Software 9
5 The SLURM scheduler 11
6 Using Containers 15
7 MPI and Infiniband 19
8 Compiling Your Code 21
9 Support 23
10 PEARL and Data Protection 25
11 SSH keys 27
12 PEARL Service User Documentation 31
For instructions on how to generate an SSH key and load it into an agent please see the section onSSH keys.
2 Getting started
Logging in
The primary login node is ui.pearl.scd.stfc.ac.uk. From here you can access your home directory, check the status of the cluster and submit and monitor your jobs.
Note: Please do not run computationally-intensive tasks on the login node as this can affect other users.
You are also able to log on to the dedicated data transfer node dtn.pearl.scd.stfc.ac.uk. This machine is solely for enabling the ingress and egress of data to and from the cluster and for this reason you are not able to submit jobs from here.
4 Chapter 1. Logging in
PEARL Storage Areas
PEARL’s primary storage areas are hosted on two Talyn storage appliances. These provide your home directory and and the shared work area. Given the nature of the expected workloads quotas are not enforced, but you are expected to manage your data in a responsible way so please give due consideration to other users.
The storage areas are available from all nodes that you can log into directly and they are also accessible from the DGX2 nodes.
The work area contains group-specific directories that you have access to depending on your organisational affiliation.
You are able to create additional directories here to suit your requirements.
There is also a universally-accessible directory where cluster-wide resources such as container images are stored for your use.
When you log in some environment variables are set to enable you to access the storage areas easily and without having to remember the paths.
Warning: Note that for operational reasons only the home directories are backed up and this area will be regularly assessed to ensure that user’s data is within reasonable limits. Large data sets should be stored in the workarea while only data that cannot be easily replaced, such as results, should be stored in home. Data will never be deleted without prior consultation with the data owner.
Below is a table summarising the PEARL storage areas:
Path Variable Description Backups
/mnt/beegfs/home/<username> $HOME Your home directory Nightly /mnt/beegfs/work/ati $WORK ATI shared work area No /mnt/beegfs/work/stfc $WORK STFC shared work area No /mnt/beegfs/images $IMAGES Shared image repository No
6 Chapter 2. PEARL Storage Areas
The Lmod Modules System
PEARL uses theLmodenvironment modules system to make pre-compiled software available to you.
Note: The following module paths are automatically added to your environment at logon. These give you access to optimised, pre-compiled software including the system’s default Singularity build.
/apps/modulefiles/eb/all:/apps/modulefiles/core
To view all the software that is available to you:
$ module avail
To list the currently loaded modules:
$ module list
To load software provided by a module:
$ module load [module_name]
To unload a module:
$ module unload [module_name]
To substitute one module for another:
$ module swap [module1_name] [module2_name]
To unload all currently loaded modules and revert to the system default state:
$ module restore
Lmod also has a tool called ml which provides the same functionality as the full module command but in a shorter, more convenient format. For instance, listing the currently loaded modules becomes simply:
$ ml
Showing all available modules can be done with:
$ ml av
You can load a module using:
$ ml [module_name]
You can unload a module with:
$ ml -[module_name]
See also:
Please see theLmod User Guidefor further information.
8 Chapter 3. The Lmod Modules System
Available Software
4.1 Software repositories
PEARL’s applications and libraries are generally provided using theEasybuildsoftware build and installation frame- work. This widely-used tool has been developed specifically to build and manage software on HPC systems. Packages provided this way do not rely on system software as all dependencies are compiled from source as required.
Note: Software which has been installed with Easybuild can be distinguished by this module path:
/apps/modulefiles/eb/all
Easybuild uses the concept of toolchains. A toolchain may just consist of a specific version of a compiler but usually adds one or more common HPC libraries such as MPI or numerical routines. This approach provides consistency and avoids many of the problems that can occur when using shared libraries.
Not all software is suitable for distribution using Easybuild and some packages are compiled from source manually.
An example of this is the Singularity containers application and this is reflected in the module path:
/apps/modulefiles/core/singularity/3.4.0-1
Software may also be provided from the Ubuntu repository but this generally contains older versions of packages and should be avoided unless absolutely necessary.
4.2 Location of binaries
The location of the binaries differs depending on what machine you are logged into. This has been done to ensure that the software is optimised for the DGX-2’s more recent processor generation, but can also be accessed from the head node. In normal use, this should be completely transparent to you and is only mentioned in this documentation for completeness. You may however see either of the following paths:
/mnt/beegfs/apps/eb/arch/generic/software/
/mnt/beegfs/apps/eb/arch/skylake/software/
Software will be added over time but a minimal software stack was made available at service launch. This consists of a recent version of the Singularity containers application, git, Python (plus several common modules) and recent versions of the GCC and Intel compilers.
If you would like a particular software package or library to be installed please make your request through the PEARL service desk.
10 Chapter 4. Available Software
The SLURM scheduler
The cluster uses theSLURMworkload manager for resource allocation and job management.
SLURM uses partitions which are analogous to queues in other scheduling systems. There is a single partition called pearlto which all jobs are submitted. Resources are assigned to an account and your unique username is then asso- ciated with one (or more) of these accounts depending on your organisational association. It is this association that provides you with access to a set of resources on the cluster.
5.1 Getting cluster information and health
Two of the most useful SLURM commands that are available to you are sinfo and squeue.
The sinfo command will show you the general health of the cluster, including the current state of the nodes and general information about them such as number of CPUs and how much memory they have.
This example invokes the command using the optional -N (node-centric view) and -l (long view):
$ sinfo -N -l
Tue Nov 19 11:55:23 2019
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE
˓→REASON
mn1 1 pearl* mixed 96 2:24:2 154680 0 1 (null)
˓→none
mn2 1 pearl* idle 96 2:24:2 154680 0 1 (null)
˓→none
The squeue command will show information on currently running jobs and jobs that are queued while they are waiting for resources to become available:
$ squeue -l
Tue Nov 19 13:20:50 2019
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES
˓→NODELIST(REASON)
1257 pearl bash pearl014 RUNNING 13:30:37 2-00:00:00 1 mn1 (continues on next page)
(continued from previous page) 1251 pearl bash pearl014 RUNNING 1-15:18:05 2-00:00:00 1 mn1 1264 pearl bash pearl014 RUNNING 1:00:13 2-00:00:00 1 mn1 1218 pearl run_v3.s pearl024 RUNNING 3-22:29:29 5-23:00:00 1 mn1
See also:
Please see thesinfoandsqueueman pages for further information
5.2 Requesting resources
SLURM has three commands for requesting resources; sbatch, salloc and srun. The first two allocate resources to you while the latter launches tasks across those resources.
5.2.1 The srun command
A very simple example of a job is to request an interactive session on a worker node. The following command will ask the scheduler to provide a command prompt with one CPU and one GPU:
$ srun -n 1 --gres=gpu:1 --pty /bin/bash
Once on the node, if you issue the command nvidia-smi you should see that you have access to a single V100 GPU.
You are able to submit more complex requests using command line parameters but a better approach is to combine them into a job script and then submit your request to the scheduler using the sbatch command.
5.2.2 The sbatch command
This command submits a batch script to the SLURM scheduler by passing the name of a file which contains all your job commands. The file should include a list of #SBATCH directives (also known as job submission flags) which tell the job scheduler what you want to do. There are many job submission flags but at a minimum you should include the following:
#SBATCH -p pearl #(the queue to run on)
#SBATCH -n 1 #(number of processors to run on)
#SBATCH --job-name="<job_name>"
#SBATCH --gres=gpu:1 #(number of GPUs to use)
#SBATCH -t 2:30 #(expected job run time)
The estimated run time of your job is an important parameter as it will allow the scheduler to make a better decision as to when your job. You don’t have to be exact but a rough estimate is better than nothing. If you leave this parameter out altogether the scheduler will assume a default runtime.
In this next example, we have requested 4 CPUs and 4 GPUs and have estimated the job execution time to be no more than 12 hours:
#!/bin/bash
#SBATCH -p pearl # partition (queue)
#SBATCH --job-name=mytestjob
#SBATCH -n 4 # number of CPU cores
(continues on next page)
12 Chapter 5. The SLURM scheduler
Note: In summary, srun is used to submit a job for execution in real time while sbatch is used to submit a job script for later execution
5.2.3 The salloc command
The salloc command reserves resources in real-time under a job allocation and then releases them when you are finished.
Any command that is run within the allocation is run in parallel across all nodes that nodes that have been allocted.
In the next example, the salloc command requests two nodes and once allocated, we can check that this is the case with the srun hostname command:
$ salloc -N2 -t 0-00:05
salloc: Granted job allocation 1498529 salloc: Waiting for resource configuration salloc: Nodes cn2g[17-18] are ready for job
cpu-bind=MASK - cn2g17, task 0 0 [67022]: mask 0x101 set
$ srun hostname
cpu-bind=MASK - cn2g17, task 0 0 [67069]: mask 0x101 set cn2g17.scarf.rl.ac.uk
cpu-bind=MASK - cn2g18, task 1 0 [43302]: mask 0x101 set cn2g18.scarf.rl.ac.uk
See also:
For more detailed information please see the SLURM man pages forsbatch,srunandsalloc.
5.3 Controlling your jobs
5.3.1 Monitoring jobs
Once you have submitted your job you can monitor its status with this command:
$ squeue -u <username>
The following example tells us that the job has been accepted by the scheduler. The PD field shows that the job is in a pendingstate and the NODELIST field tells us that this is because the scheduler is waiting for the requested resources to become available.
pearl004@ui:~/jobs $ squeue -u pearl004
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 2112 pearl syrinx pearl004 PD 0:00 1 (Resources)
The output for a running job would be:
pearl004@ui:~/jobs $ squeue -u pearl004
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 2112 pearl syrinx pearl004 R 20:33 1 mn1
If you need a more detailed summary of a running job you can use the following command:
$ scontrol show jobid -dd <job_id>
5.3.2 Holding, releasing and requeuing jobs
If you need to hold a pending job from being run you can do so using:
$ scontrol hold <job_id>
To release a job that is currently being held:
$ scontrol release <job_id>
To requeue a running, suspended or completed job:
$ scontrol requeue <job_id>
5.3.3 Cancelling jobs
If you need to cancel a job (either queued or already running) you use the scancel command:
To cancel a job using the job’s unique ID number:
$ scancel <job_id>
To cancel a job by name:
$ scancel <job_name>
To cancel all your pending jobs:
$ scancel -t PENDING -u <username>
To cancel all of your jobs, regardless of their state:
$ scancel -u <username>
5.3.4 When is my job going to run?
If your job is queued and you want to know roughly when it will start, use the following command:
$ squeue --start -j <jobid>
14 Chapter 5. The SLURM scheduler
Using Containers
Whilst you are able to run programs directly on the host OS, most of the workloads run on PEARL use containers and much of this documentation assumes you are doing so.
Container support on PEARL is provided throughSingularity. It is an integral part of the cluster, and Singularity commands can be executed natively on any login or compute node without the need to load any additional modules.
PEARL is connected to theNVIDIA GPU Cloudwhich is also known as NGC. This is a catalogue of pre-optimized deep learning framework containers designed to make full use of the DGX-2’s GPUs in both single and multi-GPU configurations.
Please see the NVIDIA Docker Containers for Deep Learning Frameworks User Guide for more information on running deep learning containers andRunning NGC Containers Using Singularityguide for more information.
NGC provides images in Docker format so to run these on PEARL they must first be converted to the Singularity format.
Note: PEARL maintains a collection of the most commonly used images from NGC which have already been converted to Singularity format. These can be found here:
/mnt/beegfs/images/singularity
If there is an image you would like to be made available then please request this through the PEARL helpdesk.
You are also able to pull down images from NGC and convert them to Singularity format. You will need to set up an NGC account and create an API key first which can be done by following these steps:
1. Create a personal NGC account athttps://ngc.nvidia.com/signup
2. Create an NGC API key for access to the NGC container registry. This can be found by selecting ‘SETUP’ from the left-side menu. Retain a copy of the key for use in the following step.
3. Log on to ui.pearl.scd.stfc.ac.uk using your pearlXXX account and set up the Singularity-Docker environment:
$ export SINGULARITY_DOCKER_USERNAME='$oauthtoken'
$ export SINGULARITY_DOCKER_PASSWORD=[your_api_key]
4. Request an interactive session with access to a GPU:
$ srun --gres=gpu:1 --pty /bin/bash
5. Pull a container from NGC and convert it to Singularity format:
$ singularity build cuda_10.0-base-centos7.sif docker://nvcr.io/nvidia/cuda:10.0-base-
˓→centos7
6. Test your container:
$ singularity exec --nv cuda_10.0-base-centos7.sif cat /etc/redhat-release && nvidia-
˓→smi
CentOS Linux release 7.7.1908 (Core) Thu Dec 19 14:00:26 2019
+---+
| NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 |
|---+---+---+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM3... On | 00000000:34:00.0 Off | 0 |
| N/A 31C P0 51W / 350W | 0MiB / 32480MiB | 0% Default | +---+---+---+
+---+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+---+
6.1 Building a custom container
It is possible to build a custom container from scratch but the ability to do so is enabled on a per-user basis. If you would like to be able to do this then please contact us via the PEARL Service Desk.
Custom containers can be built from a definition file, which is essentially a set of blueprints defining the OS, what software to install, environment variables, etc. At a very minimum the definition file must contain a header section which defines the base operating system that will be used in the container.
You will have to unset the SINGULARITY_DOCKER_USERNAME and SINGULARITY_DOCKER_PASSWORD en- vironment variables for this example to work - otherwise Singularity will try and connect to the Docker Hub with your NVIDIA NGC account. If you would like to use a private Docker image, set these variables to your Docker Hub username and access token.
The following simple example pulls docker layers from the Docker Hub, sets Ubuntu as the OS and defines two sections; test and help:
Bootstrap: docker From: ubuntu
%test
grep -q NAME=\"Ubuntu\" /etc/os-release if [ $? -eq 0 ]; then
(continues on next page)
16 Chapter 6. Using Containers
%help
Hello from inside the container!
You can then build your container:
$ singularity build --fakeroot hello.simg custom-container.def
Then do the following to test the functionality:
$ singularity test hello.simg
Container base is Ubuntu as expected.
$ singularity run-help hello.simg Hello from inside the container!
See also:
For a more detailed explanation of definition files please see theofficial Singularity documentation.
18 Chapter 6. Using Containers
MPI and Infiniband
To run multi-node MPI workloads over the low-latency network, some of the components within the container must closely match those of the host environment. In addition to Infiniband drivers, the container’s MPI implementation needs to be ABI-compatible with the version running on the host.
On PEARL the supported MPI implementation is OpenMPI. We recommend using OpenMPI 4.1.0 but you are able to use OpenMPI 3.1.6 if you prefer. The OpenMPI version within your container should be the same as that on the host, or newer.
Note: We have a Singularity definition files available for download that includes everything you need to run an MPI workload on the PEARL worker nodes. It is available as part of an archive which also includes some simple test programs and a SLURM submission script.
Archive pearl_openmpi.tar.gz
To build the image from the definition file, first request an interactive session on one of the worker nodes:
$ srun --pty /bin/bash
Then build the image using the following command:
$ singularity build --fakeroot ompi4.sif ompi4.def
Once complete, you can launch a shell within the container:
$ singularity shell ompi4.sif
From here you can check the following important components:
The Infiniband driver version:
Singularity> ofed_info -s OFED-internal-5.2-1.0.4:
The OpenMPI version:
Singularity> mpirun --version mpirun (Open MPI) 4.1.0
Then check that OpenMPI has been built with IB support:
Singularity> ompi_info | grep btl
MCA btl: openib (MCA v2.1.0, API v3.1.0, Component v4.1.0)
To ensure that everything works as expected, you can run the included tests by submitting them as a batch job using the job script provided:
$ sbatch submit.slurm
When the job has completed, view the output file to check the results. The first result is a bandwidth test between the two nodes running on the host OS. The second set of figures is the same test but run within the container. The third set of results is another bandwidth test running inside the container, but this time using mpi4py.
7.1 OpenMPI run-time tuning parameters
To ensure optimum performance over the low-latency network, There are a two run-time parameters that you should include when running an MPI workload.
--mca btl_openib_allow_ib 1 --mca btl_openib_if_include mlx5_0:1
In context, the full command used to run the mpi4py bandwidth test would be:
mpirun -q --mca btl_openib_allow_ib 1 --mca btl_openib_if_include mlx5_0:1
˓→singularity exec --nv ompi4.sif python tests/bandwidth.py
Note: The -q directive sets OpenMPI’s reporting to ‘quiet’ mode to supress non-error messages
20 Chapter 7. MPI and Infiniband
Compiling Your Code
Any compilation that you wish to do should only be done on the compute nodes by requesting an interactive job from the SLURM scheduler. This is done using the command:
$ srun --pty /bin/bash
The system default compiler is gcc 7.4.0 which is available to you without having to load any additional modules.
Newer versions of GCC are available and you can see what is available by issuing the following command:
$ ml av gcc
22 Chapter 8. Compiling Your Code
Support
If you have any questions about the PEARL service or experience any problems then please contact us via the PEARL Service Desk by sending an e-mail [email protected].
24 Chapter 9. Support
PEARL and Data Protection
At STFC (being a Council under, and part of, United Kingdom Research and Innovation), we intend to make sure that we maintain the UK as a world-leader in science and technology research - integral to this is our commitment to protecting and safeguarding your data privacy.
It is part of STFC’s public purpose to provide services such as PEARL. In order to effectively provide that service to you, we need to collect information about you and may need to contact you from time to time with regards to its provision.
STFC considers it in the legitimate interest of anyone who has signed up to access the PEARL service that it should be able to effectively and efficiently manage the service for their benefit. STFC therefore requests and securely holds the data which it requires for these purposes. Please ensure that, for so long as you are accessing the PEARL service, you notify us if your information needs to be updated, to do so you can use the Contact Us link at the bottom of this page
PEARL stores your name, affiliation and e-mail address in our user management and helpdesk systems. In addition our webserver and login nodes store the IP address you used to connect to them in their standard logs. We store information on the jobs you submit, including any scripts you pass to the batch system, and also which modules of our modules system you have loaded. Your e-mail address is shared with the JiscMail service - as part of our user registration process you will be registered to our PEARL-USERS mailing list hosted on JiscMail. You are free to unsubscribe from this mailing list at any time.
10.1 UKRI and Data Protection
UKRI and STFC comply with the requirements of the GDPR with regard to the collection, storage, processing and disclosure of personal information and is committed to upholding the GDPR’s core data protection principles. A full notice of UKRI’s position with regards to Freedom of Information and the General Data Protection Regulations (GDPR) (EU) (2016/679) can be found at:https://www.ukri.org/freedom-of-information/
26 Chapter 10. PEARL and Data Protection
SSH keys
You must use an SSH key to access PEARL.
An SSH key consists of two parts; a private key and a public key. Once you have generated your key pair the private key should be loaded into your SSH agent. Your public key should be sent to us using theapplication form.
Warning: It is very important that you keep your private key safe and do not share it with anyone. We will never ask you for your private key.
How you generate a key and add it to your agent will differ depending on the operating system that you are using.
11.1 Generating an SSH key
Note: For compatibility reasons, we recommend generating a key using the RSA algorithm and with a key size of at least 2048 bits.
11.1.1 On Linux and macOS
Most Linux and macOS versions will have an SSH client and key generation utility already installed.
To generate a key pair, run the following command:
$ ssh-keygen -t rsa -b 4096
When you are asked where you want to save the key, accept the default location. Please also ensure that you choose a strong passphrase to protect you private key. Make a note of this as you will need it when you load your key into the agent.
11.1.2 On Microsoft Windows
We recommend that you generate an SSH key on Windows using the graphicalPuTTYgen utility:
1. Launch the PuTTYgen utility
2. Make sure that Type of key to generate is set to RSA
3. Set the value for Number of bits in a generated key to 2048 or 4069
4. Click Generate and repeatedly move your mouse over the blank area until the progress bar is full
5. In the Key passphrase field, enter a strong passphrase and then retype it in the Confirm passphrase field 6. Click Save private key and choose a suitable name for the file, noting the location where it is saved.
7. Right-click the Public key for pasting into OpenSSH authorized_keys file field and click Select all
8. Right-click the highlighted text and select Copy. This is the text that needs to be included in your application form
9. Make a note of the Key fingerprint string so that you can verify this key at a later time
28 Chapter 11. SSH keys
1. If it is not already running, start the ssh-agent program in the background:
$ eval `ssh-agent`
Agent pid 90125
2. Load your SSH key into ssh-agent and type in the passphrase when asked to do so. If you saved your private key somewhere other than the default location then you will need to specify the path to it:
$ ssh-add ~/.ssh/id_rsa
3. Additionally, if you are using macOS, you can store your passphrase in the macOS keychain so that you don’t have to enter it every time:
$ ssh-add -K ~/.ssh/id_rsa
11.2.2 On Microsoft Windows
We recommend thePageant PuTTY authentication agent. When the program is running, it will show as an icon in the System Tray on the taskbar.
To add a key:
1. Right-click the Pageant System Tray icon 2. Select Add Key
3. Navigate to the directory where you saved your private key file, select it and click Open 4. In the Enter Passphrase dialog, enter the key’s passphrase and click OK
5. Check that the key has been loaded by right-clicking the Pageant icon again and select View Keys
6. In the Pageant Keys List dialog, check that the key’s fingerprint matches the one that you noted when you created the key
30 Chapter 11. SSH keys
PEARL Service User Documentation
12.1 Overview
PEARLis a high-performance computing cluster, designed primarily for Deep Learning and AI research. At its core are twoNVIDIA DGX-2s. Each DGX-2 utilises 16 Tesla V100 GPUs with a total of 512GB of GPU memory. Each also has 1.5TB of system RAM, two Intel Xeon Platinum CPUs and 30TB of NVME SSD local storage. The DGX-2s are connected over 100Gbit/s EDR InfiniBand to two Boston Flash-IO Talyn servers which together provide half a petabyte of NVMe storage.
It is run bySTFC’s Scientific Computing Research Infrastructures Groupon behalf of theAlan Turing Instituteand it is hosted at STFC’s Rutherford Appleton Laboratory.
12.2 Eligibility
PEARL is only available to researchers from The Alan Turing Institute and STFC and their immediate collaborators and only for scientific and academic research purposes falling under theAI for Science theme.
12.3 Requesting access
To request access, please complete theapplication form. Once we have verified your eligibility we will contact you with your account details.
You must have an SSH key to access the cluster. We recommend that you create one specifically for accessing PEARL.
Please see the section onSSH keysfor more information.
12.4 Service Announcements
As part of the registration process you will be added to the PEARL-USERS JISCMail mailing list to enable us to keep you informed of important service announcements and developments.
See also:
Please read ourdata protection policyfor the information that we retain for this purpose.
32 Chapter 12. PEARL Service User Documentation
