1 / 5
HPC system startup manual (version 1.30)
Document change log
Issue Date Change Modified
pages Notes 1 12/1/2012 New document
2 10/22/2013 Added the information of supported OS
P2
3 10/22/2013 Changed the example 1 for data download
P2
4 3/4/2014 Added the chapter of Torque/Maui batch job system
P2
5 3/4/2014 Changed text font
6 10/2/2014 Changed the example 1 for data download
P2
1
Overview
The high performance computing (HPC) system in the SACLA facility consists of 80 compute nodes, with two 6-core processors per node, for a total of 960 cores. The theoretical peak compute performance is 13 TFLOPS. The system supports a 170TB shared storage system, managed by the Lustre file system. Nodes are interconnected with Infiniband with a 40 Gbps bandwidth. This manual introduces the startup from account creation to the launch of software installed in the SACLA HPC system.
2
Account creation
After your experiment proposal accepted, a responsible researcher of the SACLA is assigned to your proposal. Contact him by e-mail and request the use of the SACLA HPC system. He will create your account and give the username and password. Note that “/home” directory is limited by quota to 100 GB per user. When you are going to use more storage area for a large amount of experimental data and the result data of the analysis, “work” directory can be used. Please talk with your contact researcher for the use of “/work” directory.
2 / 5 The access of the SACLA HPC system is limited to be within SPring-8 local network. First of all, you have to connect your computer to the SPring-8 local network by using VPN service. After that, you can login the HPC server through SSH protocol.
[ step 1. Connect the SPring-8 network with VPN service. ]
Access the web page: https://hpc.spring8.or.jp with a web browser.
Login the web page of VPN installation service. (Enter your username and password given in Sec. 2)
Download and install VPN connection software into your computer. Supported OS for VPN service is as follows.
[Windows]
Windows 7 Professional(64bit) Windows 7 Home Premium(64bit) Windows Vista Home Premium(32bit) [Mac]
Mac OS X Lion (10.7) [Linux]
Debian GNU/Linux 6.0 (x86_64) Redhat Enterprise Linux 6.2 (x86_64)
[ step 2. Login SACLA HPC system. ]
Type “ssh -X [username]@xhpcfep”. The username is given in Sec.2. Enter your password given in Sec.2.
4
Torque/Maui batch job system
The queuing system on SACLA HPC system is Torque/Maui. You need three main commands to use this system: showq, qsub, and qdel.
[showq]
“showq” displays information about active, eligible and blocked jobs. [qsub]
“qsub” submits a job. In its simple form you would run “qsub myscript”, where myscript is a shell script that runs your job. For those who don't want to write scripts, you can do an interactive qsub with the -I option. You can also use -X option for enabling X11 forwarding (e.g. “qsub –I –X –q smp”).
3 / 5 “qdel job_number” deletes your job. You can get information on job_number with “showq”.
5
Data handling in SACLA HPC system
[ Ex. 1 Download your experimental data to your storage space in the HPC. ]
This example downloads an HDF5 file with the data of run number 28930 in the tag range from 74121900 to 74121912 to the path of “/home/[username]/data/test/sample.h5” using interactive qsub. Login the SACLA HPC system as stated in Sec. 3.
Type “qsub -I” in the terminal window.
Type“mkdir -p ~/data/test” in the terminal window.
Type “DataConvert3 dir ~/data/test o sample.h5 each mode 20 r 28930 starttag 74121900 -endtag 74121960” in the terminal window.
“DataConvert3” command can be used at all compute nodes.
[ Ex. 2 Open your experimental data. ]
This example shows you can open the HDF5 file of sample.h5 generated in Ex. 1 by using software of HDFView. After procedure listed below, the 2 dimensional data is displayed as an image on the HDFView GUI.
Login the SACLA HPC system as stated in Sec. 3. Type “hdfview” in the terminal window.
Open a file open dialog by clicking “File -> Open” in the menu bar of the HDFView. Specify a filename to be “/home/[username]/data/test/sample.h5” in the file open dialog. Extract the HDF5 tree displayed in the HDFView left sidebar by clicking “run_[run number]
-> detector_2d_[detector ID] --> tag_[tag number]”
Invert and right click a data object of “detector_data” and select “Open as ” in the context menu Select “Display As Image” and click “OK” in the dataset selection dialog.
[ Ex. 3 Data transfer to your local computer. ]
This example is a way to transfer your data from the SACLA HPC to your local computer through SSH protocol.
Type “scp [username]@xhpcfep:/home/[username]/data/test/sample.h5 [favorite path]“ on your local terminal window.
[ Ex. 4 Browse SACLA Database. ]
The beamline information is recorded to a MySQL server. You can browse and plot the recorded information by using web service on the SACLA HPC network. The example written below is a
4 / 5 procedure to display the trend graph of a signal on the web page.
Login the SACLA HPC system as stated in Sec. 3. Type “firefox” in the terminal window.
Access the web page: http://xqaccdaq01.daq.xfel.cntl.local/syncdaq/xdaq.html.
Register a signal (in this example, we use beam monitor 1) to the left top sub-window by clicking links “xq-bl3-fetc -> xfel_bl_3_tc_bm_1_pd/charge” in the left bottom sub-window.
Select the registered signal of xfel_bl_3_tc_bm_1_pd/charge to be left Y-axis by inverting the signal name in list box of Left Axis in the left top sub-window.
Select Time to be X-axis in the left top sub-window.
Specify BeignTime and EndTime to the example time period: 15 23:29:29, 2012-05-15 23:29:50 in the left top sub-window.
Click plot button
[ Ex. 5 Export data of SACLA Database to a Text file. ]
You can export the beamline information registered in the MySQL server to an text file by using a command: “syncdaq_get”.
Login the SACLA HPC system as stated in Sec. 3.
Type “syncdaq_get –l200 -b"2012/05/15 23:29:29" -e"2012/05/15 23:29:50" xfel_bl_3_tc_bm_1_pd/charge > bm_data.txt” in the terminal window.
6
Installed software List and how to use
The available software is listed below.
[ Hdfview ]
Login the SACLA HPC system as stated in Sec. 3. Type “hdfview” in the terminal window.
[ MATLAB]
Login the SACLA HPC system as stated in Sec. 3. Type “qsub -I -vDISPLAY” in the terminal window. Type “matlab” in the terminal window.
Note: The concurrent client number is restricted to be the number of license key RIKEN possess. You MUST close the MATLAB after finish your analysis.
5 / 5 [ ImageJ ]
Login the SACLA HPC system as stated in Sec. 3. Type “qsub -I -X” in the terminal window.
Type “ImageJ” in the terminal window.
[ ROOT ]
Login the SACLA HPC system as stated in Sec. 3.
Type “export ROOTSYS=/home/software/root/pro” in the terminal window. Type “export PATH="$PATH:$ROOTSYS/bin” in the terminal window.
Type “export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ROOTSYS/lib/root" in the terminal window.
Type “root” in the terminal window.
[ FireFox ]
Login the SACLA HPC system as stated in Sec. 3. Type “firefox” in the terminal window.
