Linux Clusters

1. Information

2. Architecture

3. Running applications over many hosts

   1. Generating ssh keys

   2. Available workstations

   3. ARIA

      1. Version1.1 set up

      2. Version 1.2 set up

      3. Multiple Machines

      4. Benchmarks

   4. XPLOR

      1. Multiple Machines

      2. Benchmarks

   5. CNS

      1. Multiple machines

      2. Benchmarks

   6. CYANA

      1. Muliple Machines

      2. Benchmarks

   7. HADDOCK

      1. Multiple machines

      2. Benchmarks

   8. NMRPIPE

      1. Multiple Machines

      2. Benchmarks

   9. Using SGE to submit jobs

Information

We have 4 dual AMD opteron processor blades (servers) and 6 IBM workstations running off AMD opteron dual processors. In total there are 20 cpu's that can be used to submit jobs to.

The two servers consist of a primary (server01) and secondary (server02). The secondary is constantly updated and mirrors the primary (updated every second). If for any reason the primary server crashes, then the secondary server takes over (hopefully you will not notice a thing unless you were running a job on the server).

Another two blades are simply used as computational power houses. This is mainly due to space restrictions in the Linux room. These should be used if you want to submit large jobs

Finally we have 6 IBM workstations - two of these will have dual monitors to help visualization for NMR analysis. Again, you can use these if you wish to specify a big job over many nodes

We insist that all long jobs are submitted via the SGE job queuing system

Architecture

We use the NFS system, which allows you to access your home directory and applications from any machine on the cluster

When you are given a username, your home directory will be

/nfs/home/username

Softwares available to everyone are stored at /nfs/usr/

In fact your home directory and software application are both located on the server. This can cause a major problem with speed for programmes using a lot of I/O access. e.g. nmrPipe. To run these programmes it is best to have the data stored locally on your workstation (/usr/local/scratch) for details on this see below.

Disk space

The nfs system has 160 GigaBytes of space

In addition each workstation has 80 Gb of space. At present, however, the system is set up to delete any data that has been on the local disk for more than 15 days, so if you do something locally (e.g processing, then make sure you copy it over to your home directory. I'm not sure this is something I want to continue, but we shall see how this goes.

Running applications over many hosts

Using ssh keys

Most processes we use require login to other machines via ssh. It is important to set up your ssh configuration to login automatically without having to type a password. To do this perform the following steps:

cd ~/.ssh

ssh-keygen -t dsa

cp id_dsa.pdb authorized_keys

now to ensure that you can only logon from the cluster you need to edit the authorized_keys file. At the very beginning you need to insert:

from="*.iric.udem"

so it should have something like

from="*.iric.udem" ssh-dss AAAA

Finally ssh to all the machines. You will be asked for the password first time, thereafter it should login automatically.

The names of the machines are here

In my view the cluster (multiple machines) should be used for the following applications:

XPLOR
ARIA

You can also try NMRpipe but this is really slow over the network (check my stats on this below) because of the excessive I/O access. Since the NFS copies data from server01 to server02 every second, and nmrPipe writes and reads data a lot quicker than that - you can see the problem.

My advice is to use run nmrPipe with the data located locally on the /scratch directory.

Workstations and servers

The workstations and servers/pnodes are named as follows:

server01
server02
pnode01
pnode02
station01
station02
station03
station04
station05
station06

ARIA

ARIA1.1.2

To set up add the following command to your .bashrc

export PYTHONPATH=/nfs/home/osbornem/local/aria1.1.2:$PYTHONPATH

alias aria='/usr/bin/python2 /nfs/home/osbornem/local/aria1.1.2/Aria/RunAria.py'

and your .cshrc

setenv PYTHONPATH /nfs/home/osbornem/local/aria1.1.2:$PYTHONPATH

alias aria '/usr/bin/python /nfs/home/osbornem/local/aria1.1.2/Aria/RunAria.py'

ARIA1.2.

To set up add the following to your .bashrc

export PYTHONPATH=/nfs/home/osbornem/local/aria_1.2:$PYTHONPATH

alias aria='/usr/bin/python2 /nfs/home/osbornem/local/aria_1.2/Aria/RunAria.py'

and your .cshrc:

setenv PYTHONPATH /nfs/home/osbornem/local/aria_1.2:$PYTHONPATH

alias aria '/usr/bin/python /nfs/home/osbornem/local/aria_1.2/Aria/RunAria.py'

To run on multiple machines alter the run.cns script as follows:

{===>} queue_1="csh";

{===>} cns_exe_1="/nfs/home/osbornem/local/aria1.1.2/cns_solve_1.1/linux_opteron32_gnu/bin/cns";

{===>} cpunumber_1=1;

{===>} queue_2="ssh station04 csh";

{===>} cns_exe_2="/nfs/home/osbornem/local/aria1.1.2/cns_solve_1.1/linux_opteron32_gnu/bin/cns";

{===>} cpunumber_2=2;

{===>} queue_3="ssh station03 csh";

{===>} cns_exe_3="/nfs/home/osbornem/local/aria1.1.2/cns_solve_1.1/linux_opteron32_gnu/bin/cns";

{===>} cpunumber_3=2;

However, not that this is not necessary if you run ARIA via the SGE. I have scripts that will do this automatically for you. So PLEASE USE THE SGE SYSTEM. Users not doing this run the risk of their process being stopped.

see here for info on SGE and ARIA

Just csh will run on the current workstation. CPU number is the number of CPU's to use on that station.

Note that the maximum processors is 16: but as you calculate 20 structures, using 10 CPU's will take the same time as using 16..

Alternatively you can edit the appropriate forms for data conversion (top) or for the run.cns (see bottom) using forms on our server:

aria1.1

aria1.2

OR from the server at the pasteur institute:

Aria1.1

For aria1.2

Some Benchmarks:

These are based on calculating 20 structures per iteration (8 iterations in total). Since Aria does parallelization

Why no performance from 10 to 16 cpu's. This is because the next iteration cannot continue until all 20 structures have been computed. Thus if we could use 20 cpu's then the time will probably be halved. Or we could calculate 16 structures to get a similarly halving of the time.

BOTTOM LINE:

For maximum performance use only 10 CPU's...(assuming no-one else is using them).

XPLOR

Running Xplor on multiple machines:

xplor -parallel -machines XX.file -o oz.out < xplor.inp

where XXX.file contains names of the hosts

e.g.
station02
station03
station04

If you have old xplor scripts you wish to use in parallel then add the following syntax (remember the total number of structures should be greater than or equal to the number of machines specified:

Place these lines at the beginning of the script:

!total number of structures to calculate:

eval ($numStructs = 100)

!random seed:

eval ($randomSeed = 785)

! get parallel info

cpyth "from os import environ as env"

cpyth "xplor.command('eval ($proc_num=%s)' % env['XPLOR_PROCESS'])"

cpyth "xplor.command('eval ($num_procs=%s)' % env['XPLOR_NUM_PROCESSES'])"

eval ($firstStruct = ($proc_num * $numStructs) / $num_procs)

eval ($lastStruct = (($proc_num+1) * $numStructs) / $num_procs)

2) Then, the structure loop should be modified to :

eval ($count = $firstStruct)

while ($count <= $lastStruct) loop structure

eval ($seed = $randomSeed+$count)

set seed $seed end

.

.

evaluate ($file = "1gb1_" + encode($count) + ".pdb")

write coor output= $file end

eval ($count = $count + 1)

end loop structure

Within the Python interface, parallelization is automatic if one uses the StructureLoop class from the simulationTools module.

Benchmarks:

Running over 6 CPU's

Invoking the following command to calculate 50 structures:

nfs/usr/xplor-nih-2.10/bin/xplor -parallel -machines ./allhosts.list -o xplor_par < anneal1_par.inp"

NMRPipe

If you have a 2D or 3D (without too many LP) then it is probably best to store the data locally on /scratch and run on a single cpu!!

However for 3D or 4D which would normally take a long time - you can run over many machines.

1. First you need to have a file called pipe.net describing the CPU's e.g

station01.iric.udem 10

station01.iric.udem 10

station02.iric.udem 10

station02.iric.udem 10

station03.iric.udem 10

station03.iric.udem 10

station04.iric.udem 10

station04.iric.udem 10

station06.iric.udem 10

station06.iric.udem 10

station05.iric.udem 10

2. Now split your processing script into seperate files for each transform:

Here is an example of double lp using parallel processingNB the bold are the lines for parallel:

This id the ft of the 1^st and second dimension with no LP

# parallel1.com NMRPipe Data Processing Script

#! /bin/csh

echo 'Transforming 3-dimensional NMR data!'

echo 'Processing F3/F1 dimensions first'

hostname

waitFor -par pipe.net -init -ext part1 -cpu $1 -verb

xyz2pipe -in ./data/test%03d.fid -x -verb -par pipe.net -cpu $1 \

| nmrPipe -fn SOL -mode 2 -fl 64 -fs 2 -poly \

| nmrPipe -fn CBF -last 12 \

| nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -size 512 -c 0.5 \

| nmrPipe -fn ZF -size 1024 \

| nmrPipe -fn FT \

| nmrPipe -fn PS -p0 -66 -p1 0 -di \

| nmrPipe -fn TP \

| nmrPipe -fn SP -off 0.5 -end 0.95 -pow 2 -size 60 -c 0.5 \

| nmrPipe -fn ZF -size 256 \

| nmrPipe -fn FT \

| nmrPipe -fn PS -p0 0 -p1 0 -di \

| nmrPipe -fn TP \

| pipe2xyz -out lp/shit%03d.ft2 -x

waitFor -par pipe.net -ext part1 -cpu $1 -verb

Now ft the 3^rd dimnesion with LP

# parallel2.com NMRPipe Data Processing Script

#! /bin/csh

hostname

waitFor -par pipe.net -init -ext part1 -cpu $1 -verb

echo 'Processing F2'

xyz2pipe -in lp/shit%03d.ft2 -z -verb -par pipe.net -cpu $1 \

| nmrPipe -fn LP -fb -ord 10 -x1 2 -xn 28 -pred 32 -fix -fixMode 1 -after \

| nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -size 60 -c 0.5 \

| nmrPipe -fn ZF -size 128 \

| nmrPipe -fn FT \

| nmrPipe -fn PS -p0 0 -p1 0 -di \

| pipe2xyz -out lp/shit%03d.ft3 -z

waitFor -par pipe.net -ext part1 -cpu $1 -verb

Now do inv ft on the 2^nd dimension and LP

# Parallel3.com NMRPipe Data Processing Script

#! /bin/csh

echo 'Processing F3/F1 dimensions first'

hostname

waitFor -par pipe.net -init -ext part1 -cpu $1 -verb

echo 'Processing F2'

xyz2pipe -in lp/shit%03d.ft2 -z -verb -par pipe.net -cpu $1 \

| nmrPipe -fn LP -fb -ord 10 -x1 2 -xn 28 -pred 32 -fix -fixMode 1 -after \

| nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -size 60 -c 0.5 \

| nmrPipe -fn ZF -size 128 \

| nmrPipe -fn FT \

| nmrPipe -fn PS -p0 0 -p1 0 -di \

| pipe2xyz -out lp/shit%03d.ft3 -z

waitFor -par pipe.net -ext part1 -cpu $1 -verb

Due to I/O time, I find the first stage (direct dimnesion FT) is best done on one machine... only the LP steps are best done over many cpu's..

To run these three files I make a csh script such as below

nmrShell -rsh ssh -par pipe.net -sh parallel1.com -path /nfs/home/osbornem/shitter > shit

nmrShell -rsh ssh -par pipe.net -sh parallel2.com -path /nfs/home/osbornem/shitter > shit1

nmrShell -rsh ssh -par pipe.net -sh parallel3.com -path /nfs/home/osbornem/shitter > shit2

./parallel4.com > shit3

echo "COMPLETED"

YOU CAN DOWNLOAD EXAMPLE SCRIPTS here

Note the use of nmrShell to launch parallel processing. The final script parallel4.com is done on a sing;e processor..

Usage of nmrShell is given below

nmrShell -rsh ssh -par pipe.net -sh nmrproc.com -path /FULLPATHNAME -log
e.g if your script is in /nfs/home/osbornem/nmrpipe

nmrShell -rsh ssh -par pipe.net -sh nmrproc.com -path /nfs/home/osbornem/nmrpipe -log

Some pointers

The best way to do this is by using nmrShell using the following command:

Do not use any of the verbose commands in NMRPipe (-v, -verb) as displaying text over the nfs will really slow things down

your script will be identical as for a single cpu but you need to add on the line containing the input"-par

Benchmarks:

Running over 6 CPU's

Ruuning a double lp of a hcch-tocsy using the scripts shown above

CNS

CNS cannot (to my knowledge anyway) be parallelized. I guess you can submit a number of input files with differing starting trajectories. Nonetheless, CNS has been optimised (32-bit) by Savoirefaire linux, and it is pretty fast.

Check out some benchmark data here and graphs work compare our system with others.

Benchmarks for test suite programmes (time in seconds)

CYANA

After a lot of messing around this finally works.

To run of multiple processors use the mpi run command,

e.g

mpirun -np 16 cyana CALC.cya

You can specify which nodes using n (mpirun n0,1,2,3,7 or mpirun n1-5 etc)

Soon I will try to get this working via SGE

Cyana Benchmarks

Running an automated structure calculation (i.e. no NOE's are know and are auto assigned by CYANA)

HADDOCK

To set up haddock you can source the script haddock_configure at /nfs/home/osbornem/local/haddock/haddock_configure.

e.g. type the following at your Linux prompt

source /nfs/home/osbornem/local/haddock/haddock_configure

then type haddock1.3 in the directories containing the new.html this is good enough to set up you r run directories.

To run the run.cns script on multiple machines USE THE SGE SETUP detailed here which will essentially perform the following changes to you run.cns script. Failure to use SGE may result in your calculation being stopped prematurely

{===>} queue_1="csh";

{===>} cns_exe_1="/nfs/home/osbornem/local/aria_1.2/cns_solve_1.1/linux_opteron32_gnu/bin/cns";

{===>} cpunumber_1=1;

{===>} queue_2="ssh station04 csh";

{===>} cns_exe_2="/nfs/home/osbornem/local/aria_1.2/cns_solve_1.1/linux_opteron32_gnu/bin/cns";

{===>} cpunumber_2=2;

Note for haddock you can run up to 14 cpus at a time

Benchmarks

Running an example script producing 1000 structures in it0, 200 in it1 and 200 water-refined structures.