Minicluster:Mpich com torque
This is part three of a multi-part tutorial on installing and configuring MPICH2. The full tutorial includes
The "Other" Mpiexec
Congratulations! By going this route, you're saving yourself a lot of hassle by avoiding maintaining an mpd ring on the cluster, or helping your users set up and tear down rings. If that doesn't make sense, that's fine - that's the beauty of this option. MPI will take care of all the details for the users.
This functionality is added by using a new binary. Instead of the binary called mpiexec that came with the MPICH2 installation, a new mpiexec is installed. Unfortunately, they have the same name, but they work very differently. The "other" mpiexec - the one with Torque-functionality that we need to add to the cluster - is produced out of Ohio Supercomputer Center. This mpiexec only works with Torque. In other words, users will not be able to use it outside of a qsub script.
For this reason, you might want to only put the new mpiexec on the worker nodes, and keep the original binary for users to run outside of Torque on the head node.
Prep
Before starting, the original mpiexec needs to be moved (at least on the worker nodes) so that it is no longer in root's or users' paths (or just renamed). Find it with
which mpiexec
and then move it somewhere else as a backup.
Installation
To download the latest version of mpiexec, visit the OSC mpiexec page and scroll down to the downloads section. Right click on the file location. Then, from whenever your source is stored, run
wget http://www.osc.edu/~pw/mpiexec/mpiexec-0.83.tgz
or whatever the latest version is. Untar it with
tar xvzf mpiexec*.tar.gz
and then cd into the new directory. Mpiexec follows the standard source installation paradigm. Run
./configure --help
to see a list of options. Important options include
--prefix=- specify where you want to have the binaries installed. They need to be accessible by all of the worker nodes. An NFS mount would be a good choice.--with-pbs=- necessary to get the Torque functionality! Specify the location of the Torque installation. If you followed my Torque tutorial, it's located at/var/spool/pbs--with-default-comm=mpich2-pmi- used to indicate which version of MPI
Next, run ./configure with all the options necessary. My command looked like this:
./configure --prefix=/shared --with-pbs=/var/spool/pbs/ --with-default-comm=mpich2-pmi
Pbs_iff
To do this next part, Torque will need to already be installed. Mpiexec requires that a file named pbs_iff be on each one of the worker nodes. Normally, this file is only located on the head node and isn't installed as part of the pbs_mom installation, so it needs to be copied out from the head node to each of the other nodes.
There's an easy way to do this by scripting. The first requirement is to have a file with each of the worker nodes listed in it. Assuming Torque is running, this can be generated with
pbsnodes | grep -v = | grep -v '^$' >> machines
-
grep -v =excludes all lines that have an equal sign in them -
grep -v '^$'contains a regular expression to delete all empty lines
The "machines" file of all the worker node names can then be used in a quick script to copy pbs_iff to each of the worker nodes. Find the original file with
updatedb && locate pbs_iff
(If you receive an error, apt-get install locate and then try again.) Then, replacing my locations below for the location you found it on your cluster, run
for x in `cat machines`; do rsync /usr/local/sbin/pbs_iff $x:/usr/local/sbin/; done
Next, pbs_iff needs to have its permissions changed to setuid root. (This means the binary runs with root privileges, even when run by a different user.) Again, to do this across all the worker nodes at once, use a script and make sure the location is correct for your setup:
for x in `cat machines`; do ssh $x chmod 4755 /usr/local/sbin/pbs_iff; done
Without these steps, users trying to run mpiexec will see errors like these:
pbs_iff: file not setuid root, likely misconfigured pbs_iff: cannot connect to gyrfalcon:15001 - fatal error, errno=13 (Permission denied) cannot bind to reserved port in client_to_svr mpiexec: Error: get_hosts: pbs_connect: Unauthorized Request .
Testing
Trying to run a program with mpiexec outside of a Torque job, it will give an error:
mpiexec: Error: PBS_JOBID not set in environment. Code must be run from a PBS script, perhaps interactively using "qsub -I".
At least it's a helpful error! Therefore, in order to test it, mpiexec will need to be called from within a script. Continue on to the Torque and Maui: Submitting an MPI Job page to test.
