Mudanças entre as edições de "Minicluster:Mpich com torque"

Edição das 23h12min de 12 de maio de 2011

This is part three of a multi-part tutorial on installing and configuring MPICH2. The full tutorial includes

• Installing MPICH
• MPICH: Pick Your Paradigm
• MPICH with Torque Functionality

The "Other" Mpiexec

Congratulations! By going this route, you're saving yourself a lot of hassle by avoiding maintaining an mpd ring on the cluster, or helping your users set up and tear down rings. If that doesn't make sense, that's fine - that's the beauty of this option. MPI will take care of all the details for the users.

This functionality is added by using a new binary. Instead of the binary called mpiexec that came with the MPICH2 installation, a new mpiexec is installed. Unfortunately, they have the same name, but they work very differently. The "other" mpiexec - the one with Torque-functionality that we need to add to the cluster - is produced out of Ohio Supercomputer Center. This mpiexec only works with Torque. In other words, users will not be able to use it outside of a qsub script.

For this reason, you might want to only put the new mpiexec on the worker nodes, and keep the original binary for users to run outside of Torque on the head node.

Prep

Before starting, the original mpiexec needs to be moved (at least on the worker nodes) so that it is no longer in root's or users' paths (or just renamed). Find it with

which mpiexec

and then move it somewhere else as a backup.

Installation

To download the latest version of mpiexec, visit the OSC mpiexec page and scroll down to the downloads section. Right click on the file location. Then, from whenever your source is stored, run

wget http://www.osc.edu/~pw/mpiexec/mpiexec-0.83.tgz

or whatever the latest version is. Untar it with

tar xvzf mpiexec*.tar.gz

and then cd into the new directory. Mpiexec follows the standard source installation paradigm. Run

./configure --help

to see a list of options. Important options include

• --prefix= - specify where you want to have the binaries installed. They need to be accessible by all of the worker nodes. An NFS mount would be a good choice.
• --with-pbs= - necessary to get the Torque functionality! Specify the location of the Torque installation. If you followed my Torque tutorial, it's located at /var/spool/pbs
• --with-default-comm=mpich2-pmi - used to indicate which version of MPI

Next, run ./configure with all the options necessary. My command looked like this:

./configure --prefix=/shared --with-pbs=/var/spool/pbs/ --with-default-comm=mpich2-pmi

Pbs_iff

To do this next part, Torque will need to already be installed. Mpiexec requires that a file named pbs_iff be on each one of the worker nodes. Normally, this file is only located on the head node and isn't installed as part of the pbs_mom installation, so it needs to be copied out from the head node to each of the other nodes.

There's an easy way to do this by scripting. The first requirement is to have a file with each of the worker nodes listed in it. Assuming Torque is running, this can be generated with

pbsnodes | grep -v = | grep -v '^$' >> machines

• grep -v = excludes all lines that have an equal sign in them
• grep -v '^$' contains a regular expression to delete all empty lines

The "machines" file of all the worker node names can then be used in a quick script to copy pbs_iff to each of the worker nodes. Find the original file with

updatedb && locate pbs_iff

(If you receive an error, apt-get install locate and then try again.) Then, replacing my locations below for the location you found it on your cluster, run

for x in `cat machines`; do rsync /usr/local/sbin/pbs_iff $x:/usr/local/sbin/; done

Next, pbs_iff needs to have its permissions changed to setuid root. (This means the binary runs with root privileges, even when run by a different user.) Again, to do this across all the worker nodes at once, use a script and make sure the location is correct for your setup:

for x in `cat machines`; do ssh $x chmod 4755 /usr/local/sbin/pbs_iff; done

Without these steps, users trying to run mpiexec will see errors like these:

pbs_iff: file not setuid root, likely misconfigured
pbs_iff: cannot connect to gyrfalcon:15001 - fatal error, errno=13 (Permission denied)
   cannot bind to reserved port in client_to_svr

mpiexec: Error: get_hosts: pbs_connect: Unauthorized Request .

Testing

Trying to run a program with mpiexec outside of a Torque job, it will give an error:

mpiexec: Error: PBS_JOBID not set in environment.  Code must be run from a
  PBS script, perhaps interactively using "qsub -I".

At least it's a helpful error! Therefore, in order to test it, mpiexec will need to be called from within a script. Continue on to the Torque and Maui: Submitting an MPI Job page to test.

References

• http://debianclusters.org/index.php/MPICH_with_Torque_Functionality
• Mpiexec - MPI parallel job launcher for PBS
• Mpiexec Readme