Mudanças entre as edições de "Minicluster:Mpich com torque"

Edição das 23h16min de 12 de maio de 2011

O "Outro" Mpiexec

A funcionalidade Torque fornece um novo binário para o mpiexec instalado com o MPICH2 (com o mesmo nome). Este "outro" mpiexec - com a funcionalidade Torque - é produzido por Ohio Supercomputer Center e funciona somente junto com o Torque. Usuários não poderão usá-lo sem um qsub script.

Assim, mantenha o original seguro para rodar sem o Torque e instale o novo nos nós escravos.

Prep

Before starting, the original mpiexec needs to be moved (at least on the worker nodes) so that it is no longer in root's or users' paths (or just renamed). Find it with

which mpiexec

and then move it somewhere else as a backup.

Installation

To download the latest version of mpiexec, visit the OSC mpiexec page and scroll down to the downloads section. Right click on the file location. Then, from whenever your source is stored, run

wget http://www.osc.edu/~pw/mpiexec/mpiexec-0.83.tgz

or whatever the latest version is. Untar it with

tar xvzf mpiexec*.tar.gz

and then cd into the new directory. Mpiexec follows the standard source installation paradigm. Run

./configure --help

to see a list of options. Important options include

• --prefix= - specify where you want to have the binaries installed. They need to be accessible by all of the worker nodes. An NFS mount would be a good choice.
• --with-pbs= - necessary to get the Torque functionality! Specify the location of the Torque installation. If you followed my Torque tutorial, it's located at /var/spool/pbs
• --with-default-comm=mpich2-pmi - used to indicate which version of MPI

Next, run ./configure with all the options necessary. My command looked like this:

./configure --prefix=/shared --with-pbs=/var/spool/pbs/ --with-default-comm=mpich2-pmi

Pbs_iff

To do this next part, Torque will need to already be installed. Mpiexec requires that a file named pbs_iff be on each one of the worker nodes. Normally, this file is only located on the head node and isn't installed as part of the pbs_mom installation, so it needs to be copied out from the head node to each of the other nodes.

There's an easy way to do this by scripting. The first requirement is to have a file with each of the worker nodes listed in it. Assuming Torque is running, this can be generated with

pbsnodes | grep -v = | grep -v '^$' >> machines

• grep -v = excludes all lines that have an equal sign in them
• grep -v '^$' contains a regular expression to delete all empty lines

The "machines" file of all the worker node names can then be used in a quick script to copy pbs_iff to each of the worker nodes. Find the original file with

updatedb && locate pbs_iff

(If you receive an error, apt-get install locate and then try again.) Then, replacing my locations below for the location you found it on your cluster, run

for x in `cat machines`; do rsync /usr/local/sbin/pbs_iff $x:/usr/local/sbin/; done

Next, pbs_iff needs to have its permissions changed to setuid root. (This means the binary runs with root privileges, even when run by a different user.) Again, to do this across all the worker nodes at once, use a script and make sure the location is correct for your setup:

for x in `cat machines`; do ssh $x chmod 4755 /usr/local/sbin/pbs_iff; done

Without these steps, users trying to run mpiexec will see errors like these:

pbs_iff: file not setuid root, likely misconfigured
pbs_iff: cannot connect to gyrfalcon:15001 - fatal error, errno=13 (Permission denied)
   cannot bind to reserved port in client_to_svr

mpiexec: Error: get_hosts: pbs_connect: Unauthorized Request .

Testing

Trying to run a program with mpiexec outside of a Torque job, it will give an error:

mpiexec: Error: PBS_JOBID not set in environment.  Code must be run from a
  PBS script, perhaps interactively using "qsub -I".

At least it's a helpful error! Therefore, in order to test it, mpiexec will need to be called from within a script. Continue on to the Torque and Maui: Submitting an MPI Job page to test.

References

• http://debianclusters.org/index.php/MPICH_with_Torque_Functionality
• Mpiexec - MPI parallel job launcher for PBS
• Mpiexec Readme